Why databases
On This Lesson 11 topics
Lesson Overview
In the file-handling module you made your Contacts program remember its data by saving it to a CSV file. That worked, and for one person using one small program it is a perfectly good solution. This lesson asks the next question: what happens when the program grows? More data, more users, and more at stake if something goes wrong.
You will learn what a file cannot do for you, and what a database adds: fast search, safe use by many users at once, rules about what data is allowed, and protection when the computer crashes. This lesson stays at the level of ideas: no installing, no database commands yet.
The analogy used throughout this module: a paper notebook versus a well-run library with a librarian. A notebook is fine for one person. A library with a librarian is what you need when many people borrow, return, and look things up all day.
Why this matters for a backend: almost every backend service reads and writes persistent data on behalf of many users at once. Choosing where that data lives is one of the earliest and most expensive design decisions in a project.
Learning Objectives
By the end of this lesson you will be able to:
- Explain why a program must save data outside its own memory.
- Describe why a plain file works well for small, single-user data.
- Name four things a database adds over a plain file: fast search, safe simultaneous use, rules on the data, and crash safety.
- Explain a lost update with a concrete example.
- Explain why searching a large file means reading much of it.
- Say in one sentence why a shop should not keep its orders in one text file.
Written Explanation
Programs forget
A running program keeps its data in memory (RAM). When the program stops, that memory is released and everything in it is gone. You met this idea in the first module as the difference between fast, temporary memory and slower, permanent storage. To keep data between runs, a program has to write it to permanent storage.
The simplest way is a file. In the file-handling module you wrote contacts as CSV lines and read them back on the next start. A file is easy to understand, easy to open in a text editor, and needs nothing installed. For a personal to-do list or a small tool used by one person, it is often the right choice.
Where a file starts to hurt
A file is just a sequence of bytes on disk. It has no idea what the bytes mean. Your program supplies all the meaning: where a row ends, what a column is, which rows are valid. Everything else a data store might do has to be written by you. Four problems appear as the use grows.
1. Searching is slow. To find one contact by email in a CSV file, a program has to read lines from the top until it finds a match. If the match is the last line, it read the whole file. The work grows with the size of the file.
2. Simultaneous use is unsafe. Imagine two programs, or two copies of the same program, that both load the file, change their copy in memory, and save it back. Whoever saves last replaces the whole file, and the other change vanishes. This is a lost update. Example 2 below shows it with real output. A file does not coordinate its users in any helpful way: it does not make the second program wait or warn it that the content changed.
3. There are no rules. A file will accept any text you write. Two contacts with the same email, a missing name, or the word abc where an id should be a number: the file accepts them all. Every program that touches the file must remember to check, and one program that forgets damages the data for everyone.
4. A crash can damage it. Saving a file means overwriting old content with new content. If the power goes out or the program is killed halfway through, you can be left with a half-written file. A file gives you no built-in way to say "either all of this change happened or none of it".
None of these problems matter much when one person keeps ten contacts. All of them matter when a thousand customers place orders at the same moment.
What a database adds
A database is an organised collection of data, stored so that programs can search, add, change, and remove it reliably. The software that manages it is called a database management system (DBMS). Lesson 4 explains that term precisely. For now, think of the DBMS as the librarian: programs do not walk into the shelves themselves, they ask the librarian, and the librarian follows the rules of the library.
The same four problems, with the database answer:
| Problem with a file | What a database adds |
|---|---|
| Search reads much of the file | Structures built for finding data quickly (you will meet indexes in a later module) |
| Two writers overwrite each other | Coordination of simultaneous readers and writers |
| Any text is accepted | Rules the DBMS enforces: required values, correct types, no duplicate emails |
| A crash can leave a half-written file | A way to recover to a consistent state after a crash |
PostgreSQL, the database system this course uses later, documents the last two in its manual. It handles concurrent access with a technique called multiversion concurrency control, in which each statement sees a snapshot of the data, so readers and writers do not block each other in the usual case. For crashes, it writes a description of each change to a log before changing the data files, so that after a crash the changes can be replayed from the log. You do not need the mechanisms now, only the idea: the DBMS does these jobs for you.
A fair warning: files are not bad
Files are not obsolete. Configuration, logs, uploaded photos, and exports are usually files, and should be. The real question is: does this data need fast search, safe use by many users, rules, and crash safety? If yes, use a database. If it is a small file used by one program, a file may be simpler and better.
Examples
Example 1: finding one row costs a scan (simple)
This program writes 100,000 contact lines and then looks for one near the end, counting how many lines it had to read.
import java.io.IOException;
import java.nio.file.*;
import java.util.*;
public class L01b {
public static void main(String[] args) throws IOException {
Path file = Path.of("big.csv");
List<String> lines = new ArrayList<>();
for (int i = 1; i <= 100_000; i++) lines.add(i + ",user" + i + "@example.com");
Files.write(file, lines);
String wanted = "99999,user99999@example.com";
int linesRead = 0;
for (String line : Files.readAllLines(file)) {
linesRead++;
if (line.equals(wanted)) break;
}
System.out.println("Lines read to find one row: " + linesRead);
}
}Output when compiled and run with JDK 21:
Lines read to find one row: 99999To find one row the program looked at almost every line. With ten times more data it would look at about ten times more lines. A database avoids reading everything by keeping extra structures that point to rows directly.
Example 2: a lost update (practical)
Two programs share the file stock.csv. Both read it, both add a different product, and both save. We simulate the two programs inside one main by interleaving their steps in exactly the order that would cause the problem. No special tools are needed.
import java.io.IOException;
import java.nio.file.*;
import java.util.*;
public class L01a {
static List<String> load(Path p) throws IOException { return new ArrayList<>(Files.readAllLines(p)); }
public static void main(String[] args) throws IOException {
Path file = Path.of("stock.csv");
Files.write(file, List.of("pen,10", "book,5"));
List<String> a = load(file); // A reads
List<String> b = load(file); // B reads the same old state
a.add("lamp,2"); // A adds a row
b.add("desk,1"); // B adds a different row
Files.write(file, a); // A saves
Files.write(file, b); // B saves: overwrites A's save
System.out.println(Files.readAllLines(file));
}
}Output:
[pen,10, book,5, desk,1]The lamp row is gone. Nobody got an error. Program A believes it saved the lamp. Program B believes it saved the desk. Both are partly right and the file is wrong. The only reason the file lost data is that each program saved a whole copy based on an old read. With real programs the timing is rarely this tidy, so the bug appears only now and then.
A database lets programs change one row at a time and coordinates them, so adding the lamp and adding the desk are two separate, safe changes.
Example 3: a shop with real stakes (realistic)
A small online shop starts by saving orders in orders.csv. Trace what happens on a busy evening.
- Two customers press "Buy" within the same second. Both requests read the file, add their order, and save. One order disappears. The shop owes goods to a customer who has no record.
- Support asks, "show me all orders by customer 5821." The program reads every line and filters. In the first month this takes a blink. A year later the file is large and the page feels slow.
- A developer adds an order with a blank customer id. The file accepts it. Later the reporting program crashes on that line.
- The server restarts in the middle of a save. The file ends in the middle of a line.
Each failure maps to a row of the earlier table. The practical argument for a database is that the shop would otherwise spend months re-inventing those answers in its own code.
Backend Insight
In a typical backend, the program that handles requests is stateless in the sense that you should be able to stop it and start a new copy without losing anything. All lasting facts (users, orders, payments) live in the database. When traffic grows, a team can run several copies of the backend, all sharing one database. That only works if the data store handles simultaneous readers and writers safely. A CSV file on one server's disk cannot be shared safely, and it is lost if that disk fails.
So when you design a backend, ask of every piece of data: does it need to survive a restart? Will more than one request touch it at once? Must it follow rules? If you answer yes to any of these, it is a candidate for the database, not for a variable or a file.
Common Mistakes
Mistake 1: "A file is a database"
- Incorrect: "My contacts.csv is my database, so I do not need anything else."
- Correct: "My contacts.csv stores data. A database system also searches quickly, coordinates users, enforces rules, and recovers from crashes."
A file is a data store, but it lacks a DBMS's services.
Mistake 2: "Always use a database for everything"
Configuration, logs, and images are normally files. Using a database for data that needs none of its services adds complexity for no gain.
Mistake 3: Assuming a lost update will show an error
A lost update is silent. Nothing crashes. The data is simply wrong. Silent failures are the dangerous ones, so protection belongs in the data store.
Exercises
Try each exercise yourself before opening the solutions.
Level A: Foundation
A1. In your own words, why does a running program need to save data to a file or database at all?
Show solution A1
A1. Memory is released when the program ends, so anything not written to permanent storage is lost. Saving to a file or database lets the data survive restarts.
A2. List the four problems a plain file has as use grows, using one short phrase for each.
Show solution A2
A2. Slow search (reading much of the file); unsafe simultaneous writes (lost updates); no rules (any text accepted); crash damage (half-written file).
A3. True or false, with a reason: "Searching a CSV file for one contact by email is fast no matter how big the file is."
Show solution A3
A3. False. The program usually has to read lines from the start until it finds a match, so the work grows with file size.
Level B: Application
B1. A school keeps student grades in grades.txt. Two teachers open the program at the same time and each enters grades for a different class. Describe what could go wrong and which of the four problems it is.
Show solution B1
B1. Both load the file, change their copy, and save. The second save overwrites the first, so one teacher's grades are lost without any error. That is the simultaneous-use problem (a lost update).
B2. For each item, say whether you would choose a file or a database, and give one reason: (a) the app's color theme setting, (b) a library's list of borrowed books with many librarians working at once, (c) a daily server log, (d) customer accounts for a web shop.
Show solution B2
B2. (a) File: tiny, one program, no sharing. (b) Database: many users writing at once and rules such as "a book is borrowed by one person". (c) File: append-only text for humans and tools, no searching rules needed. (d) Database: accounts need unique emails, safe simultaneous use, and crash safety.
B3. The following program saves scores.csv after two players each added a score. Predict the final content before running it.
List<String> a = new ArrayList<>(List.of("Ana,50"));
List<String> b = new ArrayList<>(List.of("Ana,50"));
a.add("Sam,70");
b.add("Lee,90");
// save a, then save b to the same fileShow solution B3
B3. The file ends with Ana,50 and Lee,90 only. Sam,70 is lost because the second save writes b, which never saw it.
Level C: Challenge
C1. Sketch a way to reduce the lost-update problem in the CSV program without a database. Name one problem your fix does not solve.
Show solution C1
C1. One fix: before saving, reload the file and merge your change into the fresh content. That narrows the gap but does not close it: another program can still save between your reload and your save. Another fix: a "lock file" that a program creates before editing and deletes after. That creates new problems: if the program crashes before deleting it, everyone waits forever, and every program must follow the convention. Either answer is acceptable if the new problem is named.
C2. A friend says: "Our shop has only 50 customers, so we do not need to think about any of this." Write a short, fair reply that says when they are right and when they should reconsider.
Show solution C2
C2. They may be right that speed is not an issue at 50 customers, and a file could work for a prototype. The silent problems (lost updates when two orders arrive together, no rule against duplicates, damage after a crash) do not depend on size, and they get worse and more expensive to fix as the data grows. They should reconsider as soon as more than one person or request writes at the same time.
Mini Challenge
Pick a small app you could imagine building: a habit tracker, a bookshop, a gym booking system. Write a table with two columns: "Data it stores" and "File or database?". Include at least five kinds of data, and make sure at least one is a file and at least one is a database. For each database entry, name which of the four database benefits (fast search, safe simultaneous use, rules, crash safety) is the main reason.
Check your work: you justified each choice with a concrete need, not with "databases are better".
Summary & Next Lesson
Key takeaways
- Programs forget data when they stop, so data must be saved to permanent storage.
- A file is simple and fine for small, single-user data.
- As use grows, files cause slow search, lost updates, missing rules, and crash damage.
- A database, run by a DBMS, adds fast search, safe simultaneous use, enforced rules, and crash recovery.
- Files remain the right tool for logs, configuration, and uploads.
Next lesson
The next lesson, "Structured vs unstructured data", looks at the shape of data: which data fits neatly into fields and rows, and which does not, and what that means for where it should be stored.
Self-check
- Why does a program lose its data when it stops?
- What is a lost update?
- Name two rules a database can enforce that a plain file cannot.
- Why does searching a big CSV file get slower as it grows?
- Name one kind of data that should normally stay in a file.
Show answers
- Its data lives in memory, which is released when the program ends.
- Two writers change the same data based on an old read, and the second save overwrites the first without any error.
- For example: a value is required, a value has the right type, and an email is unique.
- The program usually has to read lines from the start until it finds a match, so more lines mean more work.
- For example logs, configuration, or uploaded photos.
References
- PostgreSQL documentation, Concurrency Control: Introduction. Supports the statement that PostgreSQL manages concurrent access with multiversion concurrency control, where each statement sees a snapshot, and that reading and writing do not normally block each other.
- PostgreSQL documentation, Write-Ahead Logging (WAL). Supports the statement that changes are logged before data files are changed so that a database can be recovered after a crash.
- The Java code in this lesson was compiled and run with JDK 21. The notebook-and-librarian analogy is this course's own teaching device, not from a source.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.