عرض عملي: لماذا ينهار ملف CSV لجهات الاتصال مع زيادة الحجم
في هذا الدرس 11 موضوعات
هذا الدرس متاح حاليًا باللغة الإنجليزية فقط، ولم تُنشر نسخته العربية بعد.
Lesson Overview
This is the demo lesson of the module. You have heard the arguments for a database in lessons 1 to 4. Now you will see the problems with your own hands: you will take the Contacts CSV idea from the file-handling module, make it large, and watch it slow down, lose data, accept bad rows, and misread a name with a comma in it. At the end you will have a list of problems that doubles as a start on the module lab, BEG-048-LAB1.
Follow along by typing the code yourself in a scratch folder, outside your real Contacts project. We will make one deliberate mistake on purpose (a name with a comma) and then discuss the fix and its limits. Timings depend on your computer, so yours will differ from ours.
Why this matters for a backend: the point of the demo is not that CSV is bad. It is that you can now name the exact property a database must provide, which is the skill needed to choose a data store later.
The previous lesson, "DBMS and schemas", named the parts of a database system. This lesson closes the module.
Learning Objectives
By the end of this lesson you will be able to:
- Generate a one-million-row CSV file with a short Java program.
- Show that finding one row requires reading much of the file, and time it.
- Reproduce a lost update with the read-modify-save pattern.
- Show that a CSV file accepts duplicates and invalid rows without complaint.
- Explain why a comma inside a name shifts the columns, and what quoting does and does not fix.
- Write a list of at least five distinct problems of the file approach, each with the database feature that answers it.
Written Explanation
The demo plan
The demo has five steps, each exposing one problem. Every step uses only Path, Files, lists, and strings from earlier modules.
- Make it big. Write one million contact lines.
- Search it. Look for one email near the end and time the search.
- Break it with two writers. Show the lost update.
- Feed it bad data. Add a duplicate, and a name with a comma.
- List the problems and match each to a database feature.
Step 1 and 2: size and search
The generator writes lines such as 5,Name5,user5@example.com. The search reads the file and checks each line until it finds the wanted email. Because the email is near the end, almost every line is read. We measure with System.nanoTime(), which gives a high-resolution time suitable for measuring durations.
Step 3: two writers
You saw this in lesson 1: two programs each load, change, and save a whole copy. The second save overwrites the first. In the demo you will rerun that example, because a bug you have seen is easier to remember than one you have only read about.
Step 4: bad data
A CSV file stores lines of text. It does not know that two lines have the same email, or that a name contains a comma. The program that reads it decides what a line means, here by splitting on commas.
The RFC that describes common CSV practice says that a field containing commas, double quotes, or line breaks should be wrapped in double quotes, and that a quote inside a quoted field is written as two quotes. So the correct line for Smith, John is 6,"Smith, John",john@example.com. But a quick split(",") does not know about quoting, so the reader must be written more carefully, or a CSV library used. Note also that the same RFC describes itself as informational: it records common practice and is not an Internet standard, so different programs may handle edge cases differently. A column type and a proper storage format in a database remove this guesswork: the value Smith, John is simply a value of the name column, and the comma means nothing special.
Examples
Example 1: generate and search one million rows (simple)
import java.io.IOException;
import java.nio.file.*;
import java.util.*;
public class L05a {
public static void main(String[] args) throws IOException {
Path file = Path.of("contacts.csv");
List<String> lines = new ArrayList<>();
for (int i = 1; i <= 1_000_000; i++) lines.add(i + ",Name" + i + ",user" + i + "@example.com");
Files.write(file, lines);
System.out.println("File size in MB: " + Files.size(file) / 1_000_000);
String wanted = "user999999@example.com";
long start = System.nanoTime();
int found = -1;
for (String line : Files.readAllLines(file)) {
if (line.endsWith("," + wanted)) { found = Integer.parseInt(line.substring(0, line.indexOf(','))); break; }
}
long ms = (System.nanoTime() - start) / 1_000_000;
System.out.println("Found id " + found + " after reading the file; took about " + ms + " ms");
}
}Output from our run with JDK 21 on one laptop (your size will match, your time will not):
File size in MB: 41
Found id 999999 after reading the file; took about 393 msRoughly 0.4 seconds is not alarming for one search. Now picture 100 users each doing searches all day: each search re-reads the file, so the work per search grows with the data, and the machine's memory has to hold the whole file each time (readAllLines loads all lines). Our timing is one measurement on one machine, so treat it as an illustration, not a benchmark. A database that has an index on email reaches the row without reading the whole table; you will meet indexes in a later module.
Example 2: duplicates, invalid rows, and a comma (practical)
The program below appends a duplicate row to the file from Example 1, counts the rows with that email, and then splits a line containing a comma in the name. (Add this code at the end of main in L05a, after the search.)
Files.writeString(file, "5,Fake,user1@example.com" + System.lineSeparator(), StandardOpenOption.APPEND);
long dup = Files.readAllLines(file).stream().filter(l -> l.endsWith(",user1@example.com")).count();
System.out.println("Rows with user1@example.com: " + dup);
String bad = "6,Smith, John,john@example.com";
String[] f = bad.split(",");
System.out.println("fields = " + f.length + ", email column holds: '" + f[2] + "'");Added output:
Rows with user1@example.com: 2
fields = 4, email column holds: ' John'(The filter uses a stream. If streams are new to you, read it as "keep the lines that end with this email, then count them".) The file accepted a second row for the same email, with no warning. The comma in Smith, John produced four fields instead of three, so every later column moved: ' John' landed in the email column. That is the planned mistake of this demo. The fix is to quote the field in the file and to read it with a quote-aware reader, which narrows the problem for names but does not give the file the other protections: it still would not stop duplicates or a missing name.
Example 3: the problem list for 1 million contacts and 100 users (realistic)
Now put it together. Here are the problems of the Contacts CSV at one million contacts and 100 users at once, each paired with what a database offers. This is a worked answer for the style of the lab; your own list should use your own observations.
| # | Problem in the CSV file | Seen in | Database answer |
|---|---|---|---|
| 1 | Search reads much of the file, and gets slower as it grows | Example 1 | Indexes: direct lookup |
| 2 | Two users saving at once can lose an update | Lesson 1, Example 2 | Coordinated simultaneous access |
| 3 | Nothing stops duplicate emails | Example 2 | Uniqueness rules in the schema |
| 4 | Nothing checks required fields or types | Lesson 4 toy checker | Column types and required values |
| 5 | A comma or quote in a value can shift columns | Example 2 | Typed columns; values are not delimited by commas |
| 6 | A crash during a save can leave a half-written file | Lesson 1 | Crash recovery |
| 7 | Everyone who can open the file can read and change all of it | Not demonstrated | Per-user access control |
| 8 | To change one contact, the program rewrites the whole file | Lesson 1, save-on-exit pattern | Update one row at a time |
Problem 7 is a reasoned point, not something this demo showed: a plain file has only the permissions of the operating system on the whole file, while a DBMS usually lets you grant different rights to different users. Hedge such claims for your own system: say what you observed and what you inferred.
Backend Insight
Notice what the table is: a requirements list. When a team evaluates a data store, it writes down needs like these (search speed at the expected size, simultaneous writers, uniqueness, crash safety, access control) and checks each candidate against them. That is the habit the next modules build on. It is also the sensible answer to the question "do we need a database?": count how many rows of the table apply to your system.
Note also what the demo did not prove. It did not show that a database is always faster, and the previous lessons warned against that belief. It showed that a file has no built-in answer to these specific needs. If your data fits on one page and one person edits it, the table would have no rows for you, and a file is the right choice.
Common Mistakes
Mistake 1: Drawing the wrong conclusion from one timing
- Incorrect: "The search took 393 ms, so CSV is fast enough."
- Correct: "One search on one machine took about that long. The cost grows with the data, and many users repeat it."
Mistake 2: Testing with data that is too tidy
Demo data with no commas, no duplicates, and no blanks hides most of the real problems. Real data is messier.
Mistake 3: Fixing only the visible symptom
Quoting the comma fixes one problem. It does not give uniqueness, crash safety, or safe simultaneous use.
Mistake 4: Running demos in the real project folder
A demo that writes a one-million-line file belongs in a scratch folder. Do not leave a 41 MB file in a project you commit.
Exercises
Level A: Foundation
A1. In the demo, why did the search have to read almost the whole file?
أظهر الحل A1
A1. The program has no way to jump to the wanted row. It reads lines from the start and checks each one, and the wanted email was near the end.
A2. What does it mean that the CSV accepted a duplicate email "without complaint"? Which database feature prevents it?
أظهر الحل A2
A2. The file stores text and does not check anything: it wrote a second line with the same email. A uniqueness rule in the schema, enforced by the DBMS, prevents it.
A3. How should the name Smith, John be written in a CSV line, according to the common practice described in the RFC?
أظهر الحل A3
A3. In double quotes: 6,"Smith, John",john@example.com. A quote inside a quoted value is doubled. The reader must understand quoting; a plain split(",") does not.
Level B: Application
B1. Change the generator to write 2,000,000 lines and search for the last email. Predict whether the time will be about the same, double, or much smaller, then run it and compare. Explain any difference. (Your time depends on your computer.)
أظهر الحل B1
B1. The time should grow roughly in proportion to the file, so about double, because the program reads about twice as many lines. Garbage collection, caching, and memory use can change the exact figure, so "roughly proportional, not exactly" is a correct answer. Very large files may also hit memory limits with readAllLines.
B2. Write a short program (or add to the demo) that appends a row with a blank name to the CSV and then counts the lines whose second field is blank. What does this show?
أظهر الحل B2
B2. For example, Files.writeString(file, "7,,x@example.com" + System.lineSeparator(), StandardOpenOption.APPEND) followed by counting lines where line.split(",", -1)[1].isEmpty(). The count is 1: the file accepts a blank required value. A NOT NULL or required-column rule would refuse it. (The -1 keeps trailing empty values when splitting.)
B3. Rerun the lost-update example with a third program C that also adds a row. Which rows survive? Why?
أظهر الحل B3
B3. Only the rows from the program that saved last survive, so the earlier saves of the other two programs are lost. For example if C saves last, only the original rows plus C's row remain. Each program saved a whole copy based on an old read.
Level C: Challenge
C1. For the lab: write your own list of at least five distinct problems the Contacts CSV would have at 1 million contacts and 100 simultaneous users. For each, name one thing you observed or reasoned and the database feature that answers it. Do not copy the table above: use your own words and, if you can, at least one problem not in it.
أظهر الحل C1
C1. Any five distinct, well-explained problems from this list or your own: slow search; lost updates; duplicates; missing or invalid values; delimiter problems; crash damage; no per-user access control; rewriting the whole file for one change; the whole file held in memory; no way to ask questions other than by writing code. Each should be paired with a plausible database feature.
C2. A friend says: "Just keep the CSV, and add a lock file so only one program writes at a time." Which problems from your list does that solve, and which remain?
أظهر الحل C2
C2. A lock file reduces lost updates, because only one writer runs at a time. It does not help with slow search, duplicates, invalid values, delimiter problems, or crash damage. It also adds new problems: if a program crashes while holding the lock, others may wait forever, and 100 users now queue for one lock.
Mini Challenge
Write a one-page "decision note" for a small team: "Should the Contacts app move from a CSV file to a database?" State the expected data size and number of users, list which problems from your table apply and which do not, and finish with a clear recommendation. Include one thing you would measure before final approval.
Check your work: the recommendation follows from your list, and it says what would change your mind.
Summary & Next Lesson
Key takeaways
- A one-million-row CSV is about 41 MB in our test, and finding one row means reading most of it.
- Simultaneous saves can lose data silently, and the file does not stop duplicates or invalid rows.
- A comma inside a value shifts the columns unless the field is quoted and the reader understands quoting.
- A list of problems is a requirements list for choosing a data store.
- A file is still right for small, single-user, low-stakes data.
Next lesson
This was the last lesson of the module on data fundamentals. The module lab, BEG-048-LAB1, asks you to list at least five distinct problems of the Contacts CSV with 1 million contacts and 100 users; use the demo as your evidence. The next lessons come from the next module in the course outline.
Self-check
- Why does searching a big CSV file get slower as it grows?
- What is a lost update, and why is it hard to notice?
- Why did
Smith, Johnbreak the columns? - Name two problems a lock file does not solve.
- Does this demo prove that databases are always faster?
أظهر الإجابات
- The program reads lines from the start until it finds a match, so more lines mean more work.
- Two writers save whole copies and the second overwrites the first. No error is raised, so the data is silently wrong.
- The comma was treated as a column separator, so the name was split into two fields and later columns moved.
- For example slow search, duplicate emails, invalid values, delimiter problems, crash damage.
- No. It shows that a file has no built-in answer to these specific needs; speed depends on the workload.
References
- Internet Engineering Task Force, RFC 4180: Common Format and MIME Type for CSV Files. Supports the rule that fields containing commas, double quotes, or line breaks should be enclosed in double quotes (with inner quotes doubled), and the note that the document is informational and not an Internet standard.
- PostgreSQL documentation, Concurrency Control: Introduction and Write-Ahead Logging. Support the database answers named in the table: coordinated concurrent access and recovery after a crash.
- The 393 ms timing was measured once on the author's machine with JDK 21 and is an illustration only. Per-user access control in a DBMS is stated from general knowledge and was not checked against a fetched page. The Java code was compiled and run with JDK 21 (the B2 solution snippet was not run).
التطبيق
طبّق هذا الدرس بمختبر الوحدة وتمارينها واختبارها.
تعليم الدرس كمكتمل اختياري ويمكنك التراجع عنه في أي وقت. يبقى تقدمك محفوظًا في هذا المتصفح.