ترميز الأحرف (UTF-8)

Written Explanation

Characters versus bytes

A String in Java holds characters. A file holds bytes. Encoding turns characters into bytes when writing; decoding turns bytes into characters when reading. Both directions must use the same encoding.

For plain English letters, nearly every encoding produces the same single byte per letter, so mismatches go unnoticed. Trouble starts with anything outside that basic set: accented letters, the euro sign, scripts such as Greek or Cyrillic, and emoji.

Two encodings side by side

Java gives the common charsets as constants in StandardCharsets. Two matter for this lesson:

• UTF-8 is a variable-width encoding of Unicode. According to its specification (RFC 3629), it uses 1 to 4 bytes per character, it can represent every Unicode character, and the plain ASCII characters keep their one-byte values, so ASCII-only text is already valid UTF-8.
• ISO-8859-1 ("Latin-1") is an older eight-bit encoding that covers Western European characters only, one byte each.

The word café shows the difference (the é is the Unicode character U+00E9):

The first three letters are identical. Only the é differs: two bytes in UTF-8, one in Latin-1. So the number of characters (4) is not the number of bytes (5 or 4). Keep these two ideas apart: String.length() counts characters, Files.size counts bytes.

What goes wrong when they differ

There are two typical symptoms:

1. Garbled text. Bytes written as UTF-8 and read as Latin-1: the two bytes c3 a9 become two separate characters, Ã and ©, and the text shows the well-known "mojibake" like café. Nothing crashes; the data is silently wrong.
2. A read error. Bytes written as Latin-1 and read as UTF-8: the single byte e9 is not a valid UTF-8 sequence. The Files.readAllLines and readString methods report this as an IOException, because the documentation says malformed input is reported that way. By contrast, new String(bytes, UTF_8) does not throw. It silently replaces bad input with the replacement character U+FFFD, which hides the damage.

The garbled kind is worse, because nobody gets an error message.

The rule: UTF-8, and say so

Modern Java makes this easier. Since JDK 18 (JEP 400), UTF-8 is the default charset of the standard APIs, so FileReader, FileWriter, Scanner, and the like use UTF-8 unless the environment is configured otherwise. The Files methods you already know (readAllLines, readString, writeString, write, newBufferedReader, newBufferedWriter) are documented to use UTF-8 when you pass no charset, independent of the platform.

Even so, make your intention visible, for three reasons. Older Java versions used the operating system's default for classes like FileReader, and code is often run on a version you did not choose. A reader seeing StandardCharsets.UTF_8 knows the choice was deliberate. And the files you read may come from another system. So:

Files.writeString(path, text, StandardCharsets.UTF_8);
String text = Files.readString(path, StandardCharsets.UTF_8);
BufferedReader r = new BufferedReader(new FileReader(file, StandardCharsets.UTF_8));

The two-argument FileReader constructor takes a charset. Each Files text method also has a version that takes a charset as an extra argument.

A practical note: the text you see in a terminal or IDE console is a second decoding step, which depends on your console settings and may look wrong even when your file is perfect. If something looks garbled, check the bytes in the file first. Console behaviour is system-specific, so this lesson's examples print byte counts and character codes instead of accented text.

Also remember that your .java source files are text too. If you type an accented letter in a string literal, save the file as UTF-8 and compile with UTF-8 source encoding (many modern setups do this by default; check your build tool or IDE settings). An alternative is a Unicode escape such as é, which is what the examples below use.