Reading test failures

Written Explanation

The parts of a failure

A failed test shows you four things:

1. The test's name, for example threeAddsGiveSizeThree(). This is why behavior names matter.
2. The kind of failure: an assertion failure (AssertionFailedError) or an exception from the code.
3. The message: for assertEquals, expected: <3> but was: <2>.
4. A line number or stack trace pointing into your code.

Reading expected ... but was ...

For assertEquals(expected, actual), the message looks like this (real output):

org.opentest4j.AssertionFailedError: expected: <3> but was: <2>
    at FailureDemoTest.threeAddsGiveSizeThree(FailureDemoTest.java:11)

The value after expected is what your test said should happen; the value after but was is what the code actually returned. Here the code said 2. (This is exactly why expected-first, actual-second matters from lesson 3: with swapped arguments the labels would lie.) The last line says that the failing assertion is on line 11 of the test file. In an IDE, that line is usually a link that jumps to the source.

Assertion failures vs exceptions from the code

There are two broad kinds of red result:

• Assertion failure: the code ran to the end and gave an answer, but the answer was not the expected one. Read the expected and actual values.
• Exception inside the code: the code crashed before it could answer. The message is the exception's own message, plus a stack trace.

IDEs often show these with different icons or colors (sometimes called a failure and an error), but how they are labeled depends on the tool, so go by what the message says. JUnit itself marks both as failed tests.

Reading a stack trace

A stack trace lists the chain of method calls that led to the exception, most recent call first. Real output from a test whose code crashed:

java.lang.NullPointerException: Cannot invoke "String.toUpperCase()" because "name" is null
   at Greeter.shout(Greeter.java:3)
   at Badge.label(Greeter.java:9)
   at FailureDemoTest.badgeShowsShoutedName(FailureDemoTest.java:16)

(The message names the variable name because the code was compiled with debug information, which IDEs normally do; compiled without it you may see a placeholder like <parameter1>.) Read it from the top:

• The first line says what happened: a null was used where a String method was called, and which variable was null.
• Greeter.shout(Greeter.java:3) is where: line 3 of Greeter, the line that called toUpperCase().
• The next lines are the callers: Badge.label called shout, and the test called label, on line 16 of the test.

A real trace also contains many lines from JUnit and the JDK (our launcher hid them). Ignore the lines from libraries and look at the top lines that name your code. The top line is where the crash happened; the bottom line from your code is the test that triggered it.

Is the code wrong, or is the test wrong?

Before editing anything, decide which side is mistaken. Both are possible, and "fixing" the wrong one hides a bug or creates one.

Look at our first failure again: three add calls (milk, eggs, milk) and an expected size of 3. The code returned 2, because ShoppingList.add ignores an item that is already there. So whose mistake is it? It depends on the requirement:

• If a shopping list should keep duplicates, the code is wrong; fix add.
• If it is meant to ignore duplicates, the test is wrong; change the expected value to 2, and give the test a name that records the rule, like duplicateItemIsIgnored.

The test cannot answer this; the requirement does. When there is no written requirement, ask whoever owns it. Changing an expected value just to turn the bar green, without deciding which side is correct, is the testing equivalent of unplugging a smoke alarm.

A routine for failures

1. Read the message: expected, actual, and the test's name.
2. Locate the line: open the failing assertion or the first line of your code in the stack trace.
3. Decide: is the code or the test wrong? Check the requirement.
4. Fix one thing.
5. Rerun the failing test, then rerun all tests. A fix can break something that was passing, which is exactly the regression lesson 1 described.

For the question "why is list.size() 2 and not 3 after three adds?", this routine gives a short checklist: confirm the three add calls really ran, check whether any call is rejected or ignored, and look at whether add increments the count. In our case the second milk was ignored by the contains check.

Helping the failure message

Sometimes expected: <true> but was: <false> is too vague. Most assertions accept an optional message as their last argument, which JUnit prints on failure:

assertTrue(AgeRules.isAdult(18), "18 should count as an adult");

The JUnit documentation notes that the failure message is the last parameter. Use messages sparingly: a good test name often says enough.