One assertion per test?

How many times did you fight to give a test a good name? After all, as Michael Feathers said, tests are another way of understanding our code.

Here’s a story of two programmers and their struggles with naming a test case. More importantly, it’s a story about how sometimes struggling with names can be a symptom of a very different problem.

co-written by Pietro Di Bello and Joe Bew

A Friend in Need (1903) — by Cassius Marcellus Coolidge

Over the last period, we are practicing together the Poker Hands Kata, using TDD and following a strong “slicing strategy” as explained here. In our “simplified” world, a poker hand is composed of just two cards, but at the very beginning, we started with an even more simplified model where a poker hand was composed of just a single card. 😎

While developing our model incrementally and with small steps, we encountered this scenario, where we wanted to express that a poker hand with the cards four and three is lower than a poker hand with a four and a five because the latter hand has the highest card (a five).

@Test
public void compare_hands_by_highest_card() {
  PokerHand hand = new PokerHand(four(), three());
  PokerHand otherHand = new PokerHand(four(), five());
  assertEquals(IS_LOWER, hand.compareTo(otherHand));
}

As soon as we wrote the obvious implementation to make this test pass, we were ready to write our next test. Hooray!

A possible subsequent scenario could have been to compare two poker hands where the former is greater than the latter. We felt the temptation to write a new test to describe the “new” behavior, following the guideline of one assertion per test:

@Test
public void foo() {
  PokerHand hand = new PokerHand(four(), three());
  PokerHand otherHand = new PokerHand(four(), five());
  assertEquals(IS_GREATER, otherHand.compareTo(hand));
}

But then we stopped and reflect on a few idiosyncrasies we identified in this test:

• It’s hard to find a good name for this test (…how would you name it?)
• We can re-use the Poker Hands of the previous test… it’s just a matter of having them swapped.
• As we swapped the operands (hand with otherHand), we suddenly noticed we were describing the exact opposite condition, because in the end card numbers are just plain natural numbers, and they can be ordered and compared to each other.
• The test is already passing (it’s “green”) without any change in the code. This could tell us that the behavior is already documented somewhere else!
• Duplication! This test is essentially telling the same story of the previous one, from a different perspective, so the outcome will be the same (hence, the “green bar”).

So, instead of forcing ourself into writing a new test, we could rather think to move the assertion as part of the first test:

assertEquals(IS_GREATER, otherHand.compareTo(hand));

In the end, we are using the same objects and the operation is “specular”:

@Test
public void compare_hands_by_highest_card() {
  PokerHand hand = new PokerHand(four(), three());
  PokerHand otherHand = new PokerHand(four(), five());
  assertEquals(IS_LOWER, hand.compareTo(otherHand));
  assertEquals(IS_GREATER, otherHand.compareTo(hand));
}

And the same rule of thumb can be applied to describe the case where we are trying to compare the same poker hand against itself:

assertEquals(IS_TIE, hand.compareTo(hand));
assertEquals(IS_TIE, otherHand.compareTo(otherHand));

These are assertions that will improve the documentation of the first scenario:

@Test
public void compare_hands_by_highest_card() {
  PokerHand hand = new PokerHand(four(), three());
  PokerHand otherHand = new PokerHand(four(), five());
  assertEquals(IS_LOWER, hand.compareTo(otherHand));
  assertEquals(IS_GREATER, otherHand.compareTo(hand));
  assertEquals(IS_TIE, hand.compareTo(hand));
  assertEquals(IS_TIE, otherHand.compareTo(otherHand));
}

So, we wondered, are we breaking the guideline that states that “each test should have one and only one assertion”?

“One assertion per test” is a wise rule to keep in mind, because it helps you have tests that fail for a specific reason, and drives you to focus on a specific behavior at a time.

We should nevertheless remember that it’s not the rule what really matters, is the intention behind the rule, and here what matters is having well-focused tests and not testing multiple paths, so “one assertion per test” is a good rule for a novice to follow.

In our case, could we say that we are testing “different behaviors”? We don’t think so. If the previous test fails, I know that the comparison between cards is somehow broken, and I know where to look at to spot what I should fix.

In other words, quoting Kent Beck in his great writing “Test Desiderata”, this test is still “specific”, because, if it fails, “the cause of the failure should be obvious”.

It’s having “one assertion per test” meaning you should have exactly one line of code asserting something? Or should we read it as “assert just one behavior”?

As usual, software development principles and heuristics are often forces that push you into different (sometimes opposite) directions, and you should trade-off and balance each time those forces, depending on the context and on what you and your pair think makes more sense at the moment.

The baseline here is that you should listen to your tests (as stated in the “Growing Object-Oriented Software, Guided by Tests” book by Steve Freeman and Nat Pryce): they are always telling you something.

In this case, each time you struggle with finding a good name for a test, you should ask yourself why this happens, and how should the code be designed so that this struggle disappears.

If you’re curious, here’s the git repository we’re using to practice this kata: https://github.com/xpepper/poker-hands-kata.

Key Take-Aways

• Listen to your tests: why am I struggling to find a good name for a test?
• Have well-focused tests so that if one fails, the cause of its failure should be obvious.
• One behavior per test: we might need multiple assertions to verify it.
• It’s not the rule what really matters, it is the intention behind the rule.
• Trade-off and balance each time, depending on the context and on what you and your colleagues think make more sense at the moment.

References

• Dave Astel’s “One Assertion Per Test”
• Roy Osherove’s “Avoid multiple asserts in a single unit test: revisited”
• Jay Fields’ “Testing: One assertion per test”
• Kent Beck’s “Test Desiderata”
• Growing Object-Oriented Software, Guided by Tests: “Listen to the tests”