Precise Assertions

​It could be argued that one of the key roles of automated tests is to protect the existing level of code quality, preventing its decline by verifying that changes to the code comply with a set of known constraints. This protection really earns its money when working on a large and complicated codebase. Changes can be made, relatively safe in the knowledge that an unwitting mistake or omission will be flagged during development by a failing test. The reliability of this protection hinges on the quality of the tests and what they verify by assertion.

A problem I’ve recently uncovered is where a test’s assertion is not precise enough to provide adequate protection. This example is contrived but it makes the point. If we take a presenter that adds items to a list on a view, we might see something like:​​

 foreach (var item in items)  
{  
   View.AddToList(item.Name);  
}

​This functionality was verified (using NSubstitute)​ by:​​​

 fakeView.RecievedWithAnyArgs().AddListItem(null);  

This is a reasonable test. It verifies that items are added to the view using their name. However, if the use case contains acceptance criteria that states that the item must be added to list with its name displayed, we’re unable to verify this explicitly. It could be argued that the test doesn’t need to be any more precise. The view logic, in this case, is so simple that failure would be easily discovered during manual testing. I would argue that this confidence is misplaced…

Initially our presenter just adds list items. However, we add functionality to update list items. We add tests to cover the update functionality and copy the assertion regarding the addition of the list item:​​​

[Test]  
Public void Update_ValidChanges_ShouldChangeListItem()  
{  
   // Arrange  
   // Act  
   fakeView.RecievedWithAnyArgs().AddListItem(null);  
}  

There are now tests covering the addition/modification of items in the list and verifying that the items are added to the list.

A new requirement emerges: the list items should display the name followed by its id in brackets: Item 1 (345). The presenter is changed so that the add code is modified to format the name correctly. The test covering the addition doesn't change as the perception is that the existing assertion is good enough. However, for whatever reason, the update code is missed. Code changes completed, the unit tests are run and everything is green. However, the update feature now has a defect: when an item is updated, the name that's shown in the list is not correct; it doesn't feature the id in brackets.

Ultimately, the defect is introduced due to developer not fully understanding or assessing the impact of the change. However, more detailed test assertions could've done more to protect the quality of the code. While it's true that manual testing easily finds this defect, that process is not free. If the assertion regarding the addition of the list item had contained more detail about what was being added, the update tests would've failed, alerting the developer that they had unwittingly missed something:​​

fakeView.Recieved().AddListItem(Arg.Is<string>(n => n.Equals(expectedName));  

This can be taken a step further by centralising the assertion so that the attributes of a valid name are maintained in one place:​​

Private bool ListItemNameIsCorrect(string name)  
{  
   ...  
}  
   ...  
   Assert.That(ListItemNameIsCorrect(testName), Is.True);  

Now it's a lot easier to understand the scope of a change to this part of the code.
In conclusion, ensuring test assertions have the right precision and, where appropriate, are aligned with acceptance criteria can improve the quality of the test, reinforcing their role as protectors of code quality, and reduce the need to rely on manual testing.