At first I want to say, that I truly believe that it’s really important to understand unit testing and how to implement a good unit test. I’m not a unit test puritan or a professional in that area, but I think it’s very unprofessional to be a software developer without any knowledge of unit testing and how to implement a unit test.
What is this article about?
I decided to write this article because unit testing is important part of a software process. Sometimes it’s clear for a developer how to write a test for a module and sometimes it really gives you a head ache.
My humble opinion is that it’s really important to know where to draw a line what is worth to test with unit tests and what parts of a module can be left without. I know the previous sentence made unit test puritans to scream, but this article is also criticism agains unit test blindness that I have encountered in software projects during my career.
Before we start, let’s take a look at few definitions for unit testing.
Unit Testing Definitions
“In computer programming, unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual function or procedure. Unit tests are created by programmers or occasionally by white box testers.” [WikiPedia]
“The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits. Unit tests find problems early in the development cycle.” [WikiPedia]
“Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: first the developer writes a failing automated test case that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new code to acceptable standards.Kent Beck, who is credited with having developed or ‘rediscovered’ the technique, stated in 2003 that TDD encourages simple designs and inspires confidence.” [Beck, K. Test-Driven Development by Example, Addison Wesley, 2003]
What to Test with Unit Tests?
As stated above, unit testing should test an isolated module. I am C++ / Qt developer so I consider the smallest unit to be a class. So usually you write one unit test (class) per one class and it’s a pretty good rule. If you write a unit test for more than one class, then you test class integration with other classes and not the unit itself.
So the ideology of testing one unit, brings plenty of problems to us. If your class inherits a superclass, should you test the inherited functionality too? If your class uses other class (e.g. It has an instance variable) should you test that class too? I’m not sure about how to treat superclasses especially if they are abstract classes (i.e. they can’t be instantiated), but I guess it’s quite ok to test superclass’ functionality when writing a unit test for a subclass. Answer for testing a functionality that comes from a instance variable is that you should have own unit test for that class too, assuming that you have written that class.
So how to test a class that your application depends on, but that class is provided by 3rd party lib? One rule here is to provide a stub implementation for those classes. Then you will have the complete control how your unit test acts.
Stubbing or Stabbing Your Own Back
Stub is a “fake implementation” to cut the dependency off from your unit test. An example of what you could/should stub is a dependency to a network or a d-bus service. You can implement a stub which simulates real-life behavior, but removes the need to use the network or a d-bus service. Sometimes providing a stub implementation is the only way to execute unit tests in a test environment.
If you ask from a unit test puritans what you should stub, she says that “any 3rd party interface that your unit test depends”. In a perfect world this is how it (probably) should be done, but in reality stubbing everything leads to the following problems:
- If you stub a component that is under development and changes constantly you will end up to have a stub maintenance nightmare.
- A new unit test might call a stub implementation and fail. These are sometimes hard to catch especially when you don’t realize that your unit test is actually calling an empty stub implementation via a class that you are testing.
- You need to implement a logic in a stubs in order to make your code work. This takes time and makes your unit test environment to maintenance nightmare. If your other test excepts the same stub to act differently in some cases, the stub complexity increases again.
- You spend more time for updating stubs than you have spent time for writing a new functionality and unit tests. An example from a real life: fixing a bug with one line change required 1.5 day work make unit test to test that bug. Insane effort and eventually it was left without a test.
Testing UI Code
One thing that I have realized lately is, that it doesn’t make much sense to unit test UI code. You should use different tools than unit test frameworks for that. There are UI test tools which usually are scriptable and you can record e.g. events that are later posted your applications in a UI test execution environment.
The rationale behind this argument is that it’s quite hard to validate that your UI component acts as it should within a unit test. Of course there are solutions for e.g. testing if the UI component appearance is correct, but still I think it’s not a place for unit testing.
MVC (Model-View-Controller) model provides here a good division how you can decide what to test. You can always write a test which validates the Model part. For example your view might construct quite complex set of components from a model. I think that you can test your controller part with unit test. If the controller changes the visual appearance of the view, just leave the View part away from the test. If the controller changes the data, you should be able to test that too. So for example if an event effects on data (e.g. sorts it) it should be quite easily testable.
Another justification for this is that let’s think about QML. It provides a good, natural division what could be tested. Usually a QML application is constructed from C++ and QML layers. The C++ provides the business logic and data parts and the QML provides the UI layer. In my opinion, it’s natural to test the C++ side and leave the QML testing either for testers or for other test tools.
The third justification is that usually you catch the errors in the UI by yourself during the development. The UI part bugs are most easily found because the UI is always visible. If the UI is not acting as it should when you e.g. tap the button, you see that immediately, if the UI is not according the UI spec, you see that immediately, etc. But if your UI displays a list that should show 1000 items from a model, but it shows only 999 items, it’s something that is hard to detect. Then the problem might be in the lower layers or in the UI layer. It’s hard to say, but it’s simple to test if your model contains 1000 or 999 items.
How To Test Things Correctly?
The hardest part of writing a unit test is to test things correctly. As I said earlier, I’m not a unit test “professional”, I believe that it’s important to have unit test, but they should be written correctly.
My rule for this is that when you write a unit test, check the method documentation or signature first. Test the method with bad input, with good input and with special cases. I’m not sure if this is by a book, but I think it’s a pretty good start.
You shouldn’t change your production code in order to be able to test it. The exception here is, that you can add your test class as a friend class to the class which is being tested, in order to access the private members. If you can’t test your class with a unit test, make sure that you have a good design, but this should already have been taken care in the implementation phase. To make sure that your design is testable, you can always implement things using Test-driven developent (TDD) i.e. write tests first and design your interface during writing a tests. TDD actually works. Yes, I have tested it in practice (few times), but I think it can be applied best when implementing a new interfaces.
100% Test Coverage
This is a topic that projects managers probably like. It looks damn good when a project manager can show project statistics of unit testing and the test coverage is around 80%-100%. It looks green in a report, but in reality things might be differently.
At first, the test coverage usually tells the coverage of the code that tests have been executed by unit tests. If you have a one unit test, which executes 100% of code that it tests, the test coverage is 100%, but tests might cover 1% of the whole code base. Looks good in Excel sheet, but doesn’t tell anything about the reality or the quality of the testing.
Test coverage is a good way for a developer to test which parts unit tests executes and after that it’s easier to improve the unit testing coverage. There are also nice tools (lcov) that you can use for generate nice looking reports that you can display with a web brower.
This article was bit too long, but the topic has been in my mind for a long time. I know that not everyone agrees with me, but the opinions in this article have formed from a real life projects and practices.
There are three opinions that I still want to raise here:
- Be careful when you decide to use stubs. They might lead you to the maintenance hell and the gained benefit will vanish when you spend more and more time for maintaing the stubs rather than implementing new features and unit tests.
- It is really hard to test “View” parts of the UI components with unit testing. In my opinion the UI bugs will be catch faster during the development by a developer or later by testers. There are other tools for testing Uis in an automated and scriptable environments.
- It doesn’t make sense to write a test that doesn’t really test things as it should. This is the hardest part in a unit testing and I’ve seen way too many unit tests that I’m not actually sure what they test.
In the end, it’s really important to have unit tests and writing them should be part of your daily routines. No one has said that it’s easy to write unit tests, but IMHO is that it’s better to have one good unit test rather than 10 unit tests which don’t really test things properly.
Thanks for reading my blog.