Skip to end of metadata
Go to start of metadata

Introduction

The aim of this project is to cover at least 80% of instructions, lines, methods and classes and 70% of branches with test code. Components without ANTLR in use shall cover 80% of branches.

A test coverage report should be available with each component.
How the project is intend to achieve it and which principles should be noticed is explained in the following chapters.

Methodology

The developers of this project are encouraged to exercise 'Test Driven Development ' and therefore shall apply the Test First Principle.
However, it is also perfectly fine, if you code first and write tests after as long as the tests are written before commit.

Do not commit code without tests unless there is a good reason which you need to argue. 

But why do I want to encourage you to exercise 'Test First Principle '? The 'Test First Principle' enforces the developer to think about certain problems which could occur for a particular method - which exceptions could be thrown etc. One has to think about boundary conditions and finally one will hopefully think about the design of the existing code and consider a refactoring if necessary. For instance, if the developer notices that he/she has to write a lot of test methods to cover all functionalities for one particular method, he/she gets a clear sign of a smell – the method does too much. Thus the 'Test First Principle' almost automatically enhances the code quality in terms of the 'Single Responsibility Principle', 'Separation of  Concerns' and 'Low Coupling' if the developer is aware of those principles. But I do not just expect that the code quality enhances but also the motivation of each developer since one knows "Everything is fine, I have not broken old code with my new features". There are more benefits about 'Test Driven Development' and it shall be left to the developer to get more familiar with 'Test Driven Development' and find them out by him-/herself.

Principles

The developers shall especially apply the following principles:

  • If a bug is found one writes first a test which covers the bug and therefore fails, commit without push so others can reconstruct the error easily and afterwards one fixes the bug, regardless if one uses ‘Test Driven Development’ or not.
  • Each class should have (at least) a unit-test class which does not deal with other classes of the system (a real unit-test) - use mocks where necessary.
  • Make sure there are tests which cover a class if a new functionality has to be added to a class or a class has to be changed. If there are no tests or the tests do not cover the important parts of the class, the parts of the code you are going to change respectively, then create the necessary tests first.
  • Do not refactor without tests, but refactor where necessary. Ergo, create tests if there are not any for the corresponding part of the code, hence you can refactor without introducing new bugs.
  • Do write only against abstract types (interfaces or abstract classes) - if one class ignores this principle then it should be refactored. However, DTO's make an exception.
  • If you exercise ‘Test Driven Development’ then:
    • We write a unit test before the actual implementation whereby we incrementally add new unit tests and implementations (it is not wise to write first all tests and then the whole implementation - step by step; unit test, implementation, unit test implementation etc.).

Code Conventions

It has to be considered that test code underlies the Java Code Conventions as normal source code and the developer should apply all good coding principles like 'Information Hidding', 'Seperation of Concerns', 'Low Coupling', 'High Cohesion' etc. Bad written test code is not maintained in the future and will become obsolete faster than maybe expected.

Consider the following scenario: Test2 fails. The developer is going to think "what is Test2, what does this test method do?". He will look at the code and if it is well structured he can maybe see the purpose quickly and will rename the method, thus next time it is more obvious. But what if the method is long and tedious to read, and finally the developer has now idea what the test exactly does? He will just ignore the method and delete it because he will probably think "this method seems to be an old one which is no longer used and applicable for the current version of the code".

Folder Structure

Try to structure your tests according the following criterion. Is it a real unit test (only one class under test - dependencies are mocked or stubbed) or an integration test (e.g. the type checker uses the parser to parse some input code). According to this question you should put your test in one of the following folders:

/test/ch/tsphp/.../unit/
/test/ch/tsphp/.../integration/

If you are using helper classes (usually only required for integration tests), then put them into the folder testutils under the corresponding category (unit or integration).

Extensibility

In contrast to unit-tests, integration tests should be concise and should not repeat too much code. Or in other words, try to avoid code duplication but still maintain a good balance between readability and over engineering. It should be more or less clear at one glance what a test is doing. It is fine if you write integration tests which extend from an abstract class which provides helper methods etc. Never ever do this for unit-tests. Yet, you should especially consider extensibility when creating integration tests and using a test hierarchy. Try to follow the following principles (regardless if integration test or unit-test):

  1. source out the creation classes to a method (name it createXy where Xy stands for the class)
  2. make those method protected in order that a sub-class could overwrite it
  3. think about the Liskov substitution principle when writing your test class. Imagine one would extend the class under test. If your test is written in a good manner, then this person would only need to create a sub-class of your test and overwrite the corresponding createXy method to test whether he/she adhered to the Liskov substitution principle.

Continuous Integration

This project is going for a continuous integration approach in which the build server supports the developers. Test Driven Development works perfectly together with continuous integration.

What shall be tested?

The tests should cover the important parts of the implementation. Have in mind that we would like to achieve 80% coverage of lines and branches.

Do not write unit-tests for third party components such as ANTLR. They have (hopefully) written their own tests. Of course, if you think there is a bug, then feel free to write a test and contribute to the third party project directly.


So how does the developer know when he/she has tested enough? This cannot be answered easily. Make sure that:

  • You have covered normal cases, boundary conditions and error scenarios without dealing with other classes from the system (real unit-tests) - use stubs and mocks where necessary.
  • You have written integration tests which covers the main functionalities of a component (this may include the functionality of a third party component as well, that is perfectly fine).

Write an overall (blackbox) test for the whole system which covers the main functionalities.

Manual tests shall be avoided whenever possible. If some manual tests are necessary then test cases shall be written for each tests and a test protocol shall be filled in during conducting the test.