...
In the following sections we will use the word component in the broadest sense, meaning the class, module, system were are testing.
Testcase state
The definition and implementation of automatic tests go though these phazses:
- Specification: No code has been written, but a specification of the test has been written, eg. a description of the test has been made, or some of the test step have been described.
- Test-first: The test implementation is underway, but the test fails because the corresponding application functionality hasn't been implemented.
- Regression test: The Test and corresponding application functionality has been implemented, and the test is now part of the regression testsuite, eg. the test can be used to validate that the code isn't broken.
The test state is declared with the TestNG test group annotation:
| Code Block |
|---|
@Test(groups = { ""specification-only", "testfirst", "regressiontest" }) |
where only one of the groups will be used in a given test state.
Clear purpose
Test should be writing directly target at investigation a specific aspect of the component we are testing. This means a test should clearly reflect a component interface feature, which can be understood by a component user. The name of the test should directly reflect the purpose of the test. Examples are:
...
- Functionality changes not directly related to the test purpose.
- When the test are run are run.
- Race conditions.
- Environment change: This means the tests should pass on all environment variations which might be relevant for the component. Variables here could be OS, language settings, etc.
- Permissions: Tests should not assume they can change anything outside of the directory they are run in, unless explicitly configured in the test.
- Sequence of tests. Test should not be dependent on which test has run, or not run, prior to this test.
Clean output
One of the critical means of debugging the automatic test are the out generated in the applications log file and consule output. To maximize the value of these sources for runtime information, the information generated by the test should be kept separate from the application output. This includes:
- Do not commit test code with and System.out.printlns
- Use the TestLogger class for generating test log output.
- Do not log stacktraces for expected exceptions.