Testing¶

This document describes various guidelines and best practices for automated testing of Nextbit projects.

It has been inspired by the Gitlab’s guide to testing.

Why writing tests?¶

Good tests can…

Verify the code is working correctly
Prevent future regressions
Document the code’s behavior
Provide design guidance
Support refactoring

More details about these five points are described in the Five Factor Testing guide.

For more information about levels of testing, tests prioritization, and cost of tests, you should read the Principles of Automated Testing.

Testing levels¶

This diagram demonstrates the relative priority, the amount, and the speed of the tests you should write.

Unit tests¶

Formal definition: https://en.wikipedia.org/wiki/Unit_testing

These kind of tests ensure that a single unit of code (a method) works as expected (given an input, it has a predictable output). These tests should be isolated as much as possible.

These should be the most numerous tests you write, because they assure the single methods do what you what them to do.

Based on the language you are using, there are different libraries you should make use of.

Integration tests¶

Formal definition: https://en.wikipedia.org/wiki/Integration_testing

These kind of tests ensure that individual parts of the application work well together, without the overhead of the actual app environment (i.e. the browser).

You need those, because even if the single components work well, it doesn’t mean they work well together, like these two drawers: the unit tests pass, since they do their work as drawer, but the situation cleary needs an integration test.

Black-box tests at the system level, aka end-to-end tests¶

Formal definitions:

End-to-end testing is a strategy used to check whether your application works as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together.

Since they need the entire infrastracture, are quite slow to setup and run.

Note that:

knowledge of the internals of the application are not required
data needed for the tests can only be created using the GUI or the API, to be sure all the system works as expected
expectations can only be made against the browser page and API responses

Testing best practices¶

As many things in life, deciding what to test at each level of testing is a trade-off:

Unit tests are usually cheap, and you should consider them like the basement of your house: you need them to be confident that your code is behaving correctly. However if you run only unit tests without integration / system tests, you might miss the big picture!
Integration tests are a bit more expensive, but don’t abuse them. A system test is often better than an integration test that is stubbing a lot of internals.
System tests are expensive (compared to unit tests), even more if they require a JavaScript driver.

Another way to see it is to think about the cost of tests, this is well explained in this article and the basic idea is that the cost of a test includes:

The time it takes to write the test
The time it takes to run the test every time the suite runs
The time it takes to understand the test
The time it takes to fix the test if it breaks and the underlying code is OK
Maybe, the time it takes to change the code to make the code testable.

Software¶

As usual, you are free to use the software you like more. Anyhow, here you can find some suggestions of suites we are already using.

Javascript¶

Javascript world has thousands of libraries; among all of them, one that shine is Jest. If you are Vue.js and vue-cli, you can add support for Jest using the @vue/cli-plugin-unit-jest library. React supports Jest out of the box, as stated by the official guide, since they are both developed by Facebook. Jest has support for coding coverage out of the box.

Python¶

The standard ``unittest` <https://docs.python.org/3/library/unittest.html>`_ library in Python is usually more than enough; however, it doesn’t support code coverage, for which we suggest coverage.py.

Others¶

Feel free to add your suggestions for others languages / frameworks, following the contributing guide.

Gitlab CI¶

We run our tests through Gitlab CI, as we do for all the CI/CD pipelines, as described in the dedicated page.

For a general introduction to Gitlab CI, please read this guide. For any question, ask on our #help channel on Teams.