Testing ¶
The test suite is divided into broad categories of tests including:
-
Unit tests
-
Integration tests
-
Acceptance tests
-
Performance tests
The performance tests exist to aid development of performance-critical components.
Tests can be run using Pytest .
If you’re using PyCharm then tests should run directly by right clicking on the respective folder (or top-level tests folder) and clicking ‘Run pytest’.
Alternatively you can use the
pytest
.
command from the root level tests directory, or the other subdirectories.
Mocks ¶
Unit tests will often include other components acting as mocks. The intent of this is to simplify
the test suite to avoid extensive use of a mocking framework, although
MagicMock
objects are
currently used in particular cases.
Code Coverage ¶
Code coverage output is generated using
coverage
and reported using
codecov
.
High test coverage is a goal for the project however not at the expense of appropriate error handling, or causing “test induced damage” to the architecture.
There are currently areas of the codebase which are impossible to test unless there is a change to the production code. For example the last condition check of an if-else block which would catch an unrecognized value, these should be left in place in case there is a change to the production code - which these checks could then catch.
Other design-time exceptions may also be impossible to test for, and so 100% test coverage is not the ultimate goal.
Excluded code coverage ¶
The
pragma:
no
cover
comments found throughout the codebase
exclude code from test coverage
.
The reason for their use is to reduce redundant/needless tests just to keep coverage high, such as:
-
Asserting an abstract method raises
NotImplementedError
when called. -
Asserting the final condition check of an if-else block when impossible to test (as above).
These tests are expensive to maintain (as they must be kept in line with any refactorings), and
offer little to no benefit in return. However, the intention is for all abstract method
implementations to be fully covered by tests. Therefore
pragma:
no
cover
should be judiciously
removed when no longer appropriate, and its use
restricted
to the above cases.