Apertis has a slowly growing number of test cases that have been written over the years that the Apertis project has been running. Effort has been made to automate many of these tests, however many still remain as manually run testing scenarios.

Due to limited testing bandwidth, many of these manual tests are not frequently performed. Additionally there may be automated tests that are being run that, due to changes in the focus and direction of the Apertis project since it’s inception, are no longer particularly useful. When performed, such tests, both manual and automated, are consuming valuable resources that could be better utilised elsewhere.

A review of the test cases has been carried out to identify tests that may now be redundant and should therefore be removed. The following table shows the analysis performed for this review along with any suggestions for change:

General status of the test framework

Whilst reviewing the test cases a number of additional A number of additional points have been noted that are mostly out of scope for this review, though may be worth considering for future work:

• A lot of the tests seem to have needless levels of abstraction: A number of strategies have been employed over successive attempts to automate more of the test cases. In order to avoid extra overhead, generally infrastructure used to perform the previously automated tests has been kept and driven from the latest mechanism. In parallel to this, work has been done to provide a testing framework that makes steps carried out by the tests better visibly documented (leading to the test descriptions on qa.apertis.org. Due to may of these tests calling previous mechanisms, these tests are not as transparent as they could be and the (in some cases, multiple) layers of abstraction make them harder to read and maintain. It is suggested that efforts are taken to remove these layers of abstraction where possible.
• It should be possible to merge some of the AppArmor tests with their non-AppArmor counterparts: Quite a few of the AppArmor tests run up the same image and perform the same steps as the non-AppArmor test for the same functionality, only differing in what they look at to deem the test a success or failure. This adds overhead as the infrastructure is setting up and performing the tests twice. This can be reduced by checking for both the outcome of the test steps and the AppArmor status in a single test run without loss of testing coverage.
• Most tests should not be installing packages via apt: A considerable number of the tests are installing packages via apt to for use in the testing. These additions are fundamentally changing the image under test and may affect the results. We have had a mechanism to combat this for a while which needs to be universally rolled out.
• We should update tests using git clone to use wget: Rather than requiring git to be install on the test image, we have recently been downloading archives of the test harnesses with are provided by GitLab. This only requires the ability to download a resource via HTTP, which can be reasonably assumed to be possible in all the images rather than the ability to perform a git clone. Tests should be updated to use this mechanism to avoid the need for git.
• Merge some tests together: In some places we seem to have multiple tests, each testing a facet of a larger set of functionality. Whilst not viable or advisable for all the tests, merging some may be viable which would reduce the overhead of setting up the environment individually each time.