03. December 2020 4 min read

Get number of failing JUnit tests in your project under control

We can all agree that sometimes you just remember a valid test, but your implementation is not on level to pass it. Now you face a dilema: do I commit a test and fail the build, or I waste some more time to implement a solution? The usual answer is: implement a solution, mainly because most of the test frameworks fail the build, if there is a failing test. Failing test to fail the build is logical - it prevents regression, but so do the Melexis Warnings plugin, except that it enables you to set a limit of how many tests are actually expected to fail. It does not matter what you use for building, from GitHub Actions, to GitLab CI, Travis CI, Circle CI, ... there is always a place for the command line tool, that will fail your build when number of warnings is higher (or lower) then expected.

Setup warnings limit for test frameworks that output in JUnit format

First you need to install the tool through pip in your CI file (or just install it in your Docker container)

pip3 install mlx.warnings

Plugin has two options of usage. The easiest is with command line option, where you add your JUnit unit test summary right after with minimum and maximum number of warnings allowed (or you can use exact number of warnings flag). That basically means any change in number of failed JUnit results now fails the build. I even have a tendency to make them environment variables, so that they can be different among different stages of the project (maybe development has more/less warnings than release), but you can start with hard-coding.

Fail your build when there are less than 5 JUnit tests marked as failures in yourJunitFile.xml:

mlx-warnings --junit --min-warnings 5 yourJunitFile.xml

Fail your build when there are more than 5 JUnit tests marked as failures in yourJunitFile.xml:

mlx-warnings --junit --max-warnings 5 yourJunitFile.xml

Fail your build when there are not 5 JUnit tests marked as failures in yourJunitFile.xml

mlx-warnings --junit --exact-warnings 5 yourJunitFile.xml

The second option is that you use the json file. That option is mostly helpful when you have a library that is shared between number of projects and you do not want to set limits in each of the projects, but simply forward the json file from the library to your command. Then content of the json file located in your library would be:

{
    "junit": {
        "enabled": true,
        "min": 5,
        "max": 5
    },
}

and the command for the library documentation build would be:

mlx-warnings --config path/to/library/config.json yourJunitFile.xml

This is your solution of writing a test case which currently fails, but still a safety which prevents you from adding more failing tests or even worse: miss a regression. Maintain the control, but also enhance your testing with the help of a simple command line tool to keep failures at exact number.