Tests

Introduction

The design of cpp-verifier relies on different modules. In order to trust these, one can run test scripts which will test the output of the different functions of each module against the output of the reference implementation (written in python). If they both "agree", then it is highly likely that the module works as intended.

Furthermore, a test of the complete program can be ran using test data generated using the standard verificatum implementation.

Running the tests on the modules

Once the program is compiled, simply go in the directory ./tests of your cpp-verifier installation. There, run the bash script test.sh (sorry non-UNIX-fellas). It will copy the compiled test suits from the build tree, past them alongside with their test vectors and then run the python scripts using the reference implementation in Python to check that they work.

The output's structure should be as follows, where ./<module folder> goes through the list of the modules and class.method iterates over the methods whose test are implemented.

===== TESTING ./<module folder> =====
testing class.method1
[OK]
...

You can also test each module individually by running test.sh <module folder>.

Testing the complete program

When running test.sh main or at the end of test.sh, a test of the complete program is performed. It is ran in verbose mode and the output is checked against what is expected. This gives rise, if everything works smoothly, to the following output:

===== TESTING CORE VERIFICATION =====
Core verification [OK]

Testing process

No matter which module you test, things always go like this (note that you are expected in the <cppvv directory tree>/tests folder.

1. test.sh runs the script ./<module name>/test.sh
2. This last script calls a unitary test for each function tested within the module by running a Python script ./unitarytest.py <class.method>
3. The python scripts creates two files using code from the standard implementation.
   • input.txt which contains the input to be parsed by the test program.
   • ideal_output.txt contains what the test program should output.
4. The test binary is called by the python script with CLI argument <class.method>. It reads the content input.txt and writes the corresponding result in actual_output.txt.
5. The python script finishes by running a simplified diff to compare ideal_output.txt and actual_output.txt. If the files are identical, writes [OK].

Possible improvements

Right now, the tests are based on a mix of bash and Python. A good idea would probably be to go for a full Python solution (it would probably also lead to some interesting code factorization opportunities). I also considered incorporating the tests in the code itself using a C++ testing framework but, since the reference implementation is in Python, I prefered to go that way.