Every new feature requires one or more validation tests to be accepted into MPQC. This helps to detect logic regressions and to maintain the feature as the code evolves. If there is no validation test then the feature is likely to become nonfunctional; features that are not tested regularly effectively do not exist.

The architecture of the validation test suite

• The validation suite is contained in tests/validation, with the the test inputs and outputs located under tests/validation/reference/inputs and tests/validation/reference/outputs, respectively.
• The test suite is set up in tests/validation/CMakeLists.txt; e.g. this file filters tests to be run based on the configuration of the MPQC executable (see MPQC_FEATURES CMake variable). Note that to be able to do this the test names must follow particular patterns defined in this file. You will need to modify this file if you added a new major MPQC feature (i.e. it has its own library).
• Excluded tests are listed in tests/validation/CTestCustom.cmake; you should not need to modify this file.
• Test runner is defined in tests/validation/test.cmake; this is a CMake script that runs the tests, invokes the validator (see next), and signals the result to the user. You should not need to modify this file.
• Validator is defined in tests/validation/check.py. You will need to modify this file if you add a test whose target property is not one of the currently supported properties.

The following targets execute validation tests:

• check_validation runs short-duration (see below) validation tests only; this is the target invoked by the top-level target check.
• check_validation_full runs all validation tests
• check_validation_long runs long-duration validation tests only

Guidelines for constructing validation tests

• Most validation tests should have short duration, e.g. a second or two. Most validation tests should belong to this category.
• In practice this may not be possible to make all tests have short duration. e.g. computations on 3-d solids even in a minimal basis are likely to exceed this limit. To indicate that a particular validation test is long-duration add LongTest to its name. E.g., CCSDT cc-pVQZ calculation on water should be named h2o-ccpvqz-ccsdt-LongTest.json.
• Reference outputs for each test should be evaluated with higher precision than the target precision. E.g. if the test will expect energy precise to 1e-9 the reference energy should be computed to 1e-10 or ever tighter, to ensure that the deviations from the reference energy of 1e-9 are meaningful.