Commands and workflow

clerk build

clerk test

Running the tests

clerk test can be used without any arguments at the root directory of a Catala project, with the following output:

$ clerk test
┏━━━━━━━━━━━━━━━━━━━━━━━━  ALL TESTS PASSED  ━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                                                    ┃
┃             FAILED     PASSED      TOTAL                           ┃
┃   files          0         37         37                           ┃
┃   tests          0        261        261                           ┃
┃                                                                    ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

The tests line of this report counts the number of failed and passed tests. The files line displays the number of files where either all tests pass, or there is at least one failing test.

You can print the details about the files containing passing and failing tests with the --verbose flag. In addition, it is possible to run the command with the flag --xml to obtain a JUnit-compatible report.

You can restrict the scope of which tests are executed by clerk test by providing another argument:

• clerk test <file> will run only the tests in <file>;
• clerk test <folder> will run only the tests inside files in <folder> (or its sub-directories);
• clerk test <target> will run only the tests related to the build <target>.

Declaring the tests

Catala supports two distinct flavors of tests, tailored for different purposes:

• Scope tests should be the main way to write tests that validate expected results on a given computation. This is the natural way to automate the clerk run --scope=TestXxx commands you use to run your tests manually.
• CLI tests provide a way to run custom catala commands and check their output on stdout and stderr. They are sometimes useful for more specific needs, like ensuring the correct error is triggered in a given situation.

The command clerk test can be run on any file or directory holding catala files, will process both kinds of tests and print a report.

Scope tests

A test scope is a scope that is marked with the "test" attribute: simply write #[test] just before its declaration keyword.

#[test]
declaration scope TestMoneyRounding:
  output result content money

There are two requirements for a test scope:

• The scope must be public (declared in a ```catala-metadata section) so that it can be run and checked in real conditions
• It must not require any input: only internal, output and context variables are allowed

The expected output of the test should be validated with assertion statements. Without them, the only thing the test would validate is that the computation doesn't trigger an error.

scope TestMoneyRounding:
  definition result equals $100 / 3
  assertion result = $33.33

As seen in the tutorial, a test scope almost always take the form of a call to the real scope you want to test, providing it with specific inputs and an expected result:

#[test]
declaration scope Test_IncomeTaxComputation_1:
  output computation content IncomeTaxComputation

scope Test_IncomeTaxComputation_1:
  # Define the computation as IncomeTaxComputation applied to specific inputs
  definition computation equals
    output of IncomeTaxComputation with {
      -- individual:
        Individual {
          -- income: $20,000
          -- number_of_children: 0
        }
    }
  # Check that the result is as expected
  assertion computation.income_tax = $4,000

Command-line interface (CLI) tests

This second flavor of tests provides a means to validate the output of a given Catala command, which may be useful in more specific integration scenarios. It is inspired by the Cram test system, in that a single source file includes both the test command and its expected output.

For example, checking that an error happens when expected cannot be done with test scopes, which must succeed. You may want to include tests that make use of catala proof. Or you could want, for a simple test, to validate that the trace is exactly as intended. For this, a ```catala-test-cli section should be introduced in the source Catala file. The first line always starts with $ catala , followed by the Catala command to run on the current file ; the rest is the expected output from the command ; additionally, if the command terminated with an error, the last line will show the error code.

```catala-test-cli
$ catala interpret --scope=Test --trace
[LOG] ☛ Definition applied:
      ─➤ tutorial.catala_en:214.14-214.25:
          │
      214 │   definition computation equals
          │              ‾‾‾‾‾‾‾‾‾‾‾
      Test
[LOG] →  IncomeTaxComputation.direct
   ...
[LOG]   ≔  IncomeTaxComputation.direct.output: IncomeTaxComputation { -- income_tax: $4,000.00 }
[LOG] ←  IncomeTaxComputation.direct
[LOG] ≔  Test.computation: IncomeTaxComputation { -- income_tax: $4,000.00 }
┌─[RESULT]─ Test ─
│ computation = IncomeTaxComputation { -- income_tax: $4,000.00 }
└─
```

When running clerk test, the specified command is run on the file or directory (truncated to that point). If the output exactly matches what is in the file, the tests passes. Otherwise, it fails, and the precise differences are shown side-by-side.

Beware, CLI tests this cannot be used to test backend-generated code; so clerk test --backend=... won't run CLI tests.

clerk ci

clerk run

clerk clean