Testing

Overview

For a TetraScience-maintained protocol repo, e2e_test_configuration.json at the repo root declares end-to-end test cases: it runs the protocol as a real pipeline and compares its output against expected files. This page covers the file’s shape and its limitations; it does not cover the CI mechanics that consume it, which are internal.

A self-service protocol has no equivalent file — validate it with ts-cli publish --dry-run and manual pipeline runs instead.

Shape

{
    "version": "v1",
    "pipeline_configurations": {
        "configuration_A": {
            "idsFileCategory": "IDS",
            "datetimeParsingConfig": {
                "datetime_formats": ["MM/DD/YYYY h:mm:ss A", "DD-MMM-YY h:mm:ss A"]
            }
        }
    },
    "test_cases": {
        "the_test_with_the_really_big_file": {
            "pipeline_configuration": "configuration_A",
            "expected_input_files": ["input_file_A.raw"],
            "expected_output_files": [
                { "output_file_A.json": null }
            ]
        }
    }
}

• version — the file format version, currently "v1".

• pipeline_configurations (optional) — a convenience object to de-duplicate pipeline configurations shared across test cases. Each named entry (configuration_A above is an arbitrary label, not a protocol.yml id) is itself an object, and the keys under that entry must match the config IDs declared in protocol.yml exactly (see protocol.yml field reference) — a mismatch here is one of the most common causes of an e2e test failing silently as a bad pipeline configuration rather than a clear error.

• test_cases — one entry per test case:

  • pipeline_configuration — either a string key into pipeline_configurations, or an inline object of the same shape (keys are config IDs, same rule as above).

  • expected_input_files — a list of input file names, looked up in the first step’s task script repo, under its example-input folder.

  • expected_output_files — a list of {filename: value} objects, looked up in the last step’s task script repo, under its example-output folder (for a single-step protocol, the first step is also the last). value is null for an exact comparison, or an object with an ignore_keys list (below).

Skip / status-only forms

Skip all tests entirely:

{ "version": "v1", "test_cases": {} }

Status-only — confirm the pipeline runs to completion without comparing output content:

"expected_output_files": []

Ignoring specific keys in an output comparison

Useful for fields expected to vary run-to-run, such as generated IDs or non-deterministic array ordering:

"expected_output_files": [
    {
        "output_file_B.json": {
            "ignore_keys": ["methods[*].spectra[*].polarity", "datacube_metadata[*].file_id"]
        }
    }
]

Limitations

• Only JSON output can be content-compared — a non-JSON output (e.g. a raw pass-through file) can only be checked for existence, not content.

• One input file and one output file set per test case — a test case can’t assert on multiple independent output files from a single run.

• Multi-step protocols only compare the final step’s output. An intermediate step producing incorrect output but a correct final result passes undetected.

• Only the first IDS file is compared, if a step produces several.

• Secrets are not supported in pipeline_configurations — a protocol with a type: secret config can’t be fully tested this way; validate that path manually.

• Input and output file names must be globally unique across the whole configuration, even when their paths differ — two test cases both expecting a file named input.raw in different subdirectories will collide.