Testing¶

Overview¶

For a TetraScience-maintained protocol repo, e2e_test_configuration.json at the repo root declares end-to-end test cases: it runs the protocol as a real pipeline and compares its output against expected files. This page covers the file’s shape and its limitations; it does not cover the CI mechanics that consume it, which are internal.

A self-service protocol has no equivalent file — validate it with ts-cli publish --dry-run and manual pipeline runs instead.

Shape¶

{
    "version": "v1",
    "pipeline_configurations": {
        "configuration_A": {
            "idsFileCategory": "IDS",
            "datetimeParsingConfig": {
                "datetime_formats": ["MM/DD/YYYY h:mm:ss A", "DD-MMM-YY h:mm:ss A"]
            }
        }
    },
    "test_cases": {
        "the_test_with_the_really_big_file": {
            "pipeline_configuration": "configuration_A",
            "expected_input_files": ["input_file_A.raw"],
            "expected_output_files": [
                { "output_file_A.json": null }
            ]
        }
    }
}

version — the file format version, currently "v1".
pipeline_configurations (optional) — a convenience object to de-duplicate pipeline configurations shared across test cases. Each named entry (configuration_A above is an arbitrary label, not a protocol.yml id) is itself an object, and the keys under that entry must match the config IDs declared in protocol.yml exactly (see protocol.yml field reference) — a mismatch here is one of the most common causes of an e2e test failing silently as a bad pipeline configuration rather than a clear error.
test_cases — one entry per test case:
- pipeline_configuration — either a string key into pipeline_configurations, or an inline object of the same shape (keys are config IDs, same rule as above).
- expected_input_files — a list of input files. A bare name is looked up in the first step’s task script repo, under its example-input folder; a file may also live outside the repo, see Example files outside the repo.
- expected_output_files — a list of {filename: value} objects. A bare name is looked up in the last step’s task script repo, under its example-output folder (for a single-step protocol, the first step is also the last), and may likewise come from outside the repo. value is null for an exact comparison, or an object with an ignore_keys list (below).
- description (optional) — a one-line summary of what the case covers, reported with the results (below).
- associated_jira_stories (optional) — Jira issue keys this case verifies, reported with the results (below).

Skip / status-only forms¶

Skip all tests entirely:

{ "version": "v1", "test_cases": {} }

Status-only — confirm the pipeline runs to completion without comparing output content:

"expected_output_files": []

Ignoring specific keys in an output comparison¶

Useful for fields expected to vary run-to-run, such as generated IDs or non-deterministic array ordering:

"expected_output_files": [
    {
        "output_file_B.json": {
            "ignore_keys": ["methods[*].spectra[*].polarity", "datacube_metadata[*].file_id"]
        }
    }
]

Describing a test case and the work it covers¶

Two optional keys per test case say what the case is for. Neither changes what is compared — they only travel with the results, so a failure names the scenario and the story behind it instead of a file name alone:

"test_cases": {
    "the_test_with_the_really_big_file": {
        "pipeline_configuration": "configuration_A",
        "expected_input_files": ["input_file_A.raw"],
        "expected_output_files": [
            { "output_file_A.json": null }
        ],
        "description": "Multi-plate export, 384 wells per plate",
        "associated_jira_stories": ["DE-1234", "DE-1250"]
    }
}

description — one line on what this case covers. It is shown against every file the case declares, next to that file’s status and its file ID in the platform.
associated_jira_stories — the Jira issue keys this case verifies, as keys ("DE-1234"), not URLs or numeric IDs. They are reported as links, and the run is linked back to those issues so the test result is visible from the story.

Both are per test case, so a file listed in expected_input_files inherits the description of the case that lists it. A test case declaring neither reports exactly as it did before — these keys are additive, and existing configuration files need no change.

Example files outside the repo¶

An example file does not have to live in the task script repo. Large binary files and client-provided files sit badly in a git repo — especially one that may be shared outside TetraScience — and a repo often holds only a subset of the samples that exist for an instrument.

The source is decided by the shape of the string; there is no type field:

Value	Read from
`"plate 4.xls"`	the task script repo, as before
`"s3://ts-protocols-example-files/<task-script-slug>/example-input/plate 4.xls"`	AWS S3
`"s3://<bucket>/<key>?versionId=<id>"`	one exact S3 object version
`"gdrive:<file-id>"`	Google Drive, anyone-with-link
`"gdrive:<file-id>#plate 4.xls"`	the same, with a readable name in reports
`"https://drive.google.com/file/d/<id>/view"`	Google Drive share link

Any other https:// address is rejected: a test input an arbitrary web host can change is a test whose expected result that host can change.

Existing configurations need no change — a bare name means the same repo file it always did.

Uploading to S3¶

Bucket ts-protocols-example-files, laid out as:

ts-protocols-example-files/<task-script-slug>/example-input/<file>
ts-protocols-example-files/<task-script-slug>/example-output/<file>

Subfolders are kept. Input files resolve against the protocol’s first step and output files against its last — the same split as the repos they would otherwise come from. Put the full s3://… path in the configuration; the layout above is where files should go, not something inferred, so a file elsewhere in the bucket still works and only gets a less specific name in reports.

Versions, and files that change¶

The bucket has object versioning enabled. Each run records, per file, the source, the full location, the S3 object version read and a SHA-256 of the content. The next run compares against that record and warns when an example file has changed:

example file changed since last recorded run: s3:example-output/plate 4.json
  (version 3HL4kq… -> 8dRmpQ…)

A warning, not a failure — updating a sample file is legitimate. The point is that replacing an expected-output file no longer changes what a test asserts with nothing in a diff to show for it.

To freeze a file, pin the version (copy it from the S3 console, Object → Versions):

"expected_input_files": [
    "s3://ts-protocols-example-files/<slug>/example-input/plate 4.xls?versionId=3HL4kqCxf3vjVBH40Nrjfkd"
]

Google Drive¶

Drive works, with real limits:

Anyone-with-link only. No Google credentials are used, so a file shared with named people or with the organisation cannot be read; the run fails saying so. Set sharing to Anyone with the link → Viewer.
No version tracking. Drive revision ids need the Drive API, which link sharing does not reach, so provenance is the content hash alone and the version pin above is unavailable.
No paths. A reference is a file id. Use the optional #<name> suffix so reports read gdrive:example-input/plate 4.xls rather than a 33-character id.

Prefer S3 for anything sanitized or customer-derived. “Anyone with the link” means exactly that; the S3 bucket is access-controlled. Sanitize customer-derived samples before uploading either way.

When a file cannot be read¶

Failures name their cause:

Reported as	Usual fix
Example file not found at its source	the path, or the file was moved or deleted
Access denied reading an example file	the S3 bucket policy, or a Drive file not shared as anyone-with-link
Example file reference names an unsupported source	the reference form — see the table above

A pinned versionId that no longer resolves reports the same way whether it was mistyped or the version was permanently deleted; S3 does not distinguish the two.

Limitations¶

Only JSON output can be content-compared — a non-JSON output (e.g. a raw pass-through file) can only be checked for existence, not content.
One input file and one output file set per test case — a test case can’t assert on multiple independent output files from a single run.
Multi-step protocols only compare the final step’s output. An intermediate step producing incorrect output but a correct final result passes undetected.
Only the first IDS file is compared, if a step produces several.
Secrets are not supported in pipeline_configurations — a protocol with a type: secret config can’t be fully tested this way; validate that path manually.
Input and output file names must be globally unique across the whole configuration, even when their paths differ — two test cases both expecting a file named input.raw in different subdirectories will collide, because the platform stores and reports the file under its bare name. An S3 file held outside the <slug>/example-{input,output}/ layout is identified by its bare name for the same reason; two such files sharing a name stop the run when it starts, rather than one silently replacing the other.