Generate the IDS README “Raw to IDS Mapping” table

Summary

The ## Raw to IDS Mapping table in an IDS README (see ### Raw to IDS Mapping) is generated from a mapping_table.json file, never hand-edited. This page describes mapping_table.json’s structure and how to keep it, schema.json, and the README table in sync.

The structure of mapping_table.json

{
    "header": [
        {"key": "path", "column_name": "IDS Property Name"},
        {"key": "source", "column_name": "Source location"},
        {"key": "notes", "column_name": "Notes"}
    ],
    "body": [
        {
            "path": "samples[*].foo.bar",
            "visible": true,
            "internal_comment": "Something we want to know about this field, not for readme table",
            "source": "constant `42`",
            "notes": "This is a magic number"
        },
        {
            "path": "samples[*].foo.baz",
            "visible": false,
            "source": "constant `Yes`",
            "notes": "This is a magic string"
        }
    ]
}

  • header — the mapping table’s columns. Each item has a key (used in body records) and a column_name (the README column heading).

  • body — one record per schema field. path and visible are required; internal_comment is an optional note that never appears in the generated README; every other key must match a header key.

  • Array indices: use [*] when any number of elements can occur, or [0], [1], … when the index is meaningful or the array is capped at one element.

  • Set "visible": false rather than deleting a record when a schema field is never populated by the task script — this keeps the table in sync with the schema without cluttering the user-facing view.

Keeping schema.json, mapping_table.json, and the README in sync

Note

ts_ids_core.mapping_table.sync (used below) requires the mapping-table extra:

pip install "ts-ids-core[mapping-table]"

Importing it without installing the extra raises ImportError with instructions for installing the extra.

Add a pytest to the IDS repo that regenerates all three files in order — using pytest-steps to guarantee ordering — and run pytest --snapshot-update whenever the schema changes:

from pathlib import Path

from path.to.schema import IdsSchema
from pytest_steps.steps import test_steps
from ts_ids_core.mapping_table.sync import create_mapping_table_json, create_readme

ROOT_FOLDER = Path(__file__).parents[1]


@test_steps("schema.json", "mapping_table.json", "Mapping table in readme")
def test_schema_sync(snapshot) -> None:
    """The content of schema.json matches the schema JSON from the IDS model"""
    # Arrange
    snapshot.snapshot_dir = ROOT_FOLDER
    schema_path = ROOT_FOLDER.joinpath("schema.json")
    mapping_table_path = ROOT_FOLDER.joinpath("mapping_table.json")
    readme_path = ROOT_FOLDER.joinpath("README.md")

    # Check schema.json
    schema_json = IdsSchema.schema_json(indent=2)
    snapshot.assert_match(schema_json, schema_path)
    yield

    # Check mapping_table.json
    mapping_table_json = create_mapping_table_json(
        schema_file=schema_path,
        mapping_table_path=mapping_table_path,
        default_record_visibility=True,
    )
    snapshot.assert_match(mapping_table_json, mapping_table_path)
    yield

    # Check README.md
    readme = create_readme(
        readme_path=readme_path,
        mapping_table_path=mapping_table_path,
        mapping_table_section_heading="Raw to IDS Mapping",
    )
    snapshot.assert_match(readme, readme_path)
    yield

When a schema field is added, pytest --snapshot-update adds a template record to mapping_table.json (path set, visible set to default_record_visibility, other fields blank) for you to fill in; when a field is removed, its record is dropped. Renaming a field is treated as remove + add — update path in mapping_table.json to the new name before running pytest --snapshot-update to preserve the record’s content.

To add or remove a column, add or remove the corresponding item in header and run pytest --snapshot-update — the column is populated with an empty string on every record, or removed from every record.

Bootstrapping mapping_table.json from an existing README

New IDS repo with a hand-written README table but no mapping_table.json yet? Bootstrap one with ts-ids mapping-table convert-readme:

ts-ids mapping-table convert-readme

Note

ts-ids mapping-table convert-readme requires the mapping-table extra:

pip install "ts-ids-core[mapping-table]"

Trying to run it without installing the extra results in an error message explaining how to install it.

For further documentation, run ts-ids mapping-table convert-readme --help.

The resulting mapping_table.json is not checked against schema.json — that’s done separately by the sync pytest above. If there’s a bug in the README table (a non-existent path, a misspelling), data may be lost the first time pytest --snapshot-update runs afterward. To avoid that:

  1. Run ts-ids mapping-table convert-readme to create mapping_table.json.

  2. Stage it: git add mapping_table.json (or commit it).

  3. Synchronize all files: pytest --snapshot-update.

  4. Inspect with git diff:

    • No changes to mapping_table.json — only formatting changes to the README table, as expected.

    • Unintended changes to mapping_table.json — undo with git restore ., fix the relevant record (see below), stage it, and repeat from step 3.

  5. Commit the updated mapping_table.json and README.md.

Known issues and their fixes, using the recipe above:

  • Records re-ordered — row order is determined from schema.json; there’s no way to specify a custom order. To diff ignoring order, use jd: git difftool -x "jd -mset" mapping_table.json.

  • Records erroneously dropped — the path found in the README doesn’t match the path in schema.json; correct path in mapping_table.json.

  • Irrelevant rows added — hide them by setting "visible": false, or temporarily run the sync pytest with create_mapping_table_json(default_record_visibility=False) so all new fields default to hidden.

  • Spacer rows removed — only rows corresponding to a schema field are supported; there’s no way to keep other kinds of rows.

  • A cell contains a literal | — convert-readme can’t parse a table cell containing an unescaped |; edit the README to remove or reword it before converting. (mapping_table.json → README generation, the reverse direction, does support a literal | in a cell — it’s escaped automatically as \|.)