IDS versioning rules

It is important to version your IDSs in a way that is platform compliant. As mentioned in rule #7 of the schema.json platform requirements, it is required that an IDS contains the field @idsVersion whose value is a version string which follows the semantic versioning specification. Semantic versioning rules specific to IDSs follow. It’s necessary to follow these rules, otherwise you may not see changes reflected in the platform even though you made the change in your IDS.

General guidelines

These rules generally follow SemVer 2.0.0, with modifications to fit the TetraScience platform — several changes classified as non-breaking below would be breaking under plain semver (e.g. removing or renaming a field), so don’t rely on semver intuition alone.

Major version: for non-backward-compatible changes — the existing data searching and querying using ES, SQL, or IDS JSON files stops working (returning errors). If the query still works but returns a null value, it’s not a breaking change.

Minor version: for backward-compatible non-trivial feature updates. Customers’ existing queries will continue to work, but some columns’ values may change if the same raw file is reprocessed.

Patch version: for backward-compatible bug fixes or small updates.

See the IDS README changelog for the specific field-level changes in each release, to check whether a given version will impact your existing queries.

Starting from Tetra Data Platform (TDP) v4.1, each IDS has a separate set of Athena tables, backed by Delta Lake. With features like schema evolution, adding new fields is no longer a breaking change for those Athena tables — though it’s still a breaking change for the legacy CSV-based Athena tables. The rules on this page reflect the Delta Lake behavior; upgrade to TDP v4.1 or later to take advantage of it.

Breaking changes that require a major version update

schema.json

  • Data field primitive type is modified — a field cannot have more than one primitive type (string, number, boolean).

elasticsearch.json

  • Add or remove nested from a field — existing ES queries will not work.

  • Update dynamic_templates.

  • Add fields to nonSearchableFields — existing ES queries will not return the same result, which could break code or integrations expecting a consistent data interface.

athena.json

  • Any change to athena.json that causes an update of data, such as adding/updating the root table or adding/updating partitions.

Non-breaking changes that can be a minor or patch version update

schema.json

  • Add new fields

  • Remove existing fields

  • Rename existing fields

    Remove/rename is non-breaking only because ES mappings and Delta Lake schema evolution keep the old field present (as a nullable column) rather than dropping it — existing queries for that field keep working and return null, matching the “null, not error” bar for a major bump above. If a removed/renamed field can’t degrade this way (e.g. it’s a required Athena partition key, or the legacy CSV-backed Athena tables — see the ts-ids-validator warning below), treat it as a major version change instead.

  • Updating a required field to be optional or vice versa (note required here refers to the required array in an object)

  • Switch property type between nullable and non-nullable

  • Set additionalProperties to true — note rule #3 in the schema.json platform requirements table still requires every object type field to have "additionalProperties": false, so ts-ids-validator will fail validation regardless of version bump if this is set to true.

When run with --download or --git, ts-ids-validator will still warn (not fail) on a minor/patch field addition or removal, since the legacy CSV-backed Athena tables (see above) can’t reflect it without a major bump — that warning doesn’t mean the version bump is wrong. Note that -p/--previous-ids-dir alone can’t see the full version range and will fail the Lakehouse merge-compatibility check regardless; use --download or --git for real version validation.

elasticsearch.json

  • Remove fields from nonSearchableFields

  • Inclusion of newly added fields (in the current version) in nonSearchableFields, such that it’s not a data change

athena.json

  • N/A

No version bump needed

  • Update example.json

  • Update example files

See also task script versioning and protocol versioning for how these rules propagate up the stack.