Platform Requirements

One of the main purposes of ts-ids-core is to enable users to streamline IDS creation by automatically disallowing IDS definitions which are incompatible with the TetraScience platform. Publishing an IDS to the TetraScience platform can be done using the TetraScience software development kit, ts-sdk, and is often done while creating a self-service pipeline.

Each IDS project should contain the required file set that completes an IDS artifact. Upon publishing an IDS to the platform, schema.json, athena.json, and elasticsearch.json are validated that each document conforms to its respective requirements detailed below. Under the hood this validation is performed by TetraScience’s publicly available Python package, ts-ids-validator. Note that ts-ids-validator currently validates a larger rule set than ts-ids-core enforces and ts-ids-core only enforces definitions for schema.json. See the table below to understand Platform requirements of IDS artifacts and the differences between what is validated on publishing an IDS vs. what is automatically enforced using ts-ids-core.

schema.json platform requirements

#	Rule	Checked by ts-ids-validator	Enforced in ts-ids-core
1	All field names should be snake case .	Yes	No
2	Field names cannot have any special characters other than non-leading single underscores with the exception of TetraScience defined fields leading with @ (e.g. @idsType).	Yes	No
3	Every object type field should always have "additionalProperties": false to prevent random fields from being added.	Yes	Yes
4	If a field is in the required array of an object, it also needs to be defined in the object’s properties field.	Yes	Yes
5	Fields with type object or array cannot also have a second type. Fields with atomic types (i.e. number, string, boolean) may have a second type of null only.	Yes	Yes
6	@idsNamespace, @idsType, @idsVersion need to be in the top-level required field value array and must not be null and must have a constant value.	Yes	No
7	@idsVersion must follow semantic versioning. For information on how to version you schemas, see below.	Yes	Yes when inheriting from IdsSchema or TetraDataSchema
8	When updating an existing IDS the @idsVersion value change must follow the IDS versioning rules below	Yes	No
9	If datacubes is present, it has to have the required fields defined in the IDS Design Conventions - Schema Templates#datacubes-Pattern	Yes	Yes when defining the datacubes field using the DataCube component or a subclass of the component
10	If datacubes is present, maxItems has to equal to minItems, in both datacubes[*].measures and datacubes[*].dimensions	Yes	Yes when defining the datacubes field using the DataCube component or a subclass of the component
11	If datacubes is present, the shape of measures[*].value matches the number of dimensions (e.g. a datacube with 2 dimensions must have a measures[*].value array which represents a 2D array).	Yes	Yes when defining the datacubes field using the DataCube component or a subclass of the component

elasticsearch.json platform requirements

#	Rule	Checked by ts-ids-validator
1	datacubes, if defined in schema.json, should be in nonSearchableFields.	Yes
2	All fields defined in elasticsearch.json must exist in the IDS artifact’s schema.json file.	Yes
3	The mapping section in elasticsearch.json must match the auto-generated output from the elasticsearch.json generator, unless allowCustomMapping is set to true. See allowCustomMapping for details.	Yes

athena.json platform requirements

#	Rule	Checked by ts-ids-validator
1	The partition name cannot collide with any of the normalized IDS paths. See details here.	Yes
2	The partition path cannot be a field in an array of objects. See details here.	Yes
3	The partition path must correspond to a field in the IDS artifact’s schema.json file.	Yes

IDS versioning rules

It is important to version your IDSs in a way that is platform compliant. As mentioned in rule #7 above, it is required that an IDS contains the field @idsVersion whose value is a version string which follows the semantic versioning specification. Semantic versioning rules specific to IDSs follow. It’s necessary to follow these rules, otherwise you may not see changes reflected in the platform even though you made the change in your IDS.

Major version change

1. Add new fields or add new table

2. Data fields are removed, renamed, or re-structured

3. Data field primitive type is modified (e.g. "string" -> "number")

4. Disallow a value to be null (e.g. {"type": ["string", null]} -> {"type": "string"})

5. Making a non-required field required

6. Adding a constraint (e.g. "maximum": 100)

Minor version change

There are no platform-specific needs for a minor version change. Versioning change to the minor version is up to the user’s rule set.

Patch version change

1. Making a required field non-required (note required here refers to the required array in an object)

2. Add support for null type to a field (e.g. {"type": "string"} -> {"type": ["string", null]})