Platform Requirements

One of the main purposes of ts-ids-core is to enable users to streamline IDS creation by automatically disallowing IDS definitions which are incompatible with the TetraScience platform. Publishing an IDS to the TetraScience platform can be done using the TetraScience software development kit, ts-sdk, and is often done while creating a self-service pipeline.

Each IDS project should contain the required file set that completes an IDS artifact. Upon publishing an IDS to the platform, schema.json, athena.json, and elasticsearch.json are validated that each document conforms to its respective requirements detailed below. Under the hood this validation is performed by TetraScience’s publicly available Python package, ts-ids-validator. Note that ts-ids-validator currently validates a larger rule set than ts-ids-core enforces and ts-ids-core only enforces definitions for schema.json. See the table below to understand Platform requirements of IDS artifacts and the differences between what is validated on publishing an IDS vs. what is automatically enforced using ts-ids-core.

schema.json platform requirements

#	Rule	Checked by ts-ids-validator	Enforced in ts-ids-core
1	All field names should be snake case .	Yes	No
2	Field names cannot have any special characters other than non-leading single underscores with the exception of TetraScience defined fields leading with @ (e.g. @idsType).	Yes	No
3	Every object type field should always have "additionalProperties": false to prevent random fields from being added.	Yes	Yes
4	If a field is in the required array of an object, it also needs to be defined in the object’s properties field.	Yes	Yes
5	Fields with type object or array cannot also have a second type. Fields with atomic types (i.e. number, string, boolean) may have a second type of null only.	Yes	Yes
6	@idsNamespace, @idsType, @idsVersion need to be in the top-level required field value array and must not be null and must have a constant value.	Yes	No
7	@idsVersion must follow semantic versioning. For information on how to version your schemas, see the IDS versioning rules.	Yes	Yes when inheriting from IdsSchema or TetraDataSchema
8	When updating an existing IDS the @idsVersion value change must follow the IDS versioning rules	Yes	No
9	If datacubes is present, it has to have the required fields defined in the IDS Design Conventions - Schema Templates#datacubes-Pattern	Yes	Yes when defining the datacubes field using the DataCube component or a subclass of the component
10	If datacubes is present, maxItems has to equal to minItems, in both datacubes[*].measures and datacubes[*].dimensions	Yes	Yes when defining the datacubes field using the DataCube component or a subclass of the component
11	If datacubes is present, the shape of measures[*].value matches the number of dimensions (e.g. a datacube with 2 dimensions must have a measures[*].value array which represents a 2D array).	Yes	Yes when defining the datacubes field using the DataCube component or a subclass of the component
12	If both datacubes and datacube_metadata are present, minItems and maxItems must be identical between them, for both dimensions and measures. Both are ingested into one Lakehouse table, so the counts have to agree. See Datacubes and Results.	No	No

elasticsearch.json platform requirements

#	Rule	Checked by ts-ids-validator
1	datacubes, if defined in schema.json, should be in nonSearchableFields.	Yes
2	All fields defined in elasticsearch.json must exist in the IDS artifact’s schema.json file.	Yes
3	The mapping section in elasticsearch.json must match the auto-generated output from the elasticsearch.json generator, unless allowCustomMapping is set to true. See allowCustomMapping for details.	Yes

athena.json platform requirements

#	Rule	Checked by ts-ids-validator
1	The partition name cannot collide with any of the normalized IDS paths. See details here.	Yes
2	The partition path cannot be a field in an array of objects. See details here.	Yes
3	The partition path must correspond to a field in the IDS artifact’s schema.json file.	Yes