Release notes

v2.1.0

• Add semantic module with concept class to represent a semantic entity (e.g. from a controlled vocabulary) with a unique identifier.

v2.0.0

• Merge lakehouse_schema into schema to consolidate the trial lakehouse_schema models into the main schema module

  • ts_ids_core.schema.run.InstrumentStatus has been renamed to ts_ids_core.schema.run.RunStatus, and it is no longer part of the ts_ids_core.schema.run.Run definition.

  • ts_ids_core.schema.sample.Lot class has been removed.

• Documentation is no longer packaged with ts-ids-core. The ts-ids docs command now opens the public documentation site at ids.tetrascience.com.

v1.1.2

• Nest properties and labels as object arrays in the lakehouse_schema.Sample schema class consistent to schema.Sample.

v1.1.1

• Fix incorrect alias and add descriptions to set, lot, location and compound fields in the Sample schema class.

• Fix the primary_key reference in Firmware lakehouse_schema class in order to point to a systems object.

v1.1.0

• Add description text to all field definitions.

• Add modified core schema components run, sample, system and user in ts_ids_core.lakehouse_schema to support a normalized schema standard. At this release this should be considered a beta feature subject to change.

v1.0.2

• Fix is_tetra_data_schema: true being missing from TetraDataSchema’s JSON Schema metadata when using Pydantic 2.10 or later, caused by a change in Pydantic’s JSON Schema generation. This metadata indicates that a schema is following certain conventions. Any TetraDataSchema created using ts-ids-core >=0.7.0,<=1.0.1 with Pydantic >=2.10 (released Nov 20th 2024) may be missing this metadata, and can be fixed by updating to this patch.

v1.0.1

• Many Tetra Data components represent the same concepts available in Tetra’s Recommended Labels and Key Context Terms available in the TetraConnect Hub. These descriptions were updated to match their corresponding context term or label: Project, Experiment, Run, Batch, Sample, Compound and Assay.

• Deprecated the use of Lot which will be removed in a future version because it is not included in Key Context Terms or Recommended Labels.

• Improved descriptions of the differences between a system ID, serial number and name in the System component.

• Updated dependencies to exclude Pydantic versions v2.10.0 to 2.10.2 which contain a JSON Schema generation bug.

• Modified JSON Schema generation logic to fix how Enums are serialized to JSON Schema from Pydantic v2.7 onward. When an Enum contains some elements including at least one None element, its JSON Schema should include both the non-null element type and the "null" type to create a valid IDS. For example, the Modifier component includes an enum of strings like ">=" as well as None, so its JSON Schema type is "type": ["string", "null"], Pydantic 2.7 started omitting "type" from the schema in this case, so ts-ids-core has been updated to restore the expected JSON Schema type for Enums.

v1.0.0

• Release v1.0.0, no changes from v0.7.0.

v0.7.0

• Added descriptions for multiple components in runs, systems, samples, related files and quantities (ValueUnit) with no functional change.

• Created project module with recommended component usage

  • Added Assay

  • Added Project

  • Added ProjectAttributes

• Moved Experiment from the ts_ids_core.schema.experiment module to the project module and deleted the ts_ids_core.schema.experiment module

• Changed CLI interface by grouping all commands under the ts-ids group command, see the ts-ids group page for more.

  • The previous import-schema and export-schema commands are deprecated. It’s recommended users now call these commands with ts-ids preceding the command (e.g. ts-ids export-schema [options]).

• Package documentation is now distributed with the package. Review the ts-ids docs command for more.

• Removed @idsConventionVersion from TetraDataSchema in favor of designating a schema following TetraScience standards with the schema metadata is_tetra_data_schema: true. Inheriting TetraDataSchema will automatically add is_tetra_data_schema: true as top-level schema metadata.

v0.6.1

• First release to external Python package index. No changes.

v0.6.0

• Fixed issue in Python 3.8 and 3.9 where the Literal type was not recognized if Literal was imported from typing during type validation. Now typing.Literal and typing_extensions.Literal are treated equivalently.

• Fixed a case where UUIDForeignKey validation did not run if the foreign key was inside a Required IdsElement field, for example if Foo contained a foreign key field, and Bar had a field foo: Required[Foo], then foreign key validation of Foo did not run, and now it does.

• Removed implicit default values for required literal fields, e.g. field: Required[Literal["foo"]] no longer has an implicit default of "foo", instead it needs to be explicitly defined as field: Required[Literal["foo"]] = "foo" or field: Required[Literal["foo"]] = IdsField(default="foo"). This allows type checkers to recognize the default, which isn’t possible with implicitly defined defaults. This also makes it possible to define a required constant which does not have a default value, which is closer to the default Pydantic functionality.

• Changed default value of IdsElement.model_dump_json’s by_alias argument to True, which is the intended way to use ts-ids-core: fields which use an alias, such as ids_type: Required[Literal["foo"]] = IdsElement(default="foo", alias="@idsType") will be converted to a JSON instance as {"@idsType": "foo"} (by_alias=True) by default now, instead of {"ids_type": "foo"} (by_alias=False). This gives it the same default it has for the other schema and instance serialization methods (model_dump and model_json_schema).

Component updates

• Updated the Sample component to add a non-required field compound, which is an object containing an id and name fields - see the linked documentation for full details.

• Updated the Run component to add an id field - see the linked documentation for full details.

• Updated the Experiment component’s id field by renaming the variable to id_ and giving it an alias of id. This avoids clashing with the Python builtin id in code, while still appearing as "id" in JSON Schema and JSON data.

v0.5.0

Breaking changes

Pydantic v2 introduces multiple breaking changes, but the following changes cover most of what is relevant for updating programmatic IDSs:

• Defining constants

  • Previously, constants were defined on a field using default and const like this: field: str = IdsField(default="my-value", const=True). Now they are defined using a Literal type annotation like this: field: Literal["my-value"]. See Pydantic docs for more.

• Defining lists of fixed length

  • Lists of fixed length can be defined using conlist whose keyword arguments changed in Pydantic v2. Previously they were defined like this: field: conlist(str, min_items=3, max_items=3). In the Pydantic v2 syntax, this changes to field: conlist(str, min_length=3, max_length=3). See Pydantic docs for more.

  • The above is the simplest change, but there is also a new alternative syntax which provides better IDE type checking support, see Composing types via Annotated in the pydantic docs. This is equivalent to the field above: field: Annotated[List[str]], annotated_types.Len(3, 3)] which type checkers recognize as List[str]. Note that conlist implied the field’s type is a List whereas it is explicitly stated as List[str] within the Annotated type. Please see the Improvements section for an improvement in the ease of defining a constrained annotated type with a fixed length.

• JSON Schema keyword "description" changes

  • In Pydantic v2, descriptions for objects are no longer inherited from parent classes. This means, for example, creating a custom system class like class MySystem(System): ... will no longer inherit the docstring from System as a ‘description’ when it is exported as a JSON schema. Docstrings need to be explicitly added to each class in which you want to define a description within the JSON schema. When updating an existing programmatic IDS using a previous ts-ids-core version, find which descriptions need updating by taking a diff of the exported schemas after updating to v0.5.0.

• The decimal.Decimal type was previously defined as type number within the JSON schema and serialized as a float to JSON instances. In Pydantic v2, the decimal.Decimal JSON schema type is now {'anyOf': [{'type': 'number'}, {'type': 'string'}]} and the decimal value is serialized as a string to JSON instances. Since anyOf types are not compatible with the TetraScience platform, we provided two Decimal types to use in replacement of decimal.Decimal, DecimalNumber or DecimalString. The difference between these two types are their respective JSON schemas and serialization to JSON instances. DecimalNumber will have the JSON schema type number and will be serialized as a float value to JSON instances, whereas DecimalString will have the JSON schema type string and will be serialized as a string value to JSON instances. Both types will be a standard decimal.Decimal instance when initailized in an IdsElement subclass.

• Dropped support for Python 3.7. The Python version requirement is now >=3.8,<4.

Improvements

The update from Pydantic v1 to v2 brings benefits introduced by Pydantic v2 and allows ts-ids-core to keep updated with Pydantic’s latest features and bugfixes.

Parts of ts-ids-core were restructured internally to make use of new Pydantic v2 features, leading to the following improvements which don’t affect how to use ts-ids-core - no code changes needed for these:

• Default initialization of required constants (i.e. Required[Literal["foo"]])

  • As mentioned under Breaking changes above, defining constants is now done with the Literal type. Since required constants will always be the same value and to avoid defining the constant within the Literal type and as a default using IdsField (i.e. foo: Required[Literal["bar"]] = IdsField(default="bar")), required constants will be automatically set using the Literal value just as a default value would be. So, one can define the constant as foo: Required[Literal["bar"]] and the field foo will be implicitly set with the value "bar" when instantiating an IdsElement subclass without explicitly passing a value for foo.

• For convenience in defining a fixed length Annotated type (e.g. Annotated[List[str], annotated_types.Len(3, 3)]), we provided an Annotated type metadata function fixed_length so one can define the type as Annotated[List[str], fixed_length(3)]. See fixed_length for more.

• The Required type annotation now uses Annotated instead of a custom class, which means IDE type checkers can ignore it and give type hints about the underlying type, for example IDEs now recognize that Required[str] should have the same hints and autocompletion as str in code (with no change to how Required fields are validated or exported to JSON Schema).

• UUIDForeignKey and UUIDPrimaryKey were updated to use a new Pydantic v2 Annotated syntax which also allows IDE type checkers to recognize that they are str underneath all the metadata.

• Fields like ids_type in IdsSchema are “abstract” in the sense that you have to redefine them with a specific value in any subclasses of IdsSchema. The way abstract fields are defined was updated internally to use a new annotation, Abstract, simplifying how these fields are defined and validated.

• Pydantic v2 deprecated the BaseModel method schema_json which serialized the model to a JSON string representation. To preserve this behavior, we provided the same method with IdsElement, so one would not need to call model.model_json_schema() and then convert to a json string using another package like json.

• import-schema has been updated to generate models with a Pydantic v2 syntax, and a datamodel-code-generator template has been added to make the generated models require fewer manual steps to complete the conversion to programmatic IDS. Now, the Required annotation is added automatically, and Optional is only added for types which can have a value of None, whereas before it was also added to non-required types. See Generate programmatic IDS from JSON schema for full details of how to use this tool.

v0.4.0

• Modified the installation instructions for the import-schema script to convert rarely used dependencies into optional extra dependencies. Now, to use import-schema, its extra dependencies need to be installed via pip install "ts-ids-core[import-schema]" or similar - see the documentation. Now any time ts-ids-core is installed without needing import-schema, installation will be faster and come with fewer dependencies.

• Fixed a case where UUIDForeignKey fields were not being validated that they correspond to an existing UUIDPrimaryKey if the foreign key was defined in a class containing abstract fields (any field with a default value of NotImplemented). Now all foreign keys (inherited and newly defined) are validated when any class derived from IdsSchema or TetraDataSchema is defined and doesn’t contain any abstract fields.

v0.3.0

• Added UUIDStr, an annotation for UUIDs represented as strings, with validation that they follow the standard UUID string format. Updated UUIDPrimaryKey and UUIDForeignKey to use the UUIDStr type, so that they are exported as strings, making their dictionary instances compatible with jsonschema validation in Python. See the UUIDStr docs for details.

• Removed DataCubeWithPointer and added DataCubeMetadata to replace it. This component stores DataCube metadata along with a file ID of a file in the Tetra Data Lake which stores the DataCube’s dimension and measure values. See the documentation for DataCubeMetadata

v0.2.0

• Enabled intellisense for the IdsElement constructor. This enables additional code completion, tooltips and type checking in IDEs.

• SchemaExtraMetadataType is no longer a ClassVar, so the IdsElement.schema_extra_metadata field must now be explicitly given a type of ClassVar[SchemaExtraMetadataType]: to upgrade, replace schema_extra_metadata: SchemaExtraMetadataType with schema_extra_metadata: ClassVar[SchemaExtraMetadataType] as needed. This way, intellisense does not interpret schema_extra_metadata as a constructor parameter.

• Updated documentation site and started publishing documentation to https://ids.tetrascience.com, including consolidating IDS components and conventions documentation from other sources into this documentation.

• Added RawValueUnit component for storing a value and a unit along with a raw string value.

• Added Experiment component for referring to an experiment’s ID, name and description. The experiment field which was previously in Run has been removed, and its logs and instrument_status fields were added to the Run object. Now the meaning of “experiment” matches the definition of the “experiment” attribute used for instrument onboarding in TDP.

• Added DataCubeWithParquet component for storing datacube data with a pointer to a Parquet file in the data lake (note: this component was later removed in v0.3.0 and replaced with DataCubeMetadata).

• Added UUIDPrimaryKey and UUIDForeignKey for defining fields as primary or foreign keys, see their API docs for details.