Fields and Types¶

Nullable values and required fields¶

Nullable field definitions are explained here: nullable fields, and required field definitions are explained here: required fields. This section gives best practice recommendations for when to use them.

Terminologies¶

Null (opposite: non-null): a special value which can be assigned to a field to indicate that no value is available for that field.

Missing (opposite: present): when a key-value pair is omitted from the primary data it is missing, otherwise it is present. Across multiple files, if a field is sometimes present and sometimes missing, call it sometimes-missing. Otherwise call it always-present if it’s guaranteed to be present in every file.

Container type (counterpart: atomic type): A container type is an object or array - it can contain more fields. The other types are atomic types: string, number, boolean.

Required (opposite: non-required) the field must be present in every IDS JSON instance, i.e. this field must be included in the "required": [] array in JSON Schema.

Nullable (opposite: not nullable or non-nullable): the value of this field can be null in an IDS JSON instance. Note that a container type cannot be nullable due to TDP platform requirements.

Null and required design guidelines¶

By default, we recommend making IDS fields which have an atomic type (like string, number, boolean) nullable and non-required; and container types (like array, object) non-required.

The benefits of this approach are:

Variations in source data can be handled by omitting fields or using null values. Then a single schema can apply to data from a particular source regardless of the software version or export format being used.
Updating schemas to a new version causes fewer breaking changes to existing data consumers.
- When adding a non-required field, data from the previous version of the schema is still compatible with the new schema, meaning old data can easily be loaded and used according to the new schema. If a required field were added in a schema update, then all old data is now invalid and has to be reprocessed before the new schema can be used. For downstream use cases which don’t need the newly added field, making it non-required causes less disruption.
- When adding a nullable field, downstream applications which require that field can assume a default value of null when the field is missing from IDS data which hasn’t been updated yet, instead of having to wait until all data is updated to include the field.
It makes the fewest assumptions about requirements for downstream use cases.

Should a field be required? Recommended starting point: non-required, followed by the logic in this table:

Is the field always present in the primary data?	Does every downstream use-case need the field to be present?	Outcome in IDS
Always present	Needs to be required	Make it required, e.g. `field: Required[Nullable[str]]`
Always present	No restriction	Leave it as non-required, e.g. `field: Nullable[str]`
Sometimes missing	N/A	Leave it as non-Required, e.g. `field: Nullable[str]`

Should a field be non-nullable? Recommended starting point: nullable, except for container types (array, object) which must be non-nullable, followed by the logic in this table:

Is the field always non-null in the primary data?	Does every downstream use-case need the field to be non-null?	Outcome in IDS
Always non-null	Needs to be non-nullable	Make it non-nullable, e.g. `field: str`
Always non-null	No restriction	Leave it as nullable, e.g. `field: Nullable[str]`
Sometimes null	N/A	Leave it as Nullable, e.g. `field: Nullable[str]`

Basic Types¶

Boolean¶

The values True, False, true, false, Yes, No, yes, no, 1, 0 are often modeled as the type boolean. Of course, you should consider if these values should be interpreted as booleans.

Note that it is common for a field which appears to be a boolean to actually have more than two possible values, for example "True", "False" and "Unknown". Unless the source data is guaranteed to always be a boolean, it is usually better to use a nullable string type for this set of values. If additional values need to be supported, a string field can already support them without needing a schema update, avoiding a breaking change.

String with a format specification¶

Avoid using format for fields with string type except when the format is guaranteed to be followed by the upstream data source and is a requirement of all downstream use cases. Adding a format can make the schema fragile to changes in requirements. It can also make JSON Schema validation open to a redos attack in some cases.

Schema property descriptions¶

description is a JSON schema keyword.

It must be a string. description will provide more explanation about the field. This description should consist of full, grammatically correct, and formatted sentences (capitalization, period, etc).

Some common usages of description are

Explanation of the meaning of the field which an end-user of the data would understand
What this field corresponds to in the source data system and where it can be found
Give the fully-spelled out version of an abbreviation or an acronym

You can add descriptions to both container and atomic types with ts-ids-core. The docstring of an IdsElement subclass will become the description of its respective object type when exported to JSON schema. Describing a field can be done using the description argument to IdsField.

from ts_ids_core.base.ids_element import IdsElement
from ts_ids_core.base.ids_field import IdsField

class Example(IdsElement):
    """Object description"""

    foo: str = IdsField(description="Field description.")

Show JSON schema

{
  "additionalProperties": false,
  "description": "Object description",
  "properties": {
    "foo": {
      "description": "Field description.",
      "type": "string"
    }
  },
  "type": "object"
}

id and name¶

Add these standard fields to any object which needs an identifier or a name.

id should always be a string and is machine-readable. It’s usually a unique value across all the data produced by the same instrument or within a subset of related data produced by that instrument, for example a result ID.

name should human-readable. It’s usually defined by human and there is no guarantee it will be unique across all data.

IDSs often use a combination of id and name, for example as seen in the Sample component.