Units

Units are recorded as a standard unit string on the unit field (for example, "Gram"), and the unit as it was expressed in the primary data is recorded on the raw_unit field (for example, "g").

TetraScience standardizes unit spelling on QUDT 1.1 plus the Allotrope QUDT extension. The standard unit string is the fragment after the # in the unit’s QUDT URI. For example, the URI http://qudt.org/vocab/unit#SecondTime yields the unit name SecondTime (not Second), so SecondTime is what you record on the unit field.

Note that the unit field is a free-form string and is not restricted by the schema itself; using a standard unit is a modeling convention you are responsible for following, so that units stay consistent and are recognized by downstream tooling.

Special unit values

Use ArbitraryUnit when there should be a unit, but the unit doesn’t mean anything outside of instrument scope. For example, light absorbance.

Use Unitless for quantities which are unitless. For example, ratios or pH values.

Use null for both unit and raw_unit when there should be a unit, but it is not available in the raw file.

Example
{
  "ph": {
    "raw_value": "7.10",
    "raw_unit": null,
    "value": 7.1,
    "unit": "Unitless"
  },
  "weight": {
    "raw_value": "2.3",
    "raw_unit": "g",
    "value": 2.3,
    "unit": "Gram"
  },
  "numerical_variable_with_unknown_unit": {
    "raw_value": "102",
    "raw_unit": null,
    "value": 102,
    "unit": null
  }
}

These examples use the RawValueRawUnit component, which keeps the standard unit and the parsed value alongside raw_unit and raw_value — the unit and value as they were expressed in the primary data. Use RawValueRawUnit for every quantity with a unit in a new IDS. See Data Integrity for how raw_value is populated.

For the key/value pattern, use the ParameterRaw component, which records the unit as it was expressed in the primary data on its raw_numerical_value_unit field.

Raw-only data

Some IDSs keep a structure of untransformed data beside the standard fields, for example a datacubes_raw list next to datacubes that records the instrument’s own axes and values before any conversion. Such a structure is already the record of the primary data, so its quantities use ValueUnit and do not add raw_value and raw_unit.

Mark the field of the structure with IdsField(is_raw_data=True):

class MyIds(TetraDataSchema):
    datacubes_raw: List[RawDataCube] = IdsField(
        description="Datacubes as the instrument exported them.",
        is_raw_data=True,
    )

The generated schema.json then has "is_raw_data": true on that field, and the ts-ids-validator raw unit check skips the field and everything under it. Set the flag on the field, not inside the component the field uses, so that a shared component from ts-ids-components needs no change. Do not use the flag to avoid adding raw units to a transformed quantity; it is only for data that the IDS records exactly as the primary data expressed it.

Finding the right unit — ts-lib-unit

For the curated set of units, how to look up the correct spelling for a given unit, and related conventions, refer to ts-lib-unit. This library contains the units curated by TetraScience and is the source of truth for unit standardization.

Its documentation is also published at ids.tetrascience.com/ts-lib-units/main/index.html.

If you need a unit that is not available in ts-lib-unit, do two things in parallel:

  1. Record the raw unit provided in the source data verbatim on the raw_unit field, and document it alongside your IDS in the meantime. Do not invent your own unit name.

  2. Issue a support ticket to request that the unit be added to ts-lib-unit.