Datacubes and Results¶

Measured data in an IDS is stored in one of two output fields, results or datacubes. For either field, the values are stored either in the IDS JSON itself or in an accompanying parquet file. This page is the guide for making both decisions when designing an IDS.

This page exists to help store data consistently across different IDSs. Two IDSs covering the same instrument class should ideally model the same kind of measurement in the same way, so that users of multiple IDSs have a consistent interface for accessing and querying similar data.

For the components involved, see DataCube and DataCubeMetadata in Components. For writing the Parquet file in a task script, see Writing Parquet files in Task Scripts.

1. Deciding between results and datacubes, JSON and parquet¶

1. DOMAIN     What shape can this instrument class emit — not just this file?
              Pick one structure for the whole class and apply it uniformly.
                              │
                              ▼
2. DERIVED    Separate the measured signal from values derived from it.
              Derived data of the same shape? ────► stays with the measured signal
              Scalar summary of the signal?  ─────► results
                              │
                              ▼
3. STRUCTURE  N-dimensional numeric array ──────────────────────► datacubes
              Records with named fields, tabular or hierarchical ► results
                              │
                              ▼
4. FORMAT     Where are the values stored? Small ► the IDS JSON, large ► parquet
              datacubes ► datacube_metadata + parquet, queryable in Lakehouse
              results   ► an accompanying parquet file, not queryable

Step 1 — This decision should be made for a domain, not a single file or instrument¶

Choose the structure that suits the general case for this domain of instruments, then use it for every IDS in that domain, including instruments that only ever emit simpler forms of data. For example, some plate readers record a kinetic series or a spectral scan per well, while others read a single wavelength at a single time point. The convention covers the whole domain, so the reader data goes in datacubes even for a single-filter endpoint reader whose datacubes each hold a single value.

The cost is that some IDSs use a more complex schema than their data strictly needs. That is accepted in exchange for a consistent interface across a domain of IDSs. Before choosing a structure, check whether a convention for the domain already exists, for example ts_ids_components.plate_reader and ts_ids_components.chromatography, and follow it.

Step 2 — Understand the signal directly measured by the instrument, and what is derived from it¶

A derived value with the same shape as the signal belongs with the signal as another datacube. For example, a baseline-corrected or blank-subtracted trace has the same dimensions and the same cardinality as the trace it came from, so it stays with datacubes.
A derived value that summarizes or aggregates the signal to a scalar belongs in results, together with whatever metadata a consumer needs to interpret it, such as the calculation method or the parameters used.

This is a split by shape, which correlates with how derived the data is (how much processing or calculation has been done), but the level of processing is not the rule to follow. A smoothed chromatogram is heavily processed and still a datacube, while a raw single temperature reading is a scalar and belongs in results.

Step 3 — Consider the structure of the data, and what one entry describes¶

datacubes and results use two different data structures:

datacubes is for N-dimensional numeric array data. Each measures[*].value is a nested array whose depth equals the number of dimensions, indexed by the numeric values in dimensions[*].scale. The values have to fill the grid formed by those scales.
results is an array of objects, which can represent tabular data and hierarchical data. Each object is one row, and its named fields are the columns. The fields can be nested data types which allows nesting, such as one result per detector channel, each containing an array of peaks from that channel.

results and datacubes are fact fields holding the measurements, while the other top-level fields such as samples, runs and methods describe them as star-schema dimensions (see Top-level data blocks). The remaining question is the grain: what does one entry describe? This differs between the two fields, and for datacubes it differs again between the IDS JSON and the Lakehouse:

	`results`	`datacubes`
One entry of an array in the IDS JSON	one record in the `results` array	one whole datacube in the `datacubes` array: every value of one series
One row of a Lakehouse table	one record in the `results` array	one measure value (in `measures[*].value` array) at one coordinate, with the datacube’s metadata repeated on every row
Descriptive context and foreign keys	one set per record in the `results` array	one set per datacube in the `datacubes` array

A datacubes entry therefore carries many values, which the Lakehouse expands into one row each. A parquet datacube file has that same expanded shape, so the number of datacube_metadata entries and the number of parquet rows are not the same thing (see Section 6). Because a foreign key is a single scalar, one datacube links to exactly one of each thing it points at (see Describing relationships in IDSs).

Because the values in datacubes[*].dimensions[*].scales array have to be numeric (a known limitation on the data indexing infrastructure), well positions like "A1", sample identifiers, or channel names cannot be a dimension; store it as a field on datacubes[*] or behind a foreign key referenced by the datacube.

Step 4 — Choose between storing the values in the IDS JSON or in a parquet file¶

Both output fields have this choice, made for the same reason: an IDS JSON which is too large becomes impractical to consume. The difference is the access pattern in Tetra.

For datacubes, instead of storing values in JSON, the metadata is moved to the datacube_metadata field, with a separate parquet file for data. Both JSON and Parquet formats are transformed into the same datacubes Lakehouse table, so the data stays queryable in the Lakehouse either way. For file-based access, users need to download the Parquet file separately from the IDS JSON. See Section 3.
For results, the values move to an accompanying parquet file, which is not transformed into a Lakehouse table, so the data can be retrieved but not queried. For file-based access, users need to download the Parquet file separately from the IDS JSON. See Section 7.

2. Examples from two domains¶

Chromatography. The chromatogram from each detector channel is the measured signal, so it goes in datacubes, as do derived traces of the same shape such as a blank-subtracted or baseline-corrected chromatogram. Peak characteristics computed from it, such as retention time, area, height and resolution, summarize the trace as scalars, so they go in results with one record per detector channel, alongside the method and limits used to produce them.

Plate readers. The reader data goes in datacubes, with one datacube per well and fk_sample linking to the samples entry for that well. Both dimensions are reserved for the axes the measurement varies along, time and wavelength, so the well is identified by the foreign key rather than by a dimension. Values derived per well, such as a concentration from a calibration curve or a pass or fail call, go in results with the calculation method which produced them.

3. Choosing between JSON datacubes and `datacube_metadata` with parquet¶

Both formats describe the same datacubes and populate the same Lakehouse datacubes table, so this decision is about where the values are stored rather than how they are queried.

Store datacube values in the IDS JSON by default. Use datacube_metadata with a parquet file when the datacube data would make the IDS JSON larger than roughly 100 MB. Estimate this against the largest file the instrument can realistically produce rather than a typical one, because file sizes may vary considerably within a single instrument type.

Two constraints apply regardless of size:

measure_{m}_value and dimension_{d}_value are double columns in a parquet datacube, they need to match the corresponding dimension scale and measure value types which are numeric.
Sparse data and data which is not a multidimensional array are additional reasons to prefer parquet. Values which do not fill the grid have to be padded with nulls in JSON datacubes, because measures[*].value is a rectangular nested array. A parquet datacube is a flat table, which means it does not have the same densely populated grid structure that JSON datacubes have: rows for absent combinations of dimension values can simply be left out parquet file.

4. Differences between JSON datacubes and parquet datacubes¶

Rows from both formats are combined into one Delta table, <ids_type>_v<major_version>_datacubes:

  IDS JSON                              IDS JSON
  datacubes[*]                          datacube_metadata[*]
    dimensions[*].scale                   dimensions[*].name, unit
    measures[*].value                     measures[*].name, unit
                                          file_id ──┐
       │                                            ▼
       │                                  datacubes0.parquet
       │                                    datacube_index
       │                                    dimension_{d}_value
       │                                    measure_{m}_value
       │                                            │
       └──────────────── union ─────────────────────┘
                           │
                           ▼
            <ids_type>_v<major_version>_datacubes

	values in the IDS JSON	values in a parquet file
Lakehouse Delta table	`_datacubes`	the same `_datacubes` table
Legacy CSV-backed Athena tables	populated	not populated, by design
Elasticsearch	always excluded	metadata may optionally be indexed, so datacubes are searchable by name, dimensions and measures
Value types	numeric only	`double` only
Sparse data	needs padding with nulls	no padding needed

Two rules follow from both formats being combined into one table:

The IDS JSON is the single source of truth. A parquet datacube file is reachable only through datacube_metadata[*].file_id, so a parquet file with no datacube_metadata entry pointing at it is not visible to Lakehouse ingestion, and a pipeline should never be triggered directly on a parquet datacube file.
Do not store the same data twice. If the same data is written to both JSON datacubes and Parquet datacubes for 1 IDS JSON file, then both sets of data will be inserted to the same Lakehouse datacubes table, leading to duplicate data.

5. When an IDS may declare both `datacubes` and `datacube_metadata`¶

It is recommended to aim to only use one of datacubes or datacube_metadata, but it is possible to declare both when they hold different data:

Reason	When it applies
Size split	The instrument produces two classes of signal which differ in size by orders of magnitude.
Runtime toggle	Pipeline configuration selects the format per file, and only one field is ever populated in a given IDS instance.
Type split	Some signals have string values and cannot be stored in a parquet file, while the numeric ones can.
Source-version compatibility	Different instrument or agent versions emit different shapes, during a transition.

Warning

If an IDS declares both datacubes and datacube_metadata plus a parquet file, their dimensions and measures must declare identical minItems and maxItems. The rows from both are combined into one Lakehouse table, so the dimension and measure counts have to agree in order for that table to have a consistent schema. A mismatch causes Lakehouse ingestion to fail.

6. Advanced: moving per-row metadata into the parquet file¶

A field declared in the datacube_metadata schema may be omitted from the IDS JSON and supplied as a column of the parquet file instead, with one value per row. Lakehouse ingestion uses the parquet column in preference to the JSON field, so the field can vary within a datacube rather than being constant across it.

This may be used to keeps the number of datacube_metadata entries far below the number of parquet rows when metadata rows vary as much as the data rows. Conventions such as ts_ids_components.plate_reader describe one datacube per well for values stored in the IDS JSON, so applied directly to parquet a 1536-well read would need 1536 datacube_metadata entries. Moving the per-well fields into the parquet file reduces that to a single bulk pointer, without changing the grain, which is still per well because the parquet file has an fk_sample column on every row.

This applies to foreign keys as well:

from typing import Annotated

from ts_ids_core.annotations import ForeignKey, UUIDStr
from ts_ids_core.base.ids_field import IdsField
from ts_ids_core.schema import DataCubeMetadata


class ParquetDataCubeMetadata(DataCubeMetadata):
    """`fk_sample` is declared but not required, so it can be omitted from the IDS
    JSON and supplied as a per-row column of the datacube Parquet file instead."""

    fk_sample: Annotated[UUIDStr, ForeignKey()] = IdsField(
        primary_key="/properties/samples/items/properties/pk"
    )

Three rules apply:

The field must still be declared in the datacube_metadata schema, because that schema defines which columns the parquet file may contain, and an undeclared column is dropped during ingestion.
The field must be non-required, so that it can be omitted from the IDS JSON. Note that UUIDForeignKey includes Required, so declare the annotated form shown above instead.
The values must still be populated on every parquet row, because non-required in the schema does not mean nullable in the data. A null foreign key silently drops rows from an inner join and produces all-null columns from a left join.

This also allows a parquet datacube to hold per-row foreign keys which a datacube stored in the IDS JSON cannot express at all, where a link varies along one of the axes.

7. Storing large `results` data in a parquet file¶

results has the same choice of format as datacubes: the data can be stored in the IDS JSON, or written to an accompanying parquet file and referenced from the IDS by file ID. This is primarily useful for large tabular data sets.

Warning

A results parquet file is not currently transformed into Lakehouse tables. Lakehouse ingestion handles the IDS JSON and datacubes, so a results parquet file can be retrieved by downloading it using its file ID, but its data is not SQL-queryable. It is possible to get results data into a Lakehouse table using the direct-to-lakehouse pipeline. That is outside the scope of IDS design.

Unlike the datacube case, where both formats are queryable, this is a genuine trade. Only move results data into a parquet file when the data set is large enough that the size of the IDS JSON is the bigger problem, and record the choice in the IDS README. When the data does form a numeric grid indexed by physical axes, prefer datacubes with datacube_metadata.

Datacubes and Results¶

1. Deciding between results and datacubes, JSON and parquet¶

Step 1 — This decision should be made for a domain, not a single file or instrument¶

Step 2 — Understand the signal directly measured by the instrument, and what is derived from it¶

Step 3 — Consider the structure of the data, and what one entry describes¶

Step 4 — Choose between storing the values in the IDS JSON or in a parquet file¶

2. Examples from two domains¶

3. Choosing between JSON datacubes and datacube_metadata with parquet¶

4. Differences between JSON datacubes and parquet datacubes¶

5. When an IDS may declare both datacubes and datacube_metadata¶

6. Advanced: moving per-row metadata into the parquet file¶

7. Storing large results data in a parquet file¶

3. Choosing between JSON datacubes and `datacube_metadata` with parquet¶

5. When an IDS may declare both `datacubes` and `datacube_metadata`¶

7. Storing large `results` data in a parquet file¶