"""
This module contains components for plate reader IDSs.
Plate reader samples and datacubes
==================================
A common data usage pattern for plate reader data is to read measurement data, such
as absorbance or fluorescence values, and analyse them along with metadata about the
sample which was being measured.
To do this, use the :external+ts-ids-core:ref:`samples common component <component_samples>`
to store information about the sample in each well of the plate, including a
:external+ts-ids-core:py:data:`UUID primary key <ts_ids_core.annotations.UUIDPrimaryKey>`
field named ``"pk"``.
In :external+ts-ids-core:ref:`datacubes <datacubes>`, include a foreign key
to ``samples`` with the name ``fk_sample``.
This structure is defined in :py:class:`PlateReaderSchema` below, and that class can be
used as a starting point for defining plate reader schemas.
When populating data into a plate reader IDS, store data from each well in a separate
element of the ``datacubes`` array.
This convention exists to serve two goals:
#. Data from many different plate readers is stored in the same format, so downstream
code which consumes it does not need customizing per instrument.
#. Measurement data can be joined to the metadata of the sample it came from in one
standard way, which is the same for every plate reader.
The rest of this section explains how the convention achieves those goals, what it means
for queries, and why other approaches were not chosen.
One datacube per well
---------------------
Each element of ``datacubes`` holds the data from a single well, for a single measure.
Both of its dimensions are reserved for the axes the measurement varies along -- ``time``
and ``wavelength`` -- and the well it came from is identified by ``fk_sample``, not by a
dimension.
Fixing the granularity at one well is what allows a single datacube structure to cover
plate readers which differ in modality (endpoint, kinetics, spectral scan), in assay type
(absorbance, fluorescence, luminescence and others) and across different vendors and
plate reader models.
Per-well datacubes also keep wells independent of each other. A well measured over a
different set of timepoints or wavelengths than its neighbours carries its own scales,
and a well which was not read is simply absent from ``datacubes``. This allows
``measures[*].value`` to be a fully populated rectangular array matching the
dimension scales: if the well were a dimension instead, every well on the plate would
have to share one identical ``time`` scale and one identical ``wavelength`` scale, and an
unread or masked well would have to be padded with nulls to keep the array rectangular.
A cost of this convention is duplication -- the ``time`` and ``wavelength`` scales are
repeated in the datacube of every well. Another cost is that, for non-spectral endpoint
measurements, there is 1 time point and 1 wavelength per well, which can lead to a large
number of datacubes containing a single measure value. These downsides are accepted in
exchange for the consistency this approach enables.
Linking samples and datacubes with primary and foreign keys
-----------------------------------------------------------
``samples[*].pk`` and ``datacubes[*].fk_sample`` form a primary key and foreign key pair,
following the general IDS conventions for
:external+ts-ids-core:ref:`describing relationships
<creating_an_ids_artifact/design_guidelines/relationships:describing relationships in idss>`.
``samples`` is a dimension table, describing what was measured; ``datacubes`` is a fact
table, holding the measurements themselves. Foreign keys belong on the fact table, so
``datacubes`` points at ``samples`` and not the other way around.
This gives the datacube one place to record which well it came from, and gives consumers
everything about that well through a single join: ``samples`` already holds the well
``position``, ``index``, ``row`` and ``column``, along with sample identifiers and any
other sample metadata.
Because a foreign key is a single scalar value, one datacube can only relate to one well.
The linking mechanism and the one-datacube-per-well granularity therefore imply each
other -- given ``fk_sample`` on ``datacubes``, a well dimension would not be meaningful.
Foreign keys are not nullable, which keeps joins predictable: a nullable foreign key
silently drops rows from an inner join, or produces all-null joined columns from a left
join. Where a well has no sample information at all, populate an otherwise-empty
``samples`` element for it so there is still a row to link to.
A datacube can carry several foreign keys at once. :py:class:`PlateReaderDatacube2D` has
``fk_sample``, ``fk_protocol_step`` and ``fk_method`` for example. These can be modified
for each IDS to join datacubes with any relevant metadata which is useful to join.
Here is an example of defining a schema which inherits from
:py:class:`PlateReaderSchema` and populating it in Python.
This shows data being manually populated in the script itself, but in typical usage,
this data would be parsed from a raw data file.
.. literalinclude:: ../../../__tests__/unit/test_plate_reader.py
:pyobject: test_complete_plate_reader_schema
:language: python
:dedent: 4
:start-after: doc-start
:end-before: doc-end
Then, the data could be dumped to JSON by calling ``instance.model_dump_json(indent=2)``
The resulting IDS JSON looks like this:
.. literalinclude:: ../../../__tests__/unit/snapshots/plate_reader_schema_demo
:language: json
How this affects queries
------------------------
In the Lakehouse, each top-level array of an IDS becomes one table, named
``{ids_type}_v{major_version}_{field}`` with hyphens replaced by underscores, inside a
schema named after the IDS type. For an IDS type of ``plate-reader-acme-1000:v2.0.0``,
``samples`` and ``datacubes`` become
``plate_reader_acme_1000_v2_samples`` and ``plate_reader_acme_1000_v2_datacubes``. A
datacube's metadata and its numeric data are in that one ``_datacubes`` table, with the
metadata repeated on each row of data.
Joining measurement data to sample metadata is then a single join on the key pair:
.. code-block:: sql
SELECT
samples.location_position,
samples.id,
datacubes.dimension_0_value AS time_value,
datacubes.dimension_1_value AS wavelength_value,
datacubes.measure_0_value AS absorbance_value
FROM plate_reader_acme_1000_v2_datacubes AS datacubes
JOIN plate_reader_acme_1000_v2_samples AS samples
ON samples.pk = datacubes.fk_sample
The important property is that this query is not scoped to one file. Primary keys are
UUIDs generated per IDS instance, so they are unique across the whole table, and the join
above is correct over data from every file of this IDS type at once. The same query shape
works for any other plate reader IDS following this convention -- only the table names
change.
Measures and dimensions appear as ``measure_{n}_value`` and ``dimension_{n}_value``
columns, numbered by their position in the ``measures`` and ``dimensions`` arrays. This
positional naming is one reason to keep the dimensions fixed and meaningful: with the
convention above, ``dimension_0_value`` is always time and ``dimension_1_value`` is
always wavelength.
Why not other approaches
------------------------
.. raw:: html
<details>
<summary>Expand: alternative ways of linking samples and datacubes</summary>
**A sample index, well index, row or column as a dimension.** This approach was used
prior to the current convention. These are some of the reasons not to use that approach:
* An index is only unique within one file. Sample index ``0`` in one IDS instance is a
different well from sample index ``0`` in another, so joining ``samples`` to
``datacubes`` on the index alone is wrong as soon as more than one file is in scope. It
can be made to work, but only by first joining data from a single instance together using
the platform-generated ``uuid``/``parent_uuid`` columns, which do not exist in the IDS
schema and are a common source of confusion:
.. code-block:: sql
-- With a sample index dimension: correct only within one file
SELECT samples.id, datacubes.measure_0_value
FROM plate_reader_acme_1000_v2_datacubes AS datacubes
JOIN plate_reader_acme_1000_v2_samples AS samples
ON samples.parent_uuid = datacubes.parent_uuid
AND samples.location_index = datacubes.dimension_0_value
That second condition also hard-codes the knowledge that ``dimension_0`` is the sample
index, and relies on ``samples.location_index`` being available from the raw data, which
is not always possible, as described below.
* An index cannot always be derived from the raw data. If the raw file gives only a well
position such as ``B02`` and never states the plate size, there is no way to compute an
index from it. An arbitrary number could be assigned, but that is not a well index and
would not match any other system.
* Dimension scales have to be numeric -- all dimensions share a single data type, and
Lakehouse transformations require numeric dimensions -- so a well *position* like
``"B02"`` cannot be a dimension at all, only a derived index.
* Some data has no unique index to use. On instruments which report aggregates alongside
individual locations, the aggregate rows share a placeholder index, so a dimension would
have to be populated with invented values and then documented.
* Conceptually, a dimension scale describes a physical axis along which a measurement
varies. Identity and metadata are a different kind of information, and belong in fields
on ``datacubes[*]`` or behind a foreign key.
.. raw:: html
</details>
When to use 3D datacubes
------------------------
2D datacubes with ``time`` and ``wavelength`` dimensions are for data which has a single spectral
axis, as in absorbance or a single emission scan. Use :py:class:`PlateReaderDatacube3D` with
dimensions of ``excitation wavelength`` and ``emission wavelength`` when an instrument
is capable of spectrally scanning both excitation and emission wavelengths. For multi-modal
instruments which don't always need the third dimension, use the 3D datacube in the IDS and
just populate two of the dimensions for single-scan data.
Not every measurement has a single wavelength to record. Luminescence, broadband illumination
and multi-channel optical modules select an optic setting rather than a wavelength, so there
is no nanometer value for the scale. Record the channel or optical module index (depending on
the instrument) if one exists.
.. code-block:: python
PlateReaderDimension(name="channel", unit="Unitless", scale=[1])
Other plate reader components
=============================
There are components in this module which don't have a predefined top-level path in
an IDS because they may be used in multiple places throughout a plate reader schema,
and their location in the schema may vary across data sources.
Typically, a specific instrument IDS can have more fields than the ones present in
these models. For example, if the instrument filter has an additional value-unit field,
such as a reference wavelength, this field can be added by inheriting from the Filter
component:
.. code-block:: python
class ReferenceFilter(Filter):
reference: ValueUnit
"""
from enum import Enum
from typing import List
from ts_ids_core.annotations import (
Nullable,
NullableString,
Required,
UUIDForeignKey,
UUIDPrimaryKey,
fixed_length,
)
from ts_ids_core.base.ids_element import IdsElement
from ts_ids_core.base.ids_field import IdsField
from ts_ids_core.schema import (
DataCube,
Dimension,
Measure,
Sample,
System,
TetraDataSchema,
Time,
ValueUnit,
)
from typing_extensions import Annotated
from ts_ids_components.plate_reader.methods import (
PlateReaderMeasurementSetting,
PlateReaderMethod,
PlateReaderStep,
)
# Light sources
[docs]
class LightSource(IdsElement):
"""Definition of a general light source system"""
type_: Nullable[str] = IdsField(alias="type", description="Light source type")
system: System = IdsField(description="Light source system information")
[docs]
class Lamp(LightSource):
"""Lamp light source"""
power: ValueUnit = IdsField(description="Nominal lamp power")
[docs]
class LED(LightSource):
"""Light Emitting Diode light source
Related Open Microscopy Environment model:
https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#LightEmittingDiode
"""
power: ValueUnit = IdsField(description="Nominal LED power")
# Optics
[docs]
class Filter(IdsElement):
"""Optical filter properties"""
position: Nullable[str] = IdsField(
description="Position of this filter in a container like a filter wheel"
)
bandwidth: ValueUnit = IdsField(
description="The range of frequencies associated with this filter"
)
wavelength: ValueUnit = IdsField(
description="Characteristic wavelength of this filter"
)
[docs]
class BeamSplitter(System):
"""Beamsplitter properties"""
# Detection
[docs]
class DetectorSystem(System, System.Id):
"""Definition of a detector system"""
# Sample environment
[docs]
class EnvironmentRun(IdsElement):
"""Measured environment during a run"""
measured_temperature: ValueUnit = IdsField(
description="Measured temperature during a run"
)
# Measurement protocol
[docs]
class ShakingStep(IdsElement):
"""Shaker methods and metadata"""
mode: Nullable[str] = IdsField(
description="Shaking mode, such as 'orbital' or 'linear'"
)
speed: ValueUnit = IdsField(
description="Shaking speed, the angular speed or frequency of shaking"
)
time: Time = IdsField(description="Shaking timing")
[docs]
class InjectionStep(IdsElement):
"""Injection method for a single injection, including pump and volume settings"""
pump_id: NullableString = IdsField(description="Identifier for pump being used")
flow_rate: ValueUnit = IdsField(description="Flow speed of the injector pump")
volume: ValueUnit = IdsField(description="Volume of injection")
time: Time = IdsField(description="Injection timing")
[docs]
class MeasurementPattern(IdsElement):
"""The measurement pattern, including which plate is being measured, which wells are
measured, and in what order
"""
plate: Nullable[str] = IdsField("Identifier for the plate being measured")
wells: List[str] = IdsField(
description="References to the wells being measured, in order"
)
[docs]
class MeasurementPatternByArea(MeasurementPattern):
"""Measurement pattern for plate readers which specify a plate area to measure"""
area: Nullable[str] = IdsField(
description="An area of the plate as a string, e.g. 'A1-F4'"
)
reading_direction: str = IdsField(
description=("A description of the direction that wells are read from a plate")
)
# Samples
[docs]
class PlateReaderSample(Sample):
"""A sample stored in a well on a plate"""
pk: UUIDPrimaryKey
# Datacubes
[docs]
class PlateReaderDimensionNames(str, Enum):
"""A limited set of possible names for plate reader dimensions"""
TIME = "time"
WAVELENGTH = "wavelength"
EXCITATION_WAVELENGTH = "excitation wavelength"
EMISSION_WAVELENGTH = "emission wavelength"
[docs]
class PlateReaderDimension(Dimension):
"""A plate reader dimension with a limited set of possible names"""
name: Nullable[str] # Use the PlateReaderDimensionNames enum for standard names
[docs]
class Measure2D(Measure):
"""A two-dimensional datacube measure"""
value: Required[List[List[Nullable[float]]]]
[docs]
class Measure3D(Measure):
"""A three-dimensional datacube measure"""
value: Required[List[List[List[Nullable[float]]]]]
[docs]
class PlateReaderDatacube2D(DataCube):
"""A plate reader datacube containing two dimensions and one measure.
The dimensions must be `time` and `wavelength`
"""
fk_sample: UUIDForeignKey = IdsField(
description="A foreign key linking datacubes to samples[*].",
primary_key="/properties/samples/items/properties/pk",
)
fk_protocol_step: UUIDForeignKey = IdsField(
description="A foreign key linking datacubes to protocol_steps[*].",
primary_key="/properties/protocol_steps/items/properties/pk",
)
fk_method: UUIDForeignKey = IdsField(
description="A foreign key linking datacubes to methods[*].",
primary_key="/properties/methods/items/properties/pk",
)
dimensions: Required[Annotated[List[PlateReaderDimension], fixed_length(2)]]
measures: Required[Annotated[List[Measure2D], fixed_length(1)]]
[docs]
class PlateReaderDatacube3D(DataCube):
"""A plate reader datacube containing three dimensions and one measure."""
fk_sample: UUIDForeignKey = IdsField(
description="A foreign key linking datacubes to samples[*].",
primary_key="/properties/samples/items/properties/pk",
)
fk_protocol_step: UUIDForeignKey = IdsField(
description="A foreign key linking datacubes to protocol_steps[*].",
primary_key="/properties/protocol_steps/items/properties/pk",
)
fk_method: UUIDForeignKey = IdsField(
description="A foreign key linking datacubes to methods[*].",
primary_key="/properties/methods/items/properties/pk",
)
dimensions: Required[Annotated[List[PlateReaderDimension], fixed_length(3)]]
measures: Required[Annotated[List[Measure3D], fixed_length(1)]]
# Top-level schema including samples, datacubes and methods paths
[docs]
class PlateReaderSchema(TetraDataSchema):
"""A schema for a plate reader."""
methods: List[PlateReaderMethod]
protocol_steps: List[PlateReaderStep]
measurement_settings: List[PlateReaderMeasurementSetting]
samples: List[PlateReaderSample]
datacubes: List[PlateReaderDatacube2D]