High-Level Design Principles¶
For what IDS is and is not, see What is IDS?; for the detailed, enumerable conventions, see the Design Guidelines.
Introduction¶
Two frameworks inform these principles:
FAIR (Findable, Accessible, Interoperable, Reusable) — for data management
ALCOA++ (Attributable, Legible, Contemporaneous, Original, Accurate, plus Complete, Consistent, Enduring, Available) — for trustworthy data in regulated environments
IDS is an intermediate step — the Bronze layer of a Medallion architecture, not the final destination (knowledge graphs, ELN/LIMS records, use-case ontologies, agentic applications). That is why fidelity — a record, not an interpretation — matters more than cleverness: each IDS’s defects propagate to every downstream consumer. These principles make FAIR and ALCOA++ specific for IDS, in service of the autonomous lab, AI/ML training, and agentic applications.
1. Ground the schema in the science¶
A schema represents the data produced by a scientific process; it is not a passive container. Designing one means deciding which scientific entities the data contains and how they link — which result belongs to which run, which array varies along which dimension.
Hand raw data to an engineer (or an AI) and say “design a schema for this,” and what comes back models the file’s surface structure (columns, sheets, tabs) rather than the science (entities, provenance, the epistemic status of each value). Closing that gap — domain grounding through knowledge elicitation — is the designer’s core job, and it must happen before and during design, not after. It pays off twice: a confidently wrong schema and a correct one look identical without domain grounding (especially when AI-drafted), and elicitation grounds your collaborators by constraining the design toward the actual scientific domain.
Question & what it revealed |
Schema design |
|---|---|
What does a |
A |
Elicit, but attribute. Elicited knowledge can tip into over-interpretation — turning the schema from a record of what happened into an assertion of what you think happened, which IDS is not responsible for. The discipline is elicit → encode → attribute. Two kinds of elicited knowledge need special care:
Assay or experiment knowledge — facts fixed by experimental design or instrument configuration, recorded outside the input data.
String-parsed knowledge — facts inferred from human-entered labels or naming conventions, an interpretation that can be wrong (a typo, an inconsistent convention).
Avoid such interpretation where you can. Where the use case requires it, document the interpretation and always keep the source string or text.
2. Phased approach: quality over quantity¶
It is often unrealistic to schematize everything an instrument produces. Focus on a clean, well-documented v1: a use case may need only 30% of the data, and when harmonizing multiple vendors some portion (often the method) may be too divergent to unify while the results harmonize easily. Prioritize fields by the scientific use case and use IDS’ semantic versioning to add deferred fields in later releases — never retrofit silently. Whatever you model must meet the principles below, and every gap must be documented.
3. Data presence: is all the important scientific data captured?¶
All scientifically meaningful data an instrument produces should ideally be represented in the schema.
Deferring whole categories of data to a later version is fine when documented. Filtering out records within a type you have already chosen to model is not: once you model a record type, capture all of its instances, including those irrelevant to today’s use case (blanks, controls, out-of-range points, carry-over checks). Field coverage is a documented, versioned scope decision; record-level omission is silent data loss. Blank injections are irrelevant to a concentration result but essential to a carry-over audit — dropping them makes that audit impossible after the fact. A clean, well-documented 30% is worth more than a shaky 100%.
3.1 Highly unstructured data¶
Some vendor output is noisy — more a log or text blob than a real data structure, with enough firmware-to-firmware variation to make full schematization unsustainable. Preserve the raw string for provenance and extract only the few scientifically critical fields, or model open-ended key–value settings with the generic Parameter pattern rather than minting a fixed field per key. See Highly Unstructured Data for both patterns and worked examples.
4. Actionable: can the operator produce schematizable data?¶
Schema documentation should give human-readable instructions on how the operator configures the instrument control software or source system such that the source data can be engineered into the schema, and should highlight common mistakes and limitations to avoid misaligned expectations.
These tactical implementation details are crucial to real-world adoption: often only a certain export file format contains the right data, the source data must be configured to export automatically after a run, or an extra license is required. Surface them explicitly so the operator can produce schematizable data in practice.
5. Consistency: harmonize, but only when it pays¶
Schemas within a family of data sets should be consistent. But harmonizing multiple vendors or models into one instrument-family IDS, while valuable, is not always worth doing up front. Do not rush into it when no concrete scientific use case is asking for cross-vendor comparison, a single vendor/model dominates the current use cases, or you have too little sample data to generalize safely — harmonizing on thin evidence is the same over-fitting failure mode described in Highly Unstructured Data. A healthy middle ground: harmonize the results, which unify easily, and preserve instrument and method details per-vendor (see the phased approach and Preserve the unstructured string).
7. Self-describing records: avoid ambiguity¶
A record — a result row, a peak, a single measurement — should be interpretable on its own, from the data inside the schematized data set. Carry the identifiers that say what it is inline: what was measured, in what context, by which run. Records are filtered, exported, and streamed into tables independently, so one that needs the whole document to interpret is fragile and expensive to query. Test: hand a single record to someone with no other context — can they say what was measured and under what conditions? If not, it is under-specified.
Each value must also be unambiguous:
Say what a statistic means — e.g. whether a particle-size mean is weighted by
volume,number, orintensity.Put the unit in the data, as a sibling of the value — not in the field name. Prefer
{ "value": 2.5, "unit": "nM" }overconcentration_nM: 2.5(unit trapped in the key) or a bare2.5(unit unknown). Standardize to QUDT where possible (see Units) and use theRawValueUnitcomponent.Distinguish intended (nominal) from measured. A setpoint, target concentration, or expected retention time has a different epistemic status from what the instrument measured. Keep them in separate fields even when numerically close; conflating them erases the ability to compute accuracy or detect drift.
Use a standard timestamp with a timezone. Use ISO 8601 with an explicit offset (e.g.
2025-12-23T14:05:00Z); local time with no zone is ambiguous the moment data crosses sites. See Date and Time.
8. Understandable: name fields for the reader¶
A schema is understandable when someone familiar with the experiment, the instrument technique, or the source data can traverse it and know what each field means — aided by the documentation and example files, without guessing. Ambiguity in names or structure is a defect, and over-harmonization or over-abstraction works against it. Use clear, consistent names, keep units out of field names, and make each field understandable from its schema.json description and README alone. See Field Naming.
9. Fidelity: store, don’t recompute¶
A value the instrument or its vendor software exports is authoritative — store it verbatim, even when you cannot inspect the algorithm. A value you re-derive is a reconstruction: perhaps close, but not the same thing. Never store a re-computed value in the same field as an instrument-reported one. IDS is an intermediate record, not a computation engine (see What is IDS?): when the raw file omits a parameter, note that it exists but is not exported — never present a reconstruction as if the instrument had reported it. If you must re-compute, record it explicitly as a derived value with its source and method. For example, a calibration curve re-fit by a downstream script is a reconstruction; record it, if at all, as a derived value — not under a field a consumer would read as “what the instrument said.”
10. Retrieval: is the data easy to query?¶
Model the data so common access patterns are cheap and obvious — plotting or overlaying chromatograms, re-computing peaks, gating flow-cytometry data, filtering results by analyte, sample, or run across a whole dataset. Self-describing records do most of the work: they let a consumer filter and export without reconstructing the document. For large numeric signals, representing them as datacubes rather than inline scalars keeps them in an aligned, unit-bearing structure that can be plotted and re-analyzed directly (see Large Numerical Data). If a reasonable use case requires loading and parsing the entire document to answer a simple question, the model is failing retrieval.