What IDS Is (and Is Not) ¶

The conceptual framing for TetraScience’s Intermediate Data Schema (IDS): what it is, what it is deliberately not for, and how it relates to ontologies, knowledge graphs, and the semantic layer. For how to design an IDS, see the companion IDS High-Level Design Principles; cross-references to it below are written as e.g. “Design Principles §1.1”.

Table of Content ¶

1. What is IDS?
2. What IDS is not for?
- 2.1 IDS in relation to ontology, knowledge graph, and the semantic layer

1. What is IDS?¶

The “Intermediate Data Schema” (IDS) is a framework designed by TetraScience in collaboration with its partners and customers to represent heterogeneous, vendor-specific scientific data in a vendor-agnostic, structured data format.

1.1 A vendor-agnostic structure over heterogeneous data¶

This schema represents vendor-specific scientific data through a vendor-agnostic data structure and interface. The framework applies structure to heterogeneous data sets such as

data type (whether a field is a string, an integer, …)
logical data relationships and hierarchy (a single chromatogram can be processed into multiple peaks)

By doing this, different kinds of scientific data can be accessed through a consistent interface.

1.2 A shared abstraction layer¶

This abstraction layer enables a variety of applications and use cases be built without needing each application or consumer to go back to the source of the data. Because a significant portion of scientific organizations’ datasets is effectively identical, IDS provides a shared abstraction that everyone can build upon.

1.3 Defined as JSON Schema, consumed as SQL via Lakehouse¶

IDS is primarily defined as a JSON Schema: an IDS artifact centers on a schema.json, and each IDS instance is a JSON document that validates against it. JSON is how an IDS is defined and serialized — but it is not how the data is usually consumed. On the platform, every IDS instance of a given type is merged into shared Lakehouse tables (Delta / SQL): many individual JSON files become rows in common tables, so the data is queried across all of them at once rather than one JSON at a time. Once represented in IDS, the data is therefore queryable via Elasticsearch, SQL, and REST APIs, and shareable with an enterprise data mesh or data lake through the Lakehouse architecture and Delta tables / data catalogs.

2. What IDS is not for?¶

IDS is deliberately scoped and not suitable for every business use case. Below we listed the most frequently inquired topics, which are not what IDS is designed for:

Enforcing rules or controls on the values it holds. IDS is a faithful record, not a validation engine. It will store a negative concentration, an out-of-range pH, or a mislabeled sample if that is what the instrument produced. Range checks, acceptance criteria, and business rules belong in the consuming application or pipeline — not in the schema. (This is also why qualified and out-of-range values are preserved, not rejected — see Design Principles §1.3.1.)
Encoding strong semantic meaning or ontological relationships. IDS records what was measured — “analyte X had concentration Y in sample Z.” It does not assert what that means — “compound X engages target Y,” or “this batch failed release.” Those relational and ontological claims belong downstream, in a use-case ontology, a knowledge graph, or an ELN/LIMS. An IDS that starts asserting interpretations becomes an opinion rather than a record — the over-interpretation failure mode described in Design Principles §1.1.

2.1 IDS in relation to ontology, knowledge graph, and the semantic layer¶

2.1.1 Punchline: ontology-ready, not an ontology¶

IDS is ontology-ready, not an ontology. It makes scientific data machine-readable with significantly lower effort; an ontology makes it machine-interpretable and reasoning-ready. IDS is the structured record those higher layers build on — not the meaning layer itself.

2.1.2 The full picture: weak, structural semantics only¶

IDS does carry weak, structural semantics — a datacube declares which of its arrays are measures and which are dimensions; top-level arrays are named for what they hold (samples, systems, methods, results, datacubes); each record links via a foreign key to the runs entry that produced it. That light structure is exactly what makes IDS a useful abstraction layer that simplifies downstream work. What it does not do is assert meaning in the strong sense. The distinction is clearest across the two words people reach for:

“Meaning.” IDS provides representational meaning — this number is a concentration in nM, of analyte X, in sample Z. It does not provide inferential meaning — what those values imply (“compound X engages target Y,” “this batch failed release”).
“Relationship.” IDS encodes structural relationships — containment and reference links, such as which result belongs to which run. It does not encode semantic relationships — typed, reasoning-bearing links like derived-from, part-of, or realizes.

2.1.3 Ontology vs. knowledge graph vs. semantic layer¶

The strong semantic layer lives outside IDS — for TetraScience, in a separate ontology (entities such as workflow, study, experiment, instrument run, sample, and container, and the typed relationships among them) and the knowledge graphs built on it. Three neighboring concepts are easy to conflate, because they are different kinds of thing:

Concept	Kind of thing	Holds instance data?	Role
Ontology	A model / schema — entities and their typed relationships	No — the model only	Declares what exists and how it relates
Knowledge graph	A populated structure — an ontology + instances, stored as a graph	Yes	Holds the actual facts, related and queryable
Semantic layer	An architectural role — serving consistent meaning to consumers	Maps to data; need not hold it	The interface apps, analytics, and agents consume

These are not a strict stack, and the industry uses the terms inconsistently (some say “semantic layer” and mean a knowledge graph; others mean dbt-style metrics — so define the term inline when it matters). The reliable relationships: an ontology is the blueprint; a knowledge graph is that blueprint populated with instances; a semantic layer is the serving role, frequently fulfilled by a knowledge graph. A knowledge graph does not sit “inside” a semantic layer — it implements one.

2.1.4 How the pieces line up in the Tetra stack¶

IDS             → harmonized instance data — the grounded facts, structured and named
Ontology        → the model — entities and their typed relationships
Knowledge graph = Ontology + IDS-derived instances, fused into a queryable graph
Semantic layer  = the serving role that exposes that graph to apps, analytics, and agents
                  (frequently the knowledge graph itself, wearing that hat)

So the flow reads: IDS grounds the data → the ontology defines the meaning → the knowledge graph fuses the two → the semantic layer serves it. An IDS that starts asserting the ontology’s entities, relationships, or interpretations has stopped being a record and become an opinion — the over-interpretation failure mode described in Design Principles §1.1.