Highly Unstructured Data

Some vendor output is noisy — more a log or text blob than a real data structure, with enough variation across firmware revisions to make full schematization and mapping unsustainable. Over-ambitious schematization here is like over-fitting to noisy data. Use these patterns instead.

Preserve the unstructured string

When a vendor encodes configuration as an opaque, instrument-specific string (XML, JSON-in-a-string, CSV) whose schema evolves across firmware versions, do not fully parse it — a future revision will silently break the mapping. Surface the string as-is for provenance, and selectively extract only the few scientifically critical fields that need structured querying or cross-instrument comparison.

Example — Waters Empower InstrumentMethod.Setups[*].Detail. Each Setup contains a Detail field: a full XML document for one module’s method parameters, whose root element changes with module type (AcquityPDAMethod, AcquitySMDIMethod, …) and child elements vary by firmware. The IDS preserves the full XML verbatim in instrument.setups[0].detail, and surfaces a couple of critical fields as first-class structured values elsewhere:

{
  "instrument": {
    "setups": [
      { "inst_on_status": true, "detail": "<?xml ...><AcquityPDAMethod>...</AcquityPDAMethod>" }
    ]
  },
  "detection": {
    "uv_vis": [
      {
        "name": "AcquityPDAMethod",
        "wavelength_range": {
          "minimum": { "value": 210, "unit": "Nanometer" },
          "maximum": { "value": 400, "unit": "Nanometer" }
        }
      }
    ]
  }
}

Here SpectralChannel.StartWavelength / EndWavelength are lifted from the XML to detection.uv_vis[0].wavelength_range, where they can be queried across instruments. The rule of thumb: extract a field when it is needed for structured querying or cross-instrument consistency; preserve everything else verbatim to avoid over-fitting the schema to one firmware version.

The Parameter pattern

The complement to preserving the unstructured string: when a section is an open-ended or vendor-variable set of key–value settings whose keys differ across instruments, methods, or firmware, do not mint a fixed field for each. Model them as a generic list of { name, value } entries — this keeps the schema stable as the set drifts, at the cost of those values not being individually typed or validated. Extract one into a first-class field only when a consumer needs to query or compare it. Use the canonical Parameter component from ts-ids-core rather than inventing your own shape — it handles dynamically-typed values and preserves the raw string alongside the typed value.

{
  "settings": [
    { "name": "param_1", "value": "x" },
    { "name": "param_2", "value": "y" }
  ]
}