Data Integrity¶

How to preserve data from a source system, alongside a form that downstream consumers can compute with.

Rule¶

Both the display formatted value and the typed value are useful, so the recommended approach for IDS design is to store both of them in the output.

Source data is often in text-based formats like .txt or .csv which means the data is stored as text strings. For example, the string "0.100" could appear in a .txt file, and although it represents a number, it contains more information than can be stored in the floating point data format because it is a string: "0.100" contains trailing zeros which would be lost when converting to a float e.g. 0.1.

However, working with display-formatted string data is much more difficult downstream for computational tasks where tools expect structured data types like float, so it is also valuable to parse the source display format into a structured data format to improve downstream usability.

Applying the rule¶

Source representation	IDS representation
Display-formatted and parseable	Keep the original string as-is and also parse it into a typed value
Display-formatted but not parseable	Keep the original string as-is and leave the typed value as `null`
Already typed	Store it as the same structured type; leave the display-formatted field as `null`, or populate it with a documented string-formatted copy

For example, if the source representation is a text format which contains "0.100", we want to store both "0.100" as a string in the IDS, as well as the numeric typed 0.1.

If the source representation is in a structured data format like a SQLite file and it contains a float 0.1, then in IDS data we will also store it as a structured numeric type 0.1. Then the display formatted field can be left as null, or populated with a string formatted copy of the source value as long as it’s documented in the raw to IDS mapping table, see ### Raw to IDS Mapping.

Terminology¶

A source representation is how a piece of data is formatted in the source format (may be in a display format or a typed value). A target representation is how it is formatted in the output (may be in a display format or a typed value). For this guidance, the target is always IDS, but the principles are general.

The different kinds of format are:

Display format — a text value which is meant to be displayed to a person. For numeric data, this can encode more than just the magnitude of the number, e.g. it can include trailing zeros, thousands separators, scientific notation, and more (e.g. 0.100, 1,234.5, or 0,800E10). May be found in CSV, XML, TXT, PDF, etc.
Typed value — a value in a structured data type, such as a double column in a SQLite table or a float returned by a vendor SDK, or a numeric field in an IDS. There is no text-based display formatting in typed values.

Components for storing both formats¶

Several IDS components include fields for a display format and a typed value. For the full definition of each component, see Components:

Component	Display format	Typed value	Use for
`RawValueUnit`	`raw_value`	`value`, with `unit` alongside	A scalar quantity with a unit
`Time`, `SampleTime`	`raw.start`, `raw.created`, …	`start`, `created`, …	The common set of process and data datetimes
`Parameter`	`value`	`numerical_value`, `string_value`, or `boolean_value`, with `numerical_value_unit` alongside	A key/value pattern whose entries vary in type, where `value_data_type` declares the type

Aim to use these existing components where they fit, otherwise follow the guidance on this page when creating new IDS fields.

Parsing and validation responsibilities¶

Connectors extract data from a source system and put it in the Tetra Data Lake. Connectors should keep the source representation as-is (avoid narrowing/transforming it where possible), so that parsing belongs in the raw-to-IDS step. See Designing a primary model.

Comparing source data with IDS data field by field¶

Compare like with like. Compare a display format in the IDS against the source’s display format using string comparison, and compare a typed value in the IDS against the source’s typed value using the specific comparison logic for that data type.

For example, for numeric data in a typed format, use floating point arithmetic which only compares magnitude, e.g. 0.500 == 0.5 for floats.

Why display formatting can matter, e.g. significant figures¶

There are many kinds of information which can be encoded in the source’s display format and therefore should be kept for data integrity: trailing zeros, thousands separators, scientific notation, locale-specific formatting. The simplest general approach for keeping information like this is to keep the original string as-is in the IDS output, so that no information is lost.

A common reason for preserving the display format is to keep significant figures from the source representation. Trailing zeros e.g. in 0.500 can represent information about how precisely the value was reported. This guidance doesn’t address significant figures in a targeted way, e.g. by using a Decimal data type or storing precision as a separate IDS field (although these approaches may make sense in some scenarios). Instead, this guidance focuses on the more general “display format” which covers all kinds of information which can be encoded in the source’s display string.