Designing a primary model

A primary model is the first place data from the raw file should be stored in memory.

In general, the structure of the primary model should closely resemble the structure of the input file (or at least the parts of the input file we are interested in), and preserve the raw value as it was in the file. This simplifies the mental model of the task script, making it easier to follow for current and future developers and reviewers.

This is also a good place to add validation around the raw file contents. Ensuring the contents are as expected simplifies downstream mapping since data types are guaranteed, and provides a place to raise sensible exceptions when an assumption is broken.

It’s important to get this step right, since all downstream steps use this structure.

There are a few different ways to construct a primary model, depending on the raw file contents:

Pydantic

Pydantic is the most widely used data validation library for Python. Pydantic is our go-to for modelling non-tabular data because:

  • In most cases pydantic.BaseModel can be used to construct a class structure which resembles the raw file structure.

  • Specified data types are validated automatically, and custom validators can be added to specific fields if needed.

  • Aliases can be used to simplify instantiation, while providing easy to understand field names downstream.

Pandera

Pandera is a library to validate DataFrame-like objects.

When the input (or part of the input) is a table, it can be useful to keep the data in a DataFrame and perform vectorized operations on it when mapping to IDS. However e.g. pandas.DataFrame is a nondescript input type for a function argument.

pandera.DataFrameModel provides a pydantic-like way of validating the contents of a DataFrame:

  • You can specify column data types

  • You can add custom validators for specific columns

  • You can use aliases to convert to more standard names for ease-of-use

  • Additionally, the model itself can be used as an Enum for the column names, making it easy to access specific columns in the DataFrame.

Note: pandera has 2 ways of specifying table schemas: pandera.DataFrameSchema and pandera.DataFrameModel. We always prefer to use pandera.DataFrameModel.

Dataclasses

Dataclasses can be used to model input data without validation.

Though performing some validation on the input data is preferred, sometimes it can be unnecessary.

For example, in text based files, we want to preserve the raw strings of numeric values to populate the raw_value field of the ts_ids_core.schema.RawValueUnit component. So it doesn’t make sense to cast the string to a float in a pydantic model at this early stage.