Text based files

Overview

This page outlines our guidelines for parsing data from text based files to a primary model.

Strategy

Text based files usually contain some amount of structure, typically a mix between key-value sections and tabular data.

Use ts-lib-parsy to create a parsy parser which:

  • Extracts data from the non-tabular sections of the file, validating the structure & content via a pydantic.BaseModel primary model.

  • Extracts tabular data into a pandas.DataFrame, validating the structure & content via a pandera.DataFrameModel primary model.