CSV-like files

Overview

This page outlines our guidelines for parsing data from CSV-like files to a primary model.

Strategy

“Tabular” Delimited Data

If the raw data is a “traditional” delimited data file, we recommend using pandas.

  • CSVs can be read via pandas.read_csv

  • TSVs can be read via pandas.read_csv with sep='\t'

  • Other delimited files can be read via pandas.read_csv with sep='<delimiter>'

Once the data is in a pandas.DataFrame, it can be validated using a primary model built with pandera.DataFrameModel.

“Unstructured” Text Files

It is common for files with the .csv extension to actually contain unstructured text data rather than just delimited tabular data.

Parsers can be written for unstructured text sections if its content is consistent. This needs multiple example files to confirm the structure is consistent.

Use ts-lib-parsy to create a parsy parser which:

  • Extracts data from the non-tabular sections of the file, validating the structure & content via a pydantic.BaseModel primary model.

  • Extracts tabular data into a pandas.DataFrame, validating the structure & content via a pandera.DataFrameModel primary model.

  • Combines the sections into the final primary model.