Writing Parquet files in Task Scripts

Overview

This page outlines our guidelines for writing parquet files to the Tetra Data Lake.

Guidelines

Parquet files are used for datacubes or results when the data is too large to store in the IDS JSON. First a pandas.DataFrame should be created in the desired structure. Then the pandas.DataFrame can be written to a parquet file in the data lake using the task_script_utils parquet.pandas_dataframe_to_parquet function.

Writing datacubes parquet

When the IDS contains the array datacube_metadata, a parquet file containing the data for the datacube should be written to the data lake along with the IDS JSON.

When writing parquet for datacube data, there are restrictions on the allowed columns:

• datacube_index (unsigned 64 bit int): This corresponds to the datacube_metadata[*].index field in the IDS instance.

• measure_{m}_value (float): Starting at m=0 e.g. measure_0_value, measure_1_value, etc. storing the data for a particular measure in the datacube.

• dimension_{d}_value (float): Starting at d=0 e.g. dimension_0_value, dimension_1_value, etc. storing the data for a particular dimension of the datacube.

• Any fields defined in the datacube_metadata object of the IDS may optionally be included in the datacube parquet file, this makes the parquet file easier to use by including both metadata and data in one place.

Writing results parquet

When writing parquet for results, there are no restrictions on the column names. However, you must ensure that each parquet file written for results has the same schema. This means that the column names and types must be the same for each parquet file written for results.