Writing Parquet files in Task Scripts

Overview

This page outlines our guidelines for writing parquet files to the Tetra Data Lake.

Guidelines

Parquet files are used for datacubes or results when the data is too large to store in the IDS JSON. First a pandas.DataFrame should be created in the desired structure. Then the pandas.DataFrame can be written to a parquet file in the data lake using the task_script_utils parquet.pandas_dataframe_to_parquet function.

For deciding whether data belongs in a parquet file at all, see Datacubes and Results.

Writing datacubes parquet

When the IDS contains the array datacube_metadata, a parquet file containing the data for the datacube should be written to the data lake along with the IDS JSON.

Write the file with the standard path, category and compression settings:

context.write_file(
    file_name="0.json/datacubes0.parquet",
    file_category="PROCESSED",
    gzip_compress_level=0,
)

  • The path must be 0.json/datacubes0.parquet. The standard path lets Lakehouse ingestion locate the file directly. Other names are supported, but they ingest more slowly.

  • Use gzip_compress_level=0. These files are read repeatedly by Lakehouse ingestion and by consumers, and the cost of decompressing on every read outweighs the storage saved. Parquet already compresses its own column data.

When writing parquet for datacube data, there are restrictions on the allowed columns:

  • datacube_index (unsigned 64 bit int): This corresponds to the datacube_metadata[*].index field in the IDS instance.

  • measure_{m}_value (float): Starting at m=0 e.g. measure_0_value, measure_1_value, etc. storing the data for a particular measure in the datacube.

  • dimension_{d}_value (float): Starting at d=0 e.g. dimension_0_value, dimension_1_value, etc. storing the data for a particular dimension of the datacube.

  • Any fields defined in the datacube_metadata object of the IDS may optionally be included in the datacube parquet file, this makes the parquet file easier to use by including both metadata and data in one place.

Note that measure_{m}_value and dimension_{d}_value are double columns, so a signal whose values are strings cannot use datacube_metadata and must stay inline in datacubes.

Including a field as a parquet column rather than in the IDS JSON also lets that field vary per row instead of being constant across the datacube, which is how the number of datacube_metadata entries is kept far below the number of parquet rows. See Moving per-row metadata into the Parquet file.

Writing results parquet

When writing parquet for results, there are no restrictions on the column names. However, you must ensure that each parquet file written for results has the same schema. This means that the column names and types must be the same for each parquet file written for results.

Warning

Unlike datacube parquet, results parquet is not currently transformed into Lakehouse tables. The data is retrievable by downloading the file using its file ID, but it is not SQL-queryable. Support for it is being formalized, so expect this to change. Because of this, only move results data out of the IDS JSON when the data set is large enough for the size of the IDS JSON to be the bigger problem, and document the choice in the IDS README. See Datacubes and Results.