task_script_utils.parquet module#

class PandasLikeDataFrame(*args, **kwargs)[source]#

Bases: Protocol

to_parquet(path: Any, engine: Any = Ellipsis, compression: Any = Ellipsis, index: bool = Ellipsis) → Any[source]#
class PolarsLikeDataFrame(*args, **kwargs)[source]#

Bases: Protocol

write_parquet(file: BinaryIO, *, compression: Literal['lz4', 'uncompressed', 'snappy', 'gzip', 'lzo', 'brotli', 'zstd']) → None[source]#
pandas_dataframe_to_parquet(df: PandasLikeDataFrame, context: Any, file_name: str, *, engine: Literal['pyarrow', 'fastparquet'] = 'pyarrow', compression: str = 'snappy') → str[source]#

Converts a Pandas-like DataFrame to Parquet format and writes the result to storage.

Parameters:

  • df – A Pandas DataFrame instance.

  • context (Any) – The task context with a write_file method.

  • file_name (str) – The target file name for storage.

  • engine (str) – Parquet engine (“pyarrow” or “fastparquet”).

  • compression (str) – Compression method (“snappy”, “gzip”, “zstd”, etc).

Returns:

The file ID assigned by the storage system.

Return type:

str

polars_dataframe_to_parquet(df: PolarsLikeDataFrame, context: Any, file_name: str, *, compression: Literal['lz4', 'uncompressed', 'snappy', 'gzip', 'lzo', 'brotli', 'zstd'] = 'snappy') → str[source]#

Converts a Polars-like DataFrame to Parquet format and writes the result to storage.

Parameters:

  • df – A Polars DataFrame instance.

  • context (Any) – The task context with a write_file method.

  • file_name (str) – The target file name for storage.

  • compression (str) – Compression method (“snappy”, “gzip”, “zstd”, etc).

Returns:

The file ID assigned by the storage system.

Return type:

str