Context API usage

Overview

This page outlines our guidelines for using the context API in raw-to-ids task scripts.

High-level

The context object provides functions for you to read files, write files, update metadata, get pipeline configuration information, and more.

context.read_file

Takes an input file reference and returns a dictionary whose key depends on the form parameter, giving you access to the file content, a file object, or the local path of a copy of the file. This method accepts the following parameters:

• file (dict): File pointer, which is an input to the main function.

• form (string): How to read the input:

  • If form="body" (the default), then result[‘body’] holds the contents of the file as a byte array. This approach cannot handle large files that don’t fit in memory.

  • If form="file_obj", then result[‘file_obj’] is a file-like object that can be used to access the body through streaming. This object can be passed to Python libraries, like Pandas.

  • If form="download", then result[‘download’] is the file name of a local file that has been downloaded from the specified data lake file. This is useful when the data needs to be processed by native code (for example, SQLite) or an external utility program.

input_file_info = context.read_file(input_file_pointer)
file_contents = input_file_info.get("body", b"")

context.write_file

Writes an output file to the Data Lake. This method accepts the following parameters:

• content (string, bytes object, byte stream, dict): The content to be written out. The content can only be dict type when file_category is set to IDS, otherwise it will error. In this case, IDS validation will be automatically performed.

• file_name (string): The name of the file to be written. The full S3 path is determined by the platform. As of ts-sdk v1.3.2, the name cannot navigate upward in the file path (for example, ../file_name). Use "0.json" as a default.

• file_category (string): The category of the file. Can be IDS, TMP, or PROCESSED.

output_file_pointer = context.write_file(
    content=output_content,
    file_name="0.json",
    file_category="IDS",
)

context.validate_ids

Checks the validity of IDS content provided in data. Throws an error if not valid. This method accepts the following parameters:

• data (dict): The JSON content of the IDS file.

• namespace (string): A namespace defines a realm; only those with the appropriate permissions can use the artifacts in that realm. There are three main categories of namespace:

  • "common"

  • "client"

  • "private"

• slug (string): The slug of the IDS.

• version (string): The version of the IDS.

This method returns:

• boolean: Returns a boolean value indicating whether the IDS is valid (true). It throws an error if the IDS is not valid.

context.validate_ids(
    data=ids, namespace=ids_namespace, slug=ids_slug, version=ids_version
)

context.allowed_ids.get_first().to_tuple

Returns the namespace, slug, and version of the first allowed IDS.

ids_namespace, ids_slug, ids_version = context.allowed_ids.get_first().to_tuple()

context.get_logger

Returns the logger object, which currently has one method: log(data).

logger = context.get_logger()