Task script repo structure

Overview

This page outlines our guidelines for the structure of a task script repository.

Guidelines

main.py

The main.py file should contain the main function, which is the entry point for the task script. This function should take 2 arguments:

  • task_inputs: A dictionary containing input_file_pointer

  • context: The context object

This function should perform the following:

  • Read the input file using context.read_file(input_file_pointer)

  • Invoke functions from the package to parse the input file to the IDS

  • Invoke context.validate_ids to validate the output

  • Invoke context.write_file to write the output file to Tetra (the platform)

This function should return:

  • output_file_pointer: A pointer to the output file in the data lake, which is returned when calling context.write_file

README.md

The readme should contain the following sections:

  • Version

  • Table of Contents

  • Summary

  • Input

    • Supported Files

  • Output

  • Assumptions

  • Limitations

  • Supported Use Cases

  • Changelog

CONTRIBUTING.md

This file is a place to put any notes for developers working on the task script. It is not required, but is recommended.

config.json

This JSON contains details about how the task script should be run on the platform. It contains the following fields:

  • language: The language the task script is written in. Always python.

  • sdkVersion: The version of the SDK to build the task script with. Always v2.

  • secureMode: Whether the task script should be run in secure mode. Always true.

  • runtime: The runtime to use. Always python3.11.

  • functions: The entrypoint functions which are available. A list of objects with the following fields:

    • slug: The task script slug

    • function: The name of the function to invoke, typically main.main.

    • allowedIds: A list of IDS which are allowed to be written by the task script. A list of objects containing:

      • namespace: The namespace of the IDS

      • slug: The slug of the IDS

      • version: The version of the IDS

example-input directory

This directory contains example input files for the task script. There should be at least 1 file per supported input file type. Each file should be named <file-type>.<extension>, e.g. example.csv, example.json, example.txt, etc. There should be a corresponding output JSON in the example-output directory with the same name.

example-output directory

This directory contains example output files for the task script. There should be at least 1 file per supported input file type. Each file should be named <file-type>.<extension>.json, e.g. example.csv.json, example.json.json, example.txt.json, etc.

__tests__ directory

This directory contains the tests for the task script. It should contain 2 subdirectories:

  • unit: Unit tests for the task script

    • Should contain 1 test module per module in the package, containing tests for the classes and functions in that module

  • integration: Integration tests for the task script

    • Should contain 1 test module test_main.py which contains snapshot tests for the main function, using the example input and output files in the example-input and example-output directories

package directory

The package directory should be named as the task script slug in snake-case. It contains the main logic for the primary model, parsing the raw file to the primary model, and mapping the primary model to the IDS. Modules should be split in an organized fashion, keeping related functions together.