Date and Time

Use Cases

  • Time will be searchable once it’s parsed into the ISO 8601 UTC format (see ElasticSearch built-in date format) otherwise we can’t guarantee instrument native date formats can be understood by Elasticsearch.

  • Parsing datetimes into the ISO 8601 UTC format remove ambiguity of the datetime formats within and across IDSs.

  • Standardized datetimes allow lexicographic ordering.

Format and parsing

  • Datetimes should be parsed into ISO 8601 format with UTC, e.g. 2021-01-26T23:19:09Z. Use UTC (Zulu) time. Convert time zones to Zulu time.

  • Elasticsearch date detection is currently disabled by TetraScience. This means dynamic mapping is disabled. We do this because we don’t want Elasticsearch indexing to fail if a malformed date format is used. So, we index every string type as keyword type.

  • Do not use the JSON Schema format, date-time, in the IDS. If you cannot convert a timestamp to ISO 8601, then we recommend preserving the timestamp in the original, not standardized format in the IDS JSON instance. We recommend a permissive approach; standardize if you can, but don’t fail parsing if you can’t, and store the raw value.

  • If a datetime string’s time is not available, simply parse the “date” part because the value will still follow ISO 8601. Vice versa if only the “time” part is available (e.g. 12:00:38.012).

  • If an instrument does not provide the ability to distinguish ambiguous datetimes, like an offset or timezone; another option is to create a protocol config in which a user can enter the datetime format which can be used by the task script to parse the datetimes provided in the primary data.

In ts-ids-core you can model datetime values using a value-raw value pair to capture the parsed datetimes and the raw value. Capturing the raw value will allow you to be permissive in your parsing strategy.

from ts_ids_core.base.ids_element import IdsElement
from ts_ids_core.base.ids_field import IdsField

class DatetimePair(IdsElement):
    value: str = IdsField(description="datetime value standardized to ISO 8601 UTC.")
    raw_value: str = IdsField(description="The raw datetime value provided in the primary data.")