elasticsearch.json

The elasticsearch.json file is a component of the IDS artifact. It is similar to elastic’s Mapping concept and allows IDS developers to have more control over how IDS instances will be indexed into elasticsearch. elasticsearch.json is used to define:

• IDS attributes that will be of type nested in the Elasticsearch document

• IDS attributes that will not be included in the Elasticsearch document

• Additional dynamic mappings that can be included

• Whether to allow custom mappings that bypass validation (via allowCustomMapping)

Generate elasticsearch.json

TetraScience created a public python package which will automatically generate elasticsearch.json once you define your IDS in a schema.json file. To create elasticsearch.json, run the generator pointing to the directory where your schema.json file lives. By default, all arrays of objects are created as type nested, and datacubes are included as a non-searchable field. To customize after generation, you can:

1. Remove the properties from the mapping for those properties that you do not want to be of type nested.

2. Add paths to nonSearchableFields for those fields you want to exclude from the Elasticsearch document.

Note

It is a platform requirement that datacubes are non-searchable and this cannot be changed by the user.

elasticsearch.json Design Guidelines

The following sections describe things to consider when creating and customizing an elasticserach.json document.

For an example elasticsearch.json, expand the following nested example:

Example

If you have the following schema.json:

{
  "additionalProperties": false,
  "properties": {
    "results": {
      "items": {
        "additionalProperties": false,
        "properties": {
          "peaks": {
            "items": {
              "additionalProperties": false,
              "properties": {
                "number": {
                  "type": "integer"
                },
                "name": {
                  "type": "string"
                }
              },
              "type": "object"
            },
            "type": "array"
          }
        },
        "type": "object"
      },
      "type": "array"
    },
    "my_really_long_array": {
      "items": {
        "additionalProperties": false,
        "properties": {
          "foo": {
            "type": "string"
          },
          "bar": {
            "type": "string"
          }
        },
        "type": "object"
      },
      "type": "array"
    }
  },
  "type": "object"
}

Then your elasticsearch.json could look like the following:

{
  "mapping": {
    "properties": {
      "result": {
        "type": "nested",
        "properties": {
          "peaks": {
            "type": "nested"
          }
        }
      }
    },
    "dynamic_templates": []
  },
  "nonSearchableFields": ["my_really_long_array"]
}

Mapping and Defining nested Attributes

Click here for the Elasticsearch official documentation describing the nested type.

It is recommended that you define all fields which are defined as an array of objects in schema.json as nested. All fields of type nested should be defined under the mapping.properties object within elasticsearch.json. To review how properties within an array of objects is assigned as nested in elasticsearch.json, review the example above.

Do not define a field to be nested if you have any of these conditions:

• You have only a few users who are searching and may not need to use nested types

• The array only contains one item

• You want to save storage space in Elasticsearch

• You want to use ignore_malformed as it does not work with nested types

• You want to use Elasticsearch query string as it does not work with nested types

nonSearchableFields

Fields defined here will not be indexed into Elasticsearch. You want to exclude fields when:

• You will most likely not search this field

• You want to save space in Elasticsearch

• You want to speed up Elasticsearch indexing

• Your file is larger than 100 MB as Elasticsearch indexing currently has a 100 MB limit

nonSearchableFields uses a syntax similar to lodash omit to exclude. Use the official lodash syntax here: Lodash omit docs

Dynamic Mapping

It is theoretically possible to include a dynamic type mapping based on either path pattern or data type. Currently, this feature is not used extensively. For details, please contact your Customer Success representative.

allowCustomMapping

The allowCustomMapping field is an optional boolean flag at the top level of elasticsearch.json that controls validation behavior.

Default value: false

Purpose: When set to true, this field allows you to bypass the IDS validator’s check that compares your elasticsearch.json mapping section against the auto-generated output from the elasticsearch.json generator.

When to use:

• You need to customize the mapping section beyond what the auto-generator produces

• You have specific Elasticsearch mapping requirements that differ from the standard generated mappings

• You want to maintain manual control over your Elasticsearch mappings

Validation behavior:

• When allowCustomMapping is false (default): The IDS validator will run the elasticsearch.json generator and compare the output’s mapping section with your saved elasticsearch.json file. Validation fails if they don’t match.

• When allowCustomMapping is true: The IDS validator will skip the mapping comparison check, allowing you to use custom mappings.

Warning

Setting allowCustomMapping to true means you are responsible for ensuring your custom mappings are correct and compatible with your IDS schema. Incorrect mappings may cause indexing failures or unexpected search behavior in Elasticsearch. The IDS validator will still do some basic checks (e.g. checking if a field with nested properties exists in the schema and is an array of objects), but it will not catch all issues.

Example:

{
  "mapping": {
    "properties": {
      "custom_field": {
        "type": "nested",
        "properties": {
          "special_property": {
            "type": "keyword"
          }
        }
      }
    },
    "dynamic_templates": []
  },
  "nonSearchableFields": ["my_really_long_array"],
  "allowCustomMapping": true
}