elasticsearch.json

The elasticsearch.json file is a component of the IDS artifact. It is similar to elastic’s Mapping concept and allows IDS developers to have more control over how IDS instances will be indexed into elasticsearch. elasticsearch.json is used to define:

• IDS attributes that will be of type nested in the Elasticsearch document

• IDS attributes that will not be included in the Elasticsearch document

• Additional dynamic mappings that can be included

Generate elasticsearch.json

TetraScience created a public python package which will automatically generate elasticsearch.json once you define your IDS in a schema.json file. To create elasticsearch.json, run the generator pointing to the directory where your schema.json file lives. By default, all arrays of objects are created as type nested, and datacubes are included as a non-searchable field. To customize after generation, you can:

1. Remove the properties from the mapping for those properties that you do not want to be of type nested.

2. Add paths to nonSearchableFields for those fields you want to exclude from the Elasticsearch document.

Note

It is a platform requirement that datacubes are non-searchable and this cannot be changed by the user.

elasticsearch.json Design Guidelines

The following sections describe things to consider when creating and customizing an elasticserach.json document.

For an example elasticsearch.json, expand the following nested example:

Example

If you have the following schema.json:

{
  "additionalProperties": false,
  "properties": {
    "results": {
      "items": {
        "additionalProperties": false,
        "properties": {
          "peaks": {
            "items": {
              "additionalProperties": false,
              "properties": {
                "number": {
                  "type": "integer"
                },
                "name": {
                  "type": "string"
                }
              },
              "type": "object"
            },
            "type": "array"
          }
        },
        "type": "object"
      },
      "type": "array"
    },
    "my_really_long_array": {
      "items": {
        "additionalProperties": false,
        "properties": {
          "foo": {
            "type": "string"
          },
          "bar": {
            "type": "string"
          }
        },
        "type": "object"
      },
      "type": "array"
    }
  },
  "type": "object"
}

Then your elasticsearch.json could look like the following:

{
  "mapping": {
    "properties": {
      "result": {
        "type": "nested",
        "properties": {
          "peaks": {
            "type": "nested"
          }
        }
      }
    },
    "dynamic_templates": []
  },
  "nonSearchableFields": ["my_really_long_array"]
}

Mapping and Defining nested Attributes

Click here for the Elasticsearch official documentation describing the nested type.

It is recommended that you define all fields which are defined as an array of objects in schema.json as nested. All fields of type nested should be defined under the mapping.properties object within elasticsearch.json. To review how properties within an array of objects is assigned as nested in elasticsearch.json, review the example above.

Do not define a field to be nested if you have any of these conditions:

• You have only a few users who are searching and may not need to use nested types

• The array only contains one item

• You want to save storage space in Elasticsearch

• You want to use ignore_malformed as it does not work with nested types

• You want to use Elasticsearch query string as it does not work with nested types

nonSearchableFields

Fields defined here will not be indexed into Elasticsearch. You want to exclude fields when:

• You will most likely not search this field

• You want to save space in Elasticsearch

• You want to speed up Elasticsearch indexing

• Your file is larger than 100 MB as Elasticsearch indexing currently has a 100 MB limit

nonSearchableFields uses a syntax similar to lodash omit to exclude. Use the official lodash syntax here: Lodash omit docs

Dynamic Mapping

It is theoretically possible to include a dynamic type mapping based on either path pattern or data type. Currently, this feature is not used extensively. For details, please contact your Customer Success representative.