Elasticsearch: How to Set Default Value for Missing Fields

There are several ways to set default values for null or missing fields in Elasticsearch depending on how you ingest your data.

In this tutorial, I will show you 2 ways to set default values for missing fields in Elasticsearch:

• Using the null_value parameter in index mapping.
• Using the ingest pipeline.

Contents

• Set Default Values for Missing Fields in Elasticsearch using the null_value Parameter
• Set Default Values for Missing Fields in Elasticsearch using Ingest Pipeline
  • Apply Ingest Pipeline to Indexing Request
  • Apply Ingest Pipeline to Logstash
• Conclusion

Set Default Values for Missing Fields in Elasticsearch using the null_value Parameter

The null_value parameter allows you to set a default value for missing fields in Elasticsearch. However, please note that it only works if the field value is explicitly set to null.

For example, if you have the following index mapping:

{
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      }
    }
  }
}

If you want to set the name field to Unknown if it is missing, you can do so by adding the null_value parameter to the mapping:

{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "null_value": "Unknown"
      }
    }
  }
}

Now, if you index a document with the name field explicitly set to null, the name field will be set to Unknown:

PUT my_index/_doc/1
{
  "name": null
}

Then, you can get the document and see that the name field is set to Unknown:

GET my_index/_doc/1

{
  "_index": "my_index",
  "_type": "_doc",
  "_id": "1",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "name": "Unknown"
  }
}

However, if you index a document with the name field missing, the name field will not be set to Unknown, it will be ignored instead:

POST my_index/_doc/2
{
  "age": 20
}

Querying the document will return the following result:

GET my_index/_doc/2

{
  "_index": "my_index",
  "_type": "_doc",
  "_id": "2",
  "_version": 1,
  "_seq_no": 1,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "age": 20
  }
}

If you want to set the name field to Unknown if it is missing, I will show you how to use the ingest pipeline instead.

Set Default Values for Missing Fields in Elasticsearch using Ingest Pipeline

The ingest pipeline allows you to perform different operations on your data before indexing it into Elasticsearch.

One of the operations that you can perform is to set a value for a field.

We can take advantage of this operation to set a default value for missing fields in Elasticsearch.

You can create an ingest pipeline using the following request:

PUT _ingest/pipeline/my_pipeline
{
  "description": "Set default value for missing fields",
  "processors": [
    {
      "set": {
        "field": "name",
        "value": "Unknown",
        "if": "ctx.name == null"
      }
    }
  ]
}

The above pipeline will set the name field to Unknown if it is missing.

Apply Ingest Pipeline to Indexing Request

Now, if you index a document with the name field missing, the name field will be set to Unknown:

POST my_index/_doc/1?pipeline=my_pipeline
{
  "age": 20
}

Querying the document will return the following result:

GET my_index/_doc/1

{
  "_index": "my_index",
  "_type": "_doc",
  "_id": "1",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "age": 20,
    "name": "Unknown"
  }
}

Apply Ingest Pipeline to Logstash

If you are using Logstash to ingest data into Elasticsearch, you can apply the ingest pipeline to Logstash by using the pipeline option in the Elasticsearch output plugin:

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "my_index"
    pipeline => "my_pipeline"
  }
}

Conclusion

In this tutorial, you have learned how to set default values for missing fields in Elasticsearch.

If you have any questions, please feel free to leave a comment below.