Logstash: How to Remove Field using Mutate Filter

In this comprehensive guide, we'll dive into the process of streamlining your Logstash data by removing unnecessary fields, such as the temporary storage _tmp field or any other superfluous data elements that clutter your output.

In Logstash, the powerful mutate filter allows you to manipulate fields in various ways, including removing fields from your Logstash event. The remove_field option of the mutate filter is the key to removing fields in Logstash.

The examples provided in each section are a great combination of my experience as well as people's questions I collected from various forums and communities. So, hopefully, you can find the solution to your specific use case in this guide.

The Logstash version used in this guide is 8.12.2, which is the latest version as of March 3, 2024.

Contents

• Basic Syntax for Removing Fields in Logstash
• How to Remove a Single Field in Logstash
• How to Remove Multiple Fields in Logstash
• How to Remove a Nested Field in Logstash
• How to Remove Field Conditionally in Logstash
• Frequently Asked Questions
  • 1. Performance Impact of Removing Fields in Logstash
  • 2. Alternative to Mutate Filter for Removing Fields
  • 3. Impact of Removing Fields on Downstream Processes
• Conclusion

Basic Syntax for Removing Fields in Logstash

To remove a field from the event in Logstash, you can use the remove_field option inside the mutate filter.

The properties of the remove_field option are as follows:

• Value type: array
• Default value: []

The syntax of the remove_field option is as follows:

filter {
  mutate {
    remove_field => [
      "field_name",
      "[nested][field_name]",
      # ...
    ]
  }
}

Throughout this tutorial, I will use the following sample event for demonstration:

{
  "@timestamp": "2023-12-02T01:00:00.000Z",
  "event": {
    "duration": 1000
  },
  "message": "Hello world",
  "_tmp": {
    "foo": "bar"
  }
}

How to Remove a Single Field in Logstash

The most basic operation when removing fields in Logstash is to delete a single field from your Logstash event.

To remove a single field in Logstash, you can use the remove_field option of the mutate filter: mutate { remove_field => [ "field_name" ] }.

The configuration is as follows:

filter {
  mutate {
    remove_field => [
      "field_name"
    ]
  }
}

To drop the unwanted _tmp field in the previous section, you can use the following configuration:

filter {
  mutate {
    remove_field => [
      "_tmp"
    ]
  }
}

The result after removing the _tmp field is as follows:

{
  "@timestamp": "2023-12-02T01:00:00.000Z",
  "event": {
    "duration": 1000
  },
  "message": "Hello world"
}

How to Remove Multiple Fields in Logstash

To remove multiple fields in Logstash, you can add multiple field names to the remove_field option as it accepts an array of field names: mutate { remove_field => [ "field1", "field2", ... ] }.

For example, to remove the _tmp and message fields, you can use the following configuration:

filter {
  mutate {
    remove_field => [
      "_tmp",
      "message"
    ]
  }
}

The result after removing the _tmp and message fields is as follows:

{
  "@timestamp": "2023-12-02T01:00:00.000Z",
  "event": {
    "duration": 1000
  }
}

Also Read: How to Remove All Fields in Logstash using the Prune Filter

How to Remove a Nested Field in Logstash

To remove a nested field in Logstash, you can use the remove_field option with the nested field syntax using square brackets: mutate { remove_field => [ "[field][subfield]" ] }.

For example, to cancel the event.duration field out, you can use the following configuration:

filter {
  mutate {
    remove_field => [
      "[event][duration]"
    ]
  }
}

The result after removing the event.duration field is as follows:

{
  "@timestamp": "2023-12-02T01:00:00.000Z",
  "event": {}
}

How to Remove Field Conditionally in Logstash

You can also clean a field with a specified condition in Logstash using the if statement.

With the if statement, you will be able to remove a field based on a value, a range of values, or any other conditions.

For example, to remove the _tmp field if the status field is success, you can use the following configuration:

filter {
  if [status] == "success" {
    mutate {
      remove_field => [
        "_tmp"
      ]
    }
  }
}

Similarly, to remove the _tmp field if the event.duration is greater than 1000, you can use the following configuration:

filter {
  if [event][duration] > 1000 {
    mutate {
      remove_field => [
        "_tmp"
      ]
    }
  }
}

Frequently Asked Questions

1. Performance Impact of Removing Fields in Logstash

Q: How does the removal of fields using the mutate filter affect the performance and processing time of Logstash, especially in large-scale data processing environments?

A: Removing fields with the mutate filter in Logstash can enhance performance by reducing the volume of data processed and transferred, especially crucial in high-throughput environments.

2. Alternative to Mutate Filter for Removing Fields

Q: Are there any alternative methods or plugins within Logstash for field removal that might offer benefits over the mutate filter in certain scenarios?

A: While the mutate filter is a common choice for field manipulation, Logstash offers other plugins like the prune filter for more dynamic field removal based on conditions or patterns, potentially offering more flexibility in certain scenarios.

3. Impact of Removing Fields on Downstream Processes

Q: How does removing fields in Logstash impact downstream processes or integrations, especially in terms of data consistency and availability for analytics or other data processing pipelines?

A: Regarding downstream impacts, field removal can streamline data flow and improve efficiency but requires careful planning to ensure necessary data remains available for analytics and processing, avoiding disruptions in data-driven decision-making processes.

Conclusion

We have learned how to remove fields in Logstash using the mutate filter in this tutorial.

You can remove a single field, multiple fields, or nested fields in Logstash using the remove_field option. Furthermore, you can also use the if/else statement to remove a field with some condition.

For more nuanced details and specific use cases, exploring Logstash documentation and community forums can provide deeper insights.

If you have any questions or need further assistance, feel free to ask in the comments section below.