Logstash Filter Tutorial: 7 Common Options

There are some basic options supported by all Logstash filter plugins that I often use to enrich the data.

For example, we can add a new field, remove an existing field, and more.

This tutorial will show you how to do that. Let's get started!

Contents

• 7 Common Options in Logstash Filter Plugins
  • 1. add_field
  • 2. remove_field
  • 3. add_tag
  • 4. remove_tag
  • 5. id
  • 6. enable_metric
  • 7. periodic_flush
• Conclusion

7 Common Options in Logstash Filter Plugins

There are 7 options that are supported by all Logstash filter plugins, which means you can use them in any filter plugin.

In this tutorial, I will use the mutate filter plugin as an example, you can use any filter plugin based on your needs.

I will also use the following sample data for demonstration throughout this tutorial:

{
  "message": "Minh Vu says hello to the world!",
  "status": 200,
  "timestamp": "2021-11-23T17:36:00Z"
}

For the input and output part of the filter config, I will use this config to parse JSON lines log:

input {
  file {
    path => "/home/dminhvu/elastic/example.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "\n"
      what => "next"
    }
  }
}
 
filter {
  # put the filter here
}
 
output {
  file {
    path => "/home/dminhvu/elastic/output.log"
    codec => "json_lines"
  }
}

1. add_field

In Logstash, the add_field action is used to add a new field to the event.

• Value type: hash
• Default value: {}

The syntax to add a single field is as follows:

filter {
  mutate {
    add_field => {
      "new_field_name" => "new_field_value"
    }
  }
}

To add multiple fields:

filter {
  mutate {
    add_field => {
      "new_field_name_1" => "new_field_value_1"
      "new_field_name_2" => "new_field_value_2"
    }
  }
}

For example, we can add a new field fine with the value true to the event:

filter {
  mutate {
    add_field => {
      "fine" => "true"
    }
  }
}

Using the above config will yield the result:

{
  "fine": true,
  "message": "Minh Vu says hello to the world!",
  "status": 200,
  "timestamp": "2021-11-23T17:36:00Z"
}

2. remove_field

To remove a field from the event in Logstash, we can use the remove_field action.

• Value type: array
• Default value: []

The syntax is as follows:

filter {
  mutate {
    remove_field => [ "field_name" ]
  }
}

To remove multiple fields:

filter {
  mutate {
    remove_field => [ "field_name_1", "field_name_2" ]
  }
}

For example, we can remove the status field from the event:

filter {
  mutate {
    remove_field => [ "status" ]
  }
}

Here is the result:

{
  "message": "Minh Vu says hello to the world!",
  "timestamp": "2021-11-23T17:36:00Z"
}

3. add_tag

The add_tag action is used to add a tag to the event.

• Value type: array
• Default value: []

The syntax is as follows:

filter {
  mutate {
    add_tag => [ "tag_name" ]
  }
}

To add multiple tags:

filter {
  mutate {
    add_tag => [ "tag_name_1", "tag_name_2" ]
  }
}

4. remove_tag

To remove a tag from the event, we can use the remove_tag action.

• Value type: array
• Default value: []

The syntax is as follows:

filter {
  mutate {
    remove_tag => [ "tag_name" ]
  }
}

To remove multiple tags:

filter {
  mutate {
    remove_tag => [ "tag_name_1", "tag_name_2" ]
  }
}

5. id

The id action is used to set the ID of the plugin configuration, which is useful when you need to identify multiple plugins of the same type.

• Value type: string
• Default value: no default value, automatically generated if not specified

The syntax is as follows:

filter {
  mutate {
    id => "plugin_id"
  }
}

6. enable_metric

The enable_metric action is used to enable or disable the metric collection for the plugin.

• Value type: boolean
• Default value: true

The syntax is as follows:

filter {
  mutate {
    enable_metric => true
  }
}

7. periodic_flush

The periodic_flush action is used to enable or disable periodic flush for the plugin.

• Value type: boolean
• Default value: false

The syntax is as follows:

filter {
  mutate {
    periodic_flush => true
  }
}

Conclusion

In this tutorial, you have learned how to use 7 common options in all Logstash filter plugins.

These options are usually combined with other filter plugins to enrich the data. You can check the Related Posts section on the right to learn more about Logstash filter plugins.