Logstash Filter Tutorial: 7 Common Options

There are some basic options supported by all Logstash filter plugins that I often use to enrich the data.

For example, we can add a new field, remove an existing field, and more.

This tutorial will show you how to do that. Let's get started!

7 Common Options in Logstash Filter Plugins
Conclusion

7 Common Options in Logstash Filter Plugins

There are 7 options that are supported by all Logstash filter plugins, which means you can use them in any filter plugin.

In this tutorial, I will use the mutate filter plugin as an example, you can use any filter plugin based on your needs.

I will also use the following sample data for demonstration throughout this tutorial:

example.log


{
  "message": "Minh Vu says hello to the world!",
  "status": 200,
  "timestamp": "2021-11-23T17:36:00Z"
}

For the input and output part of the filter config, I will use this config to parse JSON lines log:

logstash.conf


input {
  file {
    path => "/home/dminhvu/elastic/example.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "\n"
      what => "next"
    }
  }
}
 
filter {
  # put the filter here
}
 
output {
  file {
    path => "/home/dminhvu/elastic/output.log"
    codec => "json_lines"
  }
}

1. add_field

In Logstash, the add_field action is used to add a new field to the event.

Value type: hash
Default value: {}

The syntax to add a single field is as follows:

logstash.conf


filter {
  mutate {
    add_field => {
      "new_field_name" => "new_field_value"
    }
  }
}

To add multiple fields:

logstash.conf


filter {
  mutate {
    add_field => {
      "new_field_name_1" => "new_field_value_1"
      "new_field_name_2" => "new_field_value_2"
    }
  }
}

For example, we can add a new field fine with the value true to the event:

logstash.conf


filter {
  mutate {
    add_field => {
      "fine" => "true"
    }
  }
}

Using the above config will yield the result:

output.log


{
  "fine": true,
  "message": "Minh Vu says hello to the world!",
  "status": 200,
  "timestamp": "2021-11-23T17:36:00Z"
}

2. remove_field

To remove a field from the event in Logstash, we can use the remove_field action.

Value type: array
Default value: []

The syntax is as follows:

logstash.conf


filter {
  mutate {
    remove_field => [ "field_name" ]
  }
}

To remove multiple fields:

logstash.conf


filter {
  mutate {
    remove_field => [ "field_name_1", "field_name_2" ]
  }
}

For example, we can remove the status field from the event:

logstash.conf


filter {
  mutate {
    remove_field => [ "status" ]
  }
}

Here is the result:

output.log


{
  "message": "Minh Vu says hello to the world!",
  "timestamp": "2021-11-23T17:36:00Z"
}

3. add_tag

The add_tag action is used to add a tag to the event.

Value type: array
Default value: []

The syntax is as follows:

logstash.conf


filter {
  mutate {
    add_tag => [ "tag_name" ]
  }
}

To add multiple tags:

logstash.conf


filter {
  mutate {
    add_tag => [ "tag_name_1", "tag_name_2" ]
  }
}

4. remove_tag

To remove a tag from the event, we can use the remove_tag action.

Value type: array
Default value: []

The syntax is as follows:

logstash.conf


filter {
  mutate {
    remove_tag => [ "tag_name" ]
  }
}

To remove multiple tags:

logstash.conf


filter {
  mutate {
    remove_tag => [ "tag_name_1", "tag_name_2" ]
  }
}

5. id

The id action is used to set the ID of the plugin configuration, which is useful when you need to identify multiple plugins of the same type.

Value type: string
Default value: no default value, automatically generated if not specified

The syntax is as follows:

logstash.conf


filter {
  mutate {
    id => "plugin_id"
  }
}

6. enable_metric

The enable_metric action is used to enable or disable the metric collection for the plugin.

Value type: boolean
Default value: true

The syntax is as follows:

logstash.conf


filter {
  mutate {
    enable_metric => true
  }
}

7. periodic_flush

The periodic_flush action is used to enable or disable periodic flush for the plugin.

Value type: boolean
Default value: false

The syntax is as follows:

logstash.conf


filter {
  mutate {
    periodic_flush => true
  }
}

Conclusion

In this tutorial, you have learned how to use 7 common options in all Logstash filter plugins.

These options are usually combined with other filter plugins to enrich the data. You can check the Related Posts section on the right to learn more about Logstash filter plugins.

Logstash Filter Tutorial: 7 Common Options

Contents

7 Common Options in Logstash Filter Plugins

1. add_field

2. remove_field

3. add_tag

4. remove_tag

5. id

6. enable_metric

7. periodic_flush

Conclusion

Contents

7 Common Options in Logstash Filter Plugins

1. add_field

2. remove_field

3. add_tag

4. remove_tag

5. id

6. enable_metric

7. periodic_flush

Conclusion

Most Read