There are some basic options supported by all Logstash filter plugins that I often use to enrich the data.
For example, we can add a new field, remove an existing field, and more.
This tutorial will show you how to do that. Let's get started!
There are 7 options that are supported by all Logstash filter plugins, which means you can use them in any filter plugin.
In this tutorial, I will use the mutate filter plugin as an example, you can use any filter plugin based on your needs.
I will also use the following sample data for demonstration throughout this tutorial:
example.log
{
" message " : " Minh Vu says hello to the world! " ,
" status " : 200 ,
" timestamp " : " 2021-11-23T17:36:00Z "
}
For the input and output part of the filter config, I will use this config to parse JSON lines log:
logstash.conf
input {
file {
path => " /home/dminhvu/elastic/example.log "
start_position => " beginning "
sincedb_path => " /dev/null "
codec => multiline {
pattern => "\n"
what => " next "
}
}
}
filter {
# put the filter here
}
output {
file {
path => " /home/dminhvu/elastic/output.log "
codec => " json_lines "
}
}
In Logstash, the add_field action is used to add a new field to the event.
Value type: hash
Default value: {}
The syntax to add a single field is as follows:
logstash.conf
filter {
mutate {
add_field => {
" new_field_name " => " new_field_value "
}
}
}
To add multiple fields:
logstash.conf
filter {
mutate {
add_field => {
" new_field_name_1 " => " new_field_value_1 "
" new_field_name_2 " => " new_field_value_2 "
}
}
}
For example, we can add a new field fine with the value true to the event:
logstash.conf
filter {
mutate {
add_field => {
" fine " => " true "
}
}
}
Using the above config will yield the result:
output.log
{
" fine " : true ,
" message " : " Minh Vu says hello to the world! " ,
" status " : 200 ,
" timestamp " : " 2021-11-23T17:36:00Z "
}
To remove a field from the event in Logstash, we can use the remove_field action.
Value type: array
Default value: []
The syntax is as follows:
logstash.conf
filter {
mutate {
remove_field => [ " field_name " ]
}
}
To remove multiple fields:
logstash.conf
filter {
mutate {
remove_field => [ " field_name_1 " , " field_name_2 " ]
}
}
For example, we can remove the status field from the event:
logstash.conf
filter {
mutate {
remove_field => [ " status " ]
}
}
Here is the result:
output.log
{
" message " : " Minh Vu says hello to the world! " ,
" timestamp " : " 2021-11-23T17:36:00Z "
}
The add_tag action is used to add a tag to the event.
Value type: array
Default value: []
The syntax is as follows:
logstash.conf
filter {
mutate {
add_tag => [ " tag_name " ]
}
}
To add multiple tags:
logstash.conf
filter {
mutate {
add_tag => [ " tag_name_1 " , " tag_name_2 " ]
}
}
To remove a tag from the event, we can use the remove_tag action.
Value type: array
Default value: []
The syntax is as follows:
logstash.conf
filter {
mutate {
remove_tag => [ " tag_name " ]
}
}
To remove multiple tags:
logstash.conf
filter {
mutate {
remove_tag => [ " tag_name_1 " , " tag_name_2 " ]
}
}
The id action is used to set the ID of the plugin configuration, which is useful when you need to identify multiple plugins of the same type.
Value type: string
Default value: no default value, automatically generated if not specified
The syntax is as follows:
logstash.conf
filter {
mutate {
id => " plugin_id "
}
}
The enable_metric action is used to enable or disable the metric collection for the plugin.
Value type: boolean
Default value: true
The syntax is as follows:
logstash.conf
filter {
mutate {
enable_metric => true
}
}
The periodic_flush action is used to enable or disable periodic flush for the plugin.
Value type: boolean
Default value: false
The syntax is as follows:
logstash.conf
filter {
mutate {
periodic_flush => true
}
}
In this tutorial, you have learned how to use 7 common options in all Logstash filter plugins.
These options are usually combined with other filter plugins to enrich the data. You can check the Related Posts section on the right to learn more about Logstash filter plugins.
Comments
aida
Apr 01, 2024
easy to understand