Logstash Mutate Filter: Everything You Need to Know

Note: All content on this website is derived directly from my own expertise and experiences. No AI-generated text or automated content creation tools are used.

The mutate filter in Logstash is used to perform mutations on data fields. For example, to add a field, rename a field, or copy a field to another field.

This tutorial will show you all types of mutations you can perform with the mutate filter in Logstash.

The basic structure of a Logstash pipeline config is as follows:

logstash.conf


input {
  # input plugins
  stdin {} # read from the console
}
 
filter {
  # filter plugins
  # put the mutate filter here
}
 
output {
  # output plugins
  stdout {} # output to the console
}

So you can put the mutate filter in the filter section of the Logstash pipeline.

We will go deeper into each type of mutation in the next sections.

2 Commonly Used Options of the Mutate Filter and Other Filters
- 1. add_field
- 2. remove_field
14 Options of the Mutate Filter
- 1. coerce
- 2. rename
- 3. update
- 4. replace
- 5. convert
- 6. gsub
- 7. uppercase
- 8. capitalize
- 9. lowercase
- 10. strip
- 11. split
- 12. join
- 13. merge
- 14. copy
Conclusion

2 Commonly Used Options of the Mutate Filter and Other Filters

While working with the mutate filter, you will often use the add_field and remove_field options.

Please note that those actions can also be used in other filters like the ruby filter or the grok filter, not just the mutate filter.

1. add_field

In Logstash, the add_field action is used to add a new field to the event.

Value type: hash
Default value: {}

The syntax to add a single field is as follows:

logstash.conf


filter {
  mutate {
    add_field => {
      "new_field_name" => "new_field_value"
    }
  }
}

To add multiple fields:

logstash.conf


filter {
  mutate {
    add_field => {
      "new_field_name_1" => "new_field_value_1"
      "new_field_name_2" => "new_field_value_2"
    }
  }
}

For example, we can add a new field age with the value 21 to the event:

logstash.conf


filter {
  mutate {
    add_field => {
      "age" => 21
    }
  }
}

Using the above config will yield the result:

output.log


{
  "age": 21
  // other fields
}

2. remove_field

To remove a field from the event in Logstash, we can use the remove_field action.

Value type: array
Default value: []

The syntax is as follows:

logstash.conf


filter {
  mutate {
    remove_field => [ "field_name" ]
  }
}

You can also remove multiple fields:

logstash.conf


filter {
  mutate {
    remove_field => [ "field_name_1", "field_name_2" ]
  }
}

For example, we can remove the status field from the event:

logstash.conf


filter {
  mutate {
    remove_field => [ "status" ]
  }
}

14 Options of the Mutate Filter

The mutate filter can be used in the filter section of the Logstash pipeline, and specify the mutate actions to be performed.

filter.logstash.conf


filter {
  mutate {
    # put the mutate actions here
  }
}

1. coerce

The coerce action is used to set the default value of a field when it is null.

filter.logstash.conf


filter {
  mutate {
    coerce => {
      "field_name" => "default_value" # "[user][name]" => "Minh Vu"
    }
  }
}

The code above uses the coerce action to set the value of field_name to default_value, so that when the field_name is null, it will be set to default_value.

2. rename

The rename action is used to rename a field.

filter.logstash.conf


filter {
  mutate {
    rename => {
      "source_field" => "destination_field"
    }
  }
}

There are 2 points to remember when using the rename action:

If the destination field already exists, it will be overwritten.
If the source field does not exist, the destination field will not be created.

3. update

The update action is used to update the value of a field.

filter.logstash.conf


filter {
  mutate {
    update => {
      "field_name" => "new_value"
    }
  }
}

If the field does not exist, nothing will happen.

4. replace

The replace action is used to replace the value of a field and create the field if it does not exist.

filter.logstash.conf


filter {
  mutate {
    replace => {
      "field_name" => "new_value"
    }
  }
}

The replace actions is different from the update action in that it will create the field if it does not exist.

5. convert

The convert action is used to convert the value of a field to another data type, e.g. string to integer.

filter.logstash.conf


filter {
  mutate {
    convert => {
      "field_name" => "integer"
      "field_name_2" => "boolean"
    }
  }
}

If the field is an array, the conversion will be applied to all elements of the array.

Valid data types are:

integer
integer_eu
float
float_eu
string
boolean

You can visit the docs for more information about the data types.

6. gsub

The gsub action is used to replace a string with another string.

filter.logstash.conf


filter {
  mutate {
    gsub => [
      "field_name", "pattern", "replacement"
    ]
  }
}

For example, to remove all dots from a string:

filter.logstash.conf


filter {
  mutate {
    gsub => [
      "field_name", "\.", ""
    ]
  }
}

7. uppercase

The uppercase action is used to convert every character of a string or an array of strings field to uppercase.

filter.logstash.conf


filter {
  mutate {
    uppercase => [
      "field_name", # "wisecode blog" => "WISECODE BLOG"
      "array_field_name" # ["wisecode blog", "elastic"] => ["WISECODE BLOG", "ELASTIC"]
    ]
  }
}

8. capitalize

The capitalize action is used to capitalize the first character of a string or an array of strings field.

filter.logstash.conf


filter {
  mutate {
    capitalize => [
      "field_name", # "wisecode blog" => "Wisecode blog"
      "array_field_name" # ["wisecode blog", "elastic"] => ["Wisecode blog", "Elastic"]
    ]
  }
}

9. lowercase

The lowercase action is used to convert every character of a string or an array of strings field to lowercase.

filter.logstash.conf


filter {
  mutate {
    lowercase => [
      "field_name", # "WISECODE BLOG" => "wisecode blog"
      "array_field_name" # ["WISECODE BLOG", "ELASTIC"] => ["wisecode blog", "elastic"]
    ]
  }
}

10. strip

The strip action is used to remove leading and trailing whitespaces from a string or an array of strings field.

filter.logstash.conf


filter {
  mutate {
    strip => [
      "field_name", # " wisecode blog " => "wisecode blog"
      "array_field_name" # [" wisecode blog ", " elastic "] => ["wisecode blog", "elastic"]
    ]
  }
}

11. split

The split action is used to split a string field into an array of strings.

filter.logstash.conf


filter {
  mutate {
    split => {
      "field_name" => "separator"
    }
  }
}

For example, I want to split the message field by the comma character:

filter.logstash.conf


filter {
  mutate {
    split => {
      "message" => "," # "Hello, world" => ["Hello", " world"]
    }
  }
}

12. join

The join action is used to join an array of strings into a string field.

filter.logstash.conf


filter {
  mutate {
    join => {
      "field_name" => "separator"
    }
  }
}

The field being joined must be an array of strings. Otherwise, the join action will not work.

For example, I want to join the tags field by the comma character:

filter.logstash.conf


filter {
  mutate {
    join => {
      "tags" => "," # ["elastic", "logstash"] => "elastic,logstash"
    }
  }
}

13. merge

The merge action is used to merge two fields into one field.

filter.logstash.conf


filter {
  mutate {
    merge => {
      "destination_field" => "added_field"
    }
  }
}

Valid data types are:

string + string = array of 2 strings ("elastic" + "logstash" => ["elastic", "logstash"])
array of strings + string = array of strings (["elastic", "logstash"] + "tutorial" => ["elastic", "logstash", "tutorial"])

For example, I want to merge the tags field and the category field into the tags field:

filter.logstash.conf


filter {
  mutate {
    merge => {
      "tags" => "category" # ["elastic", "logstash"] + "tutorial" => ["elastic", "logstash", "tutorial"]
    }
  }
}

14. copy

The copy action is used to copy the value of a field to another field.

filter.logstash.conf


filter {
  mutate {
    copy => {
      "source_field" => "destination_field"
    }
  }
}

If the destination field already exists, it will be overwritten.

Conclusion

The mutate filter is a powerful plugin of Logstash that helps you to perform several actions on the data.

In this tutorial, we have learned all types of mutations you can perform with the mutate filter.

I hope you find this tutorial useful. Feel free to leave a comment below if you have any questions.

Logstash Mutate Filter: Everything You Need to Know

Most Read