Logstash Mutate Filter: Everything You Need to Know

The mutate filter in Logstash is used to perform mutations on data fields. For example, to add a field, rename a field, or copy a field to another field.

This tutorial will show you all types of mutations you can perform with the mutate filter in Logstash.

The basic structure of a Logstash pipeline config is as follows:

input {
  # input plugins
  stdin {} # read from the console
}
 
filter {
  # filter plugins
  # put the mutate filter here
}
 
output {
  # output plugins
  stdout {} # output to the console
}

So you can put the mutate filter in the filter section of the Logstash pipeline.

We will go deeper into each type of mutation in the next sections.

Contents

• 2 Commonly Used Options of the Mutate Filter and Other Filters
  • 1. add_field
  • 2. remove_field
• 14 Options of the Mutate Filter
  • 1. coerce
  • 2. rename
  • 3. update
  • 4. replace
  • 5. convert
  • 6. gsub
  • 7. uppercase
  • 8. capitalize
  • 9. lowercase
  • 10. strip
  • 11. split
  • 12. join
  • 13. merge
  • 14. copy
• Conclusion

2 Commonly Used Options of the Mutate Filter and Other Filters

While working with the mutate filter, you will often use the add_field and remove_field options.

Please note that those actions can also be used in other filters like the ruby filter or the grok filter, not just the mutate filter.

1. add_field

In Logstash, the add_field action is used to add a new field to the event.

• Value type: hash
• Default value: {}

The syntax to add a single field is as follows:

filter {
  mutate {
    add_field => {
      "new_field_name" => "new_field_value"
    }
  }
}

To add multiple fields:

filter {
  mutate {
    add_field => {
      "new_field_name_1" => "new_field_value_1"
      "new_field_name_2" => "new_field_value_2"
    }
  }
}

For example, we can add a new field age with the value 21 to the event:

filter {
  mutate {
    add_field => {
      "age" => 21
    }
  }
}

Using the above config will yield the result:

{
  "age": 21
  // other fields
}

2. remove_field

To remove a field from the event in Logstash, we can use the remove_field action.

• Value type: array
• Default value: []

The syntax is as follows:

filter {
  mutate {
    remove_field => [ "field_name" ]
  }
}

You can also remove multiple fields:

filter {
  mutate {
    remove_field => [ "field_name_1", "field_name_2" ]
  }
}

For example, we can remove the status field from the event:

filter {
  mutate {
    remove_field => [ "status" ]
  }
}

14 Options of the Mutate Filter

The mutate filter can be used in the filter section of the Logstash pipeline, and specify the mutate actions to be performed.

filter {
  mutate {
    # put the mutate actions here
  }
}

1. coerce

The coerce action is used to set the default value of a field when it is null.

filter {
  mutate {
    coerce => {
      "field_name" => "default_value" # "[user][name]" => "Minh Vu"
    }
  }
}

The code above uses the coerce action to set the value of field_name to default_value, so that when the field_name is null, it will be set to default_value.

2. rename

The rename action is used to rename a field.

filter {
  mutate {
    rename => {
      "source_field" => "destination_field"
    }
  }
}

There are 2 points to remember when using the rename action:

• If the destination field already exists, it will be overwritten.
• If the source field does not exist, the destination field will not be created.

3. update

The update action is used to update the value of a field.

filter {
  mutate {
    update => {
      "field_name" => "new_value"
    }
  }
}

If the field does not exist, nothing will happen.

4. replace

The replace action is used to replace the value of a field and create the field if it does not exist.

filter {
  mutate {
    replace => {
      "field_name" => "new_value"
    }
  }
}

The replace actions is different from the update action in that it will create the field if it does not exist.

5. convert

The convert action is used to convert the value of a field to another data type, e.g. string to integer.

filter {
  mutate {
    convert => {
      "field_name" => "integer"
      "field_name_2" => "boolean"
    }
  }
}

If the field is an array, the conversion will be applied to all elements of the array.

Valid data types are:

• integer
• integer_eu
• float
• float_eu
• string
• boolean

You can visit the docs for more information about the data types.

6. gsub

The gsub action is used to replace a string with another string.

filter {
  mutate {
    gsub => [
      "field_name", "pattern", "replacement"
    ]
  }
}

For example, to remove all dots from a string:

filter {
  mutate {
    gsub => [
      "field_name", "\.", ""
    ]
  }
}

7. uppercase

The uppercase action is used to convert every character of a string or an array of strings field to uppercase.

filter {
  mutate {
    uppercase => [
      "field_name", # "wisecode blog" => "WISECODE BLOG"
      "array_field_name" # ["wisecode blog", "elastic"] => ["WISECODE BLOG", "ELASTIC"]
    ]
  }
}

8. capitalize

The capitalize action is used to capitalize the first character of a string or an array of strings field.

filter {
  mutate {
    capitalize => [
      "field_name", # "wisecode blog" => "Wisecode blog"
      "array_field_name" # ["wisecode blog", "elastic"] => ["Wisecode blog", "Elastic"]
    ]
  }
}

9. lowercase

The lowercase action is used to convert every character of a string or an array of strings field to lowercase.

filter {
  mutate {
    lowercase => [
      "field_name", # "WISECODE BLOG" => "wisecode blog"
      "array_field_name" # ["WISECODE BLOG", "ELASTIC"] => ["wisecode blog", "elastic"]
    ]
  }
}

10. strip

The strip action is used to remove leading and trailing whitespaces from a string or an array of strings field.

filter {
  mutate {
    strip => [
      "field_name", # " wisecode blog " => "wisecode blog"
      "array_field_name" # [" wisecode blog ", " elastic "] => ["wisecode blog", "elastic"]
    ]
  }
}

11. split

The split action is used to split a string field into an array of strings.

filter {
  mutate {
    split => {
      "field_name" => "separator"
    }
  }
}

For example, I want to split the message field by the comma character:

filter {
  mutate {
    split => {
      "message" => "," # "Hello, world" => ["Hello", " world"]
    }
  }
}

12. join

The join action is used to join an array of strings into a string field.

filter {
  mutate {
    join => {
      "field_name" => "separator"
    }
  }
}

The field being joined must be an array of strings. Otherwise, the join action will not work.

For example, I want to join the tags field by the comma character:

filter {
  mutate {
    join => {
      "tags" => "," # ["elastic", "logstash"] => "elastic,logstash"
    }
  }
}

13. merge

The merge action is used to merge two fields into one field.

filter {
  mutate {
    merge => {
      "destination_field" => "added_field"
    }
  }
}

Valid data types are:

• string + string = array of 2 strings ("elastic" + "logstash" => ["elastic", "logstash"])
• array of strings + string = array of strings (["elastic", "logstash"] + "tutorial" => ["elastic", "logstash", "tutorial"])

For example, I want to merge the tags field and the category field into the tags field:

filter {
  mutate {
    merge => {
      "tags" => "category" # ["elastic", "logstash"] + "tutorial" => ["elastic", "logstash", "tutorial"]
    }
  }
}

14. copy

The copy action is used to copy the value of a field to another field.

filter {
  mutate {
    copy => {
      "source_field" => "destination_field"
    }
  }
}

If the destination field already exists, it will be overwritten.

Conclusion

The mutate filter is a powerful plugin of Logstash that helps you to perform several actions on the data.

In this tutorial, we have learned all types of mutations you can perform with the mutate filter.

I hope you find this tutorial useful. Feel free to leave a comment below if you have any questions.