In this tutorial, I will show you how to perform arithmetic operations (sum, substract, multiple, divide two or more fields) in Logstash.
We will use the Ruby filter plugin inside the filter section of the Logstash configuration file so that
In short, an example of adding two fields in Logstash looks like this:
logstash.conf
# ...
filter { ruby { code => ' event.set("sum", event.get("field1").to_i + event.get("field2").to_i) ' } }
# ... Let's dive deeper into the details.
The simplest way to perform arithmetic operations in Logstash is to use the Ruby filter plugin.
I will use the following log as an example for the rest of this tutorial:
example.log
{ " field1 " : 1 , " field2 " : 2 } To add (sum) two fields in Logstash, you can use the Ruby filter and the + operator:
logstash.conf
filter { ruby { code => ' event.set("sum", event.get("field1").to_i + event.get("field2").to_i) ' } } The code above will convert field1 and field2 to integers, then add them together and store the result in a new field called sum.
So the result log will look like this:
example.log
{ " field1 " : 1 , " field2 " : 2 , " sum " : 3 }
You can learn more about summing two fields in Logstash by reading this in-depth tutorial: How to Sum Two Fields in Logstash .
To subtract two fields in Logstash, you can use the Ruby filter and the - operator:
logstash.conf
filter { ruby { code => ' event.set("difference", event.get("field1").to_i - event.get("field2").to_i) ' } } Similarly, we get the following result:
example.log
{ " field1 " : 1 , " field2 " : 2 , " difference " : -1 } To multiply two fields in Logstash, you can use the Ruby filter and the * operator:
logstash.conf
filter { ruby { code => ' event.set("product", event.get("field1").to_i * event.get("field2").to_i) ' } } The result log will look like this:
example.log
{ " field1 " : 1 , " field2 " : 2 , " product " : 2 } To divide two fields in Logstash, you can use the Ruby filter and the / operator:
logstash.conf
filter { ruby { code => ' event.set("quotient", event.get("field1").to_f / event.get("field2").to_f) ' } } In this case, we convert field1 and field2 to floats instead of integers because we want to get the decimal value of the quotient.
The result log will look like this:
example.log
{ " field1 " : 1 , " field2 " : 2 , " quotient " : 0.5 } There are some common cases I usually use arithmetic operations in Logstash, let's take a look at them.
In this section, I will use the Logstash config from this tutorial to parse JSON string logs into JSON objects.
Let's say you have a field called duration that stores the duration of an event in the format HH:MM:SS.
test.log
{ " duration " : " 21:03:02 " } { " duration " : " 09:03:00 " } To calculate the total seconds from the duration, you can use the following code:
logstash.conf
input { file { path => " /home/dminhvu/elastic/test.log " start_position => " beginning " sincedb_path => " /dev/null " codec => multiline { pattern => "\n" what => " next " } } }
filter { json { source => " message " }
ruby { code => ' hours, minutes, seconds = event.get("duration").split(":").map(&:to_i) event.set("total_seconds", hours * 3600 + minutes * 60 + seconds) ' } }
output { file { path => " /home/dminhvu/elastic/output.log " codec => " json_lines " } } The resulting log will look like this:
output.log
{ // other fieds... " duration " : " 21:03:02 " , " total_seconds " : 75782 } { // other fieds... " duration " : " 09:03:00 " , " total_seconds " : 32580 } Let's say you have a field called values that stores an array of integers.
test.log
{ " values " : [ 1 , 2 , 3 ] } { " values " : [ 4 , 5 , 6 ] } To calculate the sum of values in the values field, you can use the following code:
logstash.conf
input { file { path => " /home/dminhvu/elastic/test.log " start_position => " beginning " sincedb_path => " /dev/null " codec => multiline { pattern => "\n" what => " next " } } }
filter { json { source => " message " }
ruby { code => ' event.set("sum", event.get("values").sum) ' } }
output { file { path => " /home/dminhvu/elastic/output.log " codec => " json_lines " } } The resulting log will look like this:
output.log
{ // other fieds... " values " : [ 1 , 2 , 3 ], " sum " : 6 } { // other fieds... " values " : [ 4 , 5 , 6 ], " sum " : 15 } In this tutorial, you have learned how to perform arithmetic operations in Logstash using the Ruby filter plugin.
If you have any questions, please leave a comment below.
Comments
Be the first to comment!