Logstash Input from JSON File

Disclaimer: All content on this website is derived directly from my own expertise and experiences. No AI-generated text or automated content creation tools are used.

In this tutorial, I will show you how to parse logs from a JSON file in Logstash.

In short, there are two ways to parse logs from a JSON file in Logstash:

To parse a single JSON data, convert it to the compact JSON format, then use the multiline codec followed by the json filter.
To parse multiple JSON rows in a file, use the multiline codec followed by the json filter.

Parsing Logs from Single JSON Data in Logstash
Parsing Logs from Multiple JSON Rows in a File in Logstash

Parsing Logs from Single JSON Data in Logstash

To parse logs from a single JSON data, you will need to convert it to the compact JSON format. For example, suppose you have the following log file:

example.log

{
  "id": 1,
  "body": "This is some awesome thinking!",
  "postId": 100,
  "user": {
    "id": 63,
    "username": "eburras1q"
  }
}

You can use the following command to convert it to the compact JSON format:

console

jq -c . example.log > example-compact.log

You need to install jq to run the command above.

The example-compact.log file will look like this:

example-compact.log

{
  "id": 1,
  "body": "This is some awesome thinking!",
  "postId": 100,
  "user": { "id": 63, "username": "eburras1q" }
}

Then, you can follow the next section to parse the JSON data for both single line and multiline JSON data.

Parsing Logs from Multiple JSON Rows in a File in Logstash

Suppose you have the following log file:

example-compact.log

{"id":1,"body":"This is some awesome thinking!","postId":100,"user":{"id":63,"username":"eburras1q"}}
{"id":2,"body":"What terrific math skills you’re showing!","postId":27,"user":{"id":71,"username":"omarsland1y"}}
{"id":3,"body":"You are an amazing writer!","postId":61,"user":{"id":29,"username":"jissetts"}}
{"id":4,"body":"Wow! You have improved so much!","postId":8,"user":{"id":19,"username":"bleveragei"}}
{"id":5,"body":"Nice idea!","postId":62,"user":{"id":70,"username":"cmasurel1x"}}

Remember to add a new line at the end of the file.

In Logstash, to parse the logs from JSON file, you will need to:

Use the multiline codec to read multiple lines as one event.
Use the json filter to parse the JSON data.

logstash.conf

input {
  file {
    path => "/home/dminhvu/elastic/example-compact.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "\n"
      what => "next"
    }
  }
}
 
filter {
  json {
    source => "message" # parse the message field as JSON
  }
 
  # do anything you want with the parsed JSON data
  # ...
}
 
output {
  file {
    path => "/home/dminhvu/elastic/output.log"
    codec => "json_lines" # write to JSON lines format
  }
}

After using the json filter, the JSON data will be parsed into the following format:

parsed-event

{
  "@timestamp": "2023-12-07T15:17:26.056445100Z",
  "@version": "1",
  "body": "This is some awesome thinking!",
  "event": {
    "original": "{\"id\":1,\"body\":\"This is some awesome thinking!\",\"postId\":100,\"user\":{\"id\":63,\"username\":\"eburras1q\"}}"
  },
  "host": {
    "name": "dminhvu"
  },
  "id": 1,
  "log": {
    "file": {
      "path": "/home/dminhvu/elastic/example.log"
    }
  },
  "message": "{\"id\":1,\"body\":\"This is some awesome thinking!\",\"postId\":100,\"user\":{\"id\":63,\"username\":\"eburras1q\"}}",
  "postId": 100,
  "user": {
    "id": 63,
    "username": "eburras1q"
  }
}
{
  // other events...
}

Logstash Input from JSON File

Contents

Parsing Logs from Single JSON Data in Logstash

Parsing Logs from Multiple JSON Rows in a File in Logstash