Logstash Input from CSV File

Disclaimer: All content on this website is derived directly from my own expertise and experiences. No AI-generated text or automated content creation tools are used.

In this tutorial, I will show you how to parse data from a CSV file in Logstash using the csv filter plugin.

Parsing CSV File in Logstash
Parsing CSV File with Headers in Logstash
Parsing CSV File with Custom Separator in Logstash
Parsing CSV File without Specifying Columns in Logstash

Parsing CSV File in Logstash

To parse data from a CSV file in Logstash, we can use the csv filter plugin. For example:

logstash.conf


input {
  file {
    path => "/home/dminhvu/elastic/example.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "\n"
      what => "next"
    }
  }
}
 
filter {
  csv {
    separator => "," # specify the separator
    source => "message" # parse the message field as CSV
    columns => ["name", "age", "location", "gender"] # specify the columns
  }
 
  # do anything you want with the parsed CSV data
  # ...
}
 
output {
  file {
    path => "/home/dminhvu/elastic/output.log"
    codec => "json_lines" # write to JSON lines format
  }
}

Suppose I have the following CSV file:

example.csv


Minh Vu,21,Vietnam,Male
Desmond,25,United States,Male
Nhi,30,Hong Kong,Female

Using the above config yields the following results:

output.log


{
  "age": "21",
  "gender": "Male",
  "location": "Vietnam",
  "message": "Minh Vu,21,Vietnam,Male",
  "name": "Minh Vu"
}
{
  "age": "25",
  "gender": "Male",
  "location": "United States",
  "message": "Desmond,25,United States,Male",
  "name": "Desmond"
}
{
  "age": "30",
  "gender": "Female",
  "location": "Hong Kong",
  "message": "Nhi,30,Hong Kong,Female",
  "name": "Nhi"
}

As you can see, we parsed the CSV data into JSON format. Now, we can do anything we want with the parsed data.

Parsing CSV File with Headers in Logstash

Suppose your CSV file contains headers, you can use the skip_header option and set to true to skip the first line:

example.csv


name,age,location,gender
Minh Vu,21,Vietnam,Male
Desmond,25,United States,Male
Nhi,30,Hong Kong,Female

The config will look like this:

logstash.conf


filter {
  csv {
    skip_header => "true" # skip the first line
    separator => "," # specify the separator
    source => "message" # parse the message field as CSV
    columns => ["name", "age", "location", "gender"] # specify the columns
  }
}

Parsing CSV File with Custom Separator in Logstash

If your CSV file uses a custom separator, you can specify it using the separator option:

example.csv


Minh Vu|21|Vietnam|Male
Desmond|25|United States|Male
Nhi|30|Hong Kong|Female

The config will look like this:

logstash.conf


filter {
  csv {
    separator => "|" # specify the separator
    source => "message" # parse the message field as CSV
    columns => ["name", "age", "location", "gender"] # specify the columns
  }
}

Parsing CSV File without Specifying Columns in Logstash

If you don't want to specify the column names, you can use the autodetect_column_names option and set to true:

logstash.conf


filter {
  csv {
    autodetect_column_names => "true" # autodetect the columns
    separator => "," # specify the separator
    source => "message" # parse the message field as CSV
  }
}

This only works when your first row contains headers.

Logstash Input from CSV File

Contents

Parsing CSV File in Logstash

Parsing CSV File with Headers in Logstash

Parsing CSV File with Custom Separator in Logstash

Parsing CSV File without Specifying Columns in Logstash