In this tutorial, I will show you how to parse data from a CSV file in Logstash using the csv
filter plugin.
Contents
- Parsing CSV File in Logstash
- Parsing CSV File with Headers in Logstash
- Parsing CSV File with Custom Separator in Logstash
- Parsing CSV File without Specifying Columns in Logstash
Parsing CSV File in Logstash
To parse data from a CSV file in Logstash, we can use the csv
filter plugin. For example:
input { file { path => "/home/dminhvu/elastic/example.csv" start_position => "beginning" sincedb_path => "/dev/null" codec => multiline { pattern => "\n" what => "next" } } } filter { csv { separator => "," # specify the separator source => "message" # parse the message field as CSV columns => ["name", "age", "location", "gender"] # specify the columns } # do anything you want with the parsed CSV data # ... } output { file { path => "/home/dminhvu/elastic/output.log" codec => "json_lines" # write to JSON lines format } }
Suppose I have the following CSV file:
Minh Vu,21,Vietnam,Male Desmond,25,United States,Male Nhi,30,Hong Kong,Female
Using the above config yields the following results:
{ "age": "21", "gender": "Male", "location": "Vietnam", "message": "Minh Vu,21,Vietnam,Male", "name": "Minh Vu" } { "age": "25", "gender": "Male", "location": "United States", "message": "Desmond,25,United States,Male", "name": "Desmond" } { "age": "30", "gender": "Female", "location": "Hong Kong", "message": "Nhi,30,Hong Kong,Female", "name": "Nhi" }
As you can see, we parsed the CSV data into JSON format. Now, we can do anything we want with the parsed data.
Parsing CSV File with Headers in Logstash
Suppose your CSV file contains headers, you can use the skip_header
option and set to true
to skip the first line:
name,age,location,gender Minh Vu,21,Vietnam,Male Desmond,25,United States,Male Nhi,30,Hong Kong,Female
The config will look like this:
filter { csv { skip_header => "true" # skip the first line separator => "," # specify the separator source => "message" # parse the message field as CSV columns => ["name", "age", "location", "gender"] # specify the columns } }
Parsing CSV File with Custom Separator in Logstash
If your CSV file uses a custom separator, you can specify it using the separator
option:
Minh Vu|21|Vietnam|Male Desmond|25|United States|Male Nhi|30|Hong Kong|Female
The config will look like this:
filter { csv { separator => "|" # specify the separator source => "message" # parse the message field as CSV columns => ["name", "age", "location", "gender"] # specify the columns } }
Parsing CSV File without Specifying Columns in Logstash
If you don't want to specify the column names, you can use the autodetect_column_names
option and set to true
:
filter { csv { autodetect_column_names => "true" # autodetect the columns separator => "," # specify the separator source => "message" # parse the message field as CSV } }
This only works when your first row contains headers.
Comments
Be the first to comment!