Minh Vu
Software Engineer
Hi 👋, I'm a software engineer specializing in backend systems, distributed systems, and scalable architecture. My blog shares practical tutorials based on 3+ years of experience. LeetCode 1756 (Top 10%). Actively seeking SDE roles — let's get in touch!
Leave a Comment
Receive Latest Updates 📬
Get every new post, special offers, and more via email. No fee required.
In this tutorial, I will show you how to convert a date field to ISO8601 format in Logstash.
In general, we can use the ruby filter with the DateTime.parse() method to parse a date string to a DateTime object. Then, we can use the iso8601() method to get the ISO8601 format.
Continue reading to see it in action.
ruby Filterdate Filterruby FilterIn some cases, the date field would be yyyy-MM-dd HH:mm:ss, or some other format. But we want to convert it to ISO8601 format to normalize the date format.
To convert a date string to ISO8601 format, we can use the following Logstash configuration:
filter { ruby { code => ' date = event.get("date") event.set("iso8601_date", DateTime.parse(date).iso8601()) ' } }
For example, if I have the following sample log:
{ "date": "2024-01-03 22:00:00" }
The result after using the above Logstash configuration would be:
{ "date": "2024-01-03 22:00:00", "iso8601_date": "2024-01-03T12:03:00+00:00" }
You can specify iso8601(3) to get the ISO8601 format with milliseconds of 3 digits.
{ "date": "2024-01-03 22:00:00", "iso8601_date": "2024-01-03T12:03:00.000+00:00" }
If you need to convert date to ISO8601 with the current timezone, please go to the fourth example.
Usually, the date will be in a custom format. For example, dd.MM.yyyy,HH:mm.
We can still use the same method to parse the date string to a DateTime object by specifying the format of the date string.
filter { ruby { code => ' date = event.get("date") event.set("iso8601_date", DateTime.strptime(date, "%d.%m.%Y,%H:%M").iso8601()) ' } }
The result would be:
{ "date": "03.01.2024,12:03", "iso8601_date": "2024-01-03T12:03:00+00:00" }
If you need to convert date to ISO8601 with the current timezone, please go to the fourth example.
date FilterWe can also use the date filter to convert the date to ISO8601 format.
If the date field is Jan 03 12:03:00, we can use the following Logstash configuration:
filter { date { match => ["date", "MMM dd HH:mm:ss"] target => "iso8601_date" } }
The result would be:
{ "date": "Jan 03 12:03:00", "iso8601_date": "2024-01-03T05:03:00.000Z" }
As you can see in the first example and the second example, the timezone is +00:00. However, my current timezone is +07:00 as I'm in Vietnam. So, this is a wrong conversion.
In the third example, Logstash automatically detects my timezone and converts it to UTC time, which is correct and the one we want.
To add the timezone in the conversion, we can simply add our timezone to the end of the date string and replace iso8601() with to_time.
Let's try again with the following config:
filter { ruby { code => ' date = event.get("date") event.set("iso8601_date", DateTime.parse("#{date} +07:00").to_time) ' } }
Applying the above config to this sample log:
{ "date": "2024-01-03 22:00:00" }
The result would be:
{ "date": "2024-01-03 22:00:00", "iso8601_date": "2024-01-03T12:03:00+00:00", "iso8601_date": "2024-01-03T05:03:00.000Z" }
This is exactly what we want, the correct time with timezone.
Next.js SEO: The Complete Checklist to Boost Your Site Ranking
Next.js JSON-LD: How to Add Structured Data to Your Website
Logstash: How to Remove Field using Mutate Filter
Logstash Conditionals: Using if/else to Control Log Flow
Logstash: Add Field to Event with Mutate Filter
Comments