trivago tech tips — Kibana Edition
Happy New Year from trivago’s engineering teams! 🥂
We hope you have a great start into the new year.
Also, we missed you during the holidays, but it was important to spend some time away from the computer. Now however we’re happy to bring back our popular trivago tech tips.
Let’s kick it off with a very special Kibana edition. Kibana is a search frontend for Elasticsearch. It allows you to query a lot of log data quickly — at least that’s what we mostly use it for here at trivago.
Kibana has a lot of awesome but mostly unknown features as we’ll see this week. Let’s go and change that!
Tip #1: Don’t uselessly stress your Elasticsearch
Don’t forget the basic tips to speed up your searches. Before requesting additional hardware or tuning indexes, use some basic optimizations.
1. Use filters instead of queries for binary searches or when looking for exact values, because frequently used filters are cached by Elasticsearch.
2. Use the appropriate wildcards in your queries. Using question mark (?) would replace any single character, so don’t always use asterisk (*) if not needed, it would just waste time and resources.
Tip #2: Search across logs the easy way
Speed up searches across logs by linking different indexes, using hyperlinks in String fields. For example a unique user identifier can link from an application’s access logs index to another more specific index, directly filtering search results in the other index, based on the specific value that has been clicked in the access logs.
How to do that? In your Kibana Index Patterns select the desired field and use the URL template text field to add a link to another index with filtering applied, the field name in the other index can of course also be different from the one in the source index.
An example would be
where {{value}} is the original field value of the source index
See https://www.elastic.co/guide/en/kibana/current/field-formatters-string.html for info on String fields and URL template.
Tip #3: Try adding input controls to dashboards
Consider adding input fields to control dashboards. That way you’ll have easily accessible filters, which save time and are also user friendly for the less tech-savvy in your organization.
A range slider for example can be really useful as quick filtering option, on top of a dashboard. The input controls can be added to a dashboard by creating a visualization of type Controls. It’s marked as a feature that could be removed in the future, but it’s still there in Kibana 7.9, so chances are it will be available for longer. See (older) official documentation for additional details https://www.elastic.co/guide/en/kibana/6.8/add-input-controls.html
Tip #4: Always check split series size in visualizations
When creating visualizations with split series that have a Terms sub-aggregation (a common case), don’t overlook the split series size. It could be too limited for your use-case and showing just the 5 top values.
You can increase that, but if you suddenly fall into an opposite issue and too many values make the visualization slow and unresponsive, try checking “Group other values in separate bucket” to solve it without losing the global picture. We know from experience that not seeing all the data can really make the difference sometimes…
Tip #5: Know your query language
Version 7 brought Kibana Query Language into the spotlight, making it the default, even though Lucene can still be activated through the toggle next to the search field.
Lucene might also be active on your existing saved searches and visualizations, so always remember that the differences between the two can significantly alter your results. Some examples: `400 or 500` is ok in KQL, but you need either `400 500` or `400 OR 500` in Lucene, because Lucene would consider `or` as a search term, not an operator! KQL brings you some more advantages: you can for example exploit the power of range queries made simple; something like `response>=400`. KQL also supports querying nested fields, a specialized version of the object data type that allows arrays of objects to be indexed in a way that they can be queried independently of each other. That means that a query like `operating_system: {name:windows and version=> 10}` will get all nested documents that have logged Windows version 10 or higher in our operating_system nested field.
Aaand that’s about it for this week’s Kibana edition. We hope you learned something and we’re looking forward to the next round of tips. Have a great weekend!
For regular frontend and backend tech tips, join our trivago tech community at https://trivago.substack.com.
