Taming PUNKs: How ES|QL queries Elasticsearch fields it was never told about

5, ES|QL can query unmapped fields. It reads them from _source or returns nulls, so a query keeps working when a field drops out of the mapping and you avoid a reindex that takes hours. Get hands-on with Elasticsearch: Dive into our sample notebooks in the Elasticsearch Labs repo , start a free cloud trial , or try Elastic on your local machine now . How do you make an analytical query engine use data that it cannot know exists? You “just” read the query, since everything that the user asks for is right there. 5, Elasticsearch Query Language (ES|QL) queries no longer fail when a field isn't in the mapping.

The new unmapped_fields setting lets queries load values from _source or fill with nulls , so queries keep working even when a backing index changes and a field goes missing, and can use unmapped data without reindexing. Here’s how we built that: the design choices and the edge cases (including a class of fields we nicknamed PUNKs), along with the testing strategies that gave us the confidence to ship it in general availability (GA). You built a visualization using an ES|QL query. You refined it, and the query grew. You’re at 15 chained commands and counting, but it does just the right thing.

It works, and your dashboard is useful . Your query uses an index from a remote cluster, say my-remote:logs-foo . But actually, logs-foo is an alias, and at some point, the remote cluster makes it point to a different backing index. The new index is missing a field that’s used in your query, and your query and visualization break. Or maybe you have an already fairly large index, and while building ES|QL queries on top of it, you realize that you’d like to use a field in the indexed documents that unfortunately never made it into the index mapping. You could reindex the data, but that would take hours.

ES|QL’s unmapped_fields setting is meant to deal with these types of situations. and some_field is unmapped, ES|QL’s default behavior is to fail with a verification exception. You can use the unmapped_fields setting to instead either fill some_field with null s or read it from the document’s _source , like so: Before we jump into the inner workings of unmapped_fields , we have to look into how ES|QL resolves queries regularly. Let’s consider the above query: We said that if some_field isn’t in the mapping for index , ES|QL wil