A second rule makes empty-string filters 1. It reads string lengths straight from the offset array and never touches the compressed bytes. Both rules came from the same habit of running real queries and hunting for the special case. Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running! You can start a free cloud trial or try Elastic on your local machine now. Lucene query rewrite rules make two string scan queries in Elasticsearch 's columnar mode 2. Both rules spot a query shape at runtime and swap in a cheaper implementation.
For a wildcard query like *google* , that's a substring search in place of the automaton. A filter like SearchPhrase ! = '' can skip Zstd decompression, because it only needs string lengths that are sitting in an offset array. Columnar mode is Elasticsearch's analytics-optimized columnar storage mode, built for scan-heavy workloads, like log analytics. In this mode, keyword fields don't get an inverted index by default, so term and wildcard queries scan doc values. DocValuesSkippers (zone maps) already trim how much data a scan touches, but these rewrites cut the cost of what's left.
In Lucene, every query has the option to implement a rewrite method that returns another query. This method returns a query with the same semantics but a different implementation. The query engine repeatedly calls the rewrite method until the returned query doesn’t change. This final query is the one that’s actually evaluated. Importantly, the rewrite can see the actual query arguments and specialize the implementation based on these. For example, in a query looking for documents where a string field contains the value "foo", the rewrite method knows that the term we’re searching for is "foo".
In theory, rewrite could replace the general query class with something specific to "foo". For example, the original query class ScanningBinaryDocValuesTermQuery could be replaced with FooQuery . Now this rule probably wouldn't be helpful, but it gives a sense for the level of specialization that’s achievable with rewrite rules. It's worth placing rewrite rules in the larger context of database systems. Lucene and Elasticsearch aren’t the first systems to use transformation rules to optimize queries. Most (or maybe all) database systems use some kind of rule system during query optimization.