quickwit-oss/quickwit

use JSON field for concat field

Closed this issue · 0 comments

The concat fields introduced in this PR #4773 uses a multi-valued text field to concatenate the data into one field. Non-string types are converted and indexed as string type.

We could use the JSON field instead of a multivalued text field for the concat field.
That way we keep the type info and there are some synergies since JSON has the same problem space when indexing and querying data with mixed types.

e.g. how would we know if the user input in a query is a text, number, date, etc. and how to generate queries that can handle different types with potential ambiguous cases.
On the indexing side it is similar, that we may encounter e.g. "2024-04-28T17:31:59.893Z" and should convert that to a date.

The change should be straightforward: Change the field type to JSON and remove the conversion logic.

Impact

Impact should be minimal, since the feature is pretty new and not yet released.
But may be some customers are already using it. @trinity-1686a @fmassot
The field type would change, but existing usage should still work, since we check the fields type anyways?

Update: There is one user using it, and it's okay to break it (which I think doesn't even happen).