Search fails when selecting time range

Hi :D

Describe the bug
When I try selecting a time range I get empty results from the toast notifications. I get this error message in Kuiper.log

"2023-06-21 14:25:40.987342","[ERROR]","elkdb.py.query[Lin.295]","elasticsearch","Query [TEST_CASE] failed [RequestError]","Cannot parse '!(data_type:"tag") AND (machine:"TEST_CASE_TEST_MACHINE" AND Data.@timestamp:[2023\-05\-01T16\:58\:00\ TO\ 2023\-05\-04T16\:58\:00])': Encountered " <RANGE_GOOP> "TO\\ "" at line 1, column 131.
Was expecting:
    "TO" ...
    "

To Reproduce
Use the latest commit in this repo (commit id. dcb09077799da9ee9d063f87af8619b7683f0bad)

Expected behavior
To retrieve and show records in the time range

Additional context
I was able to trace the issue to the function build_search_query in the file browse_artifacts.html

Kuiper/kuiper/app/templates/case/browse_artifacts.html

Line 1592 in 745cee7

    
           var special_chars       = [ '\\' , '/' , ':' , '-' , '{' , '}' , '(', ')' , ' ' , '@' ]

This function removes bad character and prepare ES query. However, it escapes the space characters between the timestamp and the keyword TO which is not a valid ES query.

When I remove the space from this list (i.e ' ') it works as expected. However, any search query with a space will fail

From git blam this line was added 3 years ago. However, the issue is very recent.

I tried this commit 13d7488719b020059f4c22cca9e36336ebfe9cb1 (commit was on Feb this year) and it works without issues but the commit dcb09077799da9ee9d063f87af8619b7683f0bad have this issue. So, the bug was introduced somewhere between these commits

Thanks!

could you check the data type of the field, it should be date
the escape should only be applied to text fields
https://github.com/DFIRKuiper/Kuiper/blob/745cee7f82961738ef9f76306f2914d5b0847c0d/kuiper/app/templates/case/browse_artifacts.html#L1588C46-L1588C59

Ok, I will check that next week

Happy Eid 🥳🎉

Hala @salehmuhaysin ✋

Yes the type of the field @timestamp is text, here is the command I executed:

curl http://127.0.0.1:9200/test/_mapping -s | jq '.test.mappings.properties.Data.properties."@timestamp"'

output:

{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  },
  "copy_to": [
    "catch_all"
  ],
  "analyzer": "default"
}

I noticed something, after sometime the field type will change to date. I had to remove the index and all data then parse the artifact again for it to show me the error.

NOTE: I am parsing kjson file for this error to show, However in some cases I noticed the error while parsing registry files (I am not sure which parser is it REGTimeline or AutoParser @muteb & @mayHamad might be able to help). The kjson file has timestamps in the field @timestamp in the format YYYY-MM-DD HH:MM:SS but some of the records contain an empty string (i.e "" not null) in the field @timestamp

hello
it is data type auto-detect, so it will consider the first records with the field @timestamp and then decide what is the field type, maybe these records that does not include seconds corrupted the field type and make it text.
yes if the field parsed as text you need to delete the index to fix the mapping

Could we add a check to Kuiper before it pushes the record to the index to check for @timestamp field and if it can't parse it then it should default to 1700-01-01T00:00:00? I see here you already do that for empty fields (i.e null):

Kuiper/kuiper/app/controllers/parser_management.py

Line 660 in dcb0907

    
           if '@timestamp' not in data[d]['Data'] or data[d]['Data']['@timestamp'] is None:

could we add one more check like this:

if '@timestamp' not in data[d]['Data'] or data[d]['Data']['@timestamp'] is None or data[d]['Data']['@timestamp'] is "":

done, fixed the issue
it will check for both empty string and if date should be ISO format
r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?Z?$'