wanglingsong/JsonSurfer

Add ability to parse 'live' JSON files

Closed this issue · 2 comments

grimlock81 commented

Many applications log to JSON by writing an array of JSON objects, with each object representing a log entry. As each log file is being written to, the opening array [ marker is at the start of the file, but the closing array ] marker is only written when the file is closed and a new file is opened.

This means that while the file is being written to, the contents are not well-formed JSON.

An example of this is MySQL:
https://dev.mysql.com/doc/refman/8.0/en/audit-log-file-formats.html#audit-log-file-json-format
Sample:

[
  {
    "timestamp": "2019-10-03 13:50:01",
    "id": 0,
    "class": "audit",
    "event": "startup"
  },
  {
    "timestamp": "2019-10-03 15:02:32",
    "id": 0,
    "class": "connection",
    "event": "connect"
  },
  {
    "timestamp": "2019-10-03 17:37:26",
    "id": 0,
    "class": "table_access",
    "event": "insert"
  }
]

I tried using the JsonSurfer.iterator(InputSream, JsonPath) method, hoping that calling the iterator.hasNext() method would return true when more JSON objects are added to the file, but it failed. The GSON and Jackson instances parsed the 3 objects but after that threw Exceptions complaining about parsing errors (I assume to do with non well formed JSON). The JsonSimple instance threw an UnsupportedOperationException when trying to create a resumable parser.

The FastJson instance didn't throw any exceptions and parsed the example above, but when more JSON objects were appended to the file, the Iterator.hasNext() still returned false.

Is there a way to achieve this?

Some test code for the above JSON sample:

public static void main(String[] args) {
        File f = new File("D:\\temp\\json.json");
        JsonSurfer fastJsonInstance = JsonSurferFastJson.INSTANCE; // works
        JsonSurfer gsonInstance = JsonSurferGson.INSTANCE; // fails with EOFException
        JsonSurfer jacksonInstance = JsonSurferJackson.INSTANCE; //fails with JsonParseException
        JsonSurfer jsonSimpleInstance = JsonSurferJsonSimple.INSTANCE; // unsupported

        JsonSurfer surfer = jsonSimpleInstance;
        try (InputStream currentInputStream = new FileInputStream(f)) {
            Iterator iterator = surfer.iterator(currentInputStream, compileToJsonPath("$[*]"));

            while (true) {
                while (iterator.hasNext()) {
                    System.out.println("iterator.next() = '" + iterator.next() + "'");
                }
                try {
                    System.out.println("going to sleep");
                    Thread.sleep(5000);
                }
                catch (InterruptedException ie) {
                    ie.printStackTrace();
                }
            }
        }
        catch (IOException fnfe) {
            fnfe.printStackTrace();
        }
        catch (InvalidJsonPathException ijpe) {
            ijpe.printStackTrace();
        }
    }
grimlock81 commented

Thanks that works great.