json-pull

A streaming JSON pull parser in C.

Does the world really need another JSON parser? Maybe not. But what distinguishes this one is the desire to let you read through gigantic files larger than you can comfortably fit in memory and still find the parts you need.

It builds a parse tree, but you can tear down as much of the tree as you don't need while it is still in the middle of parsing.

Setup

You can create a json_pull parser object by calling json_begin_file() to begin reading from a file or json_begin_string() to begin reading from a string.

You can also read from anything you want by calling json_begin() with your own read() function that copies UTF-8 bytes from your input stream.

Reading full JSON trees

The simplest form is json_read_tree(), which reads a complete JSON object from the stream, or NULL if there was an error or on end of file. Call json_free() on the object when you are done with it.

You can continue calling json_read_tree() to read additional objects from the same stream. This is not standard JSON, but is useful for something like the Twitter filter stream that contains a series of JSON objects separated by newlines, without a wrapper array that contains them all. (The previous parse tree will automatically have json_free() called on it to prevent memory leaks, unless you have called json_disconnect() to disconnect it from the parser.)

In addition, extra commas at the top level are no longer flagged as an error, so a series of comma-separated items can be read without their container array.

Reading JSON streams

With something like GeoJSON, where it is common to have a large wrapper array that contains many smaller items that are often useful to consider separately, you can read the stream one token at a time.

The normal form is json_read(), which returns the next complete object from the stream. This can be a single string, number, true, false, or null, or it can be an array or hash that contains other primitive or compound objects.

Note that each array or hash will be returned following all the objects that it contains. The outermost object will be the same one that json_read_tree() would have returned, and you can tell that it is the outer object because its parent field is null.

You can call json_free() on each object as you are finished with it, or wait until the end and call json_free() on the outer object which will recursively free everything that it contains. Freeing an object before its container is complete also removes it from its parent array or hash so that there are not dangling references left to it.

If you want to keep an individual JSON object around for later processing while allowing its parents and siblings to be freed, json_disconnect() will detach an object from its container. You should call json_free() later to free the detached object once you are finished with it.

Reading JSON streams with callbacks

If you are outputting a new stream as you read instead of just looking for the sub-objects that interest you, you also need to know when arrays and hashes begin, not just when they end, so you can output the opening bracket or brace. For this purpose there is an additional streaming reader function, json_read_separators(), which takes an additional argument for a function to call when brackets, braces, commas, and colons are read. Other object types and the closing of arrays and hashes are still sent through the normal return value.

The types that can be sent to the callback function are JSON_ARRAY, JSON_HASH, JSON_COMMA, and JSON_COLON.

Cleanup

If there was an error while parsing, the parser will have returned NULL before all the containers were closed. You will probably want to call json_free() on the root elemement of the parser, which should contain the full parse tree so far, to avoid leaking memory.

Shutdown

To free the parser object, call json_end() on it.

Object format

JSON objects are represented as the C struct json_object. It contains a type field to indicate its type, a parent pointer to the container that encloses it, and additional fields per type.

Types JSON_NULL, JSON_TRUE, and JSON_FALSE have no additional data.

Strings have type JSON_STRING, with null-terminated UTF-8 text in string and length in length.

Numbers have type JSON_NUMBER, with value in number, and also preserve the original representation of the number in string and length in length.

Arrays have type JSON_ARRAY. There are length elements in the array, and the elements are in array.

Hashes have type JSON_HASH. There are length key-value pairs, and the keys are in keys and the values in values.

Parser format

The parser object has two fields of public interest: error is a string describing any errors found in the JSON, and line is the current line number being read from the input to make it easier to find the errors.

The root field points to the outer object of the current parse tree.

Utility function

There is a function json_hash_get() that looks up the JSON object hash value corresponding to a C string hash key in a JSON hash object. If the object specified is NULL or not a JSON hash or has no matching key, it returns NULL.

Another function json_stringify() converts a JSON parse tree back to its string representation. The caller is responsible for calling free() on the return value once it is no longer needed.

Test program

The jsoncat program reads JSON from the standard input or from named files and pretty-prints it to the standard output.

Normally it uses the callback interface to avoid memory overhead. It has three options to exercise different aspects of the library:

-t reads the whole tree and then prints it.
-i reads incrementally, but still keeps it all in memory.
-s reads the file into a string before parsing it with callbacks.

GeoJSON feature extraction

The geojson2nd program reads JSON from the standard input input or from named files and outputs each object where object.type === "Feature' to the standard output on a separate line. The resulting stream can then conveniently be used as input to the ndjson-cli tools.

e-n-f/json-pull