Common Lisp JSON reading and writing for the Kzinti.
The primary interface to parsing and reading JSON is the read-json function.
(read-json &optional input-stream-or-string (eof-error-p t) eof-value single-value-p)The argument input-stream-or-string can be an stream, a string to read from,
or nil to use *standard-input*. The arguments eof-error-p and eof-value
have the same affect as they do in the Common Lisp function read. If the
single-value-p argument is true then the input to read-json is assumed to
be a single value, which means that extra tokens at the end will cause an
error to be generated.
There are a number of dynamic variables that will influence the parsing of JSON data.
common-lisp:*read-default-float-format*— Controls the floating-point format that is to be used when reading a floating-point number.*read-default-true-value*— The default value to return when reading a true token. Initially set tot.*read-default-false-value*— The default value to return when reading a false token. Initially set tonil.*read-default-null-value*— The default value to return when reading a null token. Initially set to:null.*read-default-array-format*— The default format to use when reading an array. Current supported formats are:vectoror:list. Initially set to:vector.*read-default-object-format*— The default format to use when reading an object. Current supported formats are:hash-table,:alistor:plist. Initially set to:hash-table.*read-length*— The maximum number of values in an array or an object. Initially set tonilwhich disables length checking.*read-level*— The maximum number of levels to allow during reading for arrays and objects. Initially set tonilwhich disables level checking.
There is also a keyword variant read-json* which will set the various dynamic
variables from supplied keywords.
(read-json* :stream nil
:eof-error t
:eof-value nil
:single-value nil
:true-value t
:false-value nil
:null-value :null
:array-format :vector
:object-format :hash-table
:float-format 'single-float
:length nil
:level nil)The primary interface to serializing and writing JSON is the write-json
function.
(write-json value &optional (output-stream t))The output-stream argument can be a stream, t for *standard-output*, or
nil for output to a string. If the output is to a string then this string will
be returned, otherwise the original value will be returned.
There are a number of dynamic variables that will influence the serialization of JSON data.
common-lisp:*print-pretty*— If true then a simple indentation algorithm will be used.*write-indent-string*— The string to use when indenting objects and arrays. Initially set to#\space.*write-ascii-encoding*— If true then any non ASCII values will be encoded using Unicode escape sequences. Initially set tonil.*write-true-values*— Values that will be written as a true token. Initially set to'(t :true).*write-false-values*— Values that will be written as a false token. Initially set to'(nil :false).*write-null-values*— Values that will be written as a null token. Initially set to(:null).*write-alist-as-object*— If true then assocation lists will be written as an object. Initially set tonil.*write-plist-as-object*— If true then property lists will be written as an object. Initially set tonil.*write-empty-array-values*— A list of values that will be written as an empty array.*write-empty-object-values*— A list of values that will be written as an empty object.*write-array-tags*— A list of values whose appearance in the CAR of a list indicates the CDR of the list should be written as an array. Initially set to'(:array).*write-object-alist-tags*— A list of values whose appearance in the CAR of a list indicates the CDR of the list is an alist and should be written as an object. Initially set to'(:object-alist).*write-object-plist-tags*— A list of values whose appearance in the CAR of a list indicates the CDR of the list is a plist and should be written as an object. Initially set to'(:object-plist).
The actual serialization of JSON data is done by the generic function
print-json-value which can be specialized for additional value types.
(print-json-value value output-stream)There is also a keyword variant write-json* which will set the various dynamic
variables from supplied keywords and will default to the current dynamic value
of each keyword.
(write-json* value :stream t
:ascii-encoding nil
:true-values '(t :true)
:false-values '(nil :false)
:null-values '(:null)
:empty-array-values '(:empty-array)
:empty-object-values '(:empty-object)
:array-tags '(:array)
:object-alist-tags '(:object-alist)
:object-plist-tags '(:object-plist)
:alist-as-object nil
:plist-as-object nil
:pretty nil
:indent-string " ")In order to facilitate extending the serialization facilities of shasht there are a number of helper functions available. To aid in the printing of JSON strings there is the following.
(write-json-string value output-stream)In order to ease the serialization of objects and arrays there is
with-json-object and with-json-array. Both of these macros take an
output stream as the first argument then enable indentation and automatic
handling of all delimiter tokens. Inside the body of with-json-object
the function (print-json-key-value key value output-stream) should be used
to output a key value pair. Inside the body of with-json-array the function
(print-json-value value output-stream) should be used to output a single
value. Example usage can be seen in the source code.
Sometimes using incremental serialization via the serialization helper functions is not the best fit for the application. For example, maybe the JSON needs to be assembled from various fragments and then analyzed before serialization. This happens in the Jupyter widget protocol. Binary data value in the JSON are extracted from the JSON and transmitted in binary form in order to decrease the network load.
Since constructing hash tables purely for the purpose of JSON serialization is a bit difficult and potentially memory wasteful, there are various keyword literals that can be used to construct JSON objects in order to avoid this situation. There are also literals for JSON arrays, although they may not be as useful. These literals exist in shasht mostly for consistency. The following shows some examples of the use of these literals.
* (shasht:write-json
'(:object-alist ("a" . :empty-array)
("b" . :empty-object)
("c" . (:object-plist "d" 1
"e" (:array 1 2 3)))))
{
"a": [],
"b": {},
"c": {
"d": 1,
"e": [
1,
2,
3
]
}
}
(:OBJECT-ALIST ("a" . :EMPTY-ARRAY) ("b" . :EMPTY-OBJECT)
("c" :OBJECT-PLIST "d" 1 "e" (:ARRAY 1 2 3)))
The values and tags that indicate these literals can be configured via the
dynamic variables *write-empty-array-values*, *write-empty-object-values*,
*write-array-tags*, *write-object-alist-tags*,
and *write-object-plist-tags*.
These literals forms are only meant for serialization and not for round-trip mapping. Therefore there is no way to read JSON in the same format.
The mapping of types between Common Lisp and JSON is not one-to-one nor is it
without ambiguity due to issues such as Common Lisp's treatment of nil as
an empty list and as a false value. JSON also includes null which does not
have an obvious representation in Common Lisp. Because of this, shasht makes
certain choices in the default mapping of data types between Common Lisp and
JSON that may not be ideal for all applications. shasht was primarily designed
to be a good round-trip encoder/decoder for common-lisp-jupyter in the
network protocol of Jupyter.
A brief explanation of the default mapping is given in the table below. For more detail regarding the mapping of individual types and how to configure that mapping see the sections following this table.
| Common Lisp | JSON | |
|---|---|---|
| integer | <-> | number without decimal or exponent |
| float | <-> | number with decimal or exponent |
| ratio | -> | number with decimal or exponent |
| rational | -> | number with decimal or exponent |
| string | <-> | string |
| character | -> | string |
| pathname | -> | string |
| symbol not matching other mapping | -> | string |
| vector | <-> | array |
| multi-dimensional array | -> | nested array |
non-nil list |
-> | array |
| hash table | <-> | object |
| standard object | -> | object |
| structure object | -> | object |
t |
<-> | true |
:true |
-> | true |
nil |
<-> | false |
:false |
-> | false |
:null |
<-> | null |
:empty-array |
-> | [] |
:empty-object |
-> | {} |
'(:array 1 2 3) |
-> | [1,2,3] |
'(:object-alist ("a" . 1) ("b" . 2)) |
-> | {"a":1,"b":2} |
'(:object-plist "a" 1 "b" 2) |
-> | {"a":1,"b":2} |
The format of a number read from JSON when a decimal or an exponent is present
in the number literal can be influenced with cl:*read-default-float-format*.
This is the same behavior of cl:read. In order to read JSON numbers with large
exponents one would need do something like the following.
(shasht:read-json "[2.232e75]" :float-format 'double-float)The dynamic variables *read-default-array-format*,
*write-empty-array-values*, and *write-array-tags* all influence the mapping
of JSON arrays to Common Lisp vectors and lists. Common Lisp vectors and
multi-dimensional arrays are always writen as JSON arrays. By default JSON
arrays are read as Common Lisp vectors. With the default settings only non-nil
lists that don't satisfy some other mapping rule are written as JSON arrays.
If one wants to use lists as the default JSON array format then
*read-default-false-value*, *read-default-array-format*, and
*write-false-value* will need to need to be set to appropriate values since
in the default mapping nil maps to false. For example, the following
could be done.
(let ((shasht:*read-default-false-value* :false)
(shasht:*read-default-array-format* :list)
(shasht:*write-false-values* '(:false)))
(shasht:read-json ...)
(shasht:write-json ...))Lists with a CAR eql to a value in *write-array-tags*,
*write-object-alist-tags*, *write-object-plist-tags* will still be written
as an array or object as appropriate. To completely disable this behavior the
variables would need to be bound to nil. Or one could do the following.
(shasht:write-json '(1 2 3) :false-value '(:false) :array-tags nil
:object-alist-tags nil :object-plist-tags nil)In this case the mapping for array types would become:
| Common Lisp | JSON | |
|---|---|---|
| vector | -> | array |
| multi-dimensional array | -> | nested array |
| list | <-> | array |
The dynamic variables *read-default-object-format*, *write-alist-as-object*,
*write-plist-as-object*, *write-empty-object-values*,
*write-object-alist-tags*, and *write-object-plist-tags* all influence the
mapping of JSON objects to Common Lisp hash tables, alists, and plists. Common
Lisp hash tables are always written as JSON objects. By default JSON objects are
read as Common Lisp hash tables.
In order to use alists as the default JSON object format the dynamic variables
*read-default-object-format*, *write-alist-as-object*,
*read-default-false-value*, and *write-false-values* will need to be set
to appropriate values. For example, the following would use alists as the
default JSON object format and :false as the JSON false value.
(let ((shasht:*read-default-object-format* :alist)
(shasht:*write-alist-as-object* t)
(shasht:*read-default-false-value* :false)
(shasht:*write-false-values* '(:false)))
(shasht:read-json ...)
(shasht:write-json ...))In this case the mapping for object types would become:
| Common Lisp | JSON | |
|---|---|---|
| hash table | -> | object |
| alist | <-> | object |
| standard object | -> | object |
| structure object | -> | object |
The same could be accomplished for plists by doing the following.
(let ((shasht:*read-default-object-format* :plist)
(shasht:*write-plist-as-object* t)
(shasht:*read-default-false-value* :false)
(shasht:*write-false-values* '(:false)))
(shasht:read-json ...)
(shasht:write-json ...))Although concise the JSON specification is very vague on a number of points and thus accurate compliance by implementations is often substandard. Without comprehensive tests compliance is difficult to ascertain. The JSONTestSuite includes over 300 reading tests including those left ambiguous by the specification. The test suite of shasht includes all of these tests in addition to various write tests. For a comparision of the compliance of the Common Lisp implementations of JSON see Compliance Comparision.
A simple benchmark can be done with tests/bench.lisp. For SBCL the following
results are typical.
JSON Read Times
0 7.002169e-6 1.4004338e-5
˫--------------------------------+--------------------------------˧
cl-json ███████████████████████████████████▉
jonathan ████████▊
json-streams ███████████████████████████████████████████████████████████████████
jsown █████████▋
shasht █████████████▏
st-json █████████████████████████████████▊
yason ████████████████████████████████████████▉
JSON Write Times
0 6.2018808e-6 1.24037615e-5
˫--------------------------------+--------------------------------˧
cl-json ███████████████████████████████████████████████████████████████████
jonathan ████████████████████████████████████▍
json-streams ███████████████████████████████▎
jsown ████████████████████████████████████████▉
shasht ███████████▍
st-json ███████▋
yason ████████████████████████████████████▏
JSON Read/Write Times
0 1.1348141e-5 2.2696282e-5
˫--------------------------------+--------------------------------˧
cl-json ███████████████████████████████████████████████████████████████████
jonathan █████████████████████████████████████▍
json-streams ██████████████████████████████████████████████████████████████▉
jsown ██████████████████████████████▎
shasht ██████████████████████▏
st-json ███████████████████████████▋
yason ███████████████████████████████████████████████████████████▉