Rapture JSON
Rapture JSON is a comprehensive library providing support for working with JSON in Scala. Rapture JSON is part of the Rapture project.
Features
- Clean, intuitive, unintrusive, boilerplate-free Scala API
- Trivial extraction and serialization to/from primitives, collections and case classes
- Consistent and typesafe type-class-based interfaces
- Works with a choice of JSON parsers and backends
- Simple, efficient conversion between different backends
- Support for both immutable and mutable JSON
- Flexible choice of error handling strategies using modes
- Can be easily extended using composable user-defined extractors and serializers
Availability
Rapture JSON 1.1.0 is available under the Apache 2.0 License from Maven Central
with group ID com.propensive
and artifact ID rapture-json_2.11
.
SBT
You can include Rapture JSON as a dependency in your own project by adding the following library dependency to your build file:
libraryDependencies ++= Seq("com.propensive" %% "rapture-json-[backend]" % "1.1.0")
where [backend]
is one of the following JSON backends:
- Argonaut (
argonaut
) - Jackson (
jackson
) - Jawn (
jawn
) - JSON4S (
json4s
) - Lift JSON (
lift
-- Scala 2.10 only) - Play JSON (
play
) - Spray (
spray
)
You can also use Rapture JSON with the JSON parser built into Scala 2.10 with the following dependency, though this is not recommended due to the poor performance characteristics of this parser.
libraryDependencies ++= Seq("com.propensive" %% "rapture-json" % "1.1.0")
Maven
If you use Maven, include the following dependency:
<dependency>
<groupId>com.propensive</groupId>
<artifactId>rapture-json-[backend]_2.11</artifactId>
<version>1.1.0</version>
</dependency>
where [backend] is one of the backends listed above.
Building from source with SBT
To build Rapture JSON from source, follow these steps:
git clone git@github.com:propensive/rapture-json.git
cd rapture-json
sbt package
If the compilation is successful, the compiled JAR file should be found in the
directory for the appropriate Scala version in the target
directory.
Status
Rapture JSON is now stable. This means that the API represents a useful set of features, and is unlikely to change significantly in subsequent releases. All API changes will be documented with instructions on how to upgrade.
Tests
Rapture JSON has a reasonably comprehensive test suite in a separate project. Unfortunately, running the tests is not entirely straightforward yet (we are working on this), but the source code shows some typical uses of Rapture JSON, so may be instructive in learning the library.
Contributing
Rapture JSON -- like all the Rapture projects -- openly welcomes contributions! We would love to receive pull requests of bugfixes and enhancements from other developers. To avoid potential wasted effort, bugs should first be reported on the Github issue tracker, and it's normally a good idea to talk about enhancements on the Rapture mailing list before embarking on any development. Alternatively, just send Jon Pretty (@propensive) a tweet to start a conversation.
Using Rapture JSON
JSON Representation
Rapture JSON is designed to be agnostic about the JSON parser and choice of AST representation used throughout the library. This means that a choice of JSON backend must be made in order to use Rapture JSON. Whilst different backend libraries provide different features, all features of Rapture JSON are available with every backend (with the exception of Jackson, which does not yet support mutable JSON operations).
The choice of backend should therefore depend on other characteristics such as performance, memory usage, required dependencies, integration with existing libraries, licensing and policy choices.
The following backends are available:
- Argonaut (
argonaut
) - Jackson (
jackson
) - Jawn (
jawn
) - JSON4S (
json4s
) - Lift (
lift
-- Scala 2.10 only) - Play (
play
) - Scala standard library JSON (
scalaJson
) - Spray (
spray
)
Work is ongoing to make Play JSON available too. It is also possible to integrate with other JSON backends, though this is not covered by this document. Anyone interested should look at the existing integration type classes, and contact the Rapture mailing list.
In the source code, you should import rapture.json.jsonBackends.<backend>._
.
The Json type
A JSON value, whether an array, object, boolean, number or string, is
represented by an instance of type Json
. As JSON is inherently
dynamically-typed, the Json
type is used to provide a safe and immutable
wrapper around the JSON tree, whose type is not known at compile time.
Instances of Json
consist of three things:
- a reference to the root of a dynamically-typed JSON tree
- a path into a node within the JSON tree
- a reference to the parser used to create, modify and read the JSON tree
Although using Json
objects should seem very intuitive, it is important to
understand the purpose of this state.
{
"fruits": [
{
"name": "apple",
"color": "red"
},
{
"name": "banana",
"color": "yellow"
}
]
}
If we were to parse the above JSON source, we should get a tree consisting of
an object containing an array under the key "fruits", with two elements, each
of which is an object containing two fields, "name" and "color", both of which
are strings. Given this tree, we can refer to an element within with a path of
strings for indexing JSON objects, and integers for indexing JSON arrays, for
example, fruits / 0 / name
, which would refer to the string "apple"
.
We could also look into the same tree with the path fruits / 3 / mass
, though
this wouldn't exist on account of there being only two elements in the fruits
list, but we would not know this until we attempted it at runtime.
A Json
instance represents both the JSON tree, and a lazily-evaluated path
into that tree, which may or may not point to a value. If we assume the full
JSON tree is a starting point (most likely originating from being parsed from
source), Json
instances can be created which hold the same reference to the
original tree, but point -- by means of a path -- to any subtree of the
original, without the performance cost of navigating the tree, or the
requirement to safely handle missing-value or type-mismatch errors which arise
because the path attempts to access a value which isn't available.
At some later point, if the JSON is to yield some useful data which we can do interesting things with, we will need to perform the access, and assign a Scala type to it, as it passes from the dynamic to the static world. It is at this point that all access failures will arise, so by deferring them to a single point, they can be handled just once.
Additionally, every Json
instance stores a reference to the backend which was
used to create it, and which will be used to access it. As Rapture JSON permits
multiple different parsers to be used alongside each other, it is important
that the AST within each Json
instance is always handled using the right
backend.
Accessing JSON values
Json
instances implement Scala's Dynamic
trait, providing a very natural
way to refer to object fields within a Json
value just by calling that field
name as if it were a method on the Json
instance. Additionally, integers may
be applied to index into arrays.
For example, using the example JSON above, we can create a new Json
instance
pointing to the string "yellow"
as follows:
json.fruits(1).color
Remember, this is just creating a pointer into the "yellow"
value; it's not
been accessed yet. To extract a value from a Json
value, the as
method is
used. as
takes a single type parameter, and is the single point at which a
JSON type-mismatch or missing-value exception can occur.
json.fruits(1).color.as[String]
Rapture JSON uses Rapture Core's modes on the as
method, thus allowing
failure cases to be handled using the preferred exception-handling strategy,
for example by throwing an exception or returning a Try
. See the section on
error messages below.
Note that calling toString
, as happens automatically after every evaluation
in the Scala REPL, will cause the AST to be accessed, but any errors will be
suppressed, and the toString
method will return the string "undefined"
.
A variety of types may be extracted from a Json
value, including the
following:
- Primitive types, such as
String
,Int
,Double
andBoolean
- Scala collections of other extractable types, e.g.
List[Int]
orSet[Int]
- Case classes, extracted by the names of their parameters, provided the type of every parameter is extractable
Option
s, whereNone
will be extracted if the element is missing or the wrong typeJson
types -- the no-op extractor- Any other type for which an implicit
Extractor
exists in scope
These types compose, so it is possible to extract values of complex types like
Option[Vector[MyCaseClass]]
.
Creating JSON values
JSON values can be created in a number of different ways. If starting with a
JSON source, which will often be a String
(but may be another type -- the
Jawn backend can parse directly from ByteBuffers, for example), we can call
Json.parse(src)
to attempt to parse the source src
.
Json
values can also be created directly in code, using the json
string
context, like so:
json"""{
"fruit": "apple",
"variants": ["cox", "braeburn"]
}"""
Much like an interpolated string, Scala expressions may be substituted into a
json
string context, provided the expressions evaluate to a type which is
serializable to Json. Generally speaking, all types which can be extracted from
a Json
value (primitives, collections, case classes, Json) can also be serialized,
like so:
val f = "apple"
json"""{
"fruit": $f,
"variants": ${List("cox", "braeburn")}
}"""
Any serializable type can also be converted to Json
by applying it to the
Json
object, for example this,
case class Fruit(name: String, variants: Set[String])
Json(Fruit("apple", Set("cox", "braeburn")))
will produce the same JSON as the previous examples.
An instructive compile error will be displayed in the event of an attempt to
serialize a type which cannot be serialized to Json
.
Pattern matching on JSON
An alternative way of extracting values from Json
types in to use pattern
matching. A Json
value can be pattern matched against JSON literals defined
inline in the case clause, like this:
val json: Json = Json.parse(src)
json match {
case json""" { "fruit": $name }""" =>
name.as[String]
case json""" { "vegetable": $name }""" =>
name.as[String]
}
This will first attempt to match any JSON object which contains a key called
fruit
, and bind the value to the identifier name
. If that match fails, it
will attempt to match any JSON object which contains a key called vegetable
,
and likewise bind its value to name
. In each case, we return the name as a
String
. The call to .as[String]
is necessary because, as before, the
compiler will no know nothing about the nature of the type at compile time, so
it must be explicitly specified.
Literal match values may also be explicitly specified, for example:
val json: Json = Json.parse(src)
json match {
case json""" { "fruit": "apple", "variety": $vs }""" =>
vs.as[Set[String]]
case json""" { "fruit": "lemon" }""" =>
Set()
}
Multiple values may be extracted, and the pattern match expression may involve arbitrarily-deep object and array nesting.
The examples above use the default configuration for structural pattern matching, however three configuration explicits control the strictness of structural pattern matching.
- patternMatching.exactObjects
- patternMatching.exactArrays
- patternMatching.exact
If it is required that pattern matching on JSON objects should match the entire object, i.e. the existence of any keys in the object which are not specified in the pattern should result in failure to match, then include the following import somewhere within the scope of the pattern match:
import patternMatching.exactObjects
By default, array elements specified in a pattern are matched positionally, and superfluous array elements in the tail of the array are ignored, and the match will be successful. If strict array matching is required, include:
import patternMatching.exactArrays
If exact matching of both arrays and objects is required, then it is sufficient to import
import patternMatching.exact
Json
Modifying Whatever underlying backend is used, the Json
type is immutable. A small
number of methods are provided to create new Json
values from existing
values. Given a Json
value, a new key may be added using the following
syntax:
val j = json"""{ "fruit": "plum" }"""
j + (_.color, "purple")
This syntax is designed to resemble the syntax for adding a value to a
scala.collection.Map
, whereby the tuple passed to the +
operator contains a
key and a value, resulting in the addition of this value to the map. However,
with Json
types, the "key" is a path into the Json
structure, and the
"value" is the value to be serialized to JSON and stored at that position in
the JSON tree.
The ++
operator can be used to combine two JSON structures, like so:
val j1 = json"""{ "fruit": { "name": "grape" } }"""
val j2 = json"""{ "fruit": { "color": "white" } }"""
val json = j1 ++ j2
The resultant json
value would be
{
"fruit": {
"name": "grape",
"color": "white"
}
}
Currently, when merging, object keys from both sides will be merged, favoring
the right side in the event of a clash, and arrays will be concatenated. If the
types of corresponding values do not match, the value from the right side will
clobber that from the left. In a later version of Rapture JSON, more control
over mechanisms for combining and merging Json
values may be provided through
configuration implicits.
Error messages
By default, failures in Rapture JSON operations will result in an exception
being thrown, however, alternative exception handling methods can be used with
a simple import from the Rapture Core project which will determine the return
type of all fallible methods such as Json.parse
and as
. For example,
import rapture.core._
import modes.returnTry
Json.parse(src)
will result in the return type of Json.parse
changing from Json
to
Try[Json]
, safely capturing any failures which may result from the operation.
All Rapture JSON methods will throw a limited set of possible exceptions, which
are implemented as case classes inheriting from a sealed trait. If choosing to
use the captureExceptions
mode, or another mode which captures the exception
and tracks the exception type in its signature, this allows exhaustivity
checking to be performed on errors, for example:
import modes.captureExceptions
json.fruit.as[String] match {
case Right(f) =>
s"Found fruit $f"
case Left(TypeMismatchException(found, _, _)) =>
s"Fruit was the wrong type: $found"
case Left(MissingValueException(path)) =>
s"Fruit value was missing at path $path"
}
If either of the final two case clauses were omitted from this pattern match, the compiler will issue a warning that the match may not be exhaustive.
Mutable JSON
In the same way that the List
type from the Scala collections library has a
corresponding mutable ListBuffer
type, the Json
type has a corresponding
JsonBuffer
type, which supports mutability in addition to the operations
described above. The functionality of JsonBuffer
is a strict superset of the
functionality of Json
.
An empty JsonBuffer
may be created with
val jb = JsonBuffer.empty
and can be mutated with instructions such as
jb.fruit.name = "apple"
jb.fruit.color = "green"
jb.fruit.varieties = List("cox")
resulting in
{
"fruit": {
"name": "apple",
"color": "green",
"varieties": ["cox"]
}
}
Note that the top-level fruit
object gets automatically created by the first
instruction, and that the right-hand side of the assignment is serialized to
JSON using Rapture's standard serialization scheme, causing a compile-time
error if the type cannot be serialized.
It is additionally possible to append items to the end of an array using the
+=
operator, like this:
jb.fruit.varieties += "braeburn"
Mutable JSON is not yet available for all backends, though this work is in progress. Note that the underlying JSON backend does not need to be inherently mutable to support mutable JSON. If a backend which uses an immutable AST is used, Rapture JSON will efficiently perform the necessary tree manipulations, updating references as necessary to give the impression of a mutable data structure. However, using a backend which uses a mutable JSON representation will likely result in better performance.
Converting JSON
All JSON is represented by the same Scala type, Json
, regardless of what type
of value it contains, and which backend was used to create it.
FIXME: Complete this
Outputting JSON
Often, the easiest way to output JSON from the Json
type is to call
.toString
on the Json
value. Although simple, this has the disadvantages
that it does not offer any flexibility in how the JSON is formatted, and it
always returns a String
whereas other types, such as an input stream may be
more appropriate for some applications.
The more general method is to use the Json.format
method, with an appropriate
implicit Formatter
in scope. Two formatters are provided as standard:
formatters.humanReadable._
, which formats the JSON with newlines and indentation, attempting to make it as readable as possible,formatters.compact._
, which includes no unnecessary whitespace
Both formatters return String
s, though it is possible for other backends to
provide their own formatters for outputting to other types.
For example,
import formatters.compact
val out = Json.format(json)
Defining custom extractors and serializers
FIXME: Complete this