Support custom Json deserializer/serializer
volyx opened this issue · 12 comments
For example Gson instead Jackson.
Move json to another maven module and allow to add it as separate jar.
Hi @volyx. Thanks for submitting this request. The use of Jackson in the library is mostly private, meaning that Jackson classes aren't returned from the methods. Is your need for GSON so that you don't need to include Jackson in your classpath?
Yes, I already exclude all jackson libraries from my project for using only Gson, but I can't replace Jackson with Gson in fusionauth-jwt. So it would be nice to have a opportunity to somehow register custom serializer/deserializer. For example
JsonSerializer ser = new GsonSerializer()
JWT jwt = new JWT()
.setSerializer(ser)
.setIssuer("www.example.com")
.setIssuedAt(iss)
.setSubject(subject)
.setExpiration(exp);
Thanks for that additional info @volyx . In theory we could abstract the mapper interface and allow a different module / library to be bound for serializing and deserializing.
Most of that code is in Mapper.java - however - we are using some Jackson annotations in the domain which makes it a bit tough without some re-work to the library.
We are using the Jackson @AnySetter and @AnyGetter annotations from Jackson, a similar function would need to exist in Gson.
While not ideal, you could fork the repo and just rip out Jackson.
I'm open to this feature in FusionAuth JWT - we'd need to move the Jackson annotations out of the domain and into some other configuration, or perhaps we could use Jackson mixins to decorate the domain with the annotations we'd need. If that was all done, then it wouldn't be too hard to make a new interface for Guice to bind one of several implementations.
This isn't a high priority - but we can leave this open and see if we have some time to hack on it a bit. Feel free to offer some example code or PRs if you have an idea of how it would work.
Thanks!
outdated
As I see there are two options:
- First is to create a multi-module maven project.
Sofusionauth-jwt-parentwill have such structure:
fusionauth-jwt-parent
- jwt
- jacksson-serializer
- fusionauth-jwt(depends on jwt, jacksjon-serializer)
- gson-serializer
So users can still use fusionauth-jwt library and don't notice any differences after versoin update
2) Do separation in runtime. We could check in runtime availability of JSON library:
if (ObjectMapper class exists) use jackson serserializer
if (Gson class exists) use gson serserializer
So user could exclude Jackson, and add Gson library.
First approach more modular, and allow future extension but needs more work and restructurization.
Second one needs less time, but use some kind of runtime class discovery that is not a very good pattern but still a workable solution.
PoC for removing Jackson
Problem: Jackson libraries are performant but heavy weight and clash with other Jackson libraries in a project
Solution: use lightweight JSON library or do not even use any JSON library - decode/encode manually
Library sizes:
340K jackson-core-2.10.1.jar
66K jackson-annotations-2.10.1.jar
1.3M jackson-databind-2.10.1.jar
136K fusionauth-jwt-3.1.6.jar
fusionauth-jwt library is just 136 Kilobytes.
For comparison
29K nanojson-1.4.jar
33K minimal-json-0.9.5.jar
8.1K JsonReader.java(manually written json reader)
I have added JSON serializers, deserializers which can replace Jackson. I have tested three options: minimal-json, nanojson(embeded), plain(manual)
All them can be tested for compatibility with Jackson ObjectMapper like this:
./mvnw test -Dminimal.json.decode=true use minimal-java for decoding, Jackson for decoding
./mvnw test -Dminimal.json.encode=true use minimal-java for encoding, Jackson for encoding
./mvnw test -Dminimal.json.encode=true -Dminimal.json.decode=true use minimal-java for encoding and decoding
The same is for others encoders/decoders:
./mvnw test -Dnano.json.encode=true -Dnano.json.decode=true
./mvnw test -Dplain.json.encode=true -Dplain.json.decode=true
I wrote benchmark for testing the performance of all these JSON mappers - JsonBenchmark.
Benchmark (serializer) Mode Cnt Score Error Units
JsonBenchmark.testDecode Jackson thrpt 5 0,800 ± 0,329 ops/us
JsonBenchmark.testDecode MinimalJson thrpt 5 0,247 ± 0,093 ops/us
JsonBenchmark.testDecode NanoJson thrpt 5 0,063 ± 0,062 ops/us
JsonBenchmark.testDecode Plain thrpt 5 0,538 ± 0,229 ops/us
JsonBenchmark.testDecode None thrpt 5 6,863 ± 1,322 ops/us
JsonBenchmark.testEncode Jackson thrpt 5 0,235 ± 0,040 ops/us
JsonBenchmark.testEncode MinimalJson thrpt 5 0,097 ± 0,013 ops/us
JsonBenchmark.testEncode NanoJson thrpt 5 0,088 ± 0,102 ops/us
JsonBenchmark.testEncode Plain thrpt 5 0,157 ± 0,077 ops/us
JsonBenchmark.testEncode None thrpt 5 24,158 ± 21,829 ops/us
JsonBenchmark.testDecode Jackson avgt 5 1,189 ± 0,222 us/op
JsonBenchmark.testDecode MinimalJson avgt 5 4,250 ± 2,976 us/op
JsonBenchmark.testDecode NanoJson avgt 5 14,250 ± 6,014 us/op
JsonBenchmark.testDecode Plain avgt 5 1,955 ± 2,875 us/op
JsonBenchmark.testDecode None avgt 5 0,137 ± 0,033 us/op
JsonBenchmark.testEncode Jackson avgt 5 5,854 ± 10,353 us/op
JsonBenchmark.testEncode MinimalJson avgt 5 10,302 ± 10,342 us/op
JsonBenchmark.testEncode NanoJson avgt 5 9,826 ± 7,726 us/op
JsonBenchmark.testEncode Plain avgt 5 5,453 ± 0,905 us/op
JsonBenchmark.testEncode None avgt 5 0,043 ± 0,013 us/op
JsonBenchmark.testDecode Jackson sample 1201091 1,692 ± 0,097 us/op
JsonBenchmark.testDecode:testDecode·p0.00 Jackson sample 0,957 us/op
JsonBenchmark.testDecode:testDecode·p0.50 Jackson sample 1,024 us/op
JsonBenchmark.testDecode:testDecode·p0.90 Jackson sample 1,880 us/op
JsonBenchmark.testDecode:testDecode·p0.95 Jackson sample 2,184 us/op
JsonBenchmark.testDecode:testDecode·p0.99 Jackson sample 4,200 us/op
JsonBenchmark.testDecode:testDecode·p0.999 Jackson sample 78,336 us/op
JsonBenchmark.testDecode:testDecode·p0.9999 Jackson sample 786,226 us/op
JsonBenchmark.testDecode:testDecode·p1.00 Jackson sample 26869,760 us/op
JsonBenchmark.testDecode MinimalJson sample 1248740 6,599 ± 0,489 us/op
JsonBenchmark.testDecode:testDecode·p0.00 MinimalJson sample 2,860 us/op
JsonBenchmark.testDecode:testDecode·p0.50 MinimalJson sample 3,360 us/op
JsonBenchmark.testDecode:testDecode·p0.90 MinimalJson sample 5,472 us/op
JsonBenchmark.testDecode:testDecode·p0.95 MinimalJson sample 5,872 us/op
JsonBenchmark.testDecode:testDecode·p0.99 MinimalJson sample 17,280 us/op
JsonBenchmark.testDecode:testDecode·p0.999 MinimalJson sample 410,245 us/op
JsonBenchmark.testDecode:testDecode·p0.9999 MinimalJson sample 4738,070 us/op
JsonBenchmark.testDecode:testDecode·p1.00 MinimalJson sample 90177,536 us/op
JsonBenchmark.testDecode NanoJson sample 925259 14,588 ± 0,234 us/op
JsonBenchmark.testDecode:testDecode·p0.00 NanoJson sample 7,344 us/op
JsonBenchmark.testDecode:testDecode·p0.50 NanoJson sample 10,256 us/op
JsonBenchmark.testDecode:testDecode·p0.90 NanoJson sample 15,440 us/op
JsonBenchmark.testDecode:testDecode·p0.95 NanoJson sample 21,440 us/op
JsonBenchmark.testDecode:testDecode·p0.99 NanoJson sample 59,802 us/op
JsonBenchmark.testDecode:testDecode·p0.999 NanoJson sample 663,552 us/op
JsonBenchmark.testDecode:testDecode·p0.9999 NanoJson sample 2514,731 us/op
JsonBenchmark.testDecode:testDecode·p1.00 NanoJson sample 20054,016 us/op
JsonBenchmark.testDecode Plain sample 1526500 2,457 ± 0,138 us/op
JsonBenchmark.testDecode:testDecode·p0.00 Plain sample 1,480 us/op
JsonBenchmark.testDecode:testDecode·p0.50 Plain sample 1,572 us/op
JsonBenchmark.testDecode:testDecode·p0.90 Plain sample 2,788 us/op
JsonBenchmark.testDecode:testDecode·p0.95 Plain sample 3,184 us/op
JsonBenchmark.testDecode:testDecode·p0.99 Plain sample 6,416 us/op
JsonBenchmark.testDecode:testDecode·p0.999 Plain sample 98,304 us/op
JsonBenchmark.testDecode:testDecode·p0.9999 Plain sample 891,597 us/op
JsonBenchmark.testDecode:testDecode·p1.00 Plain sample 54001,664 us/op
JsonBenchmark.testDecode None sample 1371906 0,821 ± 0,246 us/op
JsonBenchmark.testDecode:testDecode·p0.00 None sample 0,139 us/op
JsonBenchmark.testDecode:testDecode·p0.50 None sample 0,163 us/op
JsonBenchmark.testDecode:testDecode·p0.90 None sample 0,297 us/op
JsonBenchmark.testDecode:testDecode·p0.95 None sample 0,311 us/op
JsonBenchmark.testDecode:testDecode·p0.99 None sample 0,554 us/op
JsonBenchmark.testDecode:testDecode·p0.999 None sample 19,744 us/op
JsonBenchmark.testDecode:testDecode·p0.9999 None sample 801,597 us/op
JsonBenchmark.testDecode:testDecode·p1.00 None sample 38010,880 us/op
JsonBenchmark.testEncode Jackson sample 1411865 4,719 ± 0,049 us/op
JsonBenchmark.testEncode:testEncode·p0.00 Jackson sample 3,468 us/op
JsonBenchmark.testEncode:testEncode·p0.50 Jackson sample 3,584 us/op
JsonBenchmark.testEncode:testEncode·p0.90 Jackson sample 5,936 us/op
JsonBenchmark.testEncode:testEncode·p0.95 Jackson sample 7,056 us/op
JsonBenchmark.testEncode:testEncode·p0.99 Jackson sample 15,568 us/op
JsonBenchmark.testEncode:testEncode·p0.999 Jackson sample 128,913 us/op
JsonBenchmark.testEncode:testEncode·p0.9999 Jackson sample 849,538 us/op
JsonBenchmark.testEncode:testEncode·p1.00 Jackson sample 4415,488 us/op
JsonBenchmark.testEncode MinimalJson sample 1263047 10,270 ± 0,091 us/op
JsonBenchmark.testEncode:testEncode·p0.00 MinimalJson sample 7,064 us/op
JsonBenchmark.testEncode:testEncode·p0.50 MinimalJson sample 7,632 us/op
JsonBenchmark.testEncode:testEncode·p0.90 MinimalJson sample 13,536 us/op
JsonBenchmark.testEncode:testEncode·p0.95 MinimalJson sample 15,920 us/op
JsonBenchmark.testEncode:testEncode·p0.99 MinimalJson sample 40,448 us/op
JsonBenchmark.testEncode:testEncode·p0.999 MinimalJson sample 265,728 us/op
JsonBenchmark.testEncode:testEncode·p0.9999 MinimalJson sample 1106,095 us/op
JsonBenchmark.testEncode:testEncode·p1.00 MinimalJson sample 10371,072 us/op
JsonBenchmark.testEncode NanoJson sample 1378816 9,582 ± 0,330 us/op
JsonBenchmark.testEncode:testEncode·p0.00 NanoJson sample 7,032 us/op
JsonBenchmark.testEncode:testEncode·p0.50 NanoJson sample 7,856 us/op
JsonBenchmark.testEncode:testEncode·p0.90 NanoJson sample 9,344 us/op
JsonBenchmark.testEncode:testEncode·p0.95 NanoJson sample 12,928 us/op
JsonBenchmark.testEncode:testEncode·p0.99 NanoJson sample 23,232 us/op
JsonBenchmark.testEncode:testEncode·p0.999 NanoJson sample 164,911 us/op
JsonBenchmark.testEncode:testEncode·p0.9999 NanoJson sample 1223,867 us/op
JsonBenchmark.testEncode:testEncode·p1.00 NanoJson sample 116391,936 us/op
JsonBenchmark.testEncode Plain sample 1085856 6,361 ± 0,711 us/op
JsonBenchmark.testEncode:testEncode·p0.00 Plain sample 4,512 us/op
JsonBenchmark.testEncode:testEncode·p0.50 Plain sample 4,848 us/op
JsonBenchmark.testEncode:testEncode·p0.90 Plain sample 5,688 us/op
JsonBenchmark.testEncode:testEncode·p0.95 Plain sample 8,192 us/op
JsonBenchmark.testEncode:testEncode·p0.99 Plain sample 15,303 us/op
JsonBenchmark.testEncode:testEncode·p0.999 Plain sample 122,752 us/op
JsonBenchmark.testEncode:testEncode·p0.9999 Plain sample 1221,105 us/op
JsonBenchmark.testEncode:testEncode·p1.00 Plain sample 228589,568 us/op
JsonBenchmark.testEncode None sample 1201647 0,136 ± 0,016 us/op
JsonBenchmark.testEncode:testEncode·p0.00 None sample 0,027 us/op
JsonBenchmark.testEncode:testEncode·p0.50 None sample 0,073 us/op
JsonBenchmark.testEncode:testEncode·p0.90 None sample 0,105 us/op
JsonBenchmark.testEncode:testEncode·p0.95 None sample 0,133 us/op
JsonBenchmark.testEncode:testEncode·p0.99 None sample 0,288 us/op
JsonBenchmark.testEncode:testEncode·p0.999 None sample 9,904 us/op
JsonBenchmark.testEncode:testEncode·p0.9999 None sample 61,941 us/op
JsonBenchmark.testEncode:testEncode·p1.00 None sample 2854,912 us/op
JsonBenchmark.testDecode Jackson ss 118276,270 us/op
JsonBenchmark.testDecode MinimalJson ss 8362,015 us/op
JsonBenchmark.testDecode NanoJson ss 21499,819 us/op
JsonBenchmark.testDecode Plain ss 3517,849 us/op
JsonBenchmark.testDecode None ss 100,837 us/op
JsonBenchmark.testEncode Jackson ss 106437,109 us/op
JsonBenchmark.testEncode MinimalJson ss 14163,948 us/op
JsonBenchmark.testEncode NanoJson ss 20346,964 us/op
JsonBenchmark.testEncode Plain ss 1746,693 us/op
JsonBenchmark.testEncode None ss 40,539 us/op
Jackson and Plain mappers are comparable by average speed and throughput. So, can we maybe completely replace Jackson with own custom JsonReader/PlainObjectMapper implementation? WDYT? @robotdan
Code is here - https://github.com/volyx/fusionauth-jwt/tree/minimal-json
PlainObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/PlainObjectMapper.java
NanoJsonObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/NanoJsonObjectMapper.java
MinimalJsonObjectMapper - https://github.com/volyx/fusionauth-jwt/blob/minimal-json/src/main/java/io/fusionauth/jwt/json/MinimalJsonObjectMapper.java
Wow, thanks for the excellent and thorough suggestion @volyx
I'll take a look at you're proposal and follow up with comments.
Can this be pluggable (i.e. done via an interface)? While I do understand that not all projects use Jackson, it's de facto standard for anyone using Spring Boot (and possibly other popular web frameworks). Meaning that it wouldn't really add any extra dependencies. Pluggability may also help with writing simpler tests (possibly allow to test the validation and JSON parsing logic separately) and allow further Jackson mapper customization via the usual module/mixin/options route.
This will not be as easy as throwing away Jackson completely (you may have to do Mixins for serialization/deserialization to keep using Jackson annotations and do default ObjectMapper configuration), but avoids depending on yet-another-custom-json-parser that's only used here.
it's de facto standard for anyone using Spring Boot (and possibly other popular web frameworks).
- For spring projects common way of integration is separate module for example
spring-fusionauth-jwt. So if we suppose that it is spring project we can manually configure JSON library. - For other frameworks it is also possible to write custom integration library -
${other_popular_web_framework}-fusionauth-jwt.
This will not be as easy as throwing away Jackson completely
Example is here - PlainObjectMapper. It is as fast as Jackson and uses just standard Java APIs.
I don't see any reason to support custom JSON serialization, if I can do encode/decode internally without any dependencies. PlainObjectMapper is fully compatible with ObjectMapper for all fusion-auth classes JWT, Header, Map, JSONWebKey. You can check it with:
// will encode with PlainObjectMapper, decode with ObjectMapper
./mvnw test -Dplain.json.encode=true
// will decode with PlainObjectMapper, encode with ObjectMapper
./mvnw test -Dplain.json.decode=true
The same approach of embedding JSON serialization/deserialization code inside library is used in jose4j - https://bitbucket.org/b_c/jose4j/src/master/src/main/java/org/jose4j/json/
First of all I don't think speed is the main reason anyone uses Jackson (or Spring for that matter). Jackson and Gson provide a wide range of serialization/deserialization customization options, are well established, with decent documentation and have plenty integrations with other libraries.
As a developer I don't want to write glue logic just to interface with the embedded serializer of a yet-another-jwt-library with my Jackson object model when I have the option of not doing so if I use a library that natively supports Jackson, even if it's a little slower/bulkier/...
I think I may even lose any gained performance doing DTO-to-claim mapping vs configuring the serializer to write claims the way I need.
I'd like to emphasize that I'm not against having a custom-very-fast-json-library even as the default option, but please don't make it hard not using it.
@virtual-machinist I'm fully agree with you point. But what if you don't even need to think about the internal serialization/deserialization logic? PlainObjectMapper are going to be fully embedded into code, and will become an internal API. So you don't need to think about internals of JWT library.
Speed benchmarks are presented there only to show that the performance will not get worth after the replacement.
Sorry, @volyx , I cannot imagine a case when I'd like to add a private claim of my own object model to the token and not think about the serialization details. You're basically saying I won't have to worry about I/O because I won't have the option. 😉
I think this discussion is getting nowhere. I am for an agnostic model with first-class parser/generator access to JSON claims and header and I'm against hand-rolled JSON processing with the necessity of writing glue logic from my claims model. Especially if there's no obvious overhead of having Jackson, cause it's together with Spring.
If the model is truly agnostic, there's no problem writing a PlainObjectMapper support for it (or Gson, or json-smart, or JSR 374, or something else).
@robotdan , I'm sorry if this has already been asked - why public fields and not getters/setters?