Incremental JSON generation

[phoe]

https://www.reddit.com/r/Common_Lisp/comments/rp5lik/what_was_your_favorite_common_lisp_release/hq7obiv/

Yason supports splitting up the generation of a json into a bunch of incremental parts, something like this:

(yason:with-object () (yason:encode-object-element key value))

This is pretty useful when you’re using generic functions because a PROGN method combination will be able to generate JSON output for all the fields of a class. It’s also useful when you’re generating a lot of JSON, because you don’t have to create a single data structure with all the data.

Just double-checking - does jzon have anything like that right now?

[Zulu-Inuoe]

@phoe No, I was not planning on supporting this yet.

I do totally get the use-cases, but that's a v2 sorta feature. For initial goals I wanted to be able to DWIM read/write JSON, without having to learn a big API.

From a design perspective, I would make a protocol based around a clone of JsonTextWriter. alongside with its counterpart - JsonTextReader, and represent the reader/writer as proper objects rather than hiding them behind dynamic bindings.

I'm actually very surprised to learn YASON does not have a streaming reader. The prospect of overtaking it in features is definitely tempting me to skirt my Real Job and irresponsibly write CL.

[Zulu-Inuoe]

I'm toying around with the writer side of this. The interface I'm planning on looks like the following:

(defclass point () 
  ((x :initform 0)
   (y :initform 0)))

(-> (jzon:make-json-writer :stream t)
  (jzon:write-value 42)
  (jzon:begin-object)
  (jzon:write-key "foo")
  (jzon:write-value 42)
  (jzon:write-key "a point")
  (jzon:write-value (make-instance 'point))
  (jzon:end-object)
  (jzon:write-value #(1 2 3 4))
  (jzon:begin-array)
  (jzon:write-value nil)
  (jzon:write-value t)
  (jzon:end-array))

which would produce output like:

42
{
    "foo": 42,
    "a point": {
        "x": 0,
        "y": 0
    }
}
[1, 2, 3, 4]
[false, true]

If anyone has any thoughts on that, let me know. Naturally I can export helper functions and macros for:

• writing key + value in one call
• yason-style with-object and with-array which automatically close the object ?
• others?

I don't want to have a huge API. But I also don't want everyone to write the same utility functions everywhere.

And notably as a difference from yason - I'd rather offer a programmatic API rather than hiding the internals with a macro. This allows you to keep the json writer in a variable, return it from a fn, etc.

[phoe]

LGTM after a brief glance, but I'm no JSON expert.

[fiddlerwoaroof]

I like the design of YASON:ENCODE-OBJECT/YASON:ENCODE-OBJECT-SLOTS here.

To make a class JSON-serializable, all you have to do is usually defmethod YASON:ENCODE-OBJECT-SLOTS like:

(fw.lu:defclass+ plaintext-message ()
  ((body :initarg :body :reader body)))

(defmethod yason:encode-slots progn ((object plaintext-message))
  (yason:encode-object-element "body" (body object)))

MATRIX> (yason:with-output (*standard-output*)
          (yason:encode-object (plaintext-message "testing")))
#| stdout => {"body":"testing"} |#
#| return => #<PLAINTEXT-MESSAGE {700559EE13}> |#

Incidentally, I think this is one of the first really compelling use-cases I've seen for the PROGN method combination :).

[Zulu-Inuoe]

Thanks for the suggestion @fiddlerwoaroof
Do you have any thoughts on using a progn method combination over calling call-next-method?
eg instead of:

(defmethod yason:encode-slots progn ((object plaintext-message))
  (yason:encode-object-element "body" (body object)))

you could just have encode-slots use standard method combination and:

(defmethod yason:encode-slots ((object plaintext-message))
  (call-next-method)
  (yason:encode-object-element "body" (body object)))

The reason I ask is because I much prefer the latter. My reasoning being that in using non-standard method combinations, as somebody writing a method I need to

1. Remember that encode-slots uses progn, so that I remember to type it out
2. Remember the semantics specific about progn
3. If I accidentally defined a method using the wrong combination, I need to go and explicitly remove-method because for some reason it's only a warning to specialize with the wrong combination, but an error when called.

Not a huge deal, but for the advantages of typing out progn vs (call-next-method) it just doesn't seem worth it to me.
Besides, with a standard method combination there's more flexibility in that you can add :before and :after when necessary. Or even 'take over' the primary method altogether, if you need.

[sabracrolleton]

I updated my review of json libraries at https://sabracrolleton.github.io/json-review.html and mentioned that this was being worked on. (I also said many nice things about this library)

[Zulu-Inuoe]

Thank you so much @sabracrolleton ! Your feedback is greatly appreciated there.
I'm booked the next couple of weeks, but I plan on addressing some of the issues after that.

Namely 1. incremental JSON generation and 2. pathname handling.

And maybe I'll finally get around to deciding what to do with #7 , which should add decoding goodies.
Given your documenting efforts, I'll make sure to tag you when any of those change.

[fiddlerwoaroof]

@Zulu-Inuoe this is sort of the ideal use-case for the progn method combination. Relying on (call-next-method) has the problem that it makes it a lot easier for a user to make a mistake that results in bad serialization.

[Zulu-Inuoe]

So, @fiddlerwoaroof I did push out a writer API. Still making a couple of additions but I've decided on requiring people call-next-method. While I understand that the semantics of progn make a sense here, I think overall it's not compelling enogh.

Pros:

1. typing progn is shorter than (call-next-method)
2. It's in the spirit of the combination so it makes sense semantically
   Cons:
3. Method combinations (other than default) are not commonly used and users are more likely to not know to use them
4. If you forget to place the method combination in, the method is still attached to the GF. Even if you fix and recompile, you must remove the erroneously attached method from the GF which is annoying for me who knows how to do that via sly. If you are not familiar with this, you'll probably restart your whole image.
5. It forces you to inherit other specializations, and it forces a more specialized method to run after its less specialized one. With (call-next-method) you can decide the order of your fields vs your 'parents', or if you even want to run the less-specialized code at all

Hopefully that all makes sense. For now I'm closing this in prep for a v1.

Thanks for the feedback!