PayloadFormat in IPersistedRawEvent and IUnpersistedRawEvent should maybe be a string to allow specification of mime contentType
Opened this issue · 4 comments
I'd prefer to have either PayloadFormat be a string or add a ContentType property to the events interfaces. When saving to a backing store, such as a raw NoSQL store it is nice to know the content type and it would be nice to be able to specify it as application/json for example.
@nvivo What are your thoughts on this?
I'm not sure about having the payload format as a string. When I added the payload format, the idea was that Even could encrypt some events using different keys while leaving regular events unencrypted. Or we could upgrade change BSON to protobuf in some version and still reading the old events. But it wasn't designed to have many values in the same database. It would probably be something like:
- BSON (current default)
- Protobuf
- Encrypted version of BSON or protobuf
- Some user defined formats
As a general guideline, I believe the payload format should be internal to the driver. It should be used to allow the driver to encode and decode the payload, and nothing else. As long as it is fast and compact, it should work. Since this should be simple and fast for the driver, an int works better than a string.
But I don't see a problem with a driver storing the data as json either. Since eerything will depend on the table format, you could have a driver that stores json and don't even use the payload format. I just don't think this would require having a content type in each event, and this information will probably be the same in the entire database.
Now, it seems to me the only benefit of json storage would be for database inspection, which is something I thought about. But after giving some it thought, browsing the data in a SQL query result is not that helpful. Most interfaces will show only the beginning of the event text which is of limited use.
For that I imagined having an external app that could use the existing drivers to inspect and present the data to the user in json format (similar to geteventstore admin panel). It could be an embedded app using the new aspnet core or nancy, allowing you to browse the global and individual streams, and possibly allowing to extract some useful information like the list of event names and a merged format of all events for a given event type.
Actually in the case where the target is a NoSQL database that handles JSON natively by default: i.e. couchbase, couchbase-lite,ElasticSearch, EventStore, Postgres, etc. having JSON messages actually enables indexing and/or map/reduce on the payload.
Also the problem I'm seeing is that just looking at the Event it wasn't possible to figure out what was in the byte arrays. Your list above helps, but having numbers that require a lookup table (i.e. what does 1 map to), becomes a bit of a pain. That being with custom store implementations I can store the extra data I want as long as I can derive it from the int. But my main issue is that Even doesn't currently even give you an enum or a listing of the different payload formats.
Maybe I thought about this field too early trying a general solution, and should have waited for a real use case.
Now, I don't think it would help to try to index the event payload directly. If you think about it, the events always need to be read by sequence order, you never search the store for anything specific and you never retrieve part of an event. So, all indexing you could possibly need is defined by the driver when you create the table. Indexing and restructuring the payload is a job for projections.
I see your pain. It would be handy to use the existing database gui to understand what is being stored. But in the end, I think there will never be enough metadata to do anything useful. Adding more metadata to help you read the tables manually will only bloat the tables while giving very little back.
What we're missing is tooling that is good enough for daily use, but that is much harder to do.