re-implement tx indexing
moul opened this issue · 2 comments
The tx indexing and query system was removed to attempt a more modular and minimal implementation.
The goal I have in mind is to simply log to file all EventSwitch events (serialized using amino:binary, using protobuf for speed), and then to make readers that can idempotent-ly index or filter these events for whatever purpose, including tx indexing, but also streaming of filtered events (from the RPC using websockets, say). These streaming readers would be independent processes from the full node, so there would be possibly no resource contention depending on the configuration. Since log files could be read by any process, the node wouldn't even have to know who are the listeners are.
We could possibly use an existing persistent pubsub system to implement this, but if we're going to use such an external dependency, it needs to be implemented in Go, be minimal and simple, and mature. Alternatively we should implement one from scratch if a suitable implementation cannot be found.
The log file could use pkgs/autofile, or a fork that doesn't automatically delete older chunk files, and we could leave it up to the user to run a background service that deletes older chunk files as needed.
I'm open to alternative solutions, but this seems to be the most conceptually simple solution. Compared to Tendermint1, instead of an asynchronous pubsub, Tendermint2 has the original synchronous simple event switch. Tendermint1 bakes in several pubsub listeners, but Tendermint2 would be more minimal and simply log all events to the filesystem.
Original issue: gnolang/gno#275
Can you please explain this part more? I'm not sure why we'd need a pubsub system in the first place
We could possibly use an existing persistent pubsub system to implement this, but if we're going to use such an external dependency, it needs to be implemented in Go, be minimal and simple, and mature. Alternatively we should implement one from scratch if a suitable implementation cannot be found.
Make sure that we can have a standalone collector feeding a pubsub system. And another standalone processor that consumes the pubsub.
PubSub could be a simple file, a database, not necessarily an advanced API.