A Substreams sink to pipe data from a Substreams endpoint into a key-value store and serve queries through either:
- Connect-Web protocol (gRPC-compatible) via the
GenericService - a User defined WASM query service
Learn about WasmEdge from its Quick Start Guide, or simply run the following to install.
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --version 0.11.2Get from the Releases tab, or from source:
go install -v github.com/streamingfast/substreams-sink-kv/cmd/substreams-sink-kv@latestThe substreams-sink-kv binary offers two mode of operations:
inject: Runs a Substreams and pipes the data into a key-value storeserve: Serves data through a query service:GenericServiceorWASMQueryService
You can run substreams-sink-kv solely in inject or serve mode or both.
Note To connect to Substreams you will need an authentication token, follow this guide to obtain one.
# Inject Mode
substreams-sink-kv inject mainnet.eth.streamingfast.io:443 <kv_dsn> <substreams_spkg_path> <kv_out>
# Serve Mode
substreams-sink-kv serve <kv_dsn> <substreams_spkg_path> --listen-addr=":9000"
# Inject and Serve Mode
substreams-sink-kv inject mainnet.eth.streamingfast.io:443 <kv_dsn> <substreams_spkg_path> <kv_out> --listen-addr=":9000"The Query Service is an API that allows you to consume data from your sinked key-value. There are 2 types of Query Services:
- Generic Service
- Wasm Query Service
the sink block of your Substreams manifest defines and configures which one to use
specVersion: v0.1.0
package:
name: "your_substreams_to_sink"
version: v0.0.1
...
sink:
module: kv_out
type: sf.substreams.sink.kv.v1.WASMQueryService
config:
wasmQueryModule: "@@./blockmeta_wasm_query/blockmeta_wasm_query.wasm"
grpcService: "eth.service.v1.Blockmeta"breaking down the sink block we get the following:
- module: The name of the module that will be used to sink the key-value store. The module should be of kind
mapwith an output type ofsf.substreams.sink.kv.v1.KVOperations - type: Support to types currently:
- config: a key-value structure that matches the attributes of the Proto object for the given
typeselected above
NOTE: the
@@notation will read the path and inject the content of the file in bytes, while the@notation will dump the file content in ascii
The Generic Query service is a Connect-Web protocol (gRPC-compatible). It exposes a browser and gRPC-compatible APIs. The API is defined in protobuf here.
You can find a detailed example with documentation here
The wasm query service is a user-defined gRPC API that is backed by WASM code, which has access to underlying key-value store.
You can find a detailed example with documentation here
Refer to the general StreamingFast contribution guide.
Bumping github.com/second-state/WasmEdge-go to a newer version requires matching WasmEdge version. For example github.com/second-state/WasmEdge-go@0.11.2 requires WasmEdge 0.11.2.
If a new version of WasmEdge, it requires also modifying the ./devel/docker/Dockerfile.goreleaser file so it pulls dependencies required for building using the correct version. Testing the build can be done using by first building the Docker image via ./devel/docker/push.sh and then running:
docker run --rm -it -v `pwd`:/work -w /work goreleaser-wasmedge:v1.20.3 release --snapshot --clean --skip-validateManual Golang compilation within the docker image can be done with:
docker run --rm -it -v `pwd`:/work -w /work --entrypoint bash goreleaser-wasmedge:v1.20.3
cd /work
# Adjust CC, CXX, C_INCLUDE_PATH, LIBRARY_PATH and GOOS/GOARCH according to .goreleaser.yaml file
CGO_ENABLED=1 CC=oa64-clang CXX=oa64-clang++ GOOS=darwin GOARCH=arm64 C_INCLUDE_PATH=/usr/local/osxcross/include/arm64 LIBRARY_PATH="/usr/local/osxcross/lib/arm64" go build -trimpath -mod=readonly -ldflags="-s -w" -o /work/substreams-sink-kv-cross-compiled ./cmd/substreams-sink-kv/You can inspect badger files written by this tool with:
go install github.com/streamingfast/kvdb/cmd/kvdb@develop