Ensuring access to notarized metadata
There are several mechanisms we directly foresee being used for incentivization feedback loops based around the metadata measurements of entities in the filecoin ecosystem. For instance, reports of miner behavior can be used by reputation systems ranking miners, which then are used by clients to select higher quality miners near them. This data doesn’t make sense to directly embed within the filecoin chain for a few reasons: It is produced by independent entities, so the data itself does not need to meet the same ‘consensus’ bar as what we would expect in a global chain, and likewise aspects of reputation and measurements may have aspects of subjectivity. It is also expected that there is diversity of data and that experimentation is a good thing. However, there are nice properties of having this sort of metadata ecosystem more tightly linked to the chain that seem desirable to encourage, and this leads to the goals for the sidechain metadata service:
- Keep included metadata consistently available
- Provide light-weight, unbiased access to metadata
- Discourage historical revisionism.
Pando uses go-legs to synchronize IPLD data from providers.
We will develop an SDK for you to integrate with Pando in a more efficient way in the future (maybe a week).
For now, you have to initialize go-legs instance on their own and publish IPLD data to the topic /pando/v0.0.1
which Pando subscribes.
Pando accepts IPLD data from metadata providers with the following required IPLD children nodes:
Provider String
- provider's peer IDSignature Bytes
- signature of the whole IPLD data (excluding the signature node itself)
and a nullable node:
PreviousID nullable Link_Metadata
- a non null PreviousID will enable a recursive sync by Pando service. With this node, a full sync of linked IPLD data will be guaranteed.
Notice that, once PreviousID is absent when there is some previous metadata, Pando makes no guarantee about the state of metadata published, as the recursive sync won't happen in this case.
Make sure you correctly link your chained metadata using PreviousID node.
The provider integration is recommended to follow the steps below:
- Refer to Build Pando server and client to build the client CLI.
- Go to the
bin
directory of the repo, and register the provider instance in Pando using the client CLI:
./pando-client -a https://pando-api.kencloud.com provider register \
--peer-id PROVIDER_PEER_ID_STR \
--private-key PROVIDER_PRIVATE_KEY_STR
Note: we will not upload any provider's private key to any network. CLI uses private key only for signing registration information, as Pando will verify that the provider peer ID matches the public key to prevent illegal provider from falsely using the peer ID of other providers.
- Create a provider instance with Pando SDK, then connect to Pando, initialize a metadata instance and append new metadata if you need.
- Push the latest metadata instance.
Check out these examples for more details.
To fetch metadata/snapshot status and content via GraphQL API, click here to get a try and dig more.
TBD
Git clone this repo, and run make
, that's all. The binaries will be built at bin
.
Initialize a config file first (default path is ~/.pando/config.yaml):
./pando-server init
Since Pando uses Estuary to upload the provider metadata to the Filecoin network, you need to set the API Key of Estuary in the configuration file:
Backup:
EstuaryGateway: https://api.estuary.tech
ShuttleGateway: https://shuttle-4.estuary.tech
ApiKey: "YOUR ESTUARY API KEY"
Now you can start a pando server:
./pando-server daemon
or start with custom listen address (in multiaddress format):
./pando-server daemon \
--http-listen-addr /ip4/0.0.0.0/tcp/8080 \
--graphql-listen-addr /ipv/0.0.0.0/tcp/8081
Pando supports three types of config resource, and it will look up config value in this order:
- CLI flag
- environment variable
- config file (yaml file)
ConfigFile (string), specify config file name(only name, not path, its parent path is PandoRoot):
- -f, --config-file
- PD_CONFIGFILE
- /
PandoRoot (string), specify pando root directory for config and data persistence:
- -r, --pando-root
- PD_PANDOROOT
- PandoRoot
PandoDisableSpeedTest (bool), disable internet speed test when init config:
- -p, --disable-speedtest
- PD_DISABLESPEEDTEST
- DisableSpeedTest
LogLevel (string, support DEBUG, INFO, WARN, ERROR, DPANIC, PANIC, FATAL), Pando core module log level
- --log-level
- PD_LOGLEVEL
- LogLevel
Identity.PeerID (string), peer ID of Pando
- /
- PD_IDENTITY_PEERID
- Identity.PeerID
Identity.PrivateKey (string), private key of Pando
- /
- PD_IDENTITY_PRIVATEKEY
- Identity.PrivateKey
ServerAddress.HttpAPIListenAddress (string, example: /ip4/127.0.0.1/tcp/9000), http api listen address in multi-address format
- --http-listen-addr
- PD_SERVERADDRESS_HTTPAPILISTENADDRESS
- ServerAddress.HttpAPIListenAddress
ServerAddress.GraphqlListenAddress (string), graphql api listen address in multi-address format
- --graphql-listen-addr
- PD_SERVERADDRESS_GRAPHQLLISTENADDRESS
- ServerAddress.GraphqlListenAddress
ServerAddress.P2PAddress (string), p2p host listen address in multi-address format
- --p2p-address
- PD_SERVERADDRESS_P2PADDRESS
- ServerAddress.P2PAddress
ServerAddress.ExternalIP (string), public IP of Pando to serve API
- --external-ip
- PD_SERVERADDRESS_EXTERNALIP
- ServerAddress.ExternalIP
DataStore.Type (string), datastore type, support "levelds" only for now
- --datastore-type
- PD_DATASTORE_TYPE
- DataStore.Type
DataStore.Dir (sting), directory stores all the datastore data, its parent dir is PandoRoot
- --datastore-dir
- PD_DATASOTRE_DIR
- DataStore.Dir
Discovery.LotusGateway (string), lotus gateway address
- --discovery-lotus-gateway
- PD_DISCOVERY_LOTUSGATEWAY
- Discovery.LotusGateway
Discovery.Policy.Allow (bool), allow all origin source host
- --discovery-policy-allow
- PD_DISCOVERY_POLICY_ALLOW
- Discovery.Policy.Allow
Discovery.Policy.Trust (bool), enable discovery white-list
- --discovery-policy-trust
- PD_DISCOVERY_POLICY_TRUST
- Discovery.Policy.Trust
Discovery.PollInterval (string, example: 24h0m0s), discovery poll interval
- /
- PD_DISCOVERY_POLLINTERVAL
- Discovery.PollInterval
Discovery.RediscoverWait (string), re-discovery wait time duration
- /
- PD_DISCOVERY_REDISCOVERYWAIT
- Discovery.RediscoverWait
Discovery.Timeout (stirng), discovery timeout
- /
- PD_DISCOVERY_TIMEOUT
- Discovery.Timeout
AccountLevel.Threshold (string slice), balance bound of provider, see details at rate-limit doc
- --account-level
- PD_ACCOUNTLEVEL_THRESHOLD
- AccountLevel.Threshold
RateLimit.Bandwidth (float64), bandwidth of Pando service environment, this value will be filled when you run
./pando-server init
- --ratelimit-bandwidth
- PD_RATELIMIT_BANDWIDTH
- RateLimit.Bandwidth
RateLimit.SingleDAGSize (float64), estimate size of single DAG structure metadata
- --ratelimit-single-dag-size
- PD_RATELIMIT_SINGLEDAGSIZE
- RateLimit.SingleDAGSize
Backup.EstuaryGateway (string), estuary gateway address
- --backup-estuary-gateway
- PD_BACKUP_ESTUARYGATEWAY
- Backup.EstuaryGateway
Backup.ShuttleGateway (string), estuary shuttle gateway address
- --backup-shuttle-dateway
- PD_BACKUP_SHUTTLEGATEWAY
- Backup.ShuttleGateway
Backup.APIKey (string), estuary api key
- --backup-apikey
- PD_BACKUP_APIKEY
- Backup.APIKey
See Pando API document for more details. Or check Pando API Specification
See graphql api documentation to query metadata/snapshot status and details.
Pando integrates redoc style documentation, visit http://127.0.0.1:9000/swagger/doc
for API documentation,
or check http://127.0.0.1:9000/swagger/specs
for swagger 2.0 specification.
Check if Pando is alive
./pando-client -a http://127.0.0.1:9000 pando info
{
"code": 200,
"message": "alive",
"Data": null
}
Show information of Pando server
./pando-client -a http://127.0.0.1:9000 pando info
{
"code": 200,
"message": "ok",
"Data": {
"PeerID": "12D3KooWKw5hu5QcbbFuokt3NrYe7gak5kKHzt8h1FJNqByHQ157",
"APIAddresses": {
"GRAPHQL_API": "/ip4/1.1.1.1/tcp/8002",
"GRAPHSYNC_API": "/ip4/1.1.1.1/tcp/8003",
"HTTP_API": "/ip4/1.1.1.1/tcp/8001"
}
}
}
Let Pando subscribe a topic with provider to start metadata synchronization
./pando-client -a http://127.0.0.1:9000 pando subscribe --provider-peerid 12D3KooWSS3sEujyAXB9SWUvVtQZmxH6vTi9NitqaaRQoUjeEk3M
{
"code": 200,
"message": "subscribe success",
"Data": null
}
Provider should be registered before using Pando service
./pando-client -a http://127.0.0.1:9000 provider register \
--peer-id 12D3KooWBckWLKiYoUX4k3HTrbrSe4DD5SPNTKgP6vKTva1NaRkJ \
--private-key CAESQLypOCKYR7HGwVl4ngNhEqMZ7opchNOUA4Qc1QDpxsARGr2pWUgkXFXKU27TgzIHXqw0tXaUVx2GIbUuLitq22c= \
--addresses /ip4/127.0.0.1/tcp/9999
{
"code": 200,
"message": "register success",
"Data": null
}
to generate an envelop-data only, run:
./pando-client -a http://127.0.0.1:9000 provider register \
--only-envelop \
--peer-id 12D3KooWBckWLKiYoUX4k3HTrbrSe4DD5SPNTKgP6vKTva1NaRkJ \
--private-key CAESQLypOCKYR7HGwVl4ngNhEqMZ7opchNOUA4Qc1QDpxsARGr2pWUgkXFXKU27TgzIHXqw0tXaUVx2GIbUuLitq22c= \
--addresses /ip4/127.0.0.1/tcp/9999
envelop data saved at ./envelop.data
List all cids of metadata snapshots
./pando-client -a http://127.0.0.1:9000 metadata list
{
"code": 200,
"message": "OK",
"Data": [
{
"/": "bafy2bzacea6qqju247jpt3udlzrafiz2zbe6tdgdxv6sm22ysujnynvz2c3uk"
}
]
}
lookup and show snapshot information by its cid
./pando-client -a http://127.0.0.1:9000 \
metadata snapshot \
--snapshot-cid bafy2bzacea6qqju247jpt3udlzrafiz2zbe6tdgdxv6sm22ysujnynvz2c3uk
{
"code": 200,
"message": "metadataSnapshot found",
"Data": {
"CreateTime": 1640992855488796000,
"ExtraInfo": {
"MultiAddrs": "/ip4/127.0.0.1/tcp/9002 ",
"PeerID": "12D3KooWDdL1G1osCHaYb27dVFeixDVoTtGJ1FpSdpBYTMcCM6dC"
},
"Height": 0,
"PrevSnapShot": "",
"Update": {
"12D3KooWSS3sEujyAXB9SWUvVtQZmxH6vTi9NitqaaRQoUjeEk3M": {
"Cidlist": [
{
"/": "QmPF6SbfekNZBwpP2zH9XbzM84jY44jBNuZUEcETq7tXdX"
}
],
"LastCommitHeight": 0
}
}
}
}
or by snapshot's height
./pando-client -a http://127.0.0.1:9000 \
metadata snapshot \
--snapshot-height 0
{
"code": 200,
"message": "metadataSnapshot found",
"Data": {
"CreateTime": 1640992855488796000,
"ExtraInfo": {
"MultiAddrs": "/ip4/127.0.0.1/tcp/9002 ",
"PeerID": "12D3KooWDdL1G1osCHaYb27dVFeixDVoTtGJ1FpSdpBYTMcCM6dC"
},
"Height": 0,
"PrevSnapShot": "",
"Update": {
"12D3KooWSS3sEujyAXB9SWUvVtQZmxH6vTi9NitqaaRQoUjeEk3M": {
"Cidlist": [
{
"/": "QmPF6SbfekNZBwpP2zH9XbzM84jY44jBNuZUEcETq7tXdX"
}
],
"LastCommitHeight": 0
}
}
}
}