aura-nw/horoscope

Horoscope crawls data from Cosmos chains, stores and indexes everything into a MongoDB for easy access from clients.

Horoscope

Horoscope is an indexing service for Cosmos-based blockchain. It crawls data from the blockchain and index it into mongoDB. Based on the data, it can provide search functionality instead of querying data from LCD or RPC directly.

Currently, it supports network built by Cosmos SDK v0.45.1 or later. Supporting network:

• Aura Network
• Osmosis
• Cosmos Hub
• Evmos

Horoscope includes 2 main components:

• Crawler: crawl data from the blockchain and index it into mongoDB
• Backend API: provide search functionality through RESTful API

Main flow crawl

The crawl system divides the data into 2 parts:

• Block data: the data in the block (block information, transaction, evidence)
• State data: the data change in each state of the block (distribution, reward, delegation, community pool)

Most of the services are small Node applications written in Typescript. The Node services are built using Moleculerjs framework. With Crawler, we use Bull to manage the queue of crawling. High scalability is required for crawling, we use Redis Stream to implement Consumer Group pattern.

Architect backend api

List services (currently)

• api-gateway: API Gateway services for backend API, query data from DB

  • network: get community pool, pool and inflation
  • block: get block information
  • transaction: get transaction information
  • proposal: get proposal information
  • validator: get validator information
  • account-info: get information of a wallet onchain
  • asset: manage, register asset

• crawl-block: get block from network and store it in redis stream

• handle-block: get block from redis stream, then get transaction, evidence in block and put them to redis stream equivalent

• crawl-transaction: get transaction by hash from network and store it in redis stream

• handle-transaction: get transaction from redis stream then handle

• crawl-param: param from module Bank, Distribution, Governance, IBC Transfer, Mint, Staking, Slashing

• crawl-proposal, crawl-tally-proposal: proposal and tally result (if proposal in voting period)

• crawl-distribution: get data community pool

• crawl-staking-validator, crawl-staking-pool: get data pool, information about validator

• crawl-mint: get inflation

• handle-address: get information of a wallet onchain

  • crawl-account-auth: get auth information
  • crawl-account-balances: get balance of all coins
  • crawl-account-delegations: get delegation information
  • crawl-account-redelegations: get redelegation information
  • crawl-account-unbonds: get unbond information
  • crawl-account-spendable-balances: get spendable balance of all coins

• graphql-service: provide GraphQL service.Read more.

Install requirements

• Redis: docker run --name redis -p 6379:6379 -d redis
• MongoDB: docker run --name mongo -p 27017:27017 -d -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=xxx mongo
• NPM: Horoscope currently use private packet aurajs to decode tx from Aura Network. To install aurajs, you must create a Personal Access Token has read package, put it to (xxx_xxx) on .npmrc file in root source code

@aura-nw:registry=https://npm.pkg.github.com/aura-nw
//npm.pkg.github.com/:_authToken=xxx_xxx

How to run

# create file env
cp .env.example .env
# then replace env field with your own value, point SERVICE to service you want to run

# run with moleculer cli
npm run dev

Configuration

Config file is located at .env, all service used moleculerJS Config, another config is custom:

environment variable	description
CHAIN_ID	Chain Id of Network want to be crawled (example: serenity-testnet-001).
ENABLE_LOADBALANCER	Client Api used when get data from network, if yes use Resilent, if no use Axios . default: true

NPM scripts

• npm run dev: Start development mode (load all services locally with hot-reload & REPL)
• npm run start: Start production mode (set SERVICES env variable to load certain services)
• npm run cli: Start a CLI and connect to production. Don't forget to set production namespace with --ns argument in script
• npm run lint: Run ESLint
• npm run ci: Run continuous test mode with watching
• npm test: Run tests & generate coverage report
• npm run dc:up: Start the stack with Docker Compose
• npm run dc:down: Stop the stack with Docker Compose

Contributing

See CONTRIBUTING.md for details on how to contribute.