/finder-spark

finder-spark

Primary LanguageScalaMIT LicenseMIT

klaytn-spark

This repository contains the Spark application code for the Klaytn blockchain data processing.

For the overall architecture of the finder, please refer to the Main Repo.


Overview

This README contains

  • Build
  • Test / Execution
  • Deployment
  • Monitoring

For the details of the application, please refer to the following READMEs.

--

Build

sbt klaytn-spark/clean klaytn-spark/assembly

Test / Execution

  • Using Spark: configure /bin/local_submit.sh and run in the local
  • Without using Spark: Use sbt test
  • Running tests locally for the Main program execution: Implement in the test directory with a name like -TestMain

You can run the test with the following command.

sbt test

If you want to run a specific test, you can run it with the following command.

sbt testOnly io.klaytn.contract.lib.ERC20MetadataReaderTestMain


Deployment

Prerequisites

  1. Login AWS CLI / aws-vault
    • aws-vault exec YOUR_AWS_PROFILE

  2. Load the target cluster environment
    • source conf/prod.env.sh

    • source conf/dev.env.sh

  3. Copy the EMR ClusterId
    • aws emr list-clusters --active | jq -r '.[] | sort_by(.Name)' | jq -r '.[] | [.Id,.Status.State,.Name] | @tsv' | awk 'BEGIN { print "——————————————— —————————————— ——————————————————————————"; print " ClusterId Status ClusterName "; print "——————————————— —————————————— ——————————————————————————" }; { printf "\033[1;32m%-15s\033[0m \033[1;36m%14s\033[0m %s\n", $1, $2, $3 };'


Run the release.sh script

  • cluster_id: EMR ClusterId to deploy
  • canonical_name: Focus on the code you want to deploy in the Intellij project tree view, then copy it with "cmd + option + shift + c".
    • Example: io.klaytn.apps.block.BlockToDBStreaming
  • chain: Enter the chain name corresponding to Chain.scala.
  • --build: If this parameter exists, it will create a jar using sbt clean assembly. If not specified, the previously created jar will be deployed.

./release.sh {cluster_id} {canonical_name} {chain} [--build]

Example

  • ./release.sh finder-streaming-prod io.klaytn.apps.block.BlockToDBStreaming cypress --build

Notes

Before the new deployment, Please delete this file to shutdown the previous job.

if-want-stop-delete-this-file.prod-{baobab | cypress}.txt

You can find the file in the following path.

s3://klaytn-prod-spark/jobs/{canonical_name}.


Monitoring


Deploy Batch Examples

./release.sh finder-batch-prod io.klaytn.apps.refining.RestoreKafkaLogBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.refining.RestoreKafkaLogBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.BlockMakeData cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.BlockMakeData baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.InternalTXMakeData cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.InternalTXMakeData baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.AccountMakeData cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.AccountMakeData baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.BlockLoadData cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.BlockLoadData baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.InternalTXLoadData cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.InternalTXLoadData baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.AccountLoadData cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.bulkload.AccountLoadData baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.approveBurn.ApproveBurnBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.approveBurn.ApproveBurnBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.transfer.TransferBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.transfer.TransferBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.adhoc.block.LoadBlockRewardBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.adhoc.block.LoadBlockRewardBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.holder.HolderBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.holder.HolderBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.holder.MinusHolderBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.holder.MinusHolderBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.count.CounterBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.count.CounterBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.es.TransactionESRecoveryBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.es.TransactionESRecoveryBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.es.AccountESRecoveryBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.es.AccountESRecoveryBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.es.ContractESRecoveryBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.es.ContractESRecoveryBatch baobab --build

./release.sh finder-batch-prod io.klaytn.apps.restore.blockBurn.BlockBurn baobab --build ./release.sh finder-batch-prod io.klaytn.apps.restore.blockBurn.BlockBurn cypress --build

./release.sh finder-batch-prod io.klaytn.apps.restore.tokenURI.TokenURI baobab --build ./release.sh finder-batch-prod io.klaytn.apps.restore.tokenURI.TokenURI cypress --build

./release.sh finder-batch-prod io.klaytn.apps.restore.account.AccountRestore baobab --build ./release.sh finder-batch-prod io.klaytn.apps.restore.account.AccountRestore cypress --build

./release.sh finder-batch-prod io.klaytn.apps.restore.transaction.TransactionRestoreBatch cypress --build ./release.sh finder-batch-prod io.klaytn.apps.restore.transaction.TransactionRestoreBatch baobab --build

Deploy Streaming Examples

./release.sh finder-streaming-prod io.klaytn.apps.block.BlockToDBStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.block.BlockToDBStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.itx.InternalTXToDBStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.itx.InternalTXToDBStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.account.AccountToDBStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.account.AccountToDBStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.transfer.TransferToDBStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.transfer.TransferToDBStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.worker.FastWorkerStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.worker.FastWorkerStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.worker.TokenWorkerStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.worker.TokenWorkerStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.worker.NFTWorkerStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.worker.NFTWorkerStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.worker.TokenURIWorkerStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.worker.TokenURIWorkerStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.worker.SlowWorkerStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.worker.SlowWorkerStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.block.BlockToESStreaming cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.block.BlockToESStreaming baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.block.DumpKafkaBlock cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.block.DumpKafkaBlock baobab --build

./release.sh finder-streaming-prod io.klaytn.apps.itx.DumpKafkaInternalTX cypress --build ./release.sh finder-streaming-prod io.klaytn.apps.itx.DumpKafkaInternalTX baobab --build