failed to synchronization.

Question

failed to synchronization.

Closed this issue 3 years ago · 18 comments

Logs

info: Tzkt.Sync.Program[0]
      Version 1.8.3.0
info: Tzkt.Sync.Program[0]
      Initialize database
info: Microsoft.EntityFrameworkCore.Infrastructure[10403]
      Entity Framework Core 5.0.10 initialized 'TzktContext' using provider 'Npgsql.EntityFrameworkCore.PostgreSQL' with options: None
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (10ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON n.oid=c.relnamespace WHERE c.relname='__EFMigrationsHistory');
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT "MigrationId", "ProductVersion"
      FROM "__EFMigrationsHistory"
      ORDER BY "MigrationId";
info: Tzkt.Sync.Program[0]
      Database initialized
warn: Tzkt.Sync.Services.Observer[0]
      Observer started
info: Microsoft.EntityFrameworkCore.Infrastructure[10403]
      Entity Framework Core 5.0.10 initialized 'TzktContext' using provider 'Npgsql.EntityFrameworkCore.PostgreSQL' with options: None
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (6ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT a."Id", a."AccountCounter", a."AccountsCount", a."ActivationOpsCount", a."BallotOpsCount", a."BigMapCounter", a."BigMapKeyCounter", a."BigMapUpdateCounter", a."BlocksCount", a."Chain", a."ChainId", a."CommitmentsCount", a."ConstantsCount", a."Cycle", a."CyclesCount", a."DelegationOpsCount", a."DoubleBakingOpsCount", a."DoubleEndorsingOpsCount", a."DoublePreendorsingOpsCount", a."EndorsementOpsCount", a."EndorsingRewardOpsCount", a."Hash", a."KnownHead", a."LastSync", a."Level", a."ManagerCounter", a."Metadata", a."MigrationOpsCount", a."NextProtocol", a."NonceRevelationOpsCount", a."OperationCounter", a."OriginationOpsCount", a."PreendorsementOpsCount", a."ProposalOpsCount", a."ProposalsCount", a."Protocol", a."ProtocolsCount", a."QuoteBtc", a."QuoteCny", a."QuoteEth", a."QuoteEur", a."QuoteGbp", a."QuoteJpy", a."QuoteKrw", a."QuoteLevel", a."QuoteUsd", a."RegisterConstantOpsCount", a."RevealOpsCount", a."RevelationPenaltyOpsCount", a."ScriptCounter", a."SetDepositsLimitOpsCount", a."StorageCounter", a."Timestamp", a."TokenBalanceCounter", a."TokenBalancesCount", a."TokenCounter", a."TokenTransfersCount", a."TokensCount", a."TransactionOpsCount", a."VotingEpoch", a."VotingPeriod"
      FROM "AppState" AS a
      LIMIT 2
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /opt
info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (3ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT a."Id", a."ActiveTokensCount", a."Address", a."Balance", a."ContractsCount", a."Counter", a."DelegateId", a."DelegationLevel", a."DelegationsCount", a."FirstLevel", a."LastLevel", a."Metadata", a."MigrationsCount", a."OriginationsCount", a."RevealsCount", a."Staked", a."TokenBalancesCount", a."TokenTransfersCount", a."TransactionsCount", a."Type", a."Activated", a."PublicKey", a."RegisterConstantsCount", a."Revealed", a."SetDepositsLimitsCount", a."ActivationLevel", a."BallotsCount", a."BlocksCount", a."DeactivationLevel", a."DelegatedBalance", a."DelegatorsCount", a."DoubleBakingCount", a."DoubleEndorsingCount", a."DoublePreendorsingCount", a."EndorsementsCount", a."EndorsingRewardsCount", a."FrozenDeposit", a."FrozenDepositLimit", a."NonceRevelationsCount", a."PreendorsementsCount", a."ProposalsCount", a."RevelationPenaltiesCount", a."SoftwareId", a."StakingBalance"
      FROM "Accounts" AS a
      WHERE a."Type" = 1
info: Tzkt.Sync.Services.Observer[0]
      State initialized: [2368491:BLYA2gBkkYgGvrHQ1VdsSWkHtPLXeWtFgh33fXxnQKWRmGVn6oq]
info: Tzkt.Sync.Services.QuotesService[0]
      Quote provider: DefaultQuotesProvider (Async)
info: Tzkt.Sync.Services.Observer[0]
      Quotes initialized: [2368491]
info: Tzkt.Sync.Services.Observer[0]
      Synchronization started
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. Specified argument was out of the range of valid values. (Parameter 'value')
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. Specified argument was out of the range of valid values. (Parameter 'value')
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. Specified argument was out of the range of valid values. (Parameter 'value')
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. Specified argument was out of the range of valid values. (Parameter 'value')
fail: Tzkt.Sync.Services.Observer[0]

I ran indexer & indexer-api altogether in the same container.

Answer 1 · 2022-05-20T18:06:48.000Z

Hi! Did you set the TezosNode.Lag in the indexer config? If yes, what value?

Answer 2 · 2022-05-23T07:50:11.000Z

Hello .
yes I setup TezosNode.Lag to 30.

Answer 3 · 2022-05-23T11:22:42.000Z

Weird, I can't reproduce it. I've made error logs more detailed, could you please update to the latest commit 7ef49de and show the logs again?

Answer 4 · 2022-05-25T07:09:20.000Z

OK I will do.
I have some questions for running application.
Any other flag for running application?
Cause tzkt looks like to aim appsettings.json automatically when running.

Answer 5 · 2022-05-25T11:52:42.000Z

Not sure I get what you mean. Do you mean how to configure the app in other way than via appsettings.json? If yes, then you can also configure it via env vars, or via command line args. For example, {"TezosNode": {"Lag": 30}} can be specified via env var: TezosNode__Lag=30 or via command line arg: --TezosNode:Lag=30.

Answer 6 · 2022-05-26T09:53:08.000Z

I mean how can I use flag for like --config {configfile.json}.

Answer 7 · 2022-05-26T13:08:43.000Z

There is no way to do that. If you want to override default settings without changing appsettings.json, pass them via env vars or command line args.

Answer 8 · 2022-05-26T13:44:53.000Z

Got it.
Thank you.

Answer 9 · 2022-05-26T13:47:59.000Z

And one more thing.
I had taken too much time for the restore data from *.backup file.
It was taken about more than 48hours.
Is it normal?

Here is the command that I used.

 docker exec -i -u postgres tezos-indexer  createuser tzkt
   docker exec -i -u postgres tezos-indexer  psql -c 'GRANT ALL PRIVILEGES ON DATABASE tzkt_db TO tzkt;'
   docker exec -i -u postgres tezos-indexer  psql -c 'GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO tzkt;'
   docker exec -i -u postgres tezos-indexer psql -c 'GRANT ALL PRIVILEGES ON DATABASE "postgres" to tzkt;'
   docker exec -i -u postgres tezos-indexer psql -c 'ALTER USER tzkt WITH SUPERUSER;'
   docker exec -i -u postgres tezos-indexer  pg_restore -U tzkt -c --if-exists -v -e -j 48 -d tzkt_db  /var/lib/postgresql/data/tzkt_v1.8_mainnet.backup
  

  docker exec -i -u postgres tezos-indexer  psql -c 'ALTER SYSTEM SET maintenance_work_mem='4GB''
  docker exec -i -u postgres tezos-indexer  psql -c 'ALTER SYSTEM SET max_wal_size = "'"16GB"'"'
  docker exec -i -u postgres tezos-indexer  psql -c 'SELECT pg_reload_conf();'

Answer 10 · 2022-05-26T13:48:28.000Z

This command took more than 48hours.

 docker exec -i -u postgres tezos-indexer  pg_restore -U tzkt -c --if-exists -v -e -j 48 -d tzkt_db  /var/lib/postgresql/data/tzkt_v1.8_mainnet.backup```

Answer 11 · 2022-05-26T15:08:48.000Z

For me it usually takes a couple of hours. It mostly depends on your hardware spec and pg config, but in this case I suspect the -j 48 parameter could be the reason. Do you have 48 CPU cores? The number of parallel workers shouldn't exceed the number of cpu cores, because otherwise you will face performance degradation.

Answer 12 · 2022-05-27T04:38:24.000Z

OK got it.
thank you for the reply,

Answer 13 · 2022-05-27T11:46:55.000Z

Now it's working.
Thank you for the guide.

Answer 14 · 2022-05-27T11:59:52.000Z

Here is the error message from indexer.

info: Tzkt.Sync.Services.Observer[0]
      Applied 2368507 of 2400914
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368508 of 2400915
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368509 of 2400916
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368510 of 2400917
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368511 of 2400919
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (32,003ms) [Parameters=[], CommandType='Text', CommandTimeout='30']

                          DELETE FROM "SnapshotBalances" WHERE "Level" = 2302976;
                                                                                                                                                                                                                 1181,10       98%
         at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
         at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /tzkt/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 83
         at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /tzkt/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 103
         at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /tzkt/Tzkt.Sync/Services/Observer/Observer.cs:line 179
         at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /tzkt/Tzkt.Sync/Services/Observer/Observer.cs:line 64
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368507 of 2400914
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368508 of 2400915
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368509 of 2400916
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368510 of 2400917
info: Tzkt.Sync.Services.Observer[0]
      Applied 2368511 of 2400919
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
      Failed executing DbCommand (32,003ms) [Parameters=[], CommandType='Text', CommandTimeout='30']

                          DELETE FROM "SnapshotBalances" WHERE "Level" = 2302976;
                          INSERT INTO "SnapshotBalances" ("Level", "Balance", "AccountId", "DelegateId", "DelegatorsCount", "DelegatedBalance", "StakingBalance")
                          SELECT 2368512, "Balance", "Id", "DelegateId", "DelegatorsCount", "DelegatedBalance", "StakingBalance"
                          FROM "Accounts"
                          WHERE "Staked" = true
fail: Microsoft.EntityFrameworkCore.Database.Transaction[20205]
      An error occurred using a transaction.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Cannot access a disposed object.
      Object name: 'NpgsqlTransaction'.
      System.ObjectDisposedException: Cannot access a disposed object.
      Object name: 'NpgsqlTransaction'.
         at Npgsql.NpgsqlTransaction.CheckDisposed()
         at Npgsql.NpgsqlTransaction.CheckReady()
         at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.RollbackAsync(CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.RollbackAsync(CancellationToken cancellationToken)
         at Tzkt.Sync.ProtocolHandler.CommitBlock(Int32 head) in /tzkt/Tzkt.Sync/Protocols/ProtocolHandler.cs:line 102
         at Tzkt.Sync.Services.Observer.ApplyUpdatesAsync(CancellationToken cancelToken) in /tzkt/Tzkt.Sync/Services/Observer/Observer.cs:line 179
         at Tzkt.Sync.Services.Observer.ExecuteAsync(CancellationToken cancelToken) in /tzkt/Tzkt.Sync/Services/Observer/Observer.cs:line 64

Is it OK to go?

Head rest api return like below.

curl http://127.0.0.1:5000/v1/head | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   675    0   675    0     0   329k      0 --:--:-- --:--:-- --:--:--  329k
{
  "chain": "mainnet",
  "chainId": "NetXdQprcVkpaWU",
  "cycle": 483,
  "level": 2368505,
  "hash": "BLJbMka431qFfKzy3bhxVk4UKLgCVnF2Rsv6DwikVhcxzLhq5tq",
  "protocol": "Psithaca2MLRFYargivpo7YvUr7wUDqyxrdhC5CQq78mRvimz6A",
  "nextProtocol": "Psithaca2MLRFYargivpo7YvUr7wUDqyxrdhC5CQq78mRvimz6A",
  "timestamp": "2022-05-16T01:28:59Z",
  "votingEpoch": 37,
  "votingPeriod": 72,
  "knownLevel": 2400909,
  "lastSync": "2022-05-27T11:51:32Z",
  "synced": false,
  "quoteLevel": 2368505,
  "quoteBtc": 6.03491734167585e-05,
  "quoteEur": 1.7983930128718153,
  "quoteUsd": 1.8724288495843768,
  "quoteCny": 12.711357731173454,
  "quoteJpy": 242.40370265276894,
  "quoteKrw": 2393.1700369422924,
  "quoteEth": 0.0008848243469709158,
  "quoteGbp": 1.525284285729134
}

Answer 15 · 2022-05-27T12:02:10.000Z

Configs are added these.

TezosNode__Endpoint=https://rpc.tzkt.io/mainnet/
ConnectionStrings__DefaultConnection=host=tezos-indexer-postgres1;port=5432;database=tzkt_db;username=tzkt;password=password;

Answer 16 · 2022-05-27T12:44:11.000Z

See Failed executing DbCommand (32,003ms) ... CommandTimeout='30'. Your DB fails with the command timeout error. You need to increase it by adding command timeout=600; to the connection string.

Answer 17 · 2022-05-27T12:45:47.000Z

Got it Thank you!

Answer 18 · 2022-05-30T10:10:16.000Z

Closing this issue as it seems resolved