yongman/tidis

Tidis + TiKV not GC for long time test

slow-zhang opened this issue · 7 comments

slow-zhang commented

hi here, i using 3pd + 3 tikv + 1 tidis cluster and using go-ycsb test. After loaded 35M keys, I start a long run test with only read and update like the following config. The cluster finally stopped duo to full disks.

Why there is no GC in my cluster?

tidis config

[tidis]

listen = "0.0.0.0:5379"
max_connection = 5000

loglevel = "info"

#transaction retry count when commit failed in case of conflict
txn_retry = 2

auth = ""

tenantid = ""

[backend]
#tikv placement driver addresses
pds = "pd-ip:2379"'

tikv config

    raftdb.max-background-jobs: 3
    raftdb.max-sub-compactions: 1
    raftdb.rate-bytes-per-sec: 60M
    raftstore.raft-base-tick-interval: 3s
    raftstore.raft-store-max-leader-lease: 27s
    readpool.coprocessor.use-unified-pool: true
    readpool.storage.use-unified-pool: false
    readpool.unified.max-thread-count: 6
    rocksdb.max-background-jobs: 3
    rocksdb.max-sub-compactions: 1
    rocksdb.rate-bytes-per-sec: 60M
    rocksdb.titan.enabled: true
    server.grpc-keepalive-time: 12s
    storage.block-cache.capacity: 40

using go-ycsb command run

readproportion=0.8
updateproportion=0.2
scanproportion=0
insertproportion=0
recordcount=35000000

nohup ./bin/go-ycsb run redis
-P myworkload
-p redis.addr="ip:5379"
-p redis.pool_size=128
-p threadcount=32
-p tikv.conncount=128

yongman commented

Can you describe it in details？
In tidis，modify a key will not cause different encoding keys in tikv(rocksdb). But rocksdb wil have size amplification due to read/write and compaction configure.
What is your disk size and initial data size after insertion 35000000 records?

slow-zhang commented

I use AWS m5.16xlarge * 3, 24disk * 1900GB, 35M records with 3-copy used ~7T in total. In the update & read workload, disk is exhausted and tikv breakdown duo to it

yongman commented

Can you run test again refer to tikv configuration in wiki https://github.com/yongman/tidis/wiki/Tidis-base-benchmark ?
RocksDB tuning is a complicated thing due to different workload.

yongman commented

@slow-zhang After checked the code, GC configuration lost in code.
GC code has been added in tag https://github.com/yongman/tidis/releases/tag/v1.1.0 .
You can test again with the newest code. Ping me if have any further questions.

slow-zhang commented

@yongman Thank you, I will test again

tmikaeld commented

Was the issue resolved? I'm curious

yongman commented

Was the issue resolved? I'm curious

Resolved.

  • ❤️2