Performance issues on dump
cterence opened this issue ยท 4 comments
Hello,
I've made some basic testing of Replibyte and I'm currently experiencing two performance issues :
- Dumping operations are taking way longer than expected, around 0.5MB/s
- The RAM consumption gets out of control after a while, progressively increasing until generating an OOMKill
Here are the testing conditions :
Hardware :
- Lenovo Thinkpad T14s Gen 1
- RAM : 16GB
- CPU : AMD Ryzen 7 PRO 4750U w/ 8 physical core, 16 virtual cores
- NVMe SSD : (300MB/s write, 700MB/s read)
- OS : Linux, PopOS 20.04
Software :
- Replibyte version : 0.10.0
- Replibyte config
replibyte.yaml
source: connection_uri: postgres://postgres:postgres@localhost:5433/demo datastore: local_disk: dir: ./dumps destination: connection_uri: postgres://postgres:postgres@localhost:5432/demo
- pg_dump version : pg_dump (PostgreSQL) 15.2 (Ubuntu 15.2-1.pgdg20.04+1)
- docker-compose.yaml
services: postgresql-prod: container_name: postgresql-prod image: postgres:11 environment: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=postgres ports: - 5433:5432 volumes: - ./sample-data:/docker-entrypoint-initdb.d postgresql-staging: container_name: postgresql-staging image: postgres:11 ports: - 5432:5432 environment: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=postgres
- Sample PostgreSQL data : https://edu.postgrespro.com/demo-small-en.zip (around 320MB of data when loaded in Postgres)
- Commands run :
replibyte --config ./replibyte.yaml dump create
&replibyte --config ./replibyte.yaml dump restore remote --value latest
With all of this, here are my testing steps :
- Deploy the stack & load the sample data
- Creating the dump with replibyte
Creating the dump takes around 11 min at 0.4~0.5 MB/s.
Is there anything that I'm doing wrong ?
Don't hesitate to ask for more details if I've gone over anything, I'll be happy to help.
I've been having the same issue in a very similar environment. Any word on this?
Yup pretty much the same happening here, maybe worse.
Running on a ubuntu cloud vm 2vCPUs/4GB RAM, with a local 200MB MySQL database.
I get OOM killed with replibyte using 3GB of memory ๐ฌ
Slow for me as well, even on local disk I'm getting < 1Mib/sec when creating a dump
Hello ๐, I'm aware of this issue and I'm working on benchmarking and performance improvements here. Thanks for your feedback