Unable to restore data from the latest dump
KShivendu opened this issue · 4 comments
Hi,
I'm trying to restore data from the official dump in a Postgres container and it's not working as expected.
wget https://ecosystems-data.s3.amazonaws.com/packages-2023-08-08.tar.gz
tar xvf packages-2023-08-08.tar.gz
cd packages-2023-08-08/
docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres --name pg -v "$(pwd)":/backup postgres
docker exec -it pg bash
# Inside pg docker container:
psql -U postgres
# Inside psql shell
CREATE DATABASE ecosystems;
\q
# Exited psql shell
pg_restore --verbose --jobs=8 -U postgres -d ecosystems packages-2023-08-08.sql
# Wait for restoration to finish - actually it won't succeed
This is the tail of the output from the pg_restore
command:
pg_restore: finished item 3245 CONSTRAINT ar_internal_metadata ar_internal_metadata_pkey
pg_restore: launching item 3438 TABLE DATA registries
pg_restore: processing data for table "public.registries"
pg_restore: WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
pg_restore: error: error returned by PQputCopyData: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
pg_restore: error: a worker process died unexpectedly
Upvote & Fund
- We're using Polar.sh so you can upvote and help fund this issue.
- We receive the funding once the issue is completed & confirmed by you.
- Thank you in advance for helping prioritize & fund our backlog.
Ohh, wait. I could be completely wrong but maybe the dump is malicious.
After this:
pg_restore: finished item 3245 CONSTRAINT ar_internal_metadata ar_internal_metadata_pkey
pg_restore: launching item 3438 TABLE DATA registries
pg_restore: processing data for table "public.registries"
It starts /tmp/kdevtmpfsi
which starts consuming a lot of CPU. It's a malware https://www.createit.com/blog/kinsing-malware-kdevtmpfsi-how-to-kill/
I was wondering where did it come from in my server and ran the script suggested by the above blog to kill it if it tries to re-spawn. The pg_restore workers seem to get killed (or lose connection) the moment /tmp/kdevtmpfsi
is killed by the script.
Please let me know what you think about this :)
@KShivendu thanks for reporting, I will attempt the restore myself, I checked the server that generated the database export and could not find any trace of that malware.
@KShivendu I was able to re-import the restore, it does take quite a bit of time, disk space and cpu to finish creating the indexes, the database takes up approxmately 189GB of disk space once fully recreated. I can't find any evidence of malware in the database export.
Okay. I'll try again to restore it from the dump. Closing this issue in the meantime. I'll reopen if I still face any issues.
Thanks a lot! :D