elliotchance/mbzdb

Load table data without decompressing first

Opened this issue · 1 comments

Currently we download, decompress and use LOAD DATA INFILE on https://github.com/elliotchance/mbzdb/blob/master/backend/mysql.pl, but I think decompressing to disk isn't really needed: we could decompress to a pipe and LOAD DATA from it, avoiding slow seek times and extra disk space needed for decompression :

mkfifo /tmp/fifo
tar -xjf mbdump.tgz artist -O > /tmp/fifo &
LOAD DATA LOCAL INFILE '/tmp/fifo' INTO TABLE ....

BTW, this should be configurable by user (fifo path, etc).

Probably for this to work we need to increase timeouts first:
set global net_write_timeout = 28800;
set global net_read_timeout = 28800;

http://blog.rasilon.net/articles/mysqldump_error_2013_lost_connection_to_mysql_server/index.html