Load table data without decompressing first
Opened this issue · 1 comments
hectoras commented
Currently we download, decompress and use LOAD DATA INFILE on https://github.com/elliotchance/mbzdb/blob/master/backend/mysql.pl, but I think decompressing to disk isn't really needed: we could decompress to a pipe and LOAD DATA from it, avoiding slow seek times and extra disk space needed for decompression :
mkfifo /tmp/fifo
tar -xjf mbdump.tgz artist -O > /tmp/fifo &
LOAD DATA LOCAL INFILE '/tmp/fifo' INTO TABLE ....
BTW, this should be configurable by user (fifo path, etc).
hectoras commented
Probably for this to work we need to increase timeouts first:
set global net_write_timeout = 28800;
set global net_read_timeout = 28800;
http://blog.rasilon.net/articles/mysqldump_error_2013_lost_connection_to_mysql_server/index.html