Import ADIF records into database in one go

Question

Import ADIF records into database in one go

ctjacobs opened this issue 9 years ago · 2 comments

Currently the slowest part of importing ADIF files seems to be the insertion of each record one-by-one into the sqlite database, with individual database commits. It would probably be faster to queue them up somehow and insert them all in one go.

Answer 1 · 2015-08-12T00:01:36.000Z

After gathering all the records to insert into one list, using executemany seems to bring a significant performance benefit when inserting all the records into the database. For an ADIF file of ~6,500 entries, the import time is reduced from ~150 seconds (with a for loop calling execute in each iteration) to ~7 seconds with one call to executemany on my laptop. Populating the actual log table (i.e. the GTK ListStore) seems to still take a while though.

Thanks to Timo OH7JHA for providing a relatively large ADIF file to test with.

Answer 2 · 2015-10-02T23:16:24.000Z

Sorted out an add_record bug introduced by 37685c9 (see 794d999). I think this is fine now, so closing.