==Work in progress==
auto_sql is very early in development, more features to come
auto_sql is a memory aware csv to sqlite converter capable of converting multi-gigabyte tabular files to sqlite databases on low memory machines. auto_sql focuses on speed by enabling multi-processing on multi-core machines.
$pip install auto_sql
from auto_sql import AutoSql
tab_obj = AutoSql(file='file.csv',
db_name='database',
sep='\t',
out_dir=".")
if __name__ == "__main__":
tab_obj.run()
Currently auto_sql only supports csv's with headers
The buffer
parameter can be reduced from it's default .3
value to avoid memory errors. Conversely the buffer
parameter can be increased to gain speed at the increased risk of a memory error.
from auto_sql import AutoSql
tab_obj = AutoSql(file='file.csv',
db_name='database',
sep='\t',
out_dir=".",
buffer=.1)
if __name__ == "__main__":
tab_obj.run()
-
Python 3.4, 3.5, or 3.6
-
pandas==0.21.1
-
psutil==5.4.7