ERROR："sqlite3.OperationalError: disk I/O error"

[Bank-tidy]

Hi there,
I would like to express my gratitude for developing such a useful tool. However, I encountered some challenges while performing batch annotations.
I’m curious about what might be causing this inconsistent behavior. Additionally, I would like to ask if there’s any way to implement or suggest a checkpoint feature that would allow the process to resume from where it left off rather than starting over entirely.
Below is a snippet of the code I am using:

source activate liftoff
sample=$1
cd /public1/home/user/projects/Avena/10_liftoff/${sample}/

ref_gff="/public1/home/user/ref.gff3"
ref_genome="/public1/home/user/ref.fasta"
qry_genome="/public1/home/user/genome/${sample}.fa"
qry_gff="/public1/home/user/${sample}/${sample}_liftoff.gff3"
chr="/public1/home/user/${sample}/chroms.txt"

liftoff -g ${ref_gff} -chroms ${chr} -o ${qry_gff} ${qry_genome} ${ref_genome} -p 64 -dir /public1/home/user/${sample}/intermediate_files

Here’s an excerpt from the log file that might provide some insight into the issue:

[M::mm_idx_stat::2598.592*2.09] distinct minimizers: 70813657 (49.08% are singletons); average occurrences: 6.646; average spacing: 5.351; total length: 2518326703
[M::worker_pipeline::2642.408*2.17] mapped 3375 sequences
[M::worker_pipeline::2644.177*2.17] mapped 3375 sequences
[M::main] Version: 2.24-r1122
[M::main] CMD: minimap2 -o /public1/home/user/projects/Avena/10_liftoff/O005765/intermediate_files/unmapped_to_expected_chrom_to_target_all.sam -a --end-bonus 5 --eqx -N 50 -p 0.5 --split-prefix /public1/home/user/projects/Avena/10_liftoff/O005765/intermediate_files/unmapped_to_expected_chrom_to_target_all_split -t 64 /public1/home/user/projects/Avena/09_genome/O005765.fa /public1/home/user/projects/Avena/10_liftoff/O005765/intermediate_files/unmapped_to_expected_chrom_genes.fa
[M::main] Real time: 2644.187 sec; CPU: 5727.828 sec; Peak RSS: 45.353 GB
Traceback (most recent call last):
  File "/public1/home/user/software/miniconda3/envs/liftoff/bin/liftoff", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/public1/home/user/software/miniconda3/envs/liftoff/lib/python3.12/site-packages/liftoff/run_liftoff.py", line 12, in main
    run_all_liftoff_steps(args)
  File "/public1/home/user/software/miniconda3/envs/liftoff/lib/python3.12/site-packages/liftoff/run_liftoff.py", line 29, in run_all_liftoff_steps
    unmapped_features = map_unmapped_features(unmapped_features, target_chroms, lifted_feature_list, feature_db,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/public1/home/user/software/miniconda3/envs/liftoff/lib/python3.12/site-packages/liftoff/run_liftoff.py", line 203, in map_unmapped_features
    return liftover_types.map_unmapped_genes_agaisnt_all(unmapped_features, ref_chroms, target_chroms,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/public1/home/user/software/miniconda3/envs/liftoff/lib/python3.12/site-packages/liftoff/liftover_types.py", line 50, in map_unmapped_genes_agaisnt_all
    align_and_lift_features(ref_chroms, target_chroms, args, feature_hierarchy, liftover_type, unmapped_features,
  File "/public1/home/user/software/miniconda3/envs/liftoff/lib/python3.12/site-packages/liftoff/liftover_types.py", line 28, in align_and_lift_features
    lift_features.lift_all_features(aligned_segments, min_cov, feature_db,  feature_hierarchy,
  File "/public1/home/user/software/miniconda3/envs/liftoff/lib/python3.12/site-packages/liftoff/lift_features.py", line 8, in lift_all_features
    feature_order = get_feature_order(feature_db)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/public1/home/user/software/miniconda3/envs/liftoff/lib/python3.12/site-packages/liftoff/lift_features.py", line 29, in get_feature_order
    feature_types = list(gene_db.featuretypes())
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/public1/home/user/software/miniconda3/envs/liftoff/lib/python3.12/site-packages/gffutils/interface.py", line 437, in featuretypes
    c.execute(
sqlite3.OperationalError: disk I/O error

I would greatly appreciate any advice or suggestions you could provide.

Thank you for your time and support.

[jonathandmoore]

I will bump this issue. It is a consistent bug I have seen for quite a long time. It happens if you try to run more than one instance of liftoff at the same time from the same directory.

I too am trying to run batch liftoffs. My conclusion is that, when supplying a gff to lift off, the software is creating a temporary sqlLite database to keep working data in.

If I run two liftoffs in parallel in the same directory on the same machine, the first one succeeds, but the second fails presumably because it can't get a lock on the database.

I think, to get around this, it should be possible for each instance to specify the path to its own sqlLite database, but this is not possible as far as I can tell - the -g option to read from a gff disallows the specification of a -db option.

The only workaround I have found so far is to ensure only one instance is running at any one time.