Getting an error running Docker test

[tcrevier]

I am running on a clean docker image with the test script. It gets far into the process and then gets and error.

log_error.txt contains :

./merge_close_targets_cfd.sh: line 34: 3282 Killed python remove_contiguous_samples_cfd.py $fileIn $fileOut $thresh $chrom $position $total $true_guide $snp_info $cfd $sort_pivot $sorting_criteria_scoring $sorting_criteria
CRISPRme ERROR: contigous SNP removal failed (script: ./merge_close_targets_cfd.sh line 31)
./merge_close_targets_cfd.sh: line 34: 3281 Killed python remove_contiguous_samples_cfd.py $fileIn $fileOut $thresh $chrom $position $total $true_guide $snp_info $cfd $sort_pivot $sorting_criteria_scoring $sorting_criteria
CRISPRme ERROR: contigous SNP removal failed (script: ./merge_close_targets_cfd.sh line 31)
Traceback (most recent call last):
File "/opt/conda/opt/crisprme/PostProcess/remove_contiguous_samples_cfd.py", line 662, in
merge_targets()
File "/opt/conda/opt/crisprme/PostProcess/remove_contiguous_samples_cfd.py", line 618, in merge_targets
int(target_data[input_args[4]]),
ValueError: invalid literal for int() with base 10: '+'
CRISPRme ERROR: contigous SNP removal failed (script: ./merge_close_targets_cfd.sh line 31)

mv: cannot stat '/Test123/Results/sg1617_2/final_results_sg1617_2.bestMerge.txt.bestCFD.txt.trimmed': No such file or directory

I can add the full verbose log file if that would help, but here is the end that corresponds to the error...

Sorting file
Sorting file
Sorting file
Sorting done in 0 seconds
Sorting done in 0 seconds
Sorting done in 0 seconds
Merging contiguous targets
Merging contiguous targets
Merging contiguous targets

Any help tracing this down would be helpful, I am looking forward to running the tool on a real sequence!

Let me know if any other information would be helpful...

Tom

[ManuelTgn]

Hi @tcrevier,

thanks for reporting this issue.

From the output you shared with us it looks like Docker ran out of memory:

./merge_close_targets_cfd.sh: line 34: 3282 Killed python remove_contiguous_samples_cfd.py $fileIn $fileOut $thresh $chrom $position $total $true_guide $snp_info $cfd $sort_pivot $sorting_criteria_scoring $sorting_criteria

Could you please check how many GBs of memory are available to Docker?

Generally, to run CRISPRme without facing potential memory issues 32 GB of memory are enough.
Let us if this helps.

Best,
Manuel

[tcrevier]

Hi @ManuelTgn ,

I came to the same conclusion and went to 64 GB system. It ran longer and I got output files, but the error log still had these lines in it

./scriptAnalisiNNN_v3.sh: line 100: 36459 Killed ./new_simple_analysis.py "$referencegenome" "$dictionaries" "$jobid.total.cluster.txt" "${pam_file}" "$jobid" "$mismatch"
CRISPRme ERROR: annotation analysis failed (script: ./scriptAnalisiNNN_v3.sh line 97)
CRISPRme ERROR: SNP analysis failed (script: ./post_analisi_snp.sh line 34)
Traceback (most recent call last):
File "/opt/conda/opt/crisprme/PostProcess/remove_contiguous_samples_cfd.py", line 662, in
merge_targets()
File "/opt/conda/opt/crisprme/PostProcess/remove_contiguous_samples_cfd.py", line 618, in merge_targets
int(target_data[input_args[4]]),
ValueError: invalid literal for int() with base 10: '+'
CRISPRme ERROR: contigous SNP removal failed (script: ./merge_close_targets_cfd.sh line 31)
Traceback (most recent call last):
File "/opt/conda/opt/crisprme/PostProcess/remove_contiguous_samples_cfd.py", line 662, in
merge_targets()
File "/opt/conda/opt/crisprme/PostProcess/remove_contiguous_samples_cfd.py", line 618, in merge_targets
int(target_data[input_args[4]]),
ValueError: invalid literal for int() with base 10: '+'
CRISPRme ERROR: contigous SNP removal failed (script: ./merge_close_targets_cfd.sh line 31)
Traceback (most recent call last):
File "/opt/conda/opt/crisprme/PostProcess/remove_contiguous_samples_cfd.py", line 662, in
merge_targets()
File "/opt/conda/opt/crisprme/PostProcess/remove_contiguous_samples_cfd.py", line 618, in merge_targets
int(target_data[input_args[4]]),
ValueError: invalid literal for int() with base 10: '+'
CRISPRme ERROR: contigous SNP removal failed (script: ./merge_close_targets_cfd.sh line 31)

Again, this is with the test data.

Tom

[tcrevier]

@ManuelTgn Quick update, I ran on a 16 core system with 128 GB of RAM and it ran to completion without errors. It did consume close to 100GB of RAM at two points in the run. I was using 14 threads, I am assuming that more threads can lead to more memory consumption at certain parts of the run?

Thanks,

Tom

[ManuelTgn]

Hi @tcrevier,

It's unexpected that CRISPRme would require more than 100 GB of RAM to complete the test search as it is defined in the test script. But since you're running it on Docker, there might be some overhead added by Docker itself.

Regarding your question, using more threads could indeed lead to higher memory consumption. Similarly, conducting permissive parameter searches, such as allowing a high number of mismatches or large bulge sizes, often results in several potential off-targets. Processing these off-targets to generate the final reports may increase memory usage, even though we've minimized RAM usage as much as possible.

However, typically using 4 to 8 threads should not cause the system to run out of memory.

If you have any further questions about our software, please don't hesitate to ask.

Thank you again for using CRISPRme.

Best,
Manuel