Running on test data fails; trimgalore encountering errors
Closed this issue · 6 comments
sbresnahan commented
python3 /storage/home/stb5321/work/mcclintock/mcclintock.py \
-r /storage/home/stb5321/scratch/mcclintock_test/test/sacCer2.fasta \
-c /storage/home/stb5321/scratch/mcclintock_test/test/sac_cer_TE_seqs.fasta \
-g /storage/home/stb5321/scratch/mcclintock_test/test/reference_TE_locations.gff \
-t /storage/home/stb5321/scratch/mcclintock_test/test/sac_cer_te_families.tsv \
-1 /storage/home/stb5321/scratch/mcclintock_test/test/SRR800842_1.fastq.gz \
-2 /storage/home/stb5321/scratch/mcclintock_test/test/SRR800842_2.fastq.gz \
-o /storage/home/stb5321/scratch/mcclintock_test/testout/
SETUP checking fasta: /storage/home/stb5321/scratch/mcclintock_test/test/sacCer2.fasta
SETUP checking fastq: /storage/home/stb5321/scratch/mcclintock_test/test/SRR800842_1.fastq.gz
SETUP checking fastq: /storage/home/stb5321/scratch/mcclintock_test/test/SRR800842_2.fastq.gz
SETUP checking fasta: /storage/home/stb5321/scratch/mcclintock_test/test/sac_cer_TE_seqs.fasta
SETUP checking locations gff: /storage/home/stb5321/scratch/mcclintock_test/test/reference_TE_locations.gff
SETUP checking taxonomy TSV: /storage/home/stb5321/scratch/mcclintock_test/test/sac_cer_te_families.tsv
SETUP McClintock Version: a393b73e09e706aa522099cf0f9badd9bfc45263
Job counts:
count jobs
1 coverage
1 index_reference_genome
1 make_consensus_fasta
1 make_coverage_fasta
1 make_popoolationte_annotations
1 make_ref_te_bed
1 make_ref_te_fasta
1 make_reference_fasta
1 make_te_annotations
1 map_reads
1 mask_reference_fasta
1 median_insert_size
1 ngs_te_mapper2_post
1 ngs_te_mapper2_pre
1 ngs_te_mapper2_run
1 ngs_te_mapper_post
1 ngs_te_mapper_run
1 popoolationTE2_post
1 popoolationTE2_preprocessing
1 popoolationTE2_run
1 popoolationTE_post
1 popoolationTE_preprocessing
1 popoolationTE_ref_fasta
1 popoolationTE_run
1 process_temp
1 process_temp2
1 reference_2bit
1 relocaTE2_post
1 relocaTE2_run
1 relocaTE_consensus
1 relocaTE_post
1 relocaTE_ref_gff
1 relocaTE_run
1 repeatmask
1 retroseq_post
1 retroseq_run
1 run_temp
1 run_temp2
1 sam_to_bam
1 setup_reads
1 summary_report
1 teflon_post
1 teflon_preprocessing
1 teflon_run
1 telocate_post
1 telocate_ref
1 telocate_run
1 telocate_sam
1 telocate_taxonomy
49
PROCESSING making consensus fasta
PROCESSING consensus fasta created
PROCESSING making reference fasta
PROCESSING reference fasta created
PROCESSING formatting the name of consensus TE fasta headers for compatibility with relocaTE
PROCESSING relocaTE consensus fasta created
PROCESSING prepping reads for McClintock
PROCESSING running trim_galore &> /storage/home/stb5321/scratch/mcclintock_test/testout/logs/20210509.150110.5719502/trimgalore.log
trim_galore --paired --fastqc -j 1 -o /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1//results/trimgalore --paired /storage/home/stb5321/scratch/mcclintock_test/test/SRR800842_1.fastq.gz /storage/home/stb5321/scratch/mcclintock_test/test/SRR800842_2.fastq.gz
Traceback (most recent call last):
File "/gpfs/scratch/stb5321/mcclintock_test/testout/snakemake/5719502/.snakemake/scripts/tmp8697nkov.setup_reads.py", line 59, in main
trimmedfq, trimmedfq2 = run_trim_galore(fq1, run_id, log, mcc_out, fq2=fq2, cores=processors, flags=flags)
File "/gpfs/scratch/stb5321/mcclintock_test/testout/snakemake/5719502/.snakemake/scripts/tmp8697nkov.setup_reads.py", line 167, in run_trim_galore
for f in os.listdir(out+"/results/trimgalore"):
FileNotFoundError: [Errno 2] No such file or directory: '/storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1//results/trimgalore'
ERROR processing of FastQ files failed...check that your FastQ files are formatted correctly...Exiting...
[Sun May 9 15:03:39 2021]
Error in rule setup_reads:
jobid: 3
output: /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/intermediate/fastq/SRR800842_1_1.fq, /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/intermediate/fastq/SRR800842_1_2.fq
conda-env: /storage/home/stb5321/work/mcclintock/install/envs/conda/25d88fae
RuleException:
CalledProcessError in line 36 of /gpfs/scratch/stb5321/mcclintock_test/testout/snakemake/5719502/Snakefile:
Command 'source /storage/home/stb5321/work/miniconda3/bin/activate '/storage/home/stb5321/work/mcclintock/install/envs/conda/25d88fae'; set -euo pipefail; python /gpfs/scratch/stb5321/mcclintock_test/testout/snakemake/5719502/.snakemake/scripts/tmp8697nkov.setup_reads.py' returned non-zero exit status 1.
File "/storage/home/stb5321/work/miniconda3/envs/mcclintock/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2340, in run_wrapper
File "/gpfs/scratch/stb5321/mcclintock_test/testout/snakemake/5719502/Snakefile", line 36, in __rule_setup_reads
File "/storage/home/stb5321/work/miniconda3/envs/mcclintock/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 568, in _callback
File "/storage/home/stb5321/work/miniconda3/envs/mcclintock/lib/python3.7/concurrent/futures/thread.py", line 57, in run
File "/storage/home/stb5321/work/miniconda3/envs/mcclintock/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
File "/storage/home/stb5321/work/miniconda3/envs/mcclintock/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2352, in run_wrapper
Exiting because a job execution failed. Look above for error message
snakemake --use-conda --conda-prefix /storage/home/stb5321/work/mcclintock/install/envs/conda --configfile /storage/home/stb5321/scratch/mcclintock_test/testout/snakemake/config/config_5719502.json --cores 1 --quiet /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/ngs_te_mapper/SRR800842_1_ngs_te_mapper_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/ngs_te_mapper2/SRR800842_1_ngs_te_mapper2_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/relocate/SRR800842_1_relocate_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/relocate2/SRR800842_1_relocate2_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/temp/SRR800842_1_temp_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/temp2/SRR800842_1_temp2_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/retroseq/SRR800842_1_retroseq_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/popoolationte/SRR800842_1_popoolationte_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/popoolationte2/SRR800842_1_popoolationte2_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/te-locate/SRR800842_1_telocate_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/teflon/SRR800842_1_teflon_nonredundant.bed /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/coverage/te_depth.csv /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/intermediate/fastq/SRR800842_1_1.fq /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/intermediate/mapped_reads/median_insert.size /storage/home/stb5321/scratch/mcclintock_test/testout/SRR800842_1/results/summary/data/run/summary_report.txt```
sbresnahan commented
Looking at the trimgalore.log file, there is an error: ModuleNotFoundError: No module named 'dnaio.readers'
pbasting commented
- Based on the
ModuleNotFoundErrorin #86 (comment), this could be due to a version ofdnaiono longer being compatible with the rest of the environment. - To confirm this, could you post the packages in your trimgalore conda environment?
- Based on the paths from your log, you can do this with the following commands:
conda activate /storage/home/stb5321/work/mcclintock/install/envs/conda/25d88fae
conda list
Thanks,
Preston
sbresnahan commented
Thanks for checking on this. Here is the output:
# packages in environment at /storage/home/stb5321/work/mcclintock/install/envs/conda/25d88fae:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
biopython 1.76 py38h516909a_0 conda-forge
brotlipy 0.7.0 py38h497a2fe_1001 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2020.12.5 ha878542_0 conda-forge
certifi 2020.12.5 py38h578d9bd_1 conda-forge
cffi 1.14.4 py38ha312104_0 conda-forge
chardet 4.0.0 py38h578d9bd_1 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
coloredlogs 15.0 py38h578d9bd_0 conda-forge
colormath 3.0.0 py_2 conda-forge
cryptography 3.4.7 py38ha5dfef3_0 conda-forge
curl 7.71.1 he644dc0_3 conda-forge
cutadapt 3.2 py38h0213d0e_0 bioconda
cycler 0.10.0 py_2 conda-forge
decorator 5.0.7 pyhd8ed1ab_0 conda-forge
dnaio 0.5.1 py38h4a8c8d9_0 bioconda
fastq_utils 0.24.1 hfb9b9cc_1 bioconda
fastqc 0.11.9 hdfd78af_1 bioconda
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
fontconfig 2.13.1 hba837de_1005 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
future 0.18.2 py38h578d9bd_3 conda-forge
htslib 1.9 h4da6232_3 bioconda
humanfriendly 9.1 py38h578d9bd_0 conda-forge
icu 68.1 h58526e2_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
importlib-metadata 4.0.1 py38h578d9bd_0 conda-forge
isa-l 2.30.0 ha770c72_4 conda-forge
jinja2 2.11.3 pyh44b312d_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
kiwisolver 1.3.1 py38h1fd1430_1 conda-forge
krb5 1.17.2 h926e7f8_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge
libblas 3.9.0 9_openblas conda-forge
libcblas 3.9.0 9_openblas conda-forge
libcurl 7.71.1 hcdd3856_3 conda-forge
libdeflate 1.2 h516909a_1 bioconda
libedit 3.1.20191231 h46ee950_2 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.3.0 h2828fa1_19 conda-forge
libgfortran-ng 9.3.0 hff62375_19 conda-forge
libgfortran5 9.3.0 hff62375_19 conda-forge
libgomp 9.3.0 h2828fa1_19 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 9_openblas conda-forge
libopenblas 0.3.15 pthreads_h8fe5266_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libssh2 1.9.0 ha56f1ee_6 conda-forge
libstdcxx-ng 9.3.0 h6de172a_19 conda-forge
libtiff 4.2.0 hdc55705_1 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.2.0 h7f98852_2 conda-forge
libxml2 2.9.10 h72842e0_4 conda-forge
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
lzstring 1.0.4 py_1001 conda-forge
markdown 3.3.4 pyhd8ed1ab_0 conda-forge
markupsafe 1.1.1 py38h497a2fe_3 conda-forge
matplotlib-base 3.4.2 py38hcc49a3a_0 conda-forge
multiqc 1.9 py_1 bioconda
ncurses 6.1 hf484d3e_1002 conda-forge
networkx 2.5 py_0 conda-forge
numpy 1.20.2 py38h9894fe3_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjdk 11.0.1 h516909a_1016 conda-forge
openjpeg 2.4.0 hf7af979_0 conda-forge
openssl 1.1.1k h7f98852_0 conda-forge
perl 5.32.0 h36c2ea0_0 conda-forge
pigz 2.6 h27826a3_0 conda-forge
pillow 8.1.2 py38ha0e1e83_1 conda-forge
pip 21.1.1 pyhd8ed1ab_0 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pysocks 1.7.1 py38h578d9bd_3 conda-forge
python 3.8.6 h852b56e_0_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-isal 0.10.0 py38h497a2fe_0 conda-forge
python_abi 3.8 1_cp38 conda-forge
pyyaml 5.4.1 py38h497a2fe_0 conda-forge
readline 8.0 h46ee950_1 conda-forge
requests 2.25.1 pyhd3deb0d_0 conda-forge
samtools 1.9 h10a08f8_12 bioconda
setuptools 49.6.0 py38h578d9bd_3 conda-forge
simplejson 3.17.2 py38h497a2fe_2 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
spectra 0.0.11 py_1 conda-forge
sqlite 3.35.4 hdfb4753_0
tk 8.6.10 h21135ba_1 conda-forge
tornado 6.1 py38h497a2fe_1 conda-forge
trim-galore 0.6.5 0 bioconda
urllib3 1.26.4 pyhd8ed1ab_0 conda-forge
wheel 0.36.2 pyhd3deb0d_0 conda-forge
xopen 1.1.0 py38h578d9bd_2 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
zipp 3.4.1 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h516909a_1010 conda-forge
zstd 1.4.9 ha95c52a_0 conda-forge```
pbasting commented
- Unfortunately, even using the exact conda environment you posted, I am not able to replicate this error and trimgalore is running fine for me.
- Can you post the entire
trimgalore.logfile for your failed run that way I can see the full traceback? - Also, I'd suggest trying to re-install the trimgalore environment to see if the error persists.
python /path/to/mcclintock.py --install -m trimgalore
cbergman commented
@sbresnahan: did you resolve this issue? If no, could you please do a clean install and try again? If yes, I'd like to close this issue. Thanks!
sbresnahan commented
@sbresnahan: did you resolve this issue? If no, could you please do a clean install and try again? If yes, I'd like to close this issue. Thanks!
Apologies for the delay - yes, the issue was resolved with a clean install.