hariszaf/pema

cannot remove 'nospace.*': No such file or directory

Tclack88 opened this issue · 8 comments

Hello, I ran through the demo data given and there wasn't a problem. I have a few fastq files that I converted to fastq.gz using the provided convertIllumunaRawDataToEnaFormat.sh. This created a directory and file in the mydata dir (namely mapping_files_for_PEMA.txt rawDataInEnaFormat/) I ran with those files in the directory and got an error (as others have gotten by keeping the README.md as I thought I addressed initially here with Akhilbiju01's question) I promptly moved them out so that I only have the fastq.gz files and am getting this error involving no "no space*". I looked through the source code in PEMA_v1.2.bds line 507 to 515 the file is created (a temp file I imagine) then deleted. It appears this non-existent folder is supposed to # merge all lines of a fastq entry into one and only one line given by this line:
sys awk 'NR==1 {print ; next} {printf /^>/ ? "\n"$0"\n" : $1} END {printf "\n"}' se.$derepl > nospace.$derepl

So at this point I'm uncertain if the data is bad or if there's bug in the code. Here's my full output from the run:

$ singularity run  -B /p/home/tclack/bio/pema-1.2/test/analysis_folder/:/mnt/analysis ./pema_v.1.1.sif
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
this ouput file already exists
1E_S29_L001_R1_001.fastq.gz1E_S29_L001_R2_001.fastq.gz2G_S42_L001_R2_001.fastq.gz2H_S48_L001_R1_001.fastq.gz3C_S17_L001_R1_001.fastq.gz3C_S17_L001_R2_001.fastq.gzperl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Started analysis of 3C_S17_L001_R1_001.fastq.gz
Started analysis of 2H_S48_L001_R1_001.fastq.gz
Started analysis of 1E_S29_L001_R1_001.fastq.gz
Started analysis of 1E_S29_L001_R2_001.fastq.gz
Started analysis of 2G_S42_L001_R2_001.fastq.gz
Approx 5% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 5% complete for 2H_S48_L001_R1_001.fastq.gz
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Started analysis of 3C_S17_L001_R2_001.fastq.gz
Approx 5% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 10% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 10% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 15% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 5% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 10% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 15% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 15% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 20% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 10% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 20% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 25% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 15% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 5% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 5% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 20% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 25% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 25% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 30% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 20% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 30% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 30% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 35% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 40% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 25% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 35% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 35% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 45% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 50% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 30% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 10% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 40% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 45% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 40% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 55% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 35% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 10% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 50% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 45% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 60% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 55% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 60% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 50% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 65% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 70% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 40% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 15% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 65% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 55% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 75% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 45% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 70% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 60% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 80% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 50% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 15% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 75% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 65% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 85% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 55% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 20% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 80% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 70% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 90% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 95% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 60% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 85% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 75% complete for 3C_S17_L001_R1_001.fastq.gz
Analysis complete for 2H_S48_L001_R1_001.fastq.gz
Approx 65% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 20% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 90% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 95% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 80% complete for 3C_S17_L001_R1_001.fastq.gz
Analysis complete for 2G_S42_L001_R2_001.fastq.gz
Approx 70% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 25% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 85% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 75% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 90% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 80% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 95% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 30% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 25% complete for 1E_S29_L001_R2_001.fastq.gz
Analysis complete for 3C_S17_L001_R1_001.fastq.gz
Approx 85% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 90% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 95% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 35% complete for 1E_S29_L001_R1_001.fastq.gz
Analysis complete for 3C_S17_L001_R2_001.fastq.gz
Approx 30% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 40% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 35% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 45% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 50% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 40% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 55% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 45% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 60% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 50% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 55% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 65% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 60% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 70% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 65% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 75% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 70% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 80% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 85% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 75% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 80% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 90% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 95% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 85% complete for 1E_S29_L001_R2_001.fastq.gz
Analysis complete for 1E_S29_L001_R1_001.fastq.gz
Approx 90% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 95% complete for 1E_S29_L001_R2_001.fastq.gz
Analysis complete for 1E_S29_L001_R2_001.fastq.gz
FastQC is completed!
readF is: 1E_S29_L001_R1_001.fastq.gz
readF is: 1E_S29_L001_R2_001.fastq.gz
readF is: 2G_S42_L001_R2_001.fastq.gz
readF is: 2H_S48_L001_R1_001.fastq.gz
readF is: 3C_S17_L001_R1_001.fastq.gz
readF is: 3C_S17_L001_R2_001.fastq.gz
Trimmomatic  is done

Error correction using BayesHammer is completed!
Merging step by SPAdes is completed
all the first steps are done! clustering is about to start!
rm: cannot remove 'se.*': No such file or directory
rm: cannot remove 'nospace.*': No such file or directory
Fatal error: /home/PEMA_v1.bds, line 517, pos 1. Exec failed.
	Exit value : 1
	Command    :  rm se.* nospace.*
PEMA_v1.bds, line 517 :	sys rm se.* nospace.*

@hariszaf

If there's anything I can provide to further debug or reproduce, I would be more than happy to

Hi there! Sorry for the late reply!

Could you tell me which pema container and version are you using and on what OS ?

All good, it's the holiday season afterall and I'm sure you're probably quit busy

I've tried on SUSE and Fedora

SLES (SUSE Linux Enterprise Server) version 12 SP3 -- x86_64 architecture if it maters
and
Red Hat Enterpise Linux Server 7.7 (Maipo)

And PEMA 1.2 using Singularity (V 3.5.3)

I can also send some sample files over if you need to recreate the error with that

Many wishes for a happy new year! :)

Could you try first to run the PEMA v.1.3.2 container?
I think this should work fine as after v1.3.1 the converting part is included in pema.

You should check on having the latest parameters.tsv file as well.

I am actually working on pema these days trying to fix some things, make the repo a bit better and also add a new feature enabling custom ref dbs, so my apologies for any inconvenience! I hope I will have everything ready quite soon! 🙂

I pulled the latest singularity container. 1.3.2. I've tried te parameters.tsv as you currently have on here in /pema/analysis_directory/parameters.tsv

That was last updated 4 months ago. I'm uncertain if you have a newer version

Same error. As I said, I can send you some sample files fastqz if it may be helpful

Yes we should go for that.

You may send me a sample at pema@hcmr.gr or at haris-zaf@hcmr.gr.

and I ll check this as soon as possible.

Great! Sent to your pema email.

Thanks again for your support

For future readers, this issue was likely caused from not compressing my .fastq files. I misread a response on gitter and thought I needed to use the convertIllumunaRawDataToEnaFormat.sh on my .fastq files. But nope, I just needed to gzip them. Strangely this "concert to ena" shell script gave me gzipped files, but resulted in the "nospace" error