Can't locate error. Somewhere around a shell script at the beginning of Blast?
RubyDrus opened this issue · 13 comments
Hi all,
just started using Transposome but it fails right after starting it. I'm sorry I can't describe it better but there seems to be no verbose option. Here is the error message printed on screen (I have shortened some paths for clarity reasons):
INFO - ======== Transposome version: 0.09.8 (started at: 04-09-2015 09:29:55) ========
INFO - Configuration - Log file for monitoring progress and errors: 2015-09-02_t_log.txt
INFO - Configuration - Sequence file: ./data/2015-09-02_transposome/test_4100314/test_4100314.fastq
INFO - Configuration - Sequence format: fastq
INFO - Configuration - Sequence number for each BLAST process: 50000
INFO - Configuration - Number of CPUs per thread: 1
INFO - Configuration - Number of threads: 2
INFO - Configuration - Output directory: ./data/2015-09-02_transposome/test_4100314/transposome_out
INFO - Configuration - In-memory analysis: 1
INFO - Configuration - Percent identity for matches: 90
INFO - Configuration - Fraction coverage for pairwise matches: 0.55
INFO - Configuration - Merge threshold for clusters: 0.001
INFO - Configuration - Minimum cluster size for annotation: 100
INFO - Configuration - BLAST e-value threshold for annotation: 10
INFO - Configuration - Repeat database for annotation: ./data/genomic_dbs/2015-06-19_element_specific_sequences/repbase_homosapiens_alu_family.fasta
INFO - Configuration - Log file for clustering/merging results: 2015-09-02_t_cluster_report.txt
INFO - Transposome::Run::Blast::run_allvall_blast started at: 04-09-2015 09:29:56.
sh: - : invalid option
Usage: sh [GNU long option] [option] ...
sh [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
--dump-po-strings
--dump-strings
--help
--init-file
--login
--import-functions
--noediting
--noprofile
--norc
--posix
--protected
--rcfile
--restricted
--verbose
--version
Shell options:
-irsD or -c command or -O shopt_option (invocation only)
-abefhkmnptuvxBCHP or -o option
Use of uninitialized value $mgblast in concatenation (.) or string at /ibios/tbi_cluster/11.4/x86_64/perl/perl-5.14.1/lib/site_perl/5.14.1/Transposome/Run/Blast.pm line 329.
Use of uninitialized value $mgblast in concatenation (.) or string at /ibios/tbi_cluster/11.4/x86_64/perl/perl-5.14.1/lib/site_perl/5.14.1/Transposome/Run/Blast.pm line 329.
sh: - : invalid option
Usage: sh [GNU long option] [option] ...
sh [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
--dump-po-strings
--dump-strings
--help
--init-file
--login
--import-functions
--noediting
--noprofile
--norc
--posix
--protected
--rcfile
--restricted
--verbose
--version
Shell options:
-irsD or -c command or -O shopt_option (invocation only)
-abefhkmnptuvxBCHP or -o option
sh: - : invalid option
Usage: sh [GNU long option] [option] ...
sh [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
--dump-po-strings
--dump-strings
--help
--init-file
--login
--import-functions
--noediting
--noprofile
--norc
--posix
--protected
--rcfile
--restricted
--verbose
--version
Shell options:
-irsD or -c command or -O shopt_option (invocation only)
-abefhkmnptuvxBCHP or -o option
[ERROR]: Could not open file: ./data/2015-09-02_transposome/test_4100314/transposome_out/test_4100314_2_yUqu_test_4100314_allvall_mgblastdb.bln
One thing seems to stand out: The name of the file it could not find looks different from the actual file it is searching for:
From the screen:
./data/2015-09-02_transposome/test_4100314/transposome_out/test_4100314_2_yUqu_test_4100314_allvall_mgblastdb.bln
Compared to the actual but empty file named:
./data/2015-09-02_transposome/test_4100314/transposome_out/test_4100314_allvall_blast.bln
Since it probably cannot start blast as wished, I assumed that our customised cluster architecture might a problem. So I tried several combinations of alias for blast like alias blast="blastn" or alias mgblast="mgblast-0.09.8" but that didn't help.
Thank you in advance!
All the best,
Ruben
Edit: Maybe I should mention that I only use 100,000 reads of a sample which contains 67m reads in total.
Thanks for the report. Can you tell me what operating system this is being run on? I have seen some situations where users are required to load "modules" to run specific programs like blastn so you would have to put module load blastn or similar in your script to run transposome. I don't know if that is the case here but something is odd with how the PATH and Perl's sitebin are being searched (and not being found, in the case of the latter).
The file name is not a problem. At that step, the input is split and blast is run in parallel and then joined to create the final output file. The problem is that mgblast couldn't be found, and the program should die before this stage if it wasn't found. Please show me the output of this command:
perl -MConfig -E 'say $Config{sitebin}'
Also, it would be helpful to see the contents of the output directory after it failed:
ls -l ./data/2015-09-02_transposome/test_4100314/transposome_out
This would also help:
which mgblast
which formatdb
Any update on this one? The output of the commands above would be necessary to move forward.
Hi sestaton,
thanks for you fast reply and sorry for the delay!
I'm running it on our cluster which has openSUSE 11.4. Here is the perl -MConfig output:
/ibios/tbi_cluster/11.4/x86_64/perl/perl-5.14.1/bin
Here the ls -l output:
-rw-rw---- 1 drews B080-MMML 1548 Sep 8 08:19 2015-09-02_t_log.txt
-rw-rw---- 1 drews B080-MMML 0 Sep 8 08:19 test_4100314_allvall_blast.bln
And finally the both which commands tell me that both programs are not found. I'm puzzled you asked for them since formatdb is obsolete for many years now and mgblast is now distributed via blast+.
Okay, the required programs were not installed. Please tell me the specific steps of how you installed Transposome because it doesn't appear to be installed correctly. That would explain the missing programs. mgblast is different than megablast (it is not distributed with blast+), and it is installed by Transposome.
Hi,
just got the following information from our IT departement:
tar xzf Transposome.tar.gz
cd Transposome
curl -L cpanmin.us | perl - --installdeps .
perl Makefile.PL
make
make test
Does it helps you?
The last step should be:
sudo make install
and that would have installed the programs, but I assume they did that. Unfortunately, I don't have an openSUSE machine to test nor can I find that available from the cloud services I use. Also, I do not know what to offer at this point since I can't see the output of any of those steps to see what went wrong. One issue I have had in the past is the Admins setting a PATH and the PERL5LIB variable for all users and this interferes with some applications.
To avoid these issues, I would recommend setting up perlbrew and installing Transposome yourself. This tool (perlbrew) will install Perl in your home directory, then you can install anything you want without admin privileges (there are other benefits too). The whole process would look like this:
\curl -L http://install.perlbrew.pl | bash
echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.bashrc
source ~/.bashrc
perlbrew install perl-5.20.0 -Dusethreads
perlbrew switch perl-5.22.0
perlbrew install-cpanm
Then, you should be able to install Transposome with one command:
cpanm git://github.com/sestaton/Transposome.git
Of course, a couple of these steps take some time (installing Perl, then Transposome), but this will create a log of steps where I can see any issues, if there are any. As an aside, this is how I use Perl and Transposome.
Hi,
talked to our IT guys again: because of the cluster structure the last step sudo make install couldn't be done since its output would only be available to the head node. So they made it manually and linked the programs to a folder where all the nodes have access to. But while doing this they renamed the links with a version (so "mgblast" became "mgblast-0.9.8")... So I set a PATH variable to the original head node program and tada! It worked. Well, it aborted after a few minutes with the message:
Error: NCBI C++ Exception:
"/data/temp/blast/ncbi-blast-2.2.28+-src/c++/src/corelib/ncbiobj.cpp", line 925: Critical: ncbi::CObject::ThrowNullPointerException() - Attempt to access NULL pointer.
I assume that this is also a problem based on our customised cluster architecture but due to a conference next week I won't have time to have a look into it right now.
I'm sorry for bothering you with a problem which eventually occurred on my site. Thanks again and enjoy the weekend!
All the best,
Ruben
Sorry you are having issues, but I don't mind helping at all. I see two major problems here. 1) The package wasn't installed by running make install so all the programs and files were not put into place. It simply won't work by trying to copy or link files where you think they should be. 2) mgblast is not megablast, as I mentioned before, and it does not use the latest NCBI toolkit written C++. The error indicates that some program from BLAST+ was executed and ran into an error, but this should never happen (if the package was installed).
It looks like they have created somewhat of a mess by generating links to programs that are not expected. My advice would be to get them to install the package, or better yet, try the solution I provided above using perlbrew and that way you can install it properly yourself. Hope that helps. Let me know if you run into any issues.
Quick update:
Our system admins are on it to correct the installation and complete it as you suggested. I hope this will work! Once they succeed I would suggest to close this issue.
The Perlbrew solution is good for one person but since there are other interested students, we need it to run on our cluster system.
Thanks again!
Okay, please keep me posted and let me know if you have any questions I can help with.
Any updates? I'd like to close this if it has been resolved, or find a solution if it has not. Thanks.
Hi Evan,
currently, I have little time to focus on Transposome and didn't solve
the problem. Being mainly a local cluster issue, I propose that you can
close the issue.
Thanks again for your help!
All the best,
Ruben
Please don't hesitate if you have other questions but I will close this since it is a cluster configuration issue.