IMPORTANT: This repository serves archival purposes only. See https://github.com/nicolerg/format_refseq for a more streamlined approach to compile a database of RefSeq genomes.
This repository is intended for the download and compilation of whole-genome microbial databases via a cluster. Genomes of interest may be specified by GenBank accession or organism name. However, GenBank accession is preferred whenever possible. Note that scripts are provided to run this process locally, but this is NOT recommended for any large number of genomes.
1.1 Install the NCBI Entrez command line tool, EDirect [3]
If this is your first time using EDirect
, you need to install the command line tool. EDirect
is compatible with UNIX and Macintosh systems. Follow the installation instructions below, copied from here). Note you must already have perl
installed:
EDirect will run on UNIX and Macintosh computers that have the Perl language installed, and under the Cygwin UNIX-emulation environment on Windows PCs. To install the EDirect software, copy the following commands and paste them into a terminal window:
cd ~
/bin/bash
perl -MNet::FTP -e \
'$ftp = new Net::FTP("ftp.ncbi.nlm.nih.gov", Passive => 1);
$ftp->login; $ftp->binary;
$ftp->get("/entrez/entrezdirect/edirect.tar.gz");'
gunzip -c edirect.tar.gz | tar xf -
rm edirect.tar.gz
builtin exit
export PATH=${PATH}:$HOME/edirect >& /dev/null || setenv PATH "${PATH}:$HOME/edirect"
./edirect/setup.sh
This downloads several scripts into an "edirect" folder in the user's home directory. The setup.sh script then downloads any missing Perl modules, and may print an additional command for updating the PATH environment variable in the user's configuration file. Copy that command, if present, and paste it into the terminal window to complete the installation process. The editing instructions will look something like:
echo "export PATH=\$PATH:\$HOME/edirect" >> $HOME/.bashrc
The line #PBS -V
at the top of the scripts called by qsub
exports environmental variables, including those in .bash_profile
.
As of May 1, 2018, you must request an API key from NCBI in order to submit multiple EDirect requests within one second. These scripts add a 5-second pause after each EDirect command to try to avoid the 429 Too Many Requests PLEASE REQUEST AN API_KEY FROM NCBI
error. However, I have still gotten this error even with the sleep
commands to avoid it. I highly recommend obtaining an API key from NCBI before attempting to run these script. Follow the instructions here: https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/
Briefly:
- Create an NCBI account: https://www.ncbi.nlm.nih.gov/
- Go to your settings: https://www.ncbi.nlm.nih.gov/account/settings/
- Click the link to create a new API key
- Replace "YOUR_API_KEY" with your API key in the line below and run the command to add it to your
.bashrc
.
echo "export NCBI_API_KEY=YOUR_API_KEY" >> $HOME/.bashrc
- Run
source ~/.bashrc
to export the variable (just this once)
Scripts are supplied for two different input types: 1) GenBank accessions; 2) organism names. GenBank accessions are preferred whenever possible. Follow the instructions below to generate input files in one of the accepted formats.
-
PATRIC .txt file (archaea, bacteria)
- Go here [1]
- BROWSE > Bacteria OR Archaea
- Select "Genomes" tab
- Filters > Select "Representative" and "Reference"
- Apply any other desired filters
- DOWNLOAD > Text
-
NCBI .nbr file (viruses)
- Go here [2]
- Download Viral Genome Data > Accession list of all viral genomes
-
List of GenBank accessions
- Custom by the user
- Must have only one column, containing only GenBank accessions
- GenBank accessions must be one per line or separated by commas (no space)
- All genomes fetched for accession on the SAME line will be concatenated into one genome file; the script assumes all sequences belong to the organism specified by the first accession in the comma-delimited list
- Rename the file with something simple; save it to the desired database location folder in addition to other required package files (see above)
-
NCBI .txt file
- Go here
- Top right: Filters > Apply desired filters
- Middle left: Download (TSV Download)
-
List of organism names
- Custom by the user
- Must have only one column, containing only complete strings of organism names
- Must have only one organism per line
- Rename the file with something simple; save it to the desired database location folder in addition to other required package files (see above)
- Create a new folder (e.g.
ncbi
). This path is referred to at${outdir}
. - Move the file generated in Step 2 to this folder. No other files should be present in this folder.
As of 20 March 2019, please skip this step until test
is updated
Run test.sh
for a quick unit test. It uses two E.coli genomes and one Cowpox virus genome to check if the outputs of the download, formatting, job split, and compilation steps are working as expected. The result of the test is output to test_result
. Please make sure to run the test before compiling the full database to avoid overwriting.
- Define a variable with the full path to the input file (generated in Step 2). Note that this file should be located in the desired output folder
${outdir}
(from Step 3).
infile=/users/my_sername/reprdb/taxid10239.nbr
- Run
bash mass_retrieve_V1.sh ${infile}
- Enter the input file type when prompted
- Wait for the genomes to finish downloading (this could take several hours depending on the number of genomes requested)
- Run
bash format.sh ${infile}
- Find your compiled sequence files and
all_lengths.txt
file in${outdir}/compiled_genomes
- Define a variable with the full path to the input file (generated in Step 2). Note that this file should be located in the desired output folder
${outdir}
(from Step 3).
infile=/users/my_sername/reprdb/names.txt
- Run
bash mass_retrieve_V2.sh ${infile}
- Enter the input file type when prompted
- Wait for the genomes to finish downloading (this could take several hours depending on the number of genomes requested)
- Run
bash format.sh ${infile}
- Find your compiled sequence files and
all_lengths.txt
file in${outdir}/compiled_genomes
- Start a
screen
session, which will not kill the script if your computer goes to sleep (e.g.screen -S reprdb
) - Define a variable with the full path to the input file (generated in Step 2). Note that this file should be located in the desired output folder
${outdir}
(from Step 3).
infile=/users/my_sername/reprdb/taxid10239.nbr
- Run
bash local_retrieve_V1.sh ${infile}
- Enter the input file type when prompted
- Detach from the screen while you wait for the script to finish running (
CTRL + A + D
) - Reattach to the screen to check progress (e.g.
screen -r reprdb
) - When the script has finished running, you may exit the screen session (type
exit
at the command line while attached to the screen) - Find your compiled sequence files and
all_lengths.txt
file in${outdir}/compiled_genomes
- Start a
screen
session, which will not kill the script if your computer goes to sleep (e.g.screen -S reprdb
) - Define a variable with the full path to the input file (generated in Step 2). Note that this file should be located in the desired output folder
${outdir}
(from Step 3).
infile=/users/my_sername/reprdb/names.txt
- Run
bash local_retrieve_V2.sh ${infile}
- Enter the input file type when prompted
- Detach from the screen while you wait for the script to finish running (
CTRL + A + D
) - Reattach to the screen to check progress (e.g.
screen -r reprdb
) - When the script has finished running, you may exit the screen session (type
exit
at the command line while attached to the screen) - Find your compiled sequence files and
all_lengths.txt
file in${outdir}/compiled_genomes
- Concatenated FASTA files (
.fa
), each <= 2.8 GB (change default max size withinformat.sh
) - An
all_lengths.txt
file with one line per genome. These lines provide the corresponding sequence name in the concatenated FASTA files and the length of the sequence, e.g.:
>ACCN:CP002059|Bacteria;Cyanobacteria;Nostocales;Nostocaceae;Trichormus;Nostoc_azollae_0708 5486745
- A
failed_download.txt
with errors from all jobs
A quick tutorial on what you need to know about UNIX in order to run this script:
pwd
: outputs the current path (present working directory)cd /$FOLDER_NAME/$SUBFOLDER
: allows you to navigate to the indicated path (i.e. folder)- e.g.
cd /data/user-name
- e.g.
cd /..
: brings you one level up to the parent directory
- e.g.
ls
: lists everything within that directory- e.g.
ls
: lists all files and folders in the current directory - e.g.
ls /data/s-gayn/database
: lists all files in thedatabase
folder
- e.g.
cat $FILE_NAME
: outputs the content of$FILE_NAME
mv $FILE $NEW_NAME
: renames$FILE
in the current working directory to$NEW_NAME
mv $FILE $FOLDER/$NEW_LOCATION
: moves$FILE
in the current working directory to$FOLDER/$NEW_LOCATION
cp $FILE $FOLDER/NEW_LOCATION
: copies$FILE
in the current working directory to$FOLDER/$NEW_LOCATION
These basic commands provide everything you need to know and more in able to use this database compilation package. Happy compiling!
[1] Wattam, A.R., D. Abraham, O. Dalay, T.L. Disz, T. Driscoll, J.L. Gabbard, J.J. Gillespie, R. Gough, D. Hix, R. Kenyon, D. Machi, C. Mao, E.K. Nordberg, R. Olson, R. Overbeek, G.D. Pusch, M. Shukla, J. Schulman, R.L. Stevens, D.E. Sullivan, V. Vonstein, A. Warren, R. Will, M.J.C. Wilson, H. Seung Yoo, C. Zhang, Y. Zhang, B.W. Sobral (2014). “PATRIC, the bacterial bioinformatics database and analysis resource.” Nucl Acids Res 42 (D1): D581-D591. doi:10.1093/nar/gkt1099. PMID: 24225323.
[2] NCBI viral genomes resource. Brister JR, Ako-Adjei D, Bao Y, Blinkova O.Nucleic Acids Res. 2015 Jan;43(Database issue):D571-7. doi: 10.1093/nar/gku1207. Epub 2014 Nov 26.
[3] Sayers E. E-utilities Quick Start. 2008 Dec 12 [Updated 2013 Aug 9]. In: Entrez Programming Utilities Help [Internet]. Bethesda (MD): National Center for Biotechnology Information (US); 2010-. Available from: http://www.ncbi.nlm.nih.gov/ books/NBK25500/