Issue when running TE-Aid in parallel

[manighanipoor]

Hi,

I need to run TE-Aid in parallel but it causes errors because of using shared resources.
I tried this command (to copy TE-Aid to a temp file for each process so it doesn't use the same database) in a HPC cluster in parallel but it does not work for all processes:

GENOME="../aipysurus_laevis.polished.fa"
TEAID="/hpcfs/users/a1177955/local/TE-Aid/"
parallel --bar --jobs 3 -a fasta_list.txt "mkdir -p ./tmp/{#}/TE-Aid && mkdir -p ./tmp/{#}/output && cp -ar $TEAID/* ./tmp/{#}/TE-Aid/ && ln -sf $(realpath $GENOME) ./tmp/{#}/genome_file && ./tmp/{#}/TE-Aid/TE-Aid -q {} -g ./tmp/{#}/genome_file -o ./tmp/{#}/output && mv ./tmp/{#}/output/* ./" && rm -r ./tmp/

and this is what I got (it just worked with process 1 and gave error for processes 2 and 3):

0% 0:3=0s fasta_3.fa query: fasta_2.fa
ref genome: ./tmp/2/genome_file
TE -> genome blastn e-value: 10e-8
full length min ratio: 0.9
hits transparency: 0.3
full length hits transparency: 0.9
no ORF detected, skipping blastp...
[1] "R: ploting genome blastn results and computing coverage..."
[1] "consensus length: 360 bp"
[1] "R: ploting self dot-plot and orf/protein hits..."
[1] "no orf to plot..."
null device
1
Done! The graph (.pdf) can be found in the output folder: ./tmp/2/output
Warning message:
In file(file, "rt") :
cannot open file './tmp/2/output/orftetable': No such file or directory
33% 1:2=31s fasta_3.fa query: fasta_1.fa
ref genome: ./tmp/1/genome_file
TE -> genome blastn e-value: 10e-8
full length min ratio: 0.9
hits transparency: 0.3
full length hits transparency: 0.9
RepeatPeps is downloaded and formatted, blastp-ing...
[1] "R: ploting genome blastn results and computing coverage..."
[1] "consensus length: 1582 bp"
[1] "R: ploting self dot-plot and orf/protein hits..."
null device
1
Done! The graph (.pdf) can be found in the output folder: ./tmp/1/output
66% 2:1=11s fasta_3.fa query: fasta_3.fa
ref genome: ./tmp/3/genome_file
TE -> genome blastn e-value: 10e-8
full length min ratio: 0.9
hits transparency: 0.3
full length hits transparency: 0.9
no ORF detected, skipping blastp...
[1] "R: ploting genome blastn results and computing coverage..."
[1] "consensus length: 541 bp"
[1] "R: ploting self dot-plot and orf/protein hits..."
[1] "no orf to plot..."
null device
1
Done! The graph (.pdf) can be found in the output folder: ./tmp/3/output
Warning message:
In file(file, "rt") :
cannot open file './tmp/3/output/orftetable': No such file or directory
100% 3:0=0s fasta_3.fa

would you please let me know what the solution is?

Cheers,
Mani

[foriin]

Hi Mani,

First of all, as far as I know, TE-Aid wasn't made for running in parallel. The basic output of this tool is a pdf plot that you have to inspect manually, which is not feasible for multitude of TEs. In other words, TE-Aid was designed to work with a specific consensus for getting an overview of its structure and genome representation.
Second, in order to maximize the speed without running TE-Aid in parallel and avoid potential collisions, you could just loop over your fastas with a bash script while using the same output folder. If your files and corresponding fasta headers have different names that should work fine and you won't download/generate BLAST databases for each fasta. I haven't worked with X laevis, but for danio, which has genome two times smaller, it takes ~15 seconds to run TE-Aid, when databases are prepared, so it shouldn't be as bad as well for your clawed friend. Anyhoo, I would just submit a bash script to your cluster that loops over your fastas:

#!/usr/bin/env bash
#SBATCH parameters or whatever HPC control system you have 
GENOME=/path/to/genome

for fa in ./*.fasta
do
    TE-Aid -q ${fa} -g ${GENOME} -o output_folder
done

And thirdly, the formatting of the parallel command you wrote in your question is broken. That makes it harder to read it and understand.

Cheers,
Artem

[manighanipoor]

Hi,
thanks, I could resolve the issue.