DrosophilaGenomeEvolution/TrEMOLO

`[SNK INFO] ERROR PIPELINE` while doing test run

Closed this issue · 8 comments

clemgoub commented

Hello!

I am encountering some errors while performing the test run for TrEMOLO, as described in the README. I built a singularity container following the documentation and everything seemed to go smoothly.

When I execute this command:

singularity exec TrEMOLO.simg snakemake --snakefile TrEMOLO/run.snk --configfile TrEMOLO/test/tmp_config.yml

I end up with these errors:

GET SV WITH Assemblytics with alignment...
AN ERROR OCCURRED

[SNK INFO] ERROR PIPELINE; snakefile used : work_test/SNAKE_USED/Snakefile_insider.snk

snake log file : work_test/log/Snakefile_insider.log
 Error file : work_test/log/Snakefile_insider.err

and later

GET INSERTION...
AN ERROR OCCURRED

[SNK INFO] ERROR PIPELINE; snakefile used : work_test/SNAKE_USED/Snakefile_outsider.snk
    Check LOG   : work_test/log/Snakefile_outsider.log
    Check ERROR : work_test/log/Snakefile_outsider.err
/usr/bin/bash: line 169: kill: (88322) - No such process

I have attached the full log, as well as the snakemake files and .log/.err associated to them TrEMOLO_logs.tar.gz.

Please let me know if you need further info. I have built the image on a ubuntu system with sudo, and transferred the image to our HPC.

  • I tried the test run both on a compute or login nodes, without success.
  • I've also added MODE_PARALLELING: True that was missing in the test config file but with no effect on the issue.
  • I verified that the test data are in a disk that is mounted in singularity (~/TrEMOLO)

Thanks a a lot for your help!

Clément

M-D75 commented

Hello,

I would need more information, could you provide me with the other log files such as work_test/log/SV_INSIDER.err, work_test/log/sniffles.err.

The MODE_PARALLELING is currently unstable, an update is in progress, I advise you to set it to False. Thanks for specifying that it was missing from the file in the test config file.

I would also like to see the output file work_test/INSIDER/VARIANT_CALLING/pm_against_ref.sam.delta. But to do this you will first need to modify the TrEMOLO/run.snk file at line 303 to add the --keep-incomplete option (useful to keep the output files during an error) like this

snakemake --keep-incomplete --snakefile ${{path_to_pipline}}/Snakefile --configfile {params.name_configfil} ...

and then restart the pipeline.

Please before sending me the log files, and restarting the pipeline, delete the work_test folder in order to have complete information in the log files.

What version of singularity is used to build the container and run it on the compute nodes ?

Have you tried running the test on your local PC (it doesn't take much time or resources) ?

Thanks for your feedback,
Mourdas

clemgoub commented

Hello Mourdas!

Thanks for your answer, I will try that today and send you all the required files. I built the image with singularity-ce version 3.10.2-jammy on a Ubuntu system. I execute it on our HPC (https://docs.alliancecan.ca/wiki/Technical_documentation).

I will also try on my local PC, which I didn't, thanks for the advice!

More to come,

Clém

clemgoub commented

Hi gain, I've followed your instructions and re-run the tests.

In addition, the version on Singularity on the HPC is singularity version 3.8.7-1.el7 (so older than the one I used to build). Will now try on my PC (I'll try on the ubuntu VM I run on my Mac).

clemgoub commented

UPDATE: it worked on my laptop's VM without issues (by the way, I love the html report, it's very neat!). I'm looking forward to your expertise for the failed run on HPC. I will try to match the singularity versions to see if it caused the error.

UPDATE2: I couldn't update singularity on the cluster, and a custom install would be quite complex. So I dug a little into the error logs of TrEMOLO:

  • I found issues related to libflexiblas.so.3 (realted to numpy) possibly missing.
  • Indeed, I found that python3.8 within the singularity image, would search for numpy in the local machine using $PYTHONPATH as defined in my local ~/.bashrc. This is probably the case because in our HPC, singularity mounts the content of our home directory by default.
  • I could then fix the numpy dep using --contain in the singularity command. I could import numpy this time (I used singularity shell --contain TrEMOLO.simg to try it interactively). I then tried singularity exec --contain TrEMOLO.simg snakemake --snakefile TrEMOLO/run.snk --configfile TrEMOLO/test/tmp_config.yml but, the pipeline could not access the TrEMOLO repos. It actually makes sense, since I used --contain.
  • I then added --contain -B /home/cgoubert/bin/:/home/cgoubert
    • /home/cgoubert/bin is the path from where I start the image. It contains the TrEMOLO repos.
    • /home/cgoubert is the default ~/ in the container. This is because it was also the home name on the VM I used to build the .simg
    • Now, it can read TrEMOLO/test/tmp_config.yml but seem not able to copy the files into work_test/
Building DAG of jobs...
MissingInputException in line 443 of /home/cgoubert/TrEMOLO/Snakefile:
Missing input files for rule SV_INSIDER:
work_test/INPUT/tmp_genome.fasta
work_test/INPUT/tmp_ref.fasta

I'm now running out of ideas, but I'll be happy to hear your thoughts!

Cheers,

Clém

M-D75 commented

You are right, I think the problem is the mounting of the home in the container. Excellent idea the --contain option but I think it's too strict I suggest you use the --no-home option instead example :

singularity exec -B /home/cgoubert/bin/ --no-home TrEMOLO.simg snakemake --snakefile TrEMOLO/run.snk --configfile TrEMOLO/test/tmp_config.yml

I think that should solve the problem.

don't worry about the version of singularity, I have version 3.6.0 on HPC and its working.

thanks about the "report.html" feedback it's just the beginning 😄

Cordialy,
Mourdas

M-D75 commented

After reflection this will not help you any further.

I may have an idea, I'll do some tests and get back to you, if it works

M-D75 commented

Ok, I think you got this error :

Building DAG of jobs...
MissingInputException in line 443 of /home/cgoubert/TrEMOLO/Snakefile:
Missing input files for rule SV_INSIDER:
work_test/INPUT/tmp_genome.fasta
work_test/INPUT/tmp_ref.fasta

Because when you restarted the pipeline with the --contain option I guess the work_test folder already existed, so the INPUT files which are symbolic links were considered as existing but based on another path, I think your symbolic link work_test/INPUT/canonical_TE.fa points to /home/cgoubert/bin/TrEMOLO/test/canonical_TE.fa instead of /home/cgoubert/TrEMOLO/test/canonical_TE.fa

I suggest you use your command after deleting the work_test directory.

clemgoub commented

That was it! Thanks a bunch, it passed all the tests!

Cheers,

Clément

  • 🎉1