dib-lab/elvers

Could not create conda environment from dammit-env.yaml and deseq2-env.yaml

johnsolk opened this issue · 7 comments

johnsolk commented

Ran this to setup:

git clone https://github.com/dib-lab/eelpond.git -b class/ggg201b
cd eelpondconda
conda env create --file environment.yml -n eelpond-sw
conda activate eelpond-sw

Then started the test example:

./run_eelpond examples/nema.yaml default

And got this (several times, on several separate instances):

CreateCondaEnvironmentException:
Could not create conda environment from /home/diblions/eelpond/rules/dammit/dammit-env.yaml:
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.7

Please update conda by running

    $ conda update -n base conda


pytest-runner-4.4    | 8 KB      | ########## | 100%
r-stringr-1.4.0      | 207 KB    | ########## | 100%
matplotlib-3.0.2     | 6 KB      | ########## | 100%
infernal-1.1.2       | 9.6 MB    | ########## | 100%
sphinx_rtd_theme-0.4 | 5.1 MB    | ########## | 100%
hmmer-3.2.1          | 7.1 MB    | ########## | 100%
r-r6-2.4.0           | 62 KB     | ########## | 100%
xorg-libxau-1.0.9    | 13 KB     | ########## | 100%
xorg-libx11-1.6.7    | 940 KB    | ########## | 100%
last-963             | 1.3 MB    | ########## | 100%
atomicwrites-1.3.0   | 9 KB      | ########## | 100%
bioconductor-seqlogo | 371 KB    | ########## | 100%
scipy-1.2.1          | 18.1 MB   | ########## | 100%
numpy-1.16.1         | 4.3 MB    | ########## | 100%
pandas-0.24.1        | 11.1 MB   | ########## | 100%
matplotlib-base-3.0. | 6.6 MB    | ########## | 100%
pytest-4.3.0         | 342 KB    | ########## | 100%
qt-5.6.2             | 44.5 MB   | ########## | 100%
cloudpickle-0.7.0    | 17 KB     | ########## | 100%
python-dateutil-2.8. | 219 KB    | ########## | 100%
py-1.8.0             | 65 KB     | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... failed


CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-1.0.0-r351h29659fb_1000
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/amd64/r-cran-testrcpppackage_0.1.0-1_amd64.deb'
specified in the package manifest cannot be found.

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-1.0.0-r351h29659fb_1000
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/i386/r-cran-testrcpppackage_0.1.0-1_i386.deb'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::libgfortran-3.0.0-1, conda-forge::libgcc-7.2.0-h69d50b8_2
  path: 'lib/libgfortran.so.3'


ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::libgfortran-3.0.0-1, conda-forge::libgcc-7.2.0-h69d50b8_2
  path: 'lib/libgfortran.so.3.0.0'


ClobberError: This transaction has incompatible packages due to a shared path.
  packages: bioconda::perl-app-cpanminus-1.7043-pl5.22.0_0, bioconda::perl-uri-1.71-pl5.22.0_1, bioconda::perl-scalar-list-utils-1.45-2, bioconda::perl-db-file-1.835-5
  path: 'lib/perl5/5.22.0/x86_64-linux-thread-multi/perllocal.pod'
johnsolk commented

This also occurs with running other parts of the pipeline:

(eelpond-sw) diblions@js-169-30:~/eelpond$ ./run_eelpond quant_DE.yml diffexp

--------
checking for required files:
--------

        Found input assembly at ../data/nema_trinity.fasta
        Added default parameters from rule-specific params files.
        Writing full params to .ep_quant_DE.yml
--------
details!
        snakefile: /home/diblions/eelpond/Snakefile
        config: quant_DE.yml
        params: .ep_quant_DE.yml
        targets: ['diffexp', 'assemblyinput']
        report: '/home/diblions/eelpond/lab7_out_quant_DE/logs/report.html'
--------
/opt/miniconda/envs/eelpond-sw/lib/python3.6/site-packages/snakemake/workflow.py:23: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from snakemake.shell import shell
Building DAG of jobs...
Creating conda environment rules/deseq2/deseq2-env.yaml...
Downloading remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/diblions/eelpond/rules/deseq2/deseq2-env.yaml:
Collecting package metadata: ...working... done
Solving environment: ...working... done

Downloading and Extracting Packages
xorg-libxau-1.0.9    | 13 KB     | ########## | 100%
xorg-libx11-1.6.7    | 940 KB    | ########## | 100%
Preparing transaction: ...working... done
Verifying transaction: ...working... failed

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-0.12.13-r3.4.1_0
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/amd64/r-cran-testrcpppackage_0.1.0-1_amd64.deb'
specified in the package manifest cannot be found.

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-0.12.13-r3.4.1_0
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/i386/r-cran-testrcpppackage_0.1.0-1_i386.deb'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::libgfortran-3.0.0-1, conda-forge::libgcc-7.2.0-h69d50b8_2
  path: 'lib/libgfortran.so.3'


ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::libgfortran-3.0.0-1, conda-forge::libgcc-7.2.0-h69d50b8_2
  path: 'lib/libgfortran.so.3.0.0'
johnsolk commented

same steps as above on 2 blank Jetstream instances (one Indiana and TACC). Ran conda update -n base conda beforehand, in case it made a difference. It didn't. Looks like a dammit environment problem.

(eelpond-sw) diblions@js-16-78:~/eelpond$ ./run_eelpond examples/nema.yaml default

--------
checking for required files:
--------

        Added default parameters from rule-specific params files.
        Writing full params to examples/.ep_nema.yaml
--------
details!
        snakefile: /home/diblions/eelpond/Snakefile
        config: examples/nema.yaml
        params: examples/.ep_nema.yaml
        targets: ['default']
        report: '/home/diblions/eelpond/nema_out/logs/report.html'
--------
/opt/miniconda/envs/eelpond-sw/lib/python3.6/site-packages/snakemake/workflow.py:23: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from snakemake.shell import shell
Building DAG of jobs...
Creating conda environment rules/dammit/dammit-env.yaml...
Downloading remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/diblions/eelpond/rules/dammit/dammit-env.yaml:
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.7

Please update conda by running

    $ conda update -n base conda


blast-2.2.31         | 131.1 MB  | ########## | 100%
qt-4.8.7             | 34.1 MB   | ########## | 100%
infernal-1.1.2       | 9.6 MB    | ########## | 100%
sip-4.18             | 263 KB    | ########## | 100%
krb5-1.16.1          | 1.4 MB    | ########## | 100%
boost-1.60.0         | 10.5 MB   | ########## | 100%
gfortran_linux-64-7. | 9 KB      | ########## | 100%
pytest-4.3.0         | 342 KB    | ########## | 100%
scipy-1.2.1          | 18.1 MB   | ########## | 100%
wheel-0.33.1         | 34 KB     | ########## | 100%
numpy-1.16.1         | 4.3 MB    | ########## | 100%
cloudpickle-0.8.0    | 17 KB     | ########## | 100%
pyqt-4.11.4          | 3.5 MB    | ########## | 100%
bamtools-2.5.1       | 841 KB    | ########## | 100%
matplotlib-base-3.0. | 6.6 MB    | ########## | 100%
gfortran_impl_linux- | 9.0 MB    | ########## | 100%
xorg-libxau-1.0.9    | 13 KB     | ########## | 100%
gcc_linux-64-7.3.0   | 10 KB     | ########## | 100%
r-stringr-1.4.0      | 207 KB    | ########## | 100%
matplotlib-3.0.2     | 6 KB      | ########## | 100%
pluggy-0.9.0         | 17 KB     | ########## | 100%
setuptools-40.8.0    | 626 KB    | ########## | 100%
fribidi-1.0.5        | 111 KB    | ########## | 100%
sphinx_rtd_theme-0.4 | 5.1 MB    | ########## | 100%
pandas-0.24.1        | 11.1 MB   | ########## | 100%
augustus-3.2.3       | 33.3 MB   | ########## | 100%
python-3.6.7         | 34.5 MB   | ########## | 100%
openssl-1.1.1a       | 4.0 MB    | ########## | 100%
binutils_impl_linux- | 16.5 MB   | ########## | 100%
gcc_impl_linux-64-7. | 73.2 MB   | ########## | 100%
pip-19.0.3           | 1.8 MB    | ########## | 100%
libuuid-1.0.3        | 34 KB     | ########## | 100%
r-base-3.5.1         | 39.8 MB   | ########## | 100%
r-r6-2.4.0           | 62 KB     | ########## | 100%
bioconductor-seqlogo | 371 KB    | ########## | 100%
gxx_impl_linux-64-7. | 18.7 MB   | ########## | 100%
python-dateutil-2.8. | 219 KB    | ########## | 100%
last-963             | 1.3 MB    | ########## | 100%
gxx_linux-64-7.3.0   | 9 KB      | ########## | 100%
binutils_linux-64-2. | 9 KB      | ########## | 100%
gsl-2.4              | 2.5 MB    | ########## | 100%
pytest-runner-4.4    | 8 KB      | ########## | 100%
py-1.8.0             | 65 KB     | ########## | 100%
curl-7.63.0          | 145 KB    | ########## | 100%
hmmer-3.2.1          | 7.1 MB    | ########## | 100%
libcurl-7.63.0       | 550 KB    | ########## | 100%
libssh2-1.8.0        | 239 KB    | ########## | 100%
atomicwrites-1.3.0   | 9 KB      | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... failed


CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-1.0.0-r351h29659fb_1000
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/amd64/r-cran-testrcpppackage_0.1.0-1_amd64.deb'
specified in the package manifest cannot be found.

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-1.0.0-r351h29659fb_1000
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/i386/r-cran-testrcpppackage_0.1.0-1_i386.deb'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: bioconda::perl-app-cpanminus-1.7043-pl5.22.0_0, bioconda::perl-uri-1.71-pl5.22.0_1, bioconda::perl-scalar-list-utils-1.45-2, bioconda::perl-db-file-1.835-5
  path: 'lib/perl5/5.22.0/x86_64-linux-thread-multi/perllocal.pod'

Second instance, same:

(eelpond-sw) diblions@js-16-78:~/eelpond$ ./run_eelpond examples/nema.yaml default

--------
checking for required files:
--------

        Added default parameters from rule-specific params files.
        Writing full params to examples/.ep_nema.yaml
--------
details!
        snakefile: /home/diblions/eelpond/Snakefile
        config: examples/nema.yaml
        params: examples/.ep_nema.yaml
        targets: ['default']
        report: '/home/diblions/eelpond/nema_out/logs/report.html'
--------
/opt/miniconda/envs/eelpond-sw/lib/python3.6/site-packages/snakemake/workflow.py:23: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from snakemake.shell import shell
Building DAG of jobs...
Creating conda environment rules/khmer/khmer-env.yaml...
Downloading remote packages.
Environment for rules/khmer/khmer-env.yaml created (location: .snakemake/conda/73fc3bf7)
Creating conda environment rules/sourmash/sourmash-env.yaml...
Downloading remote packages.
Environment for rules/sourmash/sourmash-env.yaml created (location: .snakemake/conda/76372394)
Creating conda environment rules/trinity/trinity-env.yaml...
Downloading remote packages.
Environment for rules/trinity/trinity-env.yaml created (location: .snakemake/conda/471e8fe4)
Creating conda environment rules/trimmomatic/trimmomatic-env.yaml...
Downloading remote packages.
Environment for rules/trimmomatic/trimmomatic-env.yaml created (location: .snakemake/conda/34ca9fad)
Creating conda environment rules/salmon/salmon-env.yaml...
Downloading remote packages.
Environment for rules/salmon/salmon-env.yaml created (location: .snakemake/conda/b6c11e18)
Creating conda environment rules/fastqc/fastqc-env.yaml...
Downloading remote packages.
Environment for rules/fastqc/fastqc-env.yaml created (location: .snakemake/conda/cb2e80bf)
Creating conda environment rules/dammit/dammit-env.yaml...
Downloading remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/diblions/eelpond/rules/dammit/dammit-env.yaml:
Collecting package metadata: ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... failed

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-1.0.0-r351h29659fb_1000
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/amd64/r-cran-testrcpppackage_0.1.0-1_amd64.deb'
specified in the package manifest cannot be found.

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-1.0.0-r351h29659fb_1000
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/i386/r-cran-testrcpppackage_0.1.0-1_i386.deb'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: bioconda::perl-app-cpanminus-1.7043-pl5.22.0_0, bioconda::perl-uri-1.71-pl5.22.0_1, bioconda::perl-scalar-list-utils-1.45-2, bioconda::perl-db-file-1.835-5
  path: 'lib/perl5/5.22.0/x86_64-linux-thread-multi/perllocal.pod'
johnsolk commented

Also DESeq2:

(eelpond-sw) diblions@js-16-78:~/eelpond$ ./run_eelpond DE.yml diffexp

--------
checking for required files:
--------

        Found input assembly at ../data/nema_trinity.fasta
        Added default parameters from rule-specific params files.
        Writing full params to .ep_DE.yml
--------
details!
        snakefile: /home/diblions/eelpond/Snakefile
        config: DE.yml
        params: .ep_DE.yml
        targets: ['diffexp', 'assemblyinput']
        report: '/home/diblions/eelpond/lab7_out_DE/logs/report.html'
--------
/opt/miniconda/envs/eelpond-sw/lib/python3.6/site-packages/snakemake/workflow.py:23: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from snakemake.shell import shell
Building DAG of jobs...
Creating conda environment rules/deseq2/deseq2-env.yaml...
Downloading remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/diblions/eelpond/rules/deseq2/deseq2-env.yaml:
Collecting package metadata: ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... failed

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-0.12.13-r3.4.1_0
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/amd64/r-cran-testrcpppackage_0.1.0-1_amd64.deb'
specified in the package manifest cannot be found.

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-0.12.13-r3.4.1_0
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/i386/r-cran-testrcpppackage_0.1.0-1_i386.deb'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::libgfortran-3.0.0-1, conda-forge::libgcc-7.2.0-h69d50b8_2
  path: 'lib/libgfortran.so.3'


ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::libgfortran-3.0.0-1, conda-forge::libgcc-7.2.0-h69d50b8_2
  path: 'lib/libgfortran.so.3.0.0'

On another instance, same thing:

(eelpond-sw) diblions@js-168-198:~/eelpond$ ./run_eelpond DE.yml diffexp

--------
checking for required files:
--------

        Found input assembly at ../data/nema_trinity.fasta
        Added default parameters from rule-specific params files.
        Writing full params to .ep_DE.yml
--------
details!
        snakefile: /home/diblions/eelpond/Snakefile
        config: DE.yml
        params: .ep_DE.yml
        targets: ['diffexp', 'assemblyinput']
        report: '/home/diblions/eelpond/lab7_out_DE/logs/report.html'
--------
/opt/miniconda/envs/eelpond-sw/lib/python3.6/site-packages/snakemake/workflow.py:23: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
  from snakemake.shell import shell
Building DAG of jobs...
Creating conda environment rules/trimmomatic/trimmomatic-env.yaml...
Downloading remote packages.
Environment for rules/trimmomatic/trimmomatic-env.yaml created (location: .snakemake/conda/34ca9fad)
Creating conda environment rules/deseq2/deseq2-env.yaml...
Downloading remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/diblions/eelpond/rules/deseq2/deseq2-env.yaml:
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.12
  latest version: 4.6.7

Please update conda by running

    $ conda update -n base conda


openssl-1.0.2q       | 3.2 MB    | ########## | 100%
xorg-libx11-1.6.7    | 940 KB    | ########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... failed


CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-0.12.13-r3.4.1_0
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/amd64/r-cran-testrcpppackage_0.1.0-1_amd64.deb'
specified in the package manifest cannot be found.

CondaVerificationError: The package for r-rcpp located at /opt/miniconda/pkgs/r-rcpp-0.12.13-r3.4.1_0
appears to be corrupted. The path 'lib/R/library/Rcpp/unitTests/bin/i386/r-cran-testrcpppackage_0.1.0-1_i386.deb'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::libgfortran-3.0.0-1, conda-forge::libgcc-7.2.0-h69d50b8_2
  path: 'lib/libgfortran.so.3'


ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::libgfortran-3.0.0-1, conda-forge::libgcc-7.2.0-h69d50b8_2
  path: 'lib/libgfortran.so.3.0.0'
bluegenes commented

Ok, looks like some packages may have changed locations. I'll look into it!

bluegenes commented

Update: Adding the r channel works, but it'd be best to avoid using another channel. Plus, the problem seems to be with r-rcpp, which has a more recent version in conda-forge than in r channel.

relevant: bioconda/bioconda-recipes#13578 (comment)

bluegenes commented

Could be a channel order error causing the underlying libgcc / libgfortran issues, or might just need ot pin a version. Relevant bioconda/bioconda-recipes#12100

note: conda-forge should now be first

channels:
  - conda-forge
  - bioconda
  - defaults
bluegenes commented

Seems this is an issue with instance templates pre-loaded with conda, which now contain some incompatible installations.

To solve, you need a fresh conda install. To fix on your existing instance:

  1. remove your conda installation. For the ggg jetstream template, use:
    sudo rm -rf /opt/miniconda
  2. download new miniconda & install in the same manner as previously installed. For the ggg v1.1 instances:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
sudo bash Miniconda3-latest-Linux-x86_64.sh -p /opt/miniconda

The ggg instance template will be updated using this strategy to prevent further issues.