UCL-ARC/hpc-spack

Stuck at configure stage of perl (in dependency tree of gromacs) even with binary mirror location added

krishnakumarg1984 opened this issue · 9 comments

krishnakumarg1984 commented

Sigh. I am trying to build gromacs@2023.1%gcc@12.2.0 ^openmpi@4.1.5 schedulers=sge %gcc@12.2.0 and it fails when trying to build perl from source (sigh. Why?) during the configure stage.

[uccagop@login11 hpc-spack]$ spack mirror list
buildcache      file:///shared/ucl/apps/spack/0.20/buildcache (fetch)
buildcache      file:///shared/ucl/apps/spack/0.20/buildcache (push)
spack-public    https://mirror.spack.io

I am using a clean ~/.bashrc & ~/.bash_profile (the default files provided on Micheal for new users).

krishnakumarg1984 commented 
# SPACKSITES: Now calling:  /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/share/spack/setup-env.sh spack gpg create ARCHPCSolutions rc-support@ucl.ac.uk
# SPACKSITES: have set SPACK_DISABLE_LOCAL_CONFIG=1
# SPACKSITES: have set HPC_SPACK_ROOT=/scratch/scratch/uccagop/spack/0.20/hpc-spack
# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack gpg create ARCHPCSolutions rc-support@ucl.ac.uk
gpg: keyring `/scratch/home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/opt/spack/gpg/secring.gpg' created
gpg: keyring `/scratch/home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/opt/spack/gpg/pubring.gpg' created
gpg: skipping control `%no-protection' ()
gpg: /scratch/home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/opt/spack/gpg/trustdb.gpg: trustdb created
gpg: key 98480850 marked as ultimately trusted
# SPACKSITES: Now calling:  /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/share/spack/setup-env.sh spack buildcache keys --install --trust
# SPACKSITES: have set SPACK_DISABLE_LOCAL_CONFIG=1
# SPACKSITES: have set HPC_SPACK_ROOT=/scratch/scratch/uccagop/spack/0.20/hpc-spack
# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack buildcache keys --install --trust
# SPACKSITES: Now calling:  /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/share/spack/setup-env.sh spack compiler find --scope=site
# SPACKSITES: have set SPACK_DISABLE_LOCAL_CONFIG=1
# SPACKSITES: have set HPC_SPACK_ROOT=/scratch/scratch/uccagop/spack/0.20/hpc-spack
# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack compiler find --scope=site
==> Added 4 new compilers to /scratch/home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/etc/spack/compilers.yaml
    intel@18.0.3  gcc@4.9.2  gcc@4.8.5  gcc@11.2.1
==> Compilers are defined in the following files:
    /scratch/home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/etc/spack/compilers.yaml
# SPACKSITES: Now calling: #######
 source /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/process-env-scripts/spack-deps-rhel-7.8.sh

source  /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/share/spack/setup-env.sh
spack config --scope=site get config
# SPACKSITES: ####################
# SPACKSITES: site.build_stage set to:  /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/../build_stage

# SPACKSITES: Adding  /scratch/scratch/uccagop/spack/0.20/hpc-spack  to PYTHONPATH
# SPACKSITES: in app.py function: install_env
/home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2
# SPACKSITES: Now calling: #######
 source /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/process-env-scripts/spack-deps-rhel-7.8.sh

source  /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/share/spack/setup-env.sh
spack config --scope=site get config
# SPACKSITES: ####################
# SPACKSITES: site.build_stage set to:  /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/../build_stage
# SPACKSITES: Now calling:  /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/share/spack/setup-env.sh spack env create first_compiler_spack_env /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/spack-env-templates/dev1/first_compiler.yaml
# SPACKSITES: have set SPACK_DISABLE_LOCAL_CONFIG=1
# SPACKSITES: have set HPC_SPACK_ROOT=/scratch/scratch/uccagop/spack/0.20/hpc-spack
# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack env create first_compiler_spack_env /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/spack-env-templates/dev1/first_compiler.yaml
==> Created environment 'first_compiler_spack_env' in /scratch/home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/var/spack/environments/first_compiler_spack_env
==> You can activate this environment with:
==>   spack env activate first_compiler_spack_env
# SPACKSITES: Now calling:  /scratch/scratch/uccagop/spack/0.20/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh /home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/share/spack/setup-env.sh spack -e first_compiler_spack_env install
# SPACKSITES: have set SPACK_DISABLE_LOCAL_CONFIG=1
# SPACKSITES: have set HPC_SPACK_ROOT=/scratch/scratch/uccagop/spack/0.20/hpc-spack
# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack -e first_compiler_spack_env install

$ spack gpg trust /shared/ucl/apps/spack/0.20/buildcache/build_cache/_pgp/9EDD6764E6A3D5145ABF8E6EFD20819772643654.pub
gpg: key 72643654: public key "ARCHPCSolutions (GPG created for Spack) <rc-support@ucl.ac.uk>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg: inserting ownertrust of 6
heatherkellyucl commented

I don't think it should be finding this many compilers (but that may be irrelevant):

==> Added 4 new compilers to /scratch/home/uccagop/apps/spack/0.20/kg_spacksite_michael_ucl2/spack/etc/spack/compilers.yaml
    intel@18.0.3  gcc@4.9.2  gcc@4.8.5  gcc@11.2.1

ETA: yes, that's irrelevant, it comes from the default modules and the system compiler and shouldn't be causing any issues as we tell Spack specifically what compiler to use.

You've not included any output where it is building perl.

krishnakumarg1984 commented

It's just stuck there when trying to build perl with no output (same as when Ian & I were debugging). I think the difference is that recently I closed the tmux session and started it again, and the buildcache keys are showing up as empty and it is trying to build everything from source.

krishnakumarg1984 commented

Every single dependency, it is trying to build from source:

==> Installing libsigsegv-2.13-2ih3zaeb3noq3n6f33jumfjpab4oknse
==> No binary for libsigsegv-2.13-2ih3zaeb3noq3n6f33jumfjpab4oknse found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/be/be78ee4176b05f7c75ff03298d84874db90f4b6c9d5503f0da1226b3a3c48119.tar.gz
==> No patches needed for libsigsegv
==> libsigsegv: Executing phase: 'autoreconf'
==> libsigsegv: Executing phase: 'configure'
krishnakumarg1984 commented 
gpg: key 72643654: "ARCHPCSolutions (GPG created for Spack) <rc-support@ucl.ac.uk>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

The keys are trusted, but still builds everything from source.

heatherkellyucl commented

The question is, is this first_compilers that is causing these builds or is it something else in the gromacs chain that needs Perl? There are some legitimate reasons that we'd end up building a new perl with with the gcc 12 compiler, as the one that we build in order to build the compiler is with gcc 11.

(You should be building gromacs into a site that you already set up and has first compilers done in it).

heatherkellyucl commented

From discussion, it sounds like the problem is when trying to go back to an already created site, and everything works as expected when creating a site from scratch.

Check if we are missing any steps on re-activating an existing site and making sure it can still see and trust the buildcache.

heatherkellyucl commented

Test reactivating a site

I am working in /home/cceahke/Scratch/spack-test/0.20 on Michael and have checked out hpc-spack into there.

I have git-pull-spacksites-personal.sh and init-spacksites-personal.sh in the 0.20 directory to set my personal root and sites_root.

I have updated hpc-spack/spacksites/settings/spack_sites.ini to this:
sites_root = /home/cceahke/Scratch/spack-test/sites/0.20

Then in hpc-spack:

source ../init-spacksites-personal.sh 
spacksites/spacksites create envtest
eval $(spacksites/spacksites spack-setup-env envtest)
spack gpg trust /shared/ucl/apps/spack/0.20/buildcache/build_cache/_pgp/9EDD6764E6A3D5145ABF8E6EFD20819772643654.pub
spacksites/spacksites install-env envtest compilers first_compiler.yaml

Interrupted the install during libxml2-2.10.3-lfr5oagid5up7b4czil2auycgvewyvfi - see if I can exit this shell, start in a new one, re-source and continue.

heatherkellyucl commented

(It does make sense that openmpi needs a perl built with gcc 12, btw - what I am testing is whether trying to re-initialise a shell in the same environment works).

Test continues:

In new shell, in hpc-spack:

source ../init-spacksites-personal.sh 
eval $(spacksites/spacksites spack-setup-env envtest)
spack gpg list
# shows I still have two keys in my keyring, and second id is the same as the buildcache one I trusted.

Then rerun this to continue.

spacksites/spacksites install-env envtest compilers first_compiler.yaml

Does setup, sees that compilers already exists as an environment, continues to install into it:

# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack env create compilers /scratch/scratch/cceahke/spack-test/0.20/hpc-spack/s
packsites/spack-env-templates/dev1/first_compiler.yaml
==> Error: 'compilers': environment already exists at /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/var/spack/environments/compilers
# SPACKSITES: Now calling:  /scratch/scratch/cceahke/spack-test/0.20/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh /home/cceahke/Scratch/spack-t
est/sites/0.20/envtest/spack/share/spack/setup-env.sh spack -e compilers install
# SPACKSITES: have set SPACK_DISABLE_LOCAL_CONFIG=1
# SPACKSITES: have set HPC_SPACK_ROOT=/scratch/scratch/cceahke/spack-test/0.20/hpc-spack
# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack -e compilers install

Sees the stuff it already installed, and continues with readline from the buildcache.

[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/libiconv-1.16-uaerdnaqvy2hrkk2gkiirw5
mqkq7cg44
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/libsigsegv-2.13-qw2elykliqxp2xlix4zmk
run2lmmiklv
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/berkeley-db-18.1.40-6pqhjoddsyu4zfo4x
yweynn5s2wsr7dh
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/ncurses-6.3-6nrbosy4yebkogx2wiyom6ez3
6wxiz6g
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/zlib-1.2.13-gsyujwxfnx6vvfw46b4t3x63m
c5xh3a5
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/autoconf-archive-2022.02.11-rmhkp3hczdktiv6co2buqt7gunykg7pb
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/pkgconf-1.8.0-rrab7kijpl5g2acssdn24cszudxvi3xn
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/xz-5.4.1-7ibfohuogs4khc4afhlryzrtxzkatf6l
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/zstd-1.5.2-myexac2eokkvlqpva26kef25kjwie4ju
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/gmake-4.4.1-3yeveltufgmue74npuzemeaxfiivvt6k
[+] /scratch/scratch/cceahke/spack-test/sites/0.20/envtest/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-11.2.1/diffutils-3.8-ehpjj76pln2dakw5zahbo7eztvsb3jha
==> Installing readline-8.1-jtrpaefvakqxyzbo3m72q6hcnuywt4uu
==> Fetching file:///shared/ucl/apps/spack/0.20/buildcache/build_cache/linux-rhel7-broadwell-gcc-11.2.1-readline-8.1-jtrpaefvakqxyzbo3m72q6hcnuywt4uu.spec.json.sig
gpg: Signature made Wed 02 Aug 2023 16:26:41 BST using RSA key ID 72643654
gpg: Good signature from "ARCHPCSolutions (GPG created for Spack) <rc-support@ucl.ac.uk>"
==> Fetching file:///shared/ucl/apps/spack/0.20/buildcache/build_cache/linux-rhel7-broadwell/gcc-11.2.1/readline-8.1/linux-rhel7-broadwell-gcc-11.2.1-readline-8.1-jtrpaefvakqxyzbo3m72q6hcnuywt4uu.spack
==> Extracting readline-8.1-jtrpaefvakqxyzbo3m72q6hcnuywt4uu from binary cache
==> readline: Successfully installed readline-8.1-jtrpaefvakqxyzbo3m72q6hcnuywt4uu
  Search: 0.00s.  Fetch: 0.77s.  Install: 0.35s.  Total: 1.13s

This seems fine. And it completed and has all 30 installed buildcache packages.