Unable to build Apptainer image from nvcr.io/nvidia/quantum/cuda-quantum:0.7.0

[babreu-ncsa]

Required prerequisites

• Consult the security policy. If reporting a security vulnerability, do not report the bug using this form. Use the process described in the policy to report the issue.
• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker to verify that this hasn't already been reported. +1 or comment there if it has.
• If possible, make a PR with a failing test to give us a starting point to work on!

Describe the bug

I'm trying to build an Apptainer image with the 0.7.0 tag for our HPC+QC users. I've done that multiple times in the past with older versions, but for some reason, it is not working now. I keep getting the following error:

FATAL:   While performing build: conveyor failed to get: while converting reference: reading manifest 0.7.0 in nvcr.io/nvidia/quantum/cuda-quantum: requested access to the resource is denied

Steps to reproduce the bug

I'm building the container from this recipe file (omitting some %post installs for the sake of space):

BootStrap: docker
From: nvcr.io/nvidia/quantum/cuda-quantum:0.7.0
Stage: build

%post
    pip3 install --upgrade pip
    NOW=`date`
    echo "export NOW=\"${NOW}\"" >> $APPTAINER_ENVIRONMENT

%test
    echo "containers OS flavor:"
    cat /etc/os-release
    echo "python packages:"
    pip3 list

%runscript
    echo "This NGC-based CUDA Quantum container was created $NOW"
    echo "Arguments received: $*"
    "$@"

I tried multiple different ways to authenticate, including

apptainer registry login --username \$oauthtoken docker://nvcr.io

and by setting APPTAINER_DOCKER_USERNAME="\$oauthtoken" and APPTAINER_DOCKER_PASSWORD with my NGC API Key (I made sure to generate a new one several times). I also tried generating a Personal Key without success. I cannot go through when I call apptainer build.

Expected behavior

This should build the container.

Is this a regression? If it is, put the last known working version (or commit) here.

Not a regression

Environment

• CUDA Quantum version: 0.7.0
• Python version: 3.9.18
• C++ compiler: gcc/11.4.0
• Operating system: Linux

Suggestions

No response

[bmhowe23]

Hi @babreu-ncsa - nvcr.io/nvidia/quantum/cuda-quantum should be enabled for anonymous pulls, but I've been hearing of multiple users having problems if they were logged into nvcr.io with other credentials. Have you tried running docker logout nvcr.io and/or some version of apptainer registry logout docker://nvcr.io?

While this is definitely unintuitive and an undesirable long term solution, I just wanted to give you a quick option while we work through the permissions issues w/ nvcr.io.

[bmhowe23]

Hi @babreu-ncsa - we believe this is fixed. Would you please try again and let us know if we can close this issue? Thanks!

[babreu-ncsa]

Hi @bmhowe23, Thanks for quickly replying to this. I apologize for taking so long to follow up. I was finally able to build the container using the recipe above. It looks like authentication with an NGC API Key is no longer required, which is great.

I think I was stumbling on a different problem before, though, and it is probably Apptainer-related. Following https://apptainer.org/docs/user/main/docker_and_oci.html#nvidia-ngc, I was setting the env variables mentioned above. However, it looks like I had an old API key stored in $HOME/.apptainer/docker-config.json, and for some reason whatever is stored there seems to take precedence over these variables (although I did not directly confirm this). Anyway, I removed that JSON and cleared the Apptainer cache with apptainer cache clean and everything seems to be fine now!