Is it possible to use artifacts-keyring with a Service Principal?

Question

Is it possible to use artifacts-keyring with a Service Principal?

NathanielMcVicar opened this issue a year ago · 16 comments

I'm trying to publish wheels to a DevOps Artifact feed from a GitHub action with twine. I have a Service Principal with access, based on the steps in https://github.com/MicrosoftDocs/azure-devops-docs/issues/8141#issuecomment-1548825563, but in the past we've always used PATs with artifacts-keyring (typically through VSS_NUGET_EXTERNAL_FEED_ENDPOINTS). You can't create a PAT for a Service Principal, so I'm trying to determine if there is any alternative available. Thanks!

Answer 1 · 2023-05-26T10:01:37.000Z

in trying this too. i also use vss extensions endpoints. so far that thread doesnt work on my end even with the SP setup, azure artifacts does not even accept the command.

so at this point we are still using a PAT

Answer 2 · 2023-05-30T14:11:36.000Z

We don't have specific support for SP/MI in artifacts-keyring (or artifacts-credprovider which it uses behind the scenes) at this time, but if you can get an AAD access token (e.g. from az account get-access-token), that should work as the password with an arbitrary username, in the same way a PAT would

Answer 3 · 2023-05-30T14:19:07.000Z

We don't have specific support for SP/MI in artifacts-keyring (or artifacts-credprovider which it uses behind the scenes) at this time, but if you can get an AAD access token (e.g. from az account get-access-token), that should work as the password with an arbitrary username, in the same way a PAT would

so theoretically something like this?
- name: Azure Login
uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}

- name: Get AAD Access Token
  id: get_aad_token
  run: |
    echo "::set-output name=token::$(az account get-access-token --query accessToken -o tsv)"

- name: Setup .NET
  uses: actions/setup-dotnet@v1
  with:
    dotnet-version: 3.1

- name: Config NuGet to use Azure Artifacts
  env:
    TOKEN: ${{ steps.get_aad_token.outputs.token }}
  run: |
    dotnet nuget add source https://pkgs.dev.azure.com/<YourOrganization>/<YourProject>/_packaging/<YourFeed>/nuget/v3/index.json -n azure -u azuresdk -p $TOKEN --store-password-in-clear-text

- name: Restore dependencies
  run: dotnet restore

Answer 4 · 2023-05-30T14:31:23.000Z

I'm not really familiar with GH Actions, but that looks like it should work. You might consider setting up the environment variables to use with artifacts-credprovider rather than using dotnet nuget add source. Doing so would avoid persisting the token on disk, and some systems log command lines.

Answer 5 · 2023-06-08T21:55:11.000Z

This worked great, I really appreciate the advice! However, I wonder is there any plan to set up some way to use OIDC or SPs directly with artifacts-credprovider, so we don't have to generate an intermediate bearer token? This seems like it would be a great feature, but I don't know how feasible it would be.

Answer 6 · 2023-06-08T21:58:22.000Z

This would be great and seeing as github is part of microsoft it would be extremely helpful and secure; so We just have to let them know how desired this is… imo it would be a great feature for those who do not want to migrate legacy artifact feeds…

Answer 7 · 2023-07-28T08:29:06.000Z

Hello @jmyersmsft,

Do you have any plans of supporting the SPA?

We want to run deployments from our on premise machines and this is causing a lot more issues than we could imagine.

Answer 8 · 2023-07-31T20:41:34.000Z

As @jmyersmsft mentioned, the keyring runs the artifacts-credprovider behind the scenes. There is an issue on that repo tracking this enhancement here.

Answer 9 · 2023-08-11T03:34:42.000Z

All right. I suffered enough through this that I thought to post my solution here in case anybody in the future struggles like I did. For context, I'm using Azure ML Pipelines and authenticating to Azure DevOps using a managed identity. The assumption is that you can authenticate with the managed identity using the az login --identity --username $DEFAULT_IDENTITY_CLIENT_ID command. Here's how you can get a package using pip (adding previous steps in case you need them):

echo "Creating and activating Python virtual environment"
# This is not required but a good practice
python -m venv my-env
source my-env/bin/activate
# You can use Azure CLI
echo "Installing Azure CLI"
curl -sL https://aka.ms/InstallAzureCLIDeb | bash
echo "Logging in to Azure CLI"
az login --identity --username $DEFAULT_IDENTITY_CLIENT_ID
echo "Getting token"
TOKEN=$(az account get-access-token --query accessToken -o tsv)
echo "Install packages"
pip install <package> --index-url "https://$TOKEN@<Organization>.pkgs.visualstudio.com/<Project>/_packaging/<Feed>/pypi/simple/"

This is how you can install Python packages from Azure Artifacts until artifacts-keyring supports non-interactive authentication with managed identities. Your managed identity will need Reader permissions on Azure DevOps.

Thanks @John-Donalson! Your solution helped me get mine.

Answer 10 · 2023-08-11T13:56:19.000Z

Thanks for sharing that, @novablinkicelance! I would make two minor suggestions:

Add a username (exact value doesn't matter) to the URL so that the token is the password. Systems are more likely to log usernames from Basic auth than passwords and as written, the token is the username. E.g.: https://token:$TOKEN@<org>.pkgs.... To be clear, Azure Artifacts does not log the username, but software or devices on your machine or network theoretically might.
Use the PIP_INDEX_URL environment variable instead of the --index-url command line parameter. Some systems log process command lines.

Answer 11 · 2024-06-26T20:29:31.000Z

0.4.0 now natively supports MI / SP set up instructions can be found on the microsoft/artifacts-credprovider#492. Please let me know if you have and feedback or find any issues!

Answer 12 · 2024-09-06T10:03:45.000Z

@embetten
Using 0.4.0rc0 I'm unable to download a library from azure devops feed as artifacts keyrings hangs.

I'm using a SP with a certificate and yes I've given it enough permissions to read from the devops feed :)

The environment variable is correctly set

export ARTIFACTS_CREDENTIALPROVIDER_FEED_ENDPOINTS="{\"endpointCredentials\": [{\"endpoint\":\"https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/\", \"clientId\":\"c88bbab3-XXXYYYZZZ-aa95-3e6283abb01a\", \"clientCertificateFilePath\":\"/Users/path_to_cert.pem\"}]}"

Using pip 24.2 from /opt/miniconda3/envs/giglio/lib/python3.12/site-packages/pip (python 3.12)
Non-user install because site-packages writeable
Created temporary directory: /private/var/folders/s2/9r083pc53lvf85c43mhl1bzc0000gn/T/pip-build-tracker-or1zyieu
Initialized build tracking at /private/var/folders/s2/9r083pc53lvf85c43mhl1bzc0000gn/T/pip-build-tracker-or1zyieu
Created build tracker: /private/var/folders/s2/9r083pc53lvf85c43mhl1bzc0000gn/T/pip-build-tracker-or1zyieu
Entered build tracker: /private/var/folders/s2/9r083pc53lvf85c43mhl1bzc0000gn/T/pip-build-tracker-or1zyieu
Created temporary directory: /private/var/folders/s2/9r083pc53lvf85c43mhl1bzc0000gn/T/pip-install-gk8n37kd
Created temporary directory: /private/var/folders/s2/9r083pc53lvf85c43mhl1bzc0000gn/T/pip-ephem-wheel-cache-owzqeyx8
Looking in indexes: https://pypi.org/simple, https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/
2 location(s) to search for versions of my-library:
* https://pypi.org/simple/my-library/
* https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/my-library/
Fetching project page and analyzing links: https://pypi.org/simple/my-library/
Getting page https://pypi.org/simple/my-library/
Found index url https://pypi.org/simple/
Looking up "https://pypi.org/simple/my-library/" in the cache
Request header has "max_age" as 0, cache bypassed
No cache entry available
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/my-library/ HTTP/1.1" 404 13
Status code 404 not in (200, 203, 300, 301, 308)
Could not fetch URL https://pypi.org/simple/my-library/: 404 Client Error: Not Found for url: https://pypi.org/simple/my-library/ - skipping
Fetching project page and analyzing links: https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/my-library/
Getting page https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/my-library/
Found index url https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/
Looking up "https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/my-library/" in the cache
Request header has "max_age" as 0, cache bypassed
No cache entry available
Starting new HTTPS connection (1): pkgs.dev.azure.com:443
https://pkgs.dev.azure.com:443 "GET /MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/my-library/ HTTP/1.1" 401 343
Found index url https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/
Keyring provider requested: auto
Keyring provider set: import
Getting credentials from keyring for https://pkgs.dev.azure.com/MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/
Loading ArtifactsKeyringBackend
Loading KWallet
Loading SecretService
Loading Windows
Loading chainer
Loading libsecret
Loading macOS
Starting new HTTPS connection (1): pkgs.dev.azure.com:443
https://pkgs.dev.azure.com:443 "GET /MY-ORG-NAME/my-library-feed-name/_packaging/cda-dat-ml-pypi/pypi/simple/ HTTP/11" 401 343
[Information] [CredentialProvider]VstsBuildTaskServiceEndpointCredentialProvider - Using certificate: CN=bengio, OU=DataScience, O=Prometeia, L=Milan, S=Milan, C=IT.

Then after a while

Keyring is skipped due to an exception
Traceback (most recent call last):
  File "/opt/miniconda3/envs/giglio/lib/python3.12/site-packages/pip/_internal/network/auth.py", line 272, in _get_keyring_auth
    return self.keyring_provider.get_auth_info(url, username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/giglio/lib/python3.12/site-packages/pip/_internal/network/auth.py", line 87, in get_auth_info
    cred = self.keyring.get_credential(url, username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/giglio/lib/python3.12/site-packages/keyring/core.py", line 80, in get_credential
    return get_keyring().get_credential(service_name, username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/giglio/lib/python3.12/site-packages/keyring/backends/chainer.py", line 69, in get_credential
    credential = keyring.get_credential(service, username)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/giglio/lib/python3.12/site-packages/artifacts_keyring/__init__.py", line 58, in get_credential
    username, password = provider.get_credentials(service)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/giglio/lib/python3.12/site-packages/artifacts_keyring/plugin.py", line 68, in get_credentials
    username, password = self._get_credentials_from_credential_provider(url, is_retry=False)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/giglio/lib/python3.12/site-packages/artifacts_keyring/plugin.py", line 125, in _get_credentials_from_credential_provider
    raise RuntimeError("Failed to get credentials: process with PID {pid} exited with code {code}; additional error message: {error}"
RuntimeError: Failed to get credentials: process with PID 82445 exited with code 2; additional error message: 
WARNING: Keyring is skipped due to an exception: Failed to get credentials: process with PID 82445 exited with code 2; additional error message: 
Keyring provider requested: auto
Keyring provider set: disabled
User for pkgs.dev.azure.com:

Answer 13 · 2024-09-19T01:04:12.000Z

Thanks for the feed back - looks like we forgot pem certificates and this version is only supporting .pfx at the moment. We will look into fixing this.

Answer 14 · 2024-09-19T11:59:49.000Z

Thank you @embetten ... are you also aware that current version of artifacts keyring is 0.4.0rc0 and this version is filtered out by some client like poetry ... they need a genuine stabile version like 0.4.0

Answer 15 · 2024-09-19T16:12:26.000Z

@noiano thanks for the heads up, I did not know about this poetry requirement, but the pre-release version at this time was intentional to get this kind of feedback. We are working to get a stable version out with this feature.

Answer 16 · 2024-09-20T07:38:41.000Z

Hello @embetten I have to amend my previous comment ... I found a way to force poetry to use non stable version of artifacts-keyring by running

poetry self add artifacts-keyring==0.4.0rc0

Unfortunately it crashes without providing hints about what went wrong 😞

Pip works perfectly with pfx certificate