Dependabot adding index to Pipfile.lock causes issue in newest version of pipenv
Opened this issue · 10 comments
Is there an existing issue for this?
- I have searched the existing issues
Package ecosystem
pipenv
Package manager version
No response
Language version
python 3.9
Manifest location and content before the Dependabot update
No response
dependabot.yml content
We have a private repo where we pull all our dependencies from:
version: 2
registries:
python-codeartifact:
type: python-index
url: https://REDACTED.d.codeartifact.us-east-1.amazonaws.com/pypi/REDACTED/simple/
username: aws
password: ${{ secrets.CODE_ARTIFACT_TOKEN }}
replaces-base: true
updates:
- package-ecosystem: 'pip'
directory: '/'
insecure-external-code-execution: allow
open-pull-requests-limit: 0
schedule:
interval: 'weekly'
registries:
- python-codeartifact
reviewers:
- org/security-team
Updated dependency
No response
What you expected to see, versus what you actually saw
Dependabot created a PR fixing some vulnerability, and as per #7744, we saw a new behaviour where it included an index called dependabot-inserted-index-0 in some of the transitive dependencies in our Pipfile.lock, for example:
"freezegun": {
"hashes": [
"sha256:cd22d1ba06941384410cd967d8a99d5ae2442f57dfafeff2fda5de8dc5c05446",
"sha256:ea1b963b993cb9ea195adbd893a48d573fda951b0da64f60883d7e988b606c9f"
],
"index": "dependabot-inserted-index-0",
"version": "==1.2.2"
},
It added that index to packages that were updated, but also to some that were not updated. It didn't add the index to all the packages. See diff below:
Our build pipelines were able to sync the dependencies when they were using an old version of pipenv, for example v2022.1.8 (we haven't found which version is the newest that still works):
Successfully installed pipenv-2022.1.8
..........................................................................................................
·✔ Successfully created virtual environment!
--
179 | Virtualenv location: /root/.local/share/virtualenvs/authorizer-0svWqLcN
180 | Installing dependencies from Pipfile.lock (923e0b)...
181 | To activate this project's virtualenv, run pipenv shell.
182 | Alternatively, run a command inside the virtualenv with pipenv run.
183 | All dependencies are now up-to-date!
However, when we updated to the latest version of pipenv (v2023.8.28), we encountered this error:
Successfully installed pipenv-2023.8.28
..........................................................................................................
✔ Successfully created virtual environment!
--
123 | Virtualenv location: /root/.local/share/virtualenvs/authorizer-jszj7Vks
124 | Installing dependencies from Pipfile.lock (923e0b)...
125 | Unable to find dependabot-inserted-index-0 in sources, please check
126 | dependencies: ['boto3==1.26.126
127 | --hash=sha256:a0a049d16dabf559bd86d4622a985549154c6918f779f373fb7588bd3cf1272d
128 | --hash=sha256:ea7dc518399106ee300b867596c92d2b7b498ee7721db935915cdd53339a9f92',
129 | 'cachetools==5.3.0
130 | --hash=sha256:13dfddc7b8df938c21a940dfa6557ce6e94a2f1cdfa58eb90c805721d58f2c14
131 | --hash=sha256:429e1a1e845c008ea6c85aa35d4b98b65d6a9763eeef3e37e92728a12d1de9d4',
132 | 'evertz-io-observability==3.0.0
133 | --hash=sha256:aa088284d216affc085b18dfb1ea0c68db05e49d1f7708287c472ba46c47cca1
134 | --hash=sha256:f8221c9d487792a8317872c03ad10580a4a0420044e3b8f1bb47df1ace99173a',
135 | 'opentelemetry-api==1.16.0
136 | --hash=sha256:4b0e895a3b1f5e1908043ebe492d33e33f9ccdbe6d02d3994c2f8721a63ddddb
137 | --hash=sha256:79e8f0cf88dbdd36b6abf175d2092af1efcaa2e71552d0d2b3b181a9707bf4bc',
138 | 'opentelemetry-sdk==1.16.0
139 | --hash=sha256:15f03915eec4839f885a5e6ed959cde59b8690c8c012d07c95b4b138c98dc43f
140 | --hash=sha256:4d3bb91e9e209dbeea773b5565d901da4f76a29bf9dbc1c9500be3cabb239a4e',
141 | 'pytest==7.2.1
142 | --hash=sha256:c7c6ca206e93355074ae32f7403e8ea12163b1163c976fee7d4d84027c162be5
143 | --hash=sha256:d45e0952f3727241918b8fd0f376f5ff6b301cc0777c6f9a556935c92d8a7d42',
144 | 'python-jose==3.3.0
145 | --hash=sha256:55779b5e6ad599c6336191246e95eb2293a9ddebd555f796a65f838f07e5d78a
146 | --hash=sha256:9b1376b023f8b298536eedd47ae1089bcdb848f1535ab30555cd92002d78923a',
147 | 'requests==2.31.0
148 | --hash=sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f
149 | --hash=sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1']
Let us know if there is anything else that we could provide to help troubleshooting this.
Thanks!
Native package manager behavior
After manually updating the dependencies with the latest version of pipenv, that index added by dependabot disappeared and our build pipelines were able to sync again. This is the change after doing a pipenv update manually:
Images of the diff or a link to the PR, issue, or logs
Smallest manifest that reproduces the issue
No response
@juanitosvq Does this still repro now that Dependabot is on a newer version of pipenv?
Since we got no answer for a while, let's assume this is fixed, and reopen if we're wrong!
Sorry for the late reply. I asked dependabot to recreate an old pull request and the new proposed change still shows the same wrong behaviour. I've been trying to find time to run the update with the CLI, I'll see if I can get to it soon, then I can reopen this.
We are trying to run the dependabot CLI to reproduce this, but having issues with authenticating against our private CodeArtifact repo: dependabot/cli#196
@deivid-rodriguez I am also experiencing this issue with the latest version of Dependabot. Can you please reopen?
Dependabot is adding dependabot-inserted-index-0 to our Pipfile.lock. When our CI / CD runner does a pipenv install --deploy, it does not have that index, which causes the install to fail.
Using pipenv-2023.11.15 in the Dependabot workflow
I spent a bit of time today looking at this and these are my findings:
Our Pipfile uses an env var with the CodeArtifact authentication token, so we define our sources like this:
[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@name-123456789012.d.codeartifact.us-east-1.amazonaws.com/pypi/evertz-io-repo/simple/"
verify_ssl = true
name = "my-repo"
[packages]
urllib3 = "==1.25.11"
cryptography = "==41.0.4"
[requires]
python_version = "3.9"
We pass that token to the dependabot config in our dependabot.yaml through a GH dependabot secret:
version: 2
registries:
python-codeartifact:
type: python-index
url: https://name-123456789012.d.codeartifact.us-east-1.amazonaws.com/pypi/evertz-io-repo/simple/
username: aws
password: ${{secrets.CODE_ARTIFACT_TOKEN_EVERTZ_IO}}
replaces-base: true
With this setting, the dependabot CLI always fails with a dependency_file_not_resolvable because it's unable to authenticate against our private artifact repository. So unfortunately I can't get any more details from there.
The dependabot running in the server is actually able to update my dependencies and creates a PR for it, but this PR has that problematic index ("index": "dependabot-inserted-index-0") in the Pipfile.lock:
I tried replacing that env var from our Pipfile with the hardcoded token, and it that case both the dependabot CLI was able to finish successfully and also the github dependabot was able to create a PR without the dependabot-inserted-index-0 index in our Pipfile.lock:
Because that token expires, I can't really use the hardcoded version (it would also be pretty unsecure). So unfortunately we are still affected by this issue.
Any thoughts? I could try to open an issue in the CLI repo and see if I could run the dependabot locally with the env var representing the token, but I noticed the CLI won't create a PR so I don't know if I'll be able to troubleshoot much.
@juanitosvq this is a good find! I also have run into this issue with an env var in the Pipfile url:
[[source]]
url = "${PIP_INDEX_URL}"
verify_ssl = true
name = "custom"Any traction on the fix for this issue coming soon? We have a few repositories that consistently fail on dependabot updates regarding? @jeffwidman maybe?
I believe the pipenv 2024.3.0 fixes similar sounding issue -- let me know if dependabot team needs any support testing this new release.