It looks like the groups option is not working
Closed this issue · 42 comments
Hello is it correct that this (beta) option is not supported? And if not are there any plans to add support to it?
It would be extremely helpful
Correct.
This is not yet supported.
PRs are welcome though it may require building an updater similar to the GitHub's version.
@mburumaxwell Name a bounty for this and I'll sponsor you - no joke. This has been my #1 feat missing that I've been missing from Renovate and I really need this <3
@ColinKrist I can also help to match a bounty if someone wants to pick it up. Maybe something that can be listed on Bounty Source?
This is quite an interesting proposition. Curious what the offers actually are ...
I was going to try and DM you on Twitter / GitHub but your DMs are closed. Does $75 USD sound fair - wanted to ask about difficulty because I'd be willing to get this expensed from my employer. LMK
Opened Twitter DMs incase you need that.
I have no particular inclination on the bounty besides curiosity. Maybe I will once I have a clue on the amount of effort required.
Meanwhile, maybe you should post all the bounties on Bounty Source like this one https://app.bountysource.com/issues/123710224-pnpm-support
Done,
https://app.bountysource.com/issues/123710276-it-looks-like-the-groups-option-is-not-working
The azure pipeline task ecosystem community is small. I definitely want to give back and support you guys where I can and make sure you guys feel like the work amounts to something.
FYI @mburumaxwell has set the bounty for this at $500.
I've just committed to the cause as I agree this is a much-needed feature. If you're interested in getting this feature implemented and are able to contribute to funding the bounty, please do so using the link posted by @ColinKrist above.
You may have to use GH sponsors at your own convenience. Bountysource has serious problems:
https://github.com/bountysource/core/wiki/Frequently-Asked-Questions#can-i-receive-a-refund-for-a-bounty-i-posted
https://bountysource.com/contact-us/
I've sent in my request, but who knows how long it'll be in limbo.
@mburumaxwell once I get this money back I'll consider sending this over via GH. I'm hoping for something a bit more public to prevent misuse / guarantee the work will be done.
Will you start the work before receiving the funds for this feature? I should have set an expiration time on the bounty to prevent feature work limbo like this, so I apologize to anyone who has added funds to the bounty.
@ColinKrist No worries, none of us knew of the problems. Is the intention to withdraw the bounty and fund through another mechanism?
I paid through PayPal but convinced my (apparently rightly skeptical) organisation to reimburse me - I'll open a dispute with PayPal and seek a refund, then figure out how I can pay my org back.
The more important thing is that you get your money back instead of loosing it. Sponsoring on GH will be your choice and at your convenience.
Will the work be done before that? Yes, it appears we need PR grouping internally too. However, I can't promise any timelines because I know the amount of changes required.
@mburumaxwell wrote:
" because I know the amount of changes required."
Would you be able to share a rough outline of what you think needs to be done? I'm curious to give this a go myself but I don't know this codebase, so it'd be very helpful to hear an analysis from somebody who does.
Please do not use bountysource. Many devs have had trouble getting paid there. You can check out this lemmy community as an alternative https://lemmy.ml/c/bugbounties
For statements from devs who have been unable to cash out from bountysource see:
https://github.com/bountysource/core/issues
It is now available in the stable version:
https://github.blog/2023-08-24-a-faster-way-to-manage-version-updates-with-dependabot/
@mburumaxwell Can this maybe be included now?
Not yet but probably won't decline a contribution.
@RoystonS first step is to understand how grouping works in dependabot. Then plug that into the updater script while taking into account merge conflict resolution and closing of unwanted PRs.
I pulled in code from the official updater but I haven't tested and the server side is not yet 100% ready to support it.
This still something that is being looked at implemented?
This still something that is being looked at implemented?
Yes but improving the updater to avoid the very lengthy file needs to happen first. Unfortunately, that seems to have stalled due to the amount of time it requires. Copying from the official updater does not seem to be a solution due to its complexity; meaning we need to write our own bearing in mind testability, resolution of merge conflicts for groups etc. It is also entirely possible that I am looking at this from the wrong angle and another set of eyes could go about it differently; that's why it is open source.
Hi, there are any other news on that? This would be a very nice feature
For the benefit of those looking on wondering why this isn't "just being done"... From what I can see, the difficulty is that, due to the way the original GitHub dependabot code is structured, quite a bit of logic from that codebase needs to be duplicated/forked in this package in order to make it work with ADO. It isn't simply a case of providing an ADO-specific implementation of some nice clean abstractions. This means that features like grouping require a lot more specific code in this repo than might otherwise be the case.
Really looking forward for grouping support. It is indeed the only feature we are currently missing for our configuration
Is there an update for this issue? We are really looking forward to it as well
Any update on this?
We are looking forward to have grouping support as well.
No there are no updates.
At this point, I am fairly certain that this will have to be a community contribution as we can't allocate time to anything significantly new here such as grouping support.
I will leave this issue open for tracking purposes only. Should things change, someone will report back here. Until then, hit that snooze button.
@rhyskoedijk That's an immense contribution, thank you for the time and effort you've put into those PRs. I'm sure that this work will benefit a lot of people once it has been merged.
I previously contributed to a bounty for this feature, but the bounty ended up being cancelled. I'd be more than happy to re-contribute that money over you once this feature has been merged in. If you enable sponsorship on your GitHub profile, that is probably the best way for us to sponsor you.
I'd be more than happy to re-contribute that money over you once this feature has been merged in.
I appreciate the offer, but that's not necessary. If anybody, sponsor the owner of the repo for keeping this project going and actively maintaining it.
Thanks for your work, i'm looking forward to try this out.
Should this already work (Image 1.30)?
Yes images from 1.30.0 onwards have this. If you are using the image directly replace update_script with update_script_vnext. If you are using the task:
- task: dependabot@1
inputs:
useUpdateScriptvNext: true
# optionally pin the tag
# dockerImageTag: latest # or '1.30.2-ci0005' if you are pinning some recent changesOnly pin the version of the docker image if you have specific changes you want.
I would very much to hear if there are any issues coming from non-nuget updates.
I've tried the new version. The grouping works great, thank you!
The only downside I've seen is that the PR description is empty. Maybe it was too long since I had 14 updated packages.
I've been using vNext for the last week on about 10 repos with a mixture of NuGet, NPM, and Yarn. So far it has been working well with the exception of a few issues already known and reported in dependabot-core.
The only downside I've seen is that the PR description is empty. Maybe it was too long since I had 14 updated packages.
I've had this too in large group PRs. Unfortunately Azure DevOps has a 4000 character limit for PR description which does seem to cause problems sometimes. If Dependabot truncates the description in the wrong place it can cause HTML tags to not be closed properly, code blocks to not be terminated properly, etc.
Give this an upvote or comment:
https://developercommunity.visualstudio.com/t/raise-the-character-limit-for-pull-request-descrip/365708
@rhyskoedijk Are your NPM repositories in a private Azure Artifacts feed? I've just tried implementing grouping for NPM but now it fails auth for me. NuGet is authenticating with no problems.
NuGet Credentials (working):
##[debug]/usr/bin/docker arg: ["-e","DEPENDABOT_EXTRA_CREDENTIALS=[{\"type\":\"nuget_feed\",\"token\":\"PAT:myAccessCode\",\"url\":\"[{feedUrl}/nuget/v3/index.json\"},{\"type\":\"nuget_feed\",\"url\":\"https://api.nuget.org/v3/index.json\"}]"]({feedUrl}/nuget/v3/index.json/%22%7D,%7B/%22type/%22:/%22nuget_feed/%22,/%22url/%22:/%22https://api.nuget.org/v3/index.json/%22%7D]%22)]
NPM Credentials (not working):
##[debug]/usr/bin/docker arg: ["-e","DEPENDABOT_EXTRA_CREDENTIALS=[{\"type\":\"npm_registry\",\"token\":\"PAT:myAccessCode\",\"registry\":\"${feedUrl}/npm/registry/\"},{\"type\":\"npm_registry\",\"registry\":\"registry.npmjs.org\"}]"]
Checking if @uol/cwd 7.0.0 needs updating
🌍 --> GET{feedUrl}/npm/registry/@uol%2Fcwd
🌍 <-- 401 feedUrl}/npm/registry/@uol%2Fcwd
/home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:338:in `check_npm_response': The following source could not be reached as it requires authentication (and any provided details were invalid or lacked the required permissions): feedUrl}/npm/registry (Dependabot::PrivateSourceAuthenticationFailure)
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:297:in `fetch_npm_details'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:291:in `npm_details'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:118:in `valid_npm_details?'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:41:in `latest_version_from_registry'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker.rb:303:in `latest_released_version'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker.rb:311:in `latest_version_details'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker.rb:42:in `latest_version'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-common-0.267.0/lib/dependabot/update_checkers/base.rb:314:in `numeric_version_up_to_date?'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/sorbet-runtime-0.5.11495/lib/types/private/methods/call_validation_2_7.rb:919:in `bind_call'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/sorbet-runtime-0.5.11495/lib/types/private/methods/call_validation_2_7.rb:919:in `block in create_validator_method_medium0'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-common-0.267.0/lib/dependabot/update_checkers/base.rb:267:in `version_up_to_date?'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/sorbet-runtime-0.5.11495/lib/types/private/methods/call_validation_2_7.rb:919:in `bind_call'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/sorbet-runtime-0.5.11495/lib/types/private/methods/call_validation_2_7.rb:919:in `block in create_validator_method_medium0'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-common-0.267.0/lib/dependabot/update_checkers/base.rb:82:in `up_to_date?'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/sorbet-runtime-0.5.11495/lib/types/private/methods/call_validation_2_7.rb:919:in `bind_call'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/sorbet-runtime-0.5.11495/lib/types/private/methods/call_validation_2_7.rb:919:in `block in create_validator_method_medium0'
from /home/dependabot/dependabot-updater/vendor/ruby/3.3.0/gems/dependabot-npm_and_yarn-0.267.0/lib/dependabot/npm_and_yarn/update_checker.rb:30:in `up_to_date?'
from bin/update_script.rb:598:in `block in <main>'
from bin/update_script.rb:561:in `each'
from bin/update_script.rb:561:in `<main>'
@DaleMckeown, I have not tested NPM with private feeds, no.
My understanding is that dependabot-core has recently been moving away from supporting private feed credentials in the "updater" component for security reasons, which breaks private feed auth in this Azure DevOps implementation of dependabot.
AFAIK, only NuGet private feed auth currently works and that only works because I submitted some hacks to restore the old functionality. It might be possible to restore the old auth for NPM using a similar "fix", but I haven't looked in to it.
I believe to properly restore private feed auth in all package managers, this project will need to implement behavior similar to dependabot-cli, which wraps the "updater" container in a special "proxy" container that injects the auth credentials to all outgoing HTTP requests for private feeds. This would be a lot of work to implement, so not sure if/when that would be supported.
It seems that there are only a couple of options:
- Migrate to GitHub to use dependabot natively (a huge amount of effort).
- Hope and pray that this repo implements the dependabot cli proxy wrapper in the near future.
- Make our feeds Artifacts feeds public instead of private, this comes with other risks if you want to control who can access your packages.
It seems that there are only a couple of options:
Migrate to GitHub to use dependabot natively (a huge amount of effort).
Hope and pray that this repo implements the dependabot cli proxy wrapper in the near future.
Make our feeds Artifacts feeds public instead of private, this comes with other risks if you want to control who can access your packages.
1 and 3 are quite simple. In fact, if you have that luxury of doing it, I recommend it because you will have less hustle. I recently did this for a couple of organizations/repositories. Granted that sometimes you'll get flaky PRs. Also, Azure DevOps does enterprise much better/cheaper.
As for supporting the proxy, see #1317
1 and 3 are quite simple. In fact, if you have that luxury of doing it, I recommend it because you will have less hustle. I recently did this for a couple of organizations/repositories. Granted that sometimes you'll get flaky PRs. Also, Azure DevOps does enterprise much better/cheaper.
As for supporting the proxy, see #1317
It would be good if #1293 could be tested, this would fix my auth issues at least in the interim.
With option 3, the biggest concern is that others external to the organisation would theoretically be able to access/download closed source code, so it's not an ideal solution for us.
@DaleMckeown FYI that Task V2 has been recently added which uses dependabot-cli to perform updates; It is still very experimental and has some bugs that are still being worked through, but it should resolve the NPM auth issues you were facing. If you are in a position to test it out, feedback would be appreciated.
@rhyskoedijk Nice one! I'll test this out next week and report back.
I confirm groups are working with v2
@mburumaxwell it might be worth closing this now; there hasn't been any further issues related to groups since your comment for feedback three months ago;
Groups are supported in V2 and in V1 if useUpdateScriptVNext: true is used.
See unsupported features and configurations for more.