Provide a lightweight, generic mechanism to resolve all build dependencies for a project
Closed this issue · 6 comments
The current implementation relies on invoking a particular Gradle task or tasks in order to ensure that all project dependencies are resolved. This is both error-prone (it's easy to miss some dependencies) and inefficient (we don't want to have to build the project just to determine the dependencies).
The plugin should provide a lightweight mechanism to resolve as many build dependencies as possible, similar to the way that the --write-verification-metadata switch will resolve all dependencies in order to generate the verification metadata file.
To start with , it's OK if not all dependencies are caught by this mechanism, as the existing "user-specified task" mechanism will not be removed. Over time, we can hope to improve the mechanism to be more comprehensive.
Interesting. This is different from the way that the Gradle Enterprise Plugin does it, which is to just capture the dependencies resolved as a part of that build.
My original intention was, for a given build, or set of builds, executed in, for example, a GH Actions pipeline, submit a result for each builds execution. That way, the set of dependencies submitted to GitHub for a given commit will be "eventually consistent" with what is actually resolved overall by all configurations.
This plan had the up-side of only capturing the dependencies that are actually used, and didn't incur a performance impact on builds that didn't need to resolve the entire graph. But had the downside, as you note, of not getting the full graph.
I am concerned that, on very, very large Gradle builds, resolving the entire dependency graph for every build will be expensive. But I don't know how much end-users care about this in CI.
It's also important to note that, for any given build, it may be impossible to resolve all possible dependencies, because some dependencies being added may be guarded behind OS checks, for example in the case of JavaFX which has platform-specific dependencies.
This is why the GitHub dependency submission API is designed (or it was when I was talking with GitHub) to be additive each time you upload for a given commit. That way, if you have multiple jobs that run, each can create their own view of the dependency graph, and GitHub will rationalize all of them into a single view on the back end.
This meant that we, as publishers of dependency submissions, didn't need to be concerned with GitHub actions build matricis and trying to collect results into a single submission at the end of the build pipeline.
(we don't want to have to build the project just to determine the dependencies).
FYI, that's also one of the major goals I was trying to achieve in ORT's new GradleInspector. It walks the dependency graph in a similar way like ./gradlew dependencies does, and tries to resolve only relevant metadata, not all binary artifacts.
This is why the GitHub dependency submission API is designed (or it was when I was talking with GitHub) to be additive each time you upload for a given commit. That way, if you have multiple jobs that run, each can create their own view of the dependency graph, and GitHub will rationalize all of them into a single view on the back end.
That's very interesting to know! Would you have a public reference for that?
I do not, but let me see what I can dig up.
You can submit multiple sets of dependencies to be included in your dependency graph. The REST API uses the job.correlator property and the detector.name category of the snapshot to ensure the latest submissions for each workflow get shown. The correlator property itself is the primary field you will use to keep independent submissions distinct. An example correlator could be a simple combination of two variables available in actions runs: <GITHUB_WORKFLOW> <GITHUB_JOB>.
https://docs.github.com/en/rest/dependency-graph/dependency-submission?apiVersion=2022-11-28