Stale dependencies in Dependency graph

Question

Stale dependencies in Dependency graph

Closed this issue 3 months ago · 8 comments

Hi there 👋🏻

Sorry if this is the wrong place, I could not find anything useful anywhere else.

I have a simple Gradle project (not a multi project build) with the following workflow to submit the dependencies:

name: Dependency Submission

on:
  push:
    branches:
      - master

permissions:
  contents: write

jobs:
  dependency-submission:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout sources
        uses: actions/checkout@v4

      - name: Generate and submit dependency graph
        uses: gradle/actions/dependency-submission@v3

I bumped a dependency - spring-boot - from 2.2.2.RELEASE to 2.7.18 but when I look at the dependency graph I still see both version of the dependency, with the same date:

When I re-run the action with debug log I can see that:

Both version are found from the cache:

2024-04-15T16:44:59.4590050Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot/
2024-04-15T16:44:59.4591587Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot/2.2.2.RELEASE/
2024-04-15T16:44:59.4593185Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot/2.7.18/

Only the 2.7.18 get logged in Detected dependency:

2024-04-15T16:45:19.4173768Z Detected dependency 'org.springframework.boot:spring-boot-starter-web:2.7.18': project = ':', configuration = 'runtimeClasspath'
2024-04-15T16:45:19.4176213Z Detected dependency 'org.springframework.boot:spring-boot-starter:2.7.18': project = ':', configuration = 'runtimeClasspath'
2024-04-15T16:45:19.4178404Z Detected dependency 'org.springframework.boot:spring-boot:2.7.18': project = ':', configuration = 'runtimeClasspath'

At my level I'm not sure whether the issue reside in the gradle action, or in Github. If you have any pointer I'd be grateful.

Answer 1 · 2024-04-15T17:17:52.000Z

The duplicated dependency your showing is spring-boot-actuator, but you haven't listed the log entries for that dependency. Can you please grep the logs for spring-boot-actuator to check which version(s) are present?

Assuming you find only 2.7.18 in the logs, then I would suspect that the correct dependency-graph is submitted but the data is stale on GitHub. To confirm this, you can change your workflow to save the dependency-graph file and then inspect it directly.

      - name: Generate and submit dependency graph
        uses: gradle/actions/dependency-submission@v3
        with:
          dependency-graph: generate-and-upload

If you are able to share the complete dependency-graph file I can take a look. Otherwise, just search the file for occurrences of spring-boot-actuator to see what versions are reported.

Answer 2 · 2024-04-16T07:48:08.000Z

@bigdaz Sorry for the screenshot not showing the same dependency as the log, I screenshot the actuator dependency because they happened to be next to each other but the issue is the same for all.

When resolving things from cache it find the previous version:

2024-04-15T16:44:59.4579054Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/
2024-04-15T16:44:59.4580924Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.2.2.RELEASE/
2024-04-15T16:44:59.4583012Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.7.18/

Or:

2024-04-15T16:45:06.8851017Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.2.2.RELEASE/f1f0dc60fde75ab45bce483500b1f194b4fca67/
2024-04-15T16:45:06.8852931Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.2.2.RELEASE/f1f0dc60fde75ab45bce483500b1f194b4fca67/spring-boot-starter-actuator-2.2.2.RELEASE.pom
2024-04-15T16:45:06.8854323Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.7.18/ac6d614d1d0503dfec26110c801902d48bd22628/
2024-04-15T16:45:06.8856088Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.7.18/ac6d614d1d0503dfec26110c801902d48bd22628/spring-boot-starter-actuator-2.7.18.jar
2024-04-15T16:45:06.8857490Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.7.18/be483be1af5c3b28ffc4b897d8e74fcd3c5c0779/
2024-04-15T16:45:06.8859261Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.7.18/be483be1af5c3b28ffc4b897d8e74fcd3c5c0779/spring-boot-starter-actuator-2.7.18.pom
2024-04-15T16:45:06.8860631Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.7.18/e507b547278f3ad1f01975bab15560a7566b88a4/
2024-04-15T16:45:06.8862573Z ../../../.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.7.18/e507b547278f3ad1f01975bab15560a7566b88a4/spring-boot-starter-actuator-2.7.18.module

But once it "detect dependency" only the 2.7.18 is visible:

2024-04-15T16:45:19.5951882Z Detected dependency 'org.eclipse.jetty.websocket:javax-websocket-client-impl:9.4.53.v20231009': project = ':', configuration = 'testRuntimeClasspath'
2024-04-15T16:45:19.5953160Z Detected dependency 'org.springframework.boot:spring-boot-starter-actuator:2.7.18': project = ':', configuration = 'testRuntimeClasspath'
2024-04-15T16:45:19.5954510Z Detected dependency 'org.springframework.boot:spring-boot-actuator-autoconfigure:2.7.18': project = ':', configuration = 'testRuntimeClasspath'
2024-04-15T16:45:19.5955887Z Detected dependency 'org.springframework.boot:spring-boot-actuator:2.7.18': project = ':', configuration = 'testRuntimeClasspath'
2024-04-15T16:45:19.5956931Z Detected dependency 'io.micrometer:micrometer-core:1.9.17': project = ':', configuration = 'testRuntimeClasspath'

The file generated by generate-and-upload only contains 2.7.18 so the issue seems to be on GitHub side... When I download the SBOM for the project I have:

{
  "SPDXID": "SPDXRef-DOCUMENT",
  "spdxVersion": "SPDX-2.3",
  "creationInfo": {
    "created": "2024-04-16T07:35:45Z",
    "creators": [
      "Tool: GitHub.com-Dependency-Graph",
      "Tool: GitHub Dependency Graph Gradle Plugin"
    ]
  },
  [...],
    "relationship": [
     ...
         {
      "relationshipType": "DEPENDS_ON",
      "spdxElementId": "SPDXRef-com.github.MY-PROJECT",
      "relatedSpdxElement": "SPDXRef-maven-org.springframework.boot-spring-boot-actuator-2.2.2.RELEASE"
    },
    {
      "relationshipType": "DEPENDS_ON",
      "spdxElementId": "SPDXRef-com.github.MY_PROJECT",
      "relatedSpdxElement": "SPDXRef-maven-org.springframework.boot-spring-boot-actuator-2.7.18"
    },
      ...
  ]
}

I'll try to find some time to try & submit the SBOM manually using Github API see if the API returns something useful (does not seems to have the Github API response in debug mode) and/or try to submit an empty list, see if I can somehow "reset" the state 🤷🏻

Answer 3 · 2024-04-16T16:05:28.000Z

Thanks for checking. A few things to note:

Each dependency graph submitted includes a 'correlator' value: GitHub uses this to determine if newly submitted graph should replace a previous one, or be merged with it.
- The dependency-submission action generates the correlator from the workflow name, job name and matrix: it's the same as the file name of the generated graph. So if you change the job name you can sometimes see stale graph data from the previous submission.
- You can explicitly override the correlator value by setting the GITHUB_DEPENDENCY_GRAPH_JOB_CORRELATOR env var or system property.
Talking to the GitHub team, this stale data should normally resolve itself in a matter of hours or days. So ideally if you continue to submit new graphs with a new correlator, the old data will eventually be removed. I don't really understand how this is working on the GitHub side.
There's an undocumented dependency-graph: clear option that will submit an empty dependency graph. You can use that to attempt to remove stale data, but you'll need to ensure the correlator matches with the original submission.

Answer 4 · 2024-04-17T07:14:45.000Z

@bigdaz Thanks you for those little nuggets of knowledge. I missed the part around the correlator.

I tried this:

name: Clear Dependency Submission

on:
  workflow_dispatch:

permissions:
  contents: write

jobs:
  dependency-submission:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout sources
        uses: actions/checkout@v4

      - name: Generate and submit dependency graph
        uses: gradle/actions/dependency-submission@v3
        env:
          GITHUB_DEPENDENCY_GRAPH_JOB_CORRELATOR: 'dependency_submission-dependency-submission'
        with:
          dependency-graph: clear

And now I'm left with the stale dependencies. So I might had the dependencies submitted under a different correlator at some point (I did not successfully setup the whole thing first try on all my projects).

The question is how can I know what was the correlator to clean it 🤔

Does not seems to clean itself after "hours" though since it's still here, and it seems to have been updated:

I guess I'll sit and wait a little while and see if it gets resolved by itself.

And cleanup Github Actions caches just for the peace of mind that it will stop finding the whole dependencies in gradle cache.

Answer 5 · 2024-04-17T13:47:54.000Z

You don't need to worry about the dependencies found in ~/.gradle/caches/modules-2. Those won't impact the dependency graph.

To work out the correlator that was used previously, you should be able to look back at the workflow runs to see what the dependency-graph file name was.

Failing that, I think you may need to raise an issue with GitHub about this. After you've raised an issue let me know and I can ping the GitHub dependency graph team directly.

Answer 6 · 2024-04-17T14:30:42.000Z

@bigdaz I finally found the culprit, I have another flow that I did not talk about that is used only for dependency review during PR:

name: Dependency review

on:
  pull_request:

permissions:
  contents: write
  pull-requests: write

jobs:
  dependency-submission:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout sources
        uses: actions/checkout@v4

      - name: Generate and submit dependency graph
        uses: gradle/actions/dependency-submission@v3

  dependency-review:
    needs: dependency-submission
    runs-on: ubuntu-latest
    steps:
      - name: Perform dependency review
        uses: actions/dependency-review-action@v4
        with:
          config-file: XXX
          external-repo-token: XXX

And I did not think much of it because it always says:

Submitted dependency-graph-reports/dependency_review-dependency-submission.json: The snapshot was accepted, but it is not for the default branch. It will not update dependency results for the repository.

But it turns out that in the very first run (we are talking 2 months ago) while I was iterating on the subject it did run on the default branch.

I was able to clean its mess and all is in order.

I'm not quite sure why the stale dependencies appeared as "up to date" in the list of dependents but I have other more pressing issues to attend to 😅

Thanks you very much for your help!

Answer 7 · 2024-04-17T15:55:47.000Z

Glad you got it sorted!
Note that as of v3.3.0 you should no longer require separate jobs for dependency-submission and dependency-review. The dependency-graph is uploaded during the Generate and submit dependency graph step, and not at the end of the Job (as it was before).

Answer 8 · 2024-04-18T06:45:24.000Z

@bigdaz Thanks for the hint and you overall help 🙏🏻