Ratelimiting error when downloading vulnerability db from ghcr.io

Question

Ratelimiting error when downloading vulnerability db from ghcr.io

HenrikDK opened this issue 4 months ago · 88 comments

Hi, we're using trivy to scan our containers, lately we've been seeing an increase number of rate-limiting errors when trivy is downloading the vulnerability database.

"2024-09-18T10:40:44Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:11c57f2012b2ac112256f94aa404e1feb7e1b7a5787598946b87149115cdb43d: TOOMANYREQUESTS: retry-after: 129.163µs, allowed: 44000/minute"

My guess is this is a global ratelimit as i can't imagine our low number of devs are causing 700+ requests a second.

I have in the meantime discovered that these scans are only used for SBOM generation on our end so we don't need to download the vulnerability database everytime, but i though this issue should be raised as i can't imagine we are the only ones seeing these errors.

Answer 1 · 2024-09-18T18:57:54.000Z

Thanks for the report, we will look into it.

Answer 2 · 2024-09-19T12:41:13.000Z

I also saw this right now :/ Any ideas why?

Answer 3 · 2024-09-19T16:53:40.000Z

I believe this is currently causing problems with anyone using the trivy action. We have had to turn it off on some workflows. I'm not sure what the long term solution might be - if GH cannot increase the global rate limit for the artifact pull then maybe it needs to be in a public AWS S3 bucket or something similar?

Answer 4 · 2024-09-19T16:57:49.000Z

Also see https://github.com/orgs/community/discussions/139074

Answer 5 · 2024-09-20T14:08:16.000Z

From My PR above, a workaround suggested by someone else:

- uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: <something else than ghcr.io>
    TRIVY_JAVA_DB_REPOSITORY: <something else than ghcr.io>```

Answer 6 · 2024-09-20T16:48:16.000Z

Does anyone know how to get trivy-action to auth with a privately hosted trivy-db repo? I can get it working fine with normal trivy on local, but trivy-action does not work with either docker/login-action or the usual echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

2024-09-20T16:39:01Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred: * GET https://ghcr.io/token?scope=repository%3Aprivate-github-org%2Ftrivy-db%3Apull&service=ghcr.io: UNAUTHORIZED: authentication required

Answer 7 · 2024-09-20T17:19:54.000Z

I was able to get it to work with ECR only using an OIDC login via the configure-aws-credentials action used right before the trivy action. It is not using docker to pull the artifact as it is not a docker image.

Answer 8 · 2024-09-22T05:23:38.000Z

I am poor student

Answer 9 · 2024-09-23T09:25:20.000Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:

      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Answer 10 · 2024-09-23T12:09:27.000Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:
      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I've tried logging in to GHCR via docker/login-action before running Trivy CLI (not action), and I am still getting lots of 429 errors.

Answer 11 · 2024-09-23T13:24:02.000Z

From My PR above, a workaround suggested by someone else:

- uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: <something else than ghcr.io>
    TRIVY_JAVA_DB_REPOSITORY: <something else than ghcr.io>```

So, if I understand this correctly:

I, as the consumer of this action, must download copies of these DBs and store them on my own registry. Then, I must pass environment variables to the action which point at my copies of the DBs. Is that correct?

How often are these DBs updated?

Answer 12 · 2024-09-23T14:06:32.000Z

@nnellanspdl think its at 00:00 every day? but im not sure.

But anyway this workaround is a hustle to host them self if u need to update them every day

Answer 13 · 2024-09-23T15:24:55.000Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:
      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
I've tried logging in to GHCR via docker/login-action before running Trivy CLI (not action), and I am still getting lots of 429 errors.

Same for me, it doesn't seem to have significant effects.

Answer 14 · 2024-09-23T15:35:10.000Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:
      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I'm trying with:

env:
    ACTIONS_RUNTIME_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I spawned multiple parallel ci/cd actions, and this seems more reliable.

Answer 15 · 2024-09-23T16:57:30.000Z

If anyone is going the route of uploading the Trivy DB to their own registry, I've had success using https://github.com/oras-project/setup-oras

Something like:

  vendor-trivy-db:
    runs-on: ubuntu-latest
    steps:
      - name: Vendor latest trivy db
        uses: oras-project/setup-oras@v1
      - run: |
          oras pull ghcr.io/aquasecurity/trivy-db:2
          oras login -u ${{ secrets.REGISTRY_USERNAME }} -p ${{ secrets.REGISTRY_TOKEN }} YOUR_REGISTRY
          oras push YOUR_REGISTRY \
            db.tar.gz:application/vnd.aquasec.trivy.db.layer.v1.tar+gzip \
            --artifact-type application/vnd.aquasec.trivy.config.v1+json

Answer 16 · 2024-09-23T17:57:44.000Z

I setup AWS ECR pull-throuhg cache for trivy-db and trivy-java-db , modified action:

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: ${{ env.DOCKER_IMAGE_TO_SCAN }}
          format: 'table'
          exit-code: '1' 
          ignore-unfixed: true
          vuln-type: 'os,library'
          severity: 'CRITICAL,HIGH'
        env:
          TRIVY_DB_REPOSITORY: <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db
          TRIVY_JAVA_DB_REPOSITORY: <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db
          TRIVY_DEBUG: true

but pulling of trivy-db fails with:

2024-09-23T16:16:12Z	INFO	Downloading DB...	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db"
2024-09-23T16:16:12Z	DEBUG	No metadata file
2024-09-23T16:16:17Z	DEBUG	Credential error	err="failed to get authorization token: operation error ECR: GetAuthorizationToken, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded"
2024-09-23T16:16:17Z	FATAL	Fatal error	init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred:
	* GET https://<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/v2/github/ghcr.io/aquasecurity/trivy-db/manifests/2: unexpected status code 401 Unauthorized: Not Authorized

Docker is logged-in.
If I run trivy binary locally or on runner, it works fine:

runner@runner-set-xs-djfqb-0:/tmp$ export TRIVY_DB_REPOSITORY=<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db
runner@runner-set-xs-djfqb-0:/tmp$ export TRIVY_JAVA_DB_REPOSITORY=<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db
runner@runner-set-xs-djfqb-0:/tmp$ trivy image    --format table --exit-code  1 --ignore-unfixed --vuln-type  os,library --severity  CRITICAL,HIGH  <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/my-awesome-app:1.23.0
2024-09-23T16:35:04Z	WARN	'--vuln-type' is deprecated. Use '--pkg-types' instead.
2024-09-23T16:35:04Z	INFO	Adding schema version to the DB repository for backward compatibility	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db:2"
2024-09-23T16:35:04Z	INFO	Adding schema version to the Java DB repository for backward compatibility	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db:1"
2024-09-23T16:35:04Z	INFO	[db] Need to update DB
2024-09-23T16:35:04Z	INFO	[db] Downloading DB...	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db:2"
53.56 MiB / 53.56 MiB [------------------------------------------------------------------------------------------------------

Has somebody tried to pull trivy-db from AWS ECR using action?

Answer 17 · 2024-09-23T20:23:49.000Z

Yes so you can pull from ECR pull through but only if you do an OIDC set-aws-credentials action first before the trivy action. Im not sure why yet that you cannot use anything but OIDC, or at least I can't seem to get regular role assumption to work. Docker login doesnt help you as the container doesnt try to pull the DB using docker commands.

If you try a docker pull you will get the unsupported media type error as the above post, as the artifact isnt an 'image'

Answer 18 · 2024-09-23T20:25:06.000Z

Ah, thanks. I was logged in under the incorrect account when I posted originally. That's what I was wondering, @billhammond-dev !

Answer 19 · 2024-09-23T20:25:21.000Z

This was my error, for anyone else who runs into it:

latest: Pulling from github/ghcr.io/aquasecurity/trivy-db
unsupported media type application/vnd.aquasec.trivy.config.v1+json

Answer 20 · 2024-09-23T20:35:38.000Z

@nnellanspdl think its at 00:00 every day? but im not sure.

But anyway this workaround is a hustle to host them self if u need to update them every day

Thanks. Yes, this is a lot to ask of consumers of your action.

Answer 21 · 2024-09-23T21:53:45.000Z

I'm guessing it would be too much work to update the logic for pulling the file to allow passing it the file directly? We could setup a workflow to pull and stash the image every X hours, and then in the workflow that uses the image, we pull the file from the stash to use. It'd lower the amount of hits by users, and we wouldn't need to host it in AWS and pay

Answer 22 · 2024-09-24T04:42:50.000Z

ACTIONS_RUNTIME_TOKEN

@NicholasFiorentini that's interesting, would you mind creating a PR to document this in the repo? If possible, could you also reference where this environment variable is documented?

Answer 23 · 2024-09-24T07:16:15.000Z

FWIW, here's a sample snippet for using AWS ECR pull through cache repositories using OIDC for AWS auth.

Pull through cache ECR repositories (for hosting the cached trivy DB artifacts) must be configured prior to running this workflow, see documentation.

- name: Setup AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-region: ...
    role-to-assume: <role, assumable through OIDC, that can pull from the cache ECR repositories>

- id: ecr-login
  name: Login to ECR
  uses: aws-actions/amazon-ecr-login@v2

...

- name: Run trivy scan
  uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: ${{ steps.ecr-login.outputs.registry }}/github/aquasecurity/trivy-db:2
    TRIVY_JAVA_DB_REPOSITORY: ${{ steps.ecr-login.outputs.registry }}/github/aquasecurity/trivy-java-db:1

Per AWS documentation:

When a cached image is pulled through the Amazon ECR private registry URI, Amazon ECR checks the upstream repository at least once every 24 hours to verify whether the cached image is the latest version. If there is a newer image in the upstream registry, Amazon ECR attempts to update the cached image. This timer is based off the last pull of the cached image.

Answer 24 · 2024-09-24T09:58:48.000Z

ACTIONS_RUNTIME_TOKEN
@NicholasFiorentini that's interesting, would you mind creating a PR to document this in the repo? If possible, could you also reference where this environment variable is documented?

I found this variable by inspecting the Docker command executed by the Trivy action. I deployed my changes yesterday and all my repos haven't seen the rate-limiting error so far.

It looks like an undocumented GitHub action feature: https://github.com/search?q=repo%3Aactions%2Ftoolkit%20ACTIONS_RUNTIME_TOKEN&type=code

Answer 25 · 2024-09-24T10:14:42.000Z

Same here, happening more and more often, causing to have to manually run again jobs :(

Running trivy with options: trivy image  --format table --exit-code  1 --ignore-unfixed --vuln-type  os,library --scanners  vuln --severity  CRITICAL,HIGH --timeout  10m0s  docker.io/oxsecurity/megalinter-only-api_spectral:pr-[40](https://github.com/oxsecurity/megalinter/actions/runs/11010163484/job/30576131056?pr=4037#step:9:41)37
Global options:  
2024-09-24T10:11:12Z	INFO	Need to update DB
2024-09-24T10:11:12Z	INFO	Downloading DB...	repository="ghcr.io/aquasecurity/trivy-db:2"
2024-09-24T10:11:12Z	FATAL	Fatal error	init error: DB error: failed to download vulnerability DB: database download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:92209767c55fb2c1a5b24efea1f96899a3956c38332d7685ac1f8580bc451712: TOOMANYREQUESTS: retry-after: 515.848µs, allowed: 44000/minute

Answer 26 · 2024-09-24T10:35:56.000Z

Another possibility that I'm currently experimenting with is using https://github.com/yogeshlonkar/trivy-cache-action

It isn't perfect as it only caches the main trivy-db so if your scan requires downloading an additional DB e.g. trivy-java-db you might still hit rate limits but it seems to improve things:

      - name: Trivy Cache
        uses: yogeshlonkar/trivy-cache-action@v0.1.7
        with:
          gh-token: ${{ secrets.GITHUB_TOKEN }}
          prefix: ${{ github.workflow }}

      - name: Trivy Vulnerability Scan
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: "image"
          image-ref: ${{ inputs.IMAGE_REF }}
          output: trivy-docker-report.json
          format: json
          exit-code: 0
          cache-dir: .trivy

Answer 27 · 2024-09-24T12:01:24.000Z

As we also run into this issue, could someone tell me, what media type to set for pushing the java-db oci bundle?

I tried the settings @BRONSOLO used for the "non-java-db", but that leads to an unsupported media type error.

Answer 28 · 2024-09-24T12:25:39.000Z

Is this an issue with the github container registry or on the trivy side? Is there a different CR we can pull it from without having to setup our own with some sort of caching?

For now our request come through ~ 1 in 5 times

Answer 29 · 2024-09-24T12:46:37.000Z

As we also run into this issue, could someone tell me, what media type to set for pushing the java-db oci bundle?

I tried the settings @BRONSOLO used for the "non-java-db", but that leads to an unsupported media type error.

its commented here #389 (comment)

U can use oras cli, but I would think that aquasec should setup an alternativ registry pretty fast.. either that or some cache on them side.

Answer 30 · 2024-09-24T12:55:00.000Z

ACTIONS_RUNTIME_TOKEN
@NicholasFiorentini that's interesting, would you mind creating a PR to document this in the repo? If possible, could you also reference where this environment variable is documented?

I only learned about this variable myself from the recent disclosure of the ArtiPACKED vulnerability:

"But I’m left with repos exposing their ACTIONS_RUNTIME_TOKEN, which is a JWT (JSON Web Token) with an expiration of about six hours according to the exp (expiration) property. ACTIONS_RUNTIME_TOKEN is an undocumented environment variable, used by several popular actions owned by GitHub, such as actions/cache and actions/upload-artifact, to manage caching and artifacts."

Answer 31 · 2024-09-24T13:19:55.000Z

ACTIONS_RUNTIME_TOKEN
@NicholasFiorentini that's interesting, would you mind creating a PR to document this in the repo? If possible, could you also reference where this environment variable is documented?
I only learned about this variable myself from the recent disclosure of the ArtiPACKED vulnerability:

"But I’m left with repos exposing their ACTIONS_RUNTIME_TOKEN, which is a JWT (JSON Web Token) with an expiration of about six hours according to the exp (expiration) property. ACTIONS_RUNTIME_TOKEN is an undocumented environment variable, used by several popular actions owned by GitHub, such as actions/cache and actions/upload-artifact, to manage caching and artifacts."

Thanks @nnellanspdl for sharing that interesting vulnerability.

Yes, It's a very hidden and obscure variable (I linked the references in the action's codebase here).

I can only say that passing down the variable to the Trivy action has solved the issue for me.

Answer 32 · 2024-09-24T13:21:41.000Z

Couple things I checked:

Vulnerability database download code is here.

It looks like there is no way to supply custom credentials for the vulnerability database via CLI parameters.

It also seems like credentials set by the docker/login-action is not picked up the library used under the hood or a GHA authenticated request counts towards the same rate limit.

I wonder if this issue started because of a change GitHub did on their side or Trivy became so popular we just started to run into these issues.

I'm gonna give ACTIONS_RUNTIME_TOKEN a try, but it doesn't sound like something that should work. If it does, I really don't know why (yet).

Answer 33 · 2024-09-24T13:34:52.000Z

I'm gonna give ACTIONS_RUNTIME_TOKEN a try, but it doesn't sound like something that should work. If it does, I really don't know why (yet).

I also don't understand how it works. So far, what I'm trying to figure out:

It looks like the ACTIONS_RUNTIME_TOKEN variable is automatically propagated from the action's env to the Docker container.
When Trivy calls GitHub, it is likely that the Http Client inside the action is automatically using the value available in the ACTIONS_RUNTIME_TOKEN variable.

Answer 34 · 2024-09-24T13:41:55.000Z

I tested ACTIONS_RUNTIME_TOKEN and it didn't seem to prevent DB download failures due to too many requests, though its hard to say if it might be making it less likely to fail or not.

Answer 35 · 2024-09-24T14:17:03.000Z

When Trivy calls GitHub, it is likely that the Http Client inside the action is automatically using the value available in the ACTIONS_RUNTIME_TOKEN variable.

From what I can tell, the database download happens inside Trivy itself, not the action.

Answer 36 · 2024-09-24T14:26:17.000Z

Do anyone know if there is any chance it works well again someday without having to define workarounds ? :/

Answer 37 · 2024-09-24T15:17:07.000Z

Can confirm that this fix appears to be working for us:

        env:
          ACTIONS_RUNTIME_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Answer 38 · 2024-09-24T15:40:14.000Z

I tried the settings @BRONSOLO used for the "non-java-db", but that leads to an unsupported media type error.

@PascalTurbo I believe you'll want to use https://github.com/aquasecurity/trivy/blob/37d549e5b86a1c5dce6710fbfd2310aec9abe949/docs/docs/configuration/db.md?plain=1#L108

Something like:

oras push YOUR_REGISTRY  \
  javadb.tar.gz:application/vnd.aquasec.trivy.javadb.layer.v1.tar+gzip \
  --artifact-type application/vnd.aquasec.trivy.config.v1+json

Answer 39 · 2024-09-25T08:17:19.000Z

Can confirm that this fix appears to be working for us:
        env:
          ACTIONS_RUNTIME_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I can't. Added that yesterday, still seeing the issue

Log

``` Running trivy with options: trivy fs --format table --exit-code 1 --ignore-unfixed --vuln-type os,library --severity CRITICAL,HIGH . Global options: 2024-09-25T08:05:07Z INFO Need to update DB 2024-09-25T08:05:07Z INFO Downloading DB... repository="ghcr.io/aquasecurity/trivy-db:2" 2024-09-25T08:05:07Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred: * GET https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2: TOOMANYREQUESTS: retry-after: 211.298µs, allowed: 44000/minute ```

Answer 40 · 2024-09-25T09:23:42.000Z

The trivy-db has been mirrored on a public ECR thanks to this PR aquasecurity/trivy-db#440

I suppose trivy will be updated to now use this ECR by default in a near future, but in the meantime this workaround is working for me:

      - name: Run Trivy vulnerability scanner in repo mode
        uses: aquasecurity/trivy-action@0.24.0
        env:
          TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db:2
        with:
          scan-type: 'fs'
          ignore-unfixed: true
          hide-progress: true
          format: 'table'
          severity: 'CRITICAL,HIGH'

Answer 41 · 2024-09-25T10:09:32.000Z

@damienleger Is that workaround scalable though?

AWS ECR Public Service Quotas lists the following:

Name	Default	Description
Rate of authenticated image pulls	Each supported Region: 10 per second	The maximum number of authenticated image pulls per second.
Rate of image pulls to AWS resources	Each supported Region: 10 per second	The maximum number of image pulls per second to resources running on Amazon ECS, Fargate, or Amazon EC2.
Rate of unauthenticated image pulls	Each supported Region: 1 per second	The maximum number of unauthenticated image pulls per second.

Which would seem to work out at far less than the 4400/minute that GHCR is complaining about now. So if lots of people point at the ECR repository aren't they just going to hit rate limits there even sooner?

Answer 42 · 2024-09-25T18:31:32.000Z

Has anyone considered raising a feature request for trivy itself to implement backoff/retry on rate limits for the db pull?

I moved to using the trivy bin directly in our workflow and self hosting the db in our own ghcr, but it'd be nice to handle this without requiring uptake for every consumer of trivy-action.

Answer 43 · 2024-09-26T12:24:32.000Z

Has anyone considered raising a feature request for trivy itself to implement backoff/retry on rate limits for the db pull?

Retries will cause more requests hence we will hit the rate limits more often considering we are all contributing to the same global rate limit

Answer 44 · 2024-09-26T14:19:06.000Z

I used regctl to mirror the image to local repository and it has built-in backoff in case hitting the rate limit. Something similar could be implemented here. That's why retry-after is returned on the request.

regctl image copy ...
time="2024-09-26T09:03:13Z" level=warning msg="Sleeping for backoff" Host=ghcr.io Seconds=1.999993179

Answer 45 · 2024-09-26T16:17:15.000Z

Can confirm that this fix appears to be working for us:
        env:
          ACTIONS_RUNTIME_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Not enough... I have applied this to 7 repositories, some worked, others still reporting the same error.

Answer 46 · 2024-09-27T11:23:20.000Z

In addition to this issue, within our pipelines rate limits are also triggered within the build of the actions Dockerfile
I bet people also run in this issue and due to its hardcoded values even a pull-through cache won't solve this.

FROM ghcr.io/aquasecurity/trivy:0.53.0
COPY entrypoint.sh /
RUN apk --no-cache add bash curl npm
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

Answer 47 · 2024-09-27T11:31:25.000Z

FWIW, our workaround using private ECR pull through cache repositories to pull Trivy DB artifacts, with upstream pointing to ghcr.io worked fine, until it didn't, because now the upstream pulls from GitHub started to get rate limited to the point, that pulls from our private ECR cache repository also started failing (I presume this is ECR's way of warning us that they cannot update the image from upstream, to avoid stale cached images).

Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: failed to fetch the layer: GET https://<redacted>.dkr.ecr.<redacted>.amazonaws.com/v2/github/aquasecurity/trivy-db/blobs/sha256:<redacted>: DENIED: Unable to get upstream layer matching pull through cache rule. The credential specified in the pull through cache rule has reached the upstream registry pull rate limit. You may be able to increase the limit by contacting the upstream registry provider

We are now attempting to work around this by pointing our private ECR pull through cache repositories to corresponding ECR public upstreams instead, to which both the Trivy DB and Trivy Java DB seem to be now published (see here).

Answer 48 · 2024-09-27T12:47:20.000Z

Changed here now to --db-repository public.ecr.aws/aquasecurity/trivy-db:2 .. working fine..

lets see what happens kkkk

Answer 49 · 2024-09-27T17:49:27.000Z

In addition to this issue, within our pipelines rate limits are also triggered within the build of the actions Dockerfile I bet people also run in this issue and due to its hardcoded values even a pull-through cache won't solve this.
FROM ghcr.io/aquasecurity/trivy:0.53.0
COPY entrypoint.sh /
RUN apk --no-cache add bash curl npm
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

This is exactly the error we are facing as well, @tparrot were you able to find a workaround since?

Answer 50 · 2024-09-27T19:25:09.000Z

I have the same problem with other actions that are configured as Dockerfiles. Unfortunately, there seems to be no real solution to this. I even reached out to a private GitHub contact, who offered no help.

Please check out the discussion here

These are built asynchronously to the workflow's steps and as such, putting a docker/login-action ahead of the step that requires them has no effect. This means that any docker pull is subject to Docker Hub's public rate limit for self-hosted runners

Above, replace Docker Hub with GHCR

Answer 51 · 2024-09-30T12:40:06.000Z

Changed here now to --db-repository public.ecr.aws/aquasecurity/trivy-db:2 .. working fine..

lets see what happens kkkk

We use trivy.yml to configure trivy and his fix works like a charm for now:

db:
repository: "public.ecr.aws/aquasecurity/trivy-db:2"

Answer 52 · 2024-09-30T12:46:34.000Z

Changed here now to --db-repository public.ecr.aws/aquasecurity/trivy-db:2 .. working fine..
lets see what happens kkkk

We use trivy.yml to configure trivy and his fix works like a charm for now:

db: repository: "public.ecr.aws/aquasecurity/trivy-db:2"

I set up in our harbor a proxy for aws public registry with the name awspublic and use it as a pull through proxy.
Also these settings as env vars are needed in the dockerfile where I build a trivy image with the fresh db's:

ENV TRIVY_DB_REPOSITORY=$HARBOR_REGISTRY/awspublic/aquasecurity/trivy-db
ENV TRIVY_JAVA_DB_REPOSITORY=$HARBOR_REGISTRY/awspublic/aquasecurity/trivy-java-db
ENV TRIVY_OFFLINE_SCAN="true"

I then use this image to run the ci job to do scans.

Answer 53 · 2024-09-30T13:36:47.000Z

This may be fixed by the most recent trivy release - 0.56, being incorporated into this action. Currently it's using 0.53. Can we get that updated?

Answer 54 · 2024-09-30T13:49:33.000Z

This may be fixed by the most recent trivy release - 0.56, being incorporated into this action. Currently it's using 0.53. Can we get that updated?

It seems 0.56 version which potentially addresses this issue (aquasecurity/trivy#7592) is yet to be released (aquasecurity/trivy#7447)

Once released we need a trivy-action that uses 0.56

Answer 55 · 2024-10-01T10:29:26.000Z

I'm using the action and setting env TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db:2 fixed it for me

Answer 56 · 2024-10-01T12:06:31.000Z

If anyone is going the route of uploading the Trivy DB to their own registry, I've had success using https://github.com/oras-project/setup-oras

Something like:

  vendor-trivy-db:
    runs-on: ubuntu-latest
    steps:
      - name: Vendor latest trivy db
        uses: oras-project/setup-oras@v1
      - run: |
          oras pull ghcr.io/aquasecurity/trivy-db:2
          oras login -u ${{ secrets.REGISTRY_USERNAME }} -p ${{ secrets.REGISTRY_TOKEN }} YOUR_REGISTRY
          oras push YOUR_REGISTRY \
            db.tar.gz:application/vnd.aquasec.trivy.db.layer.v1.tar+gzip \
            --artifact-type application/vnd.aquasec.trivy.config.v1+json

Thanks ! I was able to setup something like that to copy Trivy's db to our JFrog internal instance, in a Docker repository (yes JFrog accepts OCI artifacts with custom types in Docker repos). The trivy CLI is then able to fetch the db from JFrog using docker login and --db-repository flag.

Answer 57 · 2024-10-01T12:06:47.000Z

For this reason amongst others - it would be really great if we could specify trivy version in the action.... even with a fat disclaimer that sometimes the action is not compatible with new releases for some cases

Answer 58 · 2024-10-01T12:08:51.000Z

Even mirroring (or using the AWS mirror) the main trivy-db, you will still get failures on the java-db

I need to pick through a dozen workflows to find those that do not require the java-db

Answer 59 · 2024-10-01T12:33:32.000Z

At the people from aquasecurity: why not host your own "read-only" registry?

use a tool like https://github.com/NicolasT/static-container-registry
run Nginx on a host with unlimited bandwidth (like Infomaniak; I'm not related, not giving any commercial advice)

Answer 60 · 2024-10-07T17:25:04.000Z

There's a ton of items referencing this issue. What is the path forward?

Answer 61 · 2024-10-07T19:27:50.000Z

I did not see a "works-for-all" solution so I suggested one...

I hope the maintainers will consider it 🙏

aquasecurity/trivy#7668

Answer 62 · 2024-10-07T20:17:22.000Z

We've released a new version which uses Trivy v0.56.1 which has the ability to download from different sources in case of hitting rate limits https://github.com/aquasecurity/trivy-action/releases/tag/0.25.0

At the same time, we're also working on a caching solution within the action itself: #399

Answer 63 · 2024-10-07T20:43:24.000Z

We've released a new version which uses Trivy v0.56.1 which has the ability to download from different sources in case of hitting rate limits https://github.com/aquasecurity/trivy-action/releases/tag/0.25.0

At the same time, we're also working on a caching solution within the action itself: #399

Will this download from other sources automatically? As if not I agree with the solution from @nvuillam here: aquasecurity/trivy#7668 that it should be hosted at multiple mirrors which it tried automatically as well as the caching you mentioned, this would resolve.

Answer 64 · 2024-10-07T20:48:45.000Z

Indeed, in my opinion the problem is global to trivy usage, and should be solved at global level, not only on trivy-action level :/

My mirror solution would be the safest, because asking users to handle their own mirror or their own cache induces the risk that those "custom mirrors" or "custom caches" are themselves vulnerabilities.

With trivy-core-team-managed mirrors, we would eliminate such risks, and nobody would have to implement any workaround to make trivy work, including on trivy-action :)

Answer 65 · 2024-10-07T20:52:51.000Z

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:
      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
I'm trying with:
env:
    ACTIONS_RUNTIME_TOKEN: ${{ secrets.GITHUB_TOKEN }}
I spawned multiple parallel ci/cd actions, and this seems more reliable.

By the way, I tried both GITHUB_TOKEN and ACTIONS_RUNTIME_TOKEN , but it's still randomly hitting rate limit :/

Answer 66 · 2024-10-07T20:55:16.000Z

Yeah, my software has like 4 different code bases, and managing dependencies for all them is a big enough job as it is without having to maintain our own mirror of trivy and ensuring that is up to date and not a vulnerability itself. Might be okay for large companies to do this, but for small teams or solo developers its not really feasible.

Answer 67 · 2024-10-07T22:04:09.000Z

We've released a new version which uses Trivy v0.56.1 which has the ability to download from different sources in case of hitting rate limits https://github.com/aquasecurity/trivy-action/releases/tag/0.25.0
At the same time, we're also working on a caching solution within the action itself: #399

Will this download from other sources automatically? As if not I agree with the solution from @nvuillam here: aquasecurity/trivy#7668 that it should be hosted at multiple mirrors which it tried automatically as well as the caching you mentioned, this would resolve.

Yes, in the latest version of Trivy we have implemented the fallback mechanism. You can read more on that here: aquasecurity/trivy#7640

Answer 68 · 2024-10-07T22:05:13.000Z

Echoing sentiment from others: this isn't considered resolved until the action can function by default (e.g without additional config required). This should be considered as a high priority bug.

Answer 69 · 2024-10-07T22:19:20.000Z

We've released a new version which uses Trivy v0.56.1 which has the ability to download from different sources in case of hitting rate limits https://github.com/aquasecurity/trivy-action/releases/tag/0.25.0
At the same time, we're also working on a caching solution within the action itself: #399

Will this download from other sources automatically? As if not I agree with the solution from @nvuillam here: aquasecurity/trivy#7668 that it should be hosted at multiple mirrors which it tried automatically as well as the caching you mentioned, this would resolve.

Yes, in the latest version of Trivy we have implemented the fallback mechanism. You can read more on that here: aquasecurity/trivy#7640

This is not the job of user the search alternative vulnerability databases, this should be handled by default by trivy's own code (and not use a database list as argument)
Or if you do that, that's a breaking change and according to Semantic Versioning, the new trivy release should be a major version (1.0.0)

Answer 70 · 2024-10-07T22:21:18.000Z

Echoing sentiment from others: this isn't considered resolved until the action can function by default (e.g without additional config required). This should be considered as a high priority bug.

We've released a new version which uses Trivy v0.56.1 which has the ability to download from different sources in case of hitting rate limits https://github.com/aquasecurity/trivy-action/releases/tag/0.25.0
At the same time, we're also working on a caching solution within the action itself: #399

Will this download from other sources automatically? As if not I agree with the solution from @nvuillam here: aquasecurity/trivy#7668 that it should be hosted at multiple mirrors which it tried automatically as well as the caching you mentioned, this would resolve.

Yes, in the latest version of Trivy we have implemented the fallback mechanism. You can read more on that here: aquasecurity/trivy#7640

This is not the job of user the search alternative vulnerability databases, this should be handled by default by trivy's own code (and not use a database list as argument) Or if you do that, that's a breaking change and according to Semantic Versioning, the new trivy release should be a major version (1.1.0)

~~Trivy by default will try alternate registries, in this case ECR and GHCR. No user intervention is needed.~~

Trivy by default will try alternate registries, if provided.

Answer 71 · 2024-10-07T22:32:31.000Z

@simar7 that's a good news, but unfortunately it still does not work :/

Cf a job from less than 2 hours using trivy-action@main

https://github.com/oxsecurity/megalinter/actions/runs/11222990312/job/31196620975

Or this job, that failed 2mn ago :/
https://github.com/oxsecurity/megalinter/actions/runs/11224511011/job/31202824634#step:9:1

Answer 72 · 2024-10-07T22:43:07.000Z

@simar7 that's a good news, but unfortunately it still does not work :/

Cf a job from less than 2 hours using trivy-action@main

https://github.com/oxsecurity/megalinter/actions/runs/11222990312/job/31196620975

Or this job, that failed 2mn ago :/ https://github.com/oxsecurity/megalinter/actions/runs/11224511011/job/31202824634#step:9:1

Sorry, I misspoke. You are right. At the moment we don't offer the ECR fallback as an option within the action itself. The PR for adding caching is still in progress #399.

As for the automatic fallback to ECR (or another source), please standby.

Answer 73 · 2024-10-08T08:33:06.000Z

For everybody trying to piece all information together (as I had to): the following step configuration does it for me (with both regular and Java DBs pointing to AWS):

      - name: "Trivy Scan"
        uses: aquasecurity/trivy-action@0.25.0
        env:
          TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db:2
          TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db:1
        with:
          scan-type: repo
          scanners: vuln,secret,config
          format: table

Answer 74 · 2024-10-08T08:52:20.000Z

DB is only a part of the problem - as @tparrot originally mentioned, there is also an issue with Dockerfile, and this has to be solved not in trivy, but in trivy-action.

Until that is also removed (as proposed in #399), everyone on the planet who uses trivy-action will continue to see intermittent failures as there is no way of even configuring your own image, save for forking the whole action (or merging a PR like #335 but that doesn't solve it for everyone).

Failure example:

Step 1/5 : FROM ghcr.io/aquasecurity/trivy:0.56.1
  0.56.1: Pulling from aquasecurity/trivy
  43c4264eed91: Pulling fs layer
  958beee14847: Pulling fs layer
  adc406921242: Pulling fs layer
  e1d9543056a1: Pulling fs layer
  e1d9543056a1: Waiting
  error pulling image configuration: download failed after attempts=1: toomanyrequests: retry-after: 39.935µs, allowed: 44000/minute

Answer 75 · 2024-10-08T09:12:50.000Z

By the way, I tried both GITHUB_TOKEN and ACTIONS_RUNTIME_TOKEN , but it's still randomly hitting rate limit :/

Yes, after a few weeks of using this solution, I still face random rate limits.

Answer 76 · 2024-10-08T11:10:45.000Z

Adding those ENVs helped me:

env:
  TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db # default is repository="ghcr.io/aquasecurity/trivy-db:2"
  TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db # default is repository=ghcr.io/aquasecurity/trivy-java-db:1

Answer 77 · 2024-10-08T14:07:02.000Z

in all honesty, seems like the workaround's seems to happen to work just by chance you are not hitting limit at the time

Even though the different db was set (public.ecr.aws/aquasecurity/trivy-db:2), and github token and GITHUB_TOKEN & ACTIONS_RUNTIME_TOKEN on our side, I still see in CI log that original db errored ( 2024-09-27T13:02:08Z INFO Downloading DB... repository="ghcr.io/aquasecurity/trivy-db:2" )

though for a sec it was because we were running trivy-action@master but 0.24.0 does pretty much the same.

Answer 78 · 2024-10-08T20:24:20.000Z

Just to update, we've released v0.26.0 which adds support for caching https://github.com/aquasecurity/trivy-action/releases/tag/0.26.0

This should alleviate some of the pain as caching should ensure DBs are reused if cache is available. You can read more on this here: https://github.com/aquasecurity/trivy-action?tab=readme-ov-file#cache

Answer 79 · 2024-10-09T08:40:04.000Z

To update on that, trivy version 0.56.0, which is running on the latest action. Includes the option to specify multiple registries. So you can do something like

  - name: Run Trivy vulnerability scanner
    uses: aquasecurity/trivy-action@0.26.0
    env:
      TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db

Which will try the default GitHub Registry, and if too many requests is reached, will use the aws mirror

Answer 80 · 2024-10-09T09:43:07.000Z

lol dockers scans are ok for now, but the code scan now throws erros with sudo call #403.

Do not know which is worse :p

Answer 81 · 2024-10-09T12:56:12.000Z

We seem to have fixed this issue (36h+ without it so far) with ECR pull through. The catch for us was with Trivy's fs scan.

While for image scanning you can use

TRIVY_DB_REPOSITORY: <account>.dkr.<region>.amazonaws.com/<ecr_repo>/trivy-db:2
TRIVY_JAVA_DB_REPOSITORY: <account>.dkr.<region>.amazonaws.com/<ecr_repo>/trivy-java-db:1

for fs scan you have to use

TRIVY_CHECKS_BUNDLE_REPOSITORY: <account>.dkr<region>.amazonaws.com/<ecr_repo>/trivy-checks:0

Hope that helps :)

Answer 82 · 2024-10-09T13:22:24.000Z

We've moved to just installing Trivy CLI and forgetting the trivy-action for now.

We've defined the following as an action:

name: Install Trivy
description: "Installs Trviy binary from the official GitHub release"
runs:
  using: "composite"
  steps:
    - run: |
        LATEST_RELEASE=$(curl -s https://api.github.com/repos/aquasecurity/trivy/releases/latest)
        VERSION=$(echo "$LATEST_RELEASE" | jq -r .tag_name)
        VERSION_NO_V=$(echo "$VERSION" | sed 's/^v//')
        DOWNLOAD_URL=$(echo "$LATEST_RELEASE" | jq -r ".assets[] | select(.name | endswith(\"Linux-64bit.tar.gz\")) | .browser_download_url")
        echo "Latest Version: $VERSION_NO_V"
        echo "Downloading Trivy from: $DOWNLOAD_URL"
        wget -qO- "$DOWNLOAD_URL" | tar zxv -C /usr/local/bin trivy
        echo "Trivy installed version:"
        trivy --version
      shell: bash

And then use the action in other workflows:

 using: "composite"
  steps:
    - uses: actions/checkout@v4.2.0
    - uses: <org>/workflow-library/.github/actions/install-trivy@master
    # Use Trivy CLI to specify the public mirror registry and avoid GHCR toomanycalls error
    - name: Dependency Scan CLI
      shell: bash
      run: |
        trivy --format json \
         --exit-code ${{ inputs.failure_code }} \
         --db-repository public.ecr.aws/aquasecurity/trivy-db:2 \
         --pkg-types library \
         --scanners vuln \
         --severity ${{ inputs.severity }} \
         fs . > trivy-oss-report.json

Answer 83 · 2024-10-09T13:35:16.000Z

does TRIVY_CHECKS_BUNDLE_REPOSITORY work like TRIVY_DB_REPOSITORY accepting comma separated list after 0.28.0 ?release

Answer 84 · 2024-10-09T14:14:45.000Z

Surely it can t be hard @simar7 to have automatic fallback to other registrys as a feature! We shouldn't be having to edit our actions to add this config otherwise its a breaking change! How hard could it be to have it mirrored in multiple locations and then just have a list of these it trys first inside the actual action!

Answer 85 · 2024-10-09T16:39:05.000Z

This is an open source project we use for free, not a paid proprietary software you are paying to use, so there is no point in being offensive.

Go is not my strength, but I have made a small PR with the changes (I believe) needed to add this other AWS mirror Registry as part of the default registries along with GHRC. Hopefully it can pass.

aquasecurity/trivy#7697