Cannot determine latest tag for policy: version list argument cannot be empty
leptonyu opened this issue · 8 comments
I got this error.
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
name: plex
namespace: flux-system
spec:
imageRepositoryRef:
name: plex
filterTags:
pattern: '^(?P<ver>[0-9]+(\.[0-9]+){2})\.(?P<ts>[0-9]+)-.*'
extract: '$ver'
policy:
semver:
range: '1.x'
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
name: plex
namespace: flux-system
spec:
image: plexinc/pms-docker
interval: 5m0s│ Type Reason Age From Message
│ ---- ------ ---- ---- -------
│ Normal info 18m (x69 over 4h45m) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
│ Warning error 18m (x9 over 18m) image-reflector-controller Cannot determine latest tag for policy: version list argument cannot be empty
│ Normal info 16m (x5 over 18m) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
│ Warning error 15m (x14 over 16m) image-reflector-controller Cannot determine latest tag for policy: version list argument cannot be empty
│ Normal info 14m (x5 over 15m) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
│ Warning error 14m (x9 over 14m) image-reflector-controller Cannot determine latest tag for policy: version list argument cannot be empty
│ Normal info 12m (x5 over 14m) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
│ Warning error 12m (x9 over 12m) image-reflector-controller Cannot determine latest tag for policy: version list argument cannot be empty
│ Normal info 12m (x3 over 12m) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
│ Warning error 9m40s (x9 over 9m41s) image-reflector-controller Cannot determine latest tag for policy: version list argument cannot be empty
│ Normal info 8m3s (x6 over 9m39s) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
│ Warning error 4m52s (x9 over 4m53s) image-reflector-controller Cannot determine latest tag for policy: version list argument cannot be empty
│ Normal info 2m12s (x5 over 4m51s) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
│ Warning error 116s (x9 over 118s) image-reflector-controller Cannot determine latest tag for policy: version list argument cannot be empty
│ Normal info 11s (x5 over 115s) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
Hi, thanks for reporting the issue.
I tried reproducing it but it seems to be working fine.
ImagePolicy status:
status:
conditions:
- lastTransitionTime: "2022-08-03T18:56:58Z"
message: 'Latest image tag for ''plexinc/pms-docker'' resolved to: 1.28.0.5999-97678ded3'
reason: ReconciliationSucceeded
status: "True"
type: Ready
latestImage: plexinc/pms-docker:1.28.0.5999-97678ded3
observedGeneration: 1And there's just success event:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal info 2m35s (x3 over 2m35s) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
Can you share the status of your ImagePolicy object?
Also, can you provide more details about your setup? Version of image-reflector-controller and overall flux version?
Hi @darkowlzz , the ImagePolicy object status is okay.
I got this error logs from image-reflector-controller, and as you can see, it creates event for ImagePolicy.
│ {"level":"error","ts":"2022-08-04T01:10:31.072Z","logger":"controller.imagepolicy","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"Im
│ agePolicy","name":"plex","namespace":"flux-system","error":"Cannot determine latest tag for policy: version list argument cannot be empty"}
│ {"level":"error","ts":"2022-08-04T01:10:31.090Z","logger":"controller.imagepolicy","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"Im
│ agePolicy","name":"plex","namespace":"flux-system","error":"Cannot determine latest tag for policy: version list argument cannot be empty"}
➜ flux git:(main) flux version
flux: v0.31.4
helm-controller: v0.22.2
image-automation-controller: v0.23.5
image-reflector-controller: v0.19.4
kustomize-controller: v0.26.3
notification-controller: v0.24.1
source-controller: v0.25.10
Automation.
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 1m0s
sourceRef:
kind: GitRepository
name: flux-system
git:
checkout:
ref:
branch: main
commit:
author:
email: leptonyu@users.noreply.github.com
name: leptonyu
messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
push:
branch: main
update:
path: ./clusters/nuc
strategy: Setters│ Status:
│ Conditions:
│ Last Transition Time: 2022-08-03T14:31:40Z
│ Message: Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
│ Reason: ReconciliationSucceeded
│ Status: True
│ Type: Ready
│ Latest Image: plexinc/pms-docker:1.28.0.5999-97678ded3
│ Observed Generation: 12
│ Events:
│ Type Reason Age From Message
│ ---- ------ ---- ---- -------
│ Normal info 86s (x129 over 10h) image-reflector-controller Latest image tag for 'plexinc/pms-docker' resolved to: 1.28.0.5999-97678ded3
The status in the last snippet looks fine. I'm not sure why you had those failures before. If your ImageRepository has scanned the repo tags, ImagePolicy should not have any issue determining the latest tag.
Looking at the code, the error is from https://github.com/fluxcd/image-reflector-controller/blob/v0.19.4/controllers/imagepolicy_controller.go#L205, which is returned when there's an error in getting the latest tag. The inner error in that error seems to be from https://github.com/fluxcd/image-reflector-controller/blob/v0.19.4/internal/policy/semver.go#L49 , which can happen when the versions list, or the tags read from the database is empty. So, if your ImageRepository is good, based on the status, ImagePolicy should also be good.
Can you check if your ImageRepository was fine when this was happening?
Since I'm unable to reproduce it, I'm not sure what to do about it. Are you somehow running multiple replicas of image-reflector-controller? Because that's the only situation I can think of when this could happen. One instance of image-reflector has the tags in the database, but the other one doesn't and results in version list to be empty.
ImageRepository Status looks fine. And I have one instance of image-reflector.
│ Spec:
│ Image: plexinc/pms-docker
│ Interval: 5m0s
│ Status:
│ Canonical Image Name: index.docker.io/plexinc/pms-docker
│ Conditions:
│ Last Transition Time: 2022-08-04T01:43:19Z
│ Message: successful scan, found 142 tags
│ Reason: ReconciliationSucceeded
│ Status: True
│ Type: Ready
{"level":"error","ts":"2022-08-05T09:15:02.966Z","logger":"controller.imageupdateautomation","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"flux-system","namespace":"flux-system","error":"unable to fetch remote 'ssh://git@github.com/leptonyu/flux': EOF"}
And I also got error from image-automation-controller..
@leptonyu both issues could be caused by some sort of network issue, maybe related to routing or throttling.
I wonder whether you could test the same against a git server or/and container registry within the same network, to confirm whether we can rule that out.
It's working by upgrading to flux v0.32.0.
The key issue was early EOF(the SSH key may not have write access to the repository)
I deleted the secret flux-system/flux-system. And bootstrap again.