Informer.watch is not guaranteed to receive all changes during restart
DocX opened this issue · 0 comments
DocX commented
Using the Infomer.watch
method that yields notices from the watch
endpoint is not guaranteed to receive all events in case the informer restarts the watcher:
- Informer fills/replaces cache from
get
request and stores the resource version - Informer starts
watch
request with the resource version from the get - Watch request stops (timeout expires, connection drops, ...)
- Some changes happen in Kubernetes cluster before the next get request (those will not be yielded to the
watch
block) - Repeat 1: the get will receive the state with the changes happened in 4 and update list
- Repeat 2: but the watch now will start after those changes, so
watch
will not be yielded with the changes happened in 4.
Potential solutions:
- Document the behaviour of that
watch
interface can miss changes (and thatlist
is the only source of truth) - Only request
get
once wheninformer.start_worker
is called, when watch request stops, only restart the watch with the latest seen resource version