jaeger: increaese retention and use a pesistent storage type
Closed this issue · 0 comments
cirocosta commented
Hey,
At the moment, jaeger
is configured to use the memory-based storage type, with (I guess) the default retention limit, which seems to be growing more than the machines we use can really hld:
NAME READY STATUS RESTARTS AGE
jaeger-jaeger-operator-5878c646f5-fjvm7 1/1 Running 0 7d2h
simplest-6fbbdc7794-9sc8p 0/1 Evicted 0 5d
simplest-6fbbdc7794-mckk6 1/1 Running 0 4h13m
simplest-6fbbdc7794-ptz2w 0/1 Evicted 0 5d6h
simplest-6fbbdc7794-vdbfp 0/1 Evicted 0 5d14h
simplest-6fbbdc7794-w7cs7 0/1 Evicted 0 7d6h
simplest-6fbbdc7794-zxx25 0/1 Evicted 0 17h
...
Message: The node was low on resource: memory. Container jaeger was using 10083764Ki, which exceeds its request of 0.
It seems like a simple-enough approach would be to have the badger
storage type enabled, without the need of bringing an entire "full on" storage backend (cassandra/elasticsearch) up: https://www.jaegertracing.io/docs/1.13/deployment/#badger-local-storage
Aside from that, it'd be good to have the pods deployed with proper requests and resource limits, rather than consume those on a best-effort basis.
Thanks