Lookups are very slow
Closed this issue · 9 comments
I am finding that lookups are very slow, to the extent where sometimes it times out and the puppet run fails.
This could of course be down to the web server providing the json.
It does seem make an http request every time it looks up something even if it has already looked up that file during that run.
Do you have any recommendations for working around this, e.g caching?
I've thought about it - Im not sure if the caching should be implemented in hiera or done on whatever endpoint you are talking to - I've not seen any delays running hiera-http talking to couchdb for example.
Happy to discuss the validity of adding such a feature though
Well the web server hiera is contacting is on the other end of a slow connection. So I think caching is necessary in my situation.
I've thought about putting something like nginx in as a cache but I would be restricted by the settings in there. I would have to guess how long a puppet run would last and set the cache life to that number. It's do-able but adds another level of complexity.
It just seems inefficient that hiera-http makes multiple https requests for the same file every time puppet runs (can be hundreds if the modules or manifests are complex). Doing this over a slow connection causes the puppet run to time-out and fail.
Surely it could cache each unique request for the duration of each puppet run?
Yep - you have some good points.
Im not sure if caching is on the roadmap for Hiera itself - it seems more sensible to be able to set caching within hiera itself regardless of the backend that is being used rather than each backend implement it separately - maybe @hlindberg or @puppetlabs can comment on that......
Yes that makes sense.
Although there are no concrete plans to add caching, we know this is an issue.
Based on puppetlabs/hiera#98 and what I've heard in #puppet, I think it's up to the backend to support caching.
@erik-stephens PR #18 currently has a proposal for caching which I'll try and get merged into the codebase this week if it looks good.
Excellent! I've been using that version for about a week now without issues.