Allow for disabling the upstream proxy cache, and/or caching locally
briantist opened this issue · 1 comments
Right now, using an upstream always caches the upstream results in Artifactory. But this turns what could have been a purely read-only usage (available without artifactory authentication if so configured), into a usage that usually needs an API key so that the cache can be written.
This fits with one of the main design goals I had, which was to avoid 429 throttling errors from an upstream Galaxy server when hit from multiple places, but it there are valid uses (like limited command line use on local dev machines) that would benefit from being able to run without an API key, where throttling is not really a concern, but having a single upstream for both local and remote collections is important (dependency resolving for example).
Also of note is that ansible-galaxy caches request responses too, so the local workstation use case needs a cache even less than very ephemeral use cases like CI runs.
So to that end I have two ideas:
- Make caching upstream responses optional
- This is easier to implement: a user can set this when they don't have an API key and would otherwise need one to write, and the proxying will still work it will just not be caching
- sort of ironically, the hardest part of this is probably the downloads themselves, only because the way it's written now, the file is downloaded from the upstream, uploaded to artifactory, and then streamed from artifactory to the client. Shouldn't be hard to workaround, just need to give it a little thought
- The other idea is to allow caching locally, like in temp files or something. Either instead of or in addition to the Artifactory cache.
- "Instead of" is probably easier to implement
- it means more requests to the upstream when usage of galactory scales horizontally (lots of CI jobs, local workstation usage)
- it's faster for long-running processes (don't need additional requests to artifactory)
- "In addition" is a little more difficult because I need to determine precedence.
- local access is faster, but might be more out of date (not sure that's a concern actually...)
- should precedence be configurable?
- should we have a mode that defaults to remote cache but allows graceful fallback to local cache when authentication fails? that sounds like a bad idea
- "Instead of" is probably easier to implement
The options are not mutually exclusive, so I think being able to disable the cache is a better first step.