Reporter information appears to be cached and not overwritten
flikka opened this issue · 0 comments
flikka commented
Example:
- You built a machine with the
remote_logging: True
. - Then, you change to
remote_logging: False
If the cache gets a hit (ie. you havent changed other things in the machine), theremote_logging:
change is "ignored", since the reporters patched into the machine when this was set toTrue
were cached with that machine.
This would probably not be a big issue, BUT, we had a situation where the MLflow logging failed. But the machine was built, and cached with the MLflowReporter, so changing the remote_logging to False didn't stop the (failing) attempt to do remote logging, as the machine with the reporter attached was retrieved from the cache.
Solution: Either include the reporter in the cache key, or "patch" the machine with the information