Logging is not saved during federated learning experiments
Closed this issue · 1 comments
JMGaljaard commented
Bug Report
Current Behavior
The default logging directory used by the Federator in experiments is written to a directory that is not mounted by a PVC during remote training. As a result experiment data will be lost after training.
Expected behavior/code
Logging data is always written to to a non-conflicting directory or file after/during training when running in a cluster.
Environment
- Python version: 3.9
- PyTorch version: 1.9.1
- OS: *
- Kubernetes version: All
- Platform: Minikube, GKE cluster
Possible Solution
Add logging directory path to the Federator.
Additional context/Screenshots
JMGaljaard commented
Resolved in #25