aws/sagemaker-python-sdk

[Bug] Environment Variables through Estimators in Local Mode Does Not Work

Closed this issue · 2 comments

ssmall41 commented

Describe the bug
Estimators accept an argument for setting environment variables in training jobs. However providing values here does not actually set them in the container when using local mode. A workaround is to pass the values as hyperparameters and then use them appropriately in the container code or training script.

To reproduce
For example, with the PyTorch estimator:

from sagemaker.pytorch import PyTorch
estimator = PyTorch(instance_type="local", environment={"MYVAR": "got_it"}, ...)

Calls to os.environ["MYVAR"] do not see "MYVAR" defined. If instead I use instance_type="ml.g4dn.xlarge", then "MYVAR" is defined in the container, as expected.

Expected behavior
"MYVAR" should be defined in the container.

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.72.3
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch
  • Framework version: 1.9.0
  • Python version: py38
  • CPU or GPU: GPU
  • Custom Docker image (Y/N): No

I can also produce this with Tensorflow images as well, so I suspect it's not dependent on the framework.

aamster commented

There should at least be a warning that the environment variables are being ignored because of local mode

  • 👍1
mufaddal-rohawala commented

Thanks for you feedback @ssmall41 and @aamster, and agree there should have been warning.

With #3015 this issue should now be resolved.

  • 🎉2