aws/studio-lab-examples

Sagemaker Studio Lab running notebook stops when disconnected from Internet

samitChak opened this issue · 3 comments

samitChak commented

I am new to AWS Sagemaker Studio Lab.

I was testing with a looping program. I saved the program and then started running it. Then I disconnected the internet. After sometime I reconnected and found that the notebook was not running.

Sagemaker is supposed to run on cloud after I start the run and it will not require my local internet connection - right?

Then why did it stop running? Is there any setting that I need to do for the program to run for the entire session duration even if I close my local machine.

MicheleMonclova commented

Interesting....
A few clarifying questions.
I am assuming you kicked of a training job then closed laptop or something, right?
And when you returned the job never completed? or it completed but there was no connection to receive the results.

samitChak commented

Interesting.... A few clarifying questions. I am assuming you kicked of a training job then closed laptop or something, right? And when you returned the job never completed? or it completed but there was no connection to receive the results.

Yes, I started the job (a simple for loop to print something every 10 seconds)...have seen it throw some initial results while running.....then I disconnected from the internet....after a while, I reconnected to internet, I logged in to Studio Lab and saw that the program did not run.....it got interrupted.

MicheleMonclova commented

Hi, after investigating this it looks like this is the expected behavior of JupyterLab. That is, if the internet connection is disconnected it will stop. However, if you are interested in running a training job on Studio Lab, even after the session ends, then you can schedule a job.

See documentation here: https://docs.aws.amazon.com/sagemaker/latest/dg/scheduled-notebook-installation.html

The upside is you can pick different instance types, and have it run as long as you want, even if you are disconnected.
The downside is, it will need to run in your AWS account and you will incur a cost.

We haven't advertised this a lot so let me know if you have questions.