Scikit processing example - Client LogStream error
gerold-csendes-epam opened this issue · 5 comments
When running the the local scikit-learn processing srcipt on a notebook instance the training job finishes successfully but I get the following error (I substituted the user and and resource id-s with ....):
ClientError: An error occurred (AccessDeniedException) when calling the DescribeLogStreams operation: User: ... is not authorized to perform: logs:DescribeLogStreams on resource: ... -group:/aws/sagemaker/ProcessingJobs:log-stream
It seems like it is trying to write logs which is not really necessary in the local mode, is it? Maybe the issue is that I run the script on sagemaker?
Hello @gerold-csendes-epam
It could be a bug in SageMaker SDK.
Can you please attach the logs and the output of pip freeze command?
Thanks
What do you exactly mean by logs? Do you mean Cloudwatch logs? I checked and there is none written when submitting the local processing job from the tutorial notebook.
Here is the result of the pip freeze:
aiobotocore @ file:///tmp/build/80754af9/aiobotocore_1615487469391/work
aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1614280892404/work
aioitertools @ file:///home/conda/feedstock_root/build_artifacts/aioitertools_1605140085294/work
async-timeout==3.0.1
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1605083924122/work
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache==1.6.1
bcrypt==3.2.0
boto3 @ file:///home/conda/feedstock_root/build_artifacts/boto3_1616540878956/work
botocore @ file:///home/conda/feedstock_root/build_artifacts/botocore_1616536672126/work
brotlipy==0.7.0
category-encoders @ file:///home/conda/feedstock_root/build_artifacts/category_encoders_1588199510095/work/dist
certifi==2020.12.5
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1606236101993/work
chardet @ file:///home/conda/feedstock_root/build_artifacts/chardet_1610093490430/work
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1615405999564/work
decorator==4.4.2
distro==1.5.0
docker==4.4.4
docker-compose==1.28.6
dockerpty==0.4.1
docopt==0.6.2
fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1614107550004/work
google-pasta==0.2.0
idna==2.10
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1615778734765/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1613852938555/work/dist/ipykernel-5.5.0-py3-none-any.whl
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1614393573755/work
ipython-genutils==0.2.0
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1610146791023/work
jmespath @ file:///home/conda/feedstock_root/build_artifacts/jmespath_1589369830981/work
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1612898609989/work
jsonschema==3.2.0
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1615693636836/work
jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1612125275706/work
mock @ file:///home/conda/feedstock_root/build_artifacts/mock_1610094566888/work
multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1610319019165/work
numexpr @ file:///home/conda/feedstock_root/build_artifacts/numexpr_1614971340455/work
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1612953005674/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1612459636436/work
pandas==1.2.3
paramiko==2.7.2
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1607618318316/work
patsy==0.5.1
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1616432837031/work
protobuf==3.15.6
protobuf3-to-dict @ file:///home/conda/feedstock_root/build_artifacts/protobuf3-to-dict_1610115095957/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1615243893546/work
PyNaCl==1.4.0
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1608055815057/work
pyparsing==2.4.7
pyrsistent==0.17.3
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1610291447907/work
python-dateutil==2.8.1
python-dotenv==0.15.0
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1612179539967/work
PyYAML==5.4.1
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1614611703677/work
requests==2.25.1
s3fs @ file:///home/conda/feedstock_root/build_artifacts/s3fs_1608058532532/work
s3transfer @ file:///home/conda/feedstock_root/build_artifacts/s3transfer_1616166519874/work
sagemaker==2.31.1
scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1605985735074/work
scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1615955928702/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
smdebug-rulesconfig @ file:///home/conda/feedstock_root/build_artifacts/smdebug-rulesconfig_1608558075041/work
statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1612273599609/work
tables @ file:///home/conda/feedstock_root/build_artifacts/pytables_1610156075390/work
texttable==1.6.3
threadpoolctl @ file:///tmp/tmp79xdzxkt/threadpoolctl-2.1.0-py3-none-any.whl
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1610094706440/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1602771532708/work
typing-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1602702424206/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1615828766818/work
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
websocket-client==0.58.0
wrapt @ file:///home/conda/feedstock_root/build_artifacts/wrapt_1610094884173/work
yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1610354134018/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1614945704755/work
Got it. I see you are using the latest version of SageMaker SDK.
Logs I mean by the output of the Pytjon program.
Just to make sure again, the processing job finishes successfully, but you get the error message about the describe logs IAM role?
The output I get is the one that I shared in the description. The message just gets truncated for some reason. This is the last thing that gets outputted before the error:
405xfy7qlb-algo-1-xyh7a | Processing Complete
405xfy7qlb-algo-1-xyh7a exited with code 0
Aborting on container exit...
===== Job Complete =====
Yeah, your description is correct.
Additionally, I can get rid of the error by specifying logs=False argument.
OK.
You can open an issue in SageMaker Python SDK GitHub Repo, and it will be handled as per Service Team priority.