device kernel image is invalid
geraldstanje1 opened this issue · 0 comments
geraldstanje1 commented
hi,
i tried to torch.compile the model using: model.model_body[0].auto_model = torch.compile(model.model_body[0].auto_model, mode="reduce-overhead")and get the following error:
error:
W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised: | 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised:
W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
used image: 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:2.1.0-transformers4.37.0-gpu-py310-cu1
torch compile: https://pytorch.org/docs/stable/generated/torch.compile.html
full error log:
2024-05-06T22:59:03.955-04:00 Collecting scikit-learn==1.2.2 (from -r /opt/ml/model/code/requirements.txt (line 1)) Downloading scikit_learn-1.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
2024-05-06T22:59:03.955-04:00 Collecting setfit==1.0.3 (from -r /opt/ml/model/code/requirements.txt (line 2)) Downloading setfit-1.0.3-py3-none-any.whl.metadata (11 kB)
2024-05-06T22:59:03.955-04:00 Collecting pandas==1.5.3 (from -r /opt/ml/model/code/requirements.txt (line 3)) Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: torch==2.1.0 in /opt/conda/lib/python3.10/site-packages (from -r /opt/ml/model/code/requirements.txt (line 4)) (2.1.0+cu118)
2024-05-06T22:59:03.955-04:00 Collecting optimum (from -r /opt/ml/model/code/requirements.txt (line 5)) Downloading optimum-1.19.1-py3-none-any.whl.metadata (19 kB)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: numpy>=1.17.3 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.22.4)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: scipy>=1.3.2 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.12.0)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: joblib>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.3.2)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (3.2.0)
2024-05-06T22:59:03.955-04:00 Collecting datasets>=2.3.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading datasets-2.19.1-py3-none-any.whl.metadata (19 kB)
2024-05-06T22:59:03.955-04:00 Collecting sentence-transformers>=2.2.1 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading sentence_transformers-2.7.0-py3-none-any.whl.metadata (11 kB)
2024-05-06T22:59:03.955-04:00 Collecting evaluate>=0.3.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading evaluate-0.4.2-py3-none-any.whl.metadata (9.3 kB)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: huggingface-hub>=0.13.0 in /opt/conda/lib/python3.10/site-packages (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (0.20.3)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: packaging in /opt/conda/lib/python3.10/site-packages (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (23.2)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/lib/python3.10/site-packages (from pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (2.8.2)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (2023.4)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.13.1)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (4.9.0)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (1.12)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.2.1)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.1.3)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: fsspec in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (2023.12.2)
2024-05-06T22:59:03.955-04:00 Collecting coloredlogs (from optimum->-r /opt/ml/model/code/requirements.txt (line 5)) Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)
2024-05-06T22:59:04.348-04:00 Requirement already satisfied: transformers<4.41.0,>=4.26.0 in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (4.37.0)
2024-05-06T22:59:04.348-04:00 Collecting pyarrow>=12.0.0 (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading pyarrow-16.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
2024-05-06T22:59:04.348-04:00 Collecting pyarrow-hotfix (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB)
2024-05-06T22:59:04.348-04:00 Collecting dill<0.3.9,>=0.3.0 (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)
2024-05-06T22:59:04.348-04:00 Requirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (2.31.0)
2024-05-06T22:59:04.348-04:00 Requirement already satisfied: tqdm>=4.62.1 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (4.64.1)
2024-05-06T22:59:04.599-04:00 Collecting xxhash (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
2024-05-06T22:59:04.849-04:00 Collecting multiprocess (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)
2024-05-06T22:59:04.849-04:00 Collecting aiohttp (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB)
2024-05-06T22:59:04.849-04:00 Collecting huggingface-hub>=0.13.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading huggingface_hub-0.23.0-py3-none-any.whl.metadata (12 kB)
2024-05-06T22:59:05.100-04:00 Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (6.0.1)
2024-05-06T22:59:05.100-04:00 Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (1.16.0)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: Pillow in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.1->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (10.2.0)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (2023.12.25)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: tokenizers<0.19,>=0.14 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.15.1)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: safetensors>=0.3.1 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.4.2)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.1.99)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: protobuf in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (3.20.2)
2024-05-06T22:59:05.601-04:00 Collecting humanfriendly>=9.1 (from coloredlogs->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (2.1.4)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (1.3.0)
2024-05-06T22:59:05.601-04:00 Collecting aiosignal>=1.1.2 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
2024-05-06T22:59:05.852-04:00 Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (23.2.0)
2024-05-06T22:59:06.102-04:00 Collecting frozenlist>=1.1.1 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
2024-05-06T22:59:06.102-04:00 Collecting multidict<7.0,>=4.5 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
2024-05-06T22:59:06.352-04:00 Collecting yarl<2.0,>=1.0 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
2024-05-06T22:59:06.352-04:00 Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)
2024-05-06T22:59:06.352-04:00 Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (3.3.2)
2024-05-06T22:59:06.352-04:00 Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (3.4)
2024-05-06T22:59:06.352-04:00 Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (1.26.18)
2024-05-06T22:59:06.607-04:00 Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (2023.11.17)
2024-05-06T22:59:06.608-04:00 Downloading scikit_learn-1.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.6/9.6 MB 87.0 MB/s eta 0:00:00
2024-05-06T22:59:06.609-04:00 Downloading setfit-1.0.3-py3-none-any.whl (75 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.9/75.9 kB 38.4 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.1/12.1 MB 130.5 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading optimum-1.19.1-py3-none-any.whl (417 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 417.0/417.0 kB 122.1 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading datasets-2.19.1-py3-none-any.whl (542 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 542.0/542.0 kB 138.6 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading evaluate-0.4.2-py3-none-any.whl (84 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.1/84.1 kB 43.6 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading huggingface_hub-0.23.0-py3-none-any.whl (401 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.2/401.2 kB 120.6 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading sentence_transformers-2.7.0-py3-none-any.whl (171 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 171.5/171.5 kB 78.0 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 kB 24.8 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading dill-0.3.8-py3-none-any.whl (116 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 59.0 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 155.8 MB/s eta 0:00:00
2024-05-06T22:59:06.856-04:00 Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 45.1 MB/s eta 0:00:00
2024-05-06T22:59:07.130-04:00 Downloading pyarrow-16.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (40.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 MB 88.9 MB/s eta 0:00:00
2024-05-06T22:59:07.133-04:00 Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 60.0 MB/s eta 0:00:00
2024-05-06T22:59:07.363-04:00 Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)
2024-05-06T22:59:07.364-04:00 Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 82.8 MB/s eta 0:00:00
2024-05-06T22:59:07.364-04:00 Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
2024-05-06T22:59:07.364-04:00 Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
2024-05-06T22:59:07.364-04:00 Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.5/239.5 kB 71.2 MB/s eta 0:00:00
2024-05-06T22:59:07.364-04:00 Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.3/124.3 kB 49.2 MB/s eta 0:00:00
2024-05-06T22:59:08.868-04:00 Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 301.6/301.6 kB 103.5 MB/s eta 0:00:00
2024-05-06T22:59:12.959-04:00 Installing collected packages: xxhash, pyarrow-hotfix, pyarrow, multidict, humanfriendly, frozenlist, dill, async-timeout, yarl, scikit-learn, pandas, multiprocess, huggingface-hub, coloredlogs, aiosignal, aiohttp, sentence-transformers, datasets, optimum, evaluate, setfit Attempting uninstall: scikit-learn Found existing installation: scikit-learn 1.4.0 Uninstalling scikit-learn-1.4.0: Successfully uninstalled scikit-learn-1.4.0 Attempting uninstall: pandas Found existing installation: pandas 2.2.0 Uninstalling pandas-2.2.0: Successfully uninstalled pandas-2.2.0
2024-05-06T22:59:17.155-04:00 Attempting uninstall: huggingface-hub Found existing installation: huggingface-hub 0.20.3 Uninstalling huggingface-hub-0.20.3: Successfully uninstalled huggingface-hub-0.20.3
2024-05-06T22:59:17.155-04:00 Successfully installed aiohttp-3.9.5 aiosignal-1.3.1 async-timeout-4.0.3 coloredlogs-15.0.1 datasets-2.19.1 dill-0.3.8 evaluate-0.4.2 frozenlist-1.4.1 huggingface-hub-0.23.0 humanfriendly-10.0 multidict-6.0.5 multiprocess-0.70.16 optimum-1.19.1 pandas-1.5.3 pyarrow-16.0.0 pyarrow-hotfix-0.6 scikit-learn-1.2.2 sentence-transformers-2.7.0 setfit-1.0.3 xxhash-3.4.1 yarl-1.9.4
2024-05-06T22:59:17.155-04:00 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
2024-05-06T22:59:17.155-04:00 [notice] A new release of pip is available: 23.3.2 -> 24.0
2024-05-06T22:59:17.406-04:00 [notice] To update, run: pip install --upgrade pip
2024-05-06T22:59:17.656-04:00 Warning: MMS is using non-default JVM parameters: -XX:-UseContainerSupport
2024-05-06T22:59:18.157-04:00 WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2024-05-06T22:59:18.157-04:00 2024-05-07T02:59:18,138 [INFO ] main com.amazonaws.ml.mms.ModelServer -
2024-05-06T22:59:18.157-04:00 MMS Home: /opt/conda/lib/python3.10/site-packages
2024-05-06T22:59:18.157-04:00 Current directory: /
2024-05-06T22:59:18.157-04:00 Temp directory: /home/model-server/tmp
2024-05-06T22:59:18.157-04:00 Number of GPUs: 1
2024-05-06T22:59:18.157-04:00 Number of CPUs: 4
2024-05-06T22:59:18.157-04:00 Max heap size: 3934 M
2024-05-06T22:59:18.157-04:00 Python executable: /opt/conda/bin/python3.10
2024-05-06T22:59:18.157-04:00 Config file: /etc/sagemaker-mms.properties
2024-05-06T22:59:18.157-04:00 Inference address: http://0.0.0.0:8080
2024-05-06T22:59:18.157-04:00 Management address: http://0.0.0.0:8080
2024-05-06T22:59:18.157-04:00 Model Store: /
2024-05-06T22:59:18.157-04:00 Initial Models: model=/opt/ml/model
2024-05-06T22:59:18.157-04:00 Log dir: null
2024-05-06T22:59:18.157-04:00 Metrics dir: null
2024-05-06T22:59:18.157-04:00 Netty threads: 0
2024-05-06T22:59:18.157-04:00 Netty client threads: 0
2024-05-06T22:59:18.157-04:00 Default workers per model: 1
2024-05-06T22:59:18.157-04:00 Blacklist Regex: N/A
2024-05-06T22:59:18.157-04:00 Maximum Response Size: 6553500
2024-05-06T22:59:18.157-04:00 Maximum Request Size: 6553500
2024-05-06T22:59:18.157-04:00 Preload model: false
2024-05-06T22:59:18.157-04:00 Prefer direct buffer: false
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,145 [INFO ] main com.amazonaws.ml.mms.ModelServer - Loading initial models: /opt/ml/model preload_model: false
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,190 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-9000-model
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,261 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model_service_worker started with args: --sock-type unix --sock-name /home/model-server/tmp/.mms.sock.9000 --handler sagemaker_huggingface_inference_toolkit.handler_service --model-path /opt/ml/model --model-name model --preload-model false --tmp-dir /home/model-server/tmp
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,262 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Listening on port: /home/model-server/tmp/.mms.sock.9000
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [PID] 67
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - MMS worker started.
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Python runtime: 3.10.9
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,263 [INFO ] main com.amazonaws.ml.mms.wlm.ModelManager - Model model loaded.
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,267 [INFO ] main com.amazonaws.ml.mms.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,273 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.mms.sock.9000
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,324 [INFO ] main com.amazonaws.ml.mms.ModelServer - Inference API bind to: http://0.0.0.0:8080
2024-05-06T22:59:18.409-04:00 Model server started.
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,328 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Connection accepted: /home/model-server/tmp/.mms.sock.9000.
2024-05-06T22:59:21.932-04:00 2024-05-07T02:59:18,329 [WARN ] pool-3-thread-1 com.amazonaws.ml.mms.metrics.MetricCollector - worker pid is not available yet.
2024-05-06T22:59:22.179-04:00 2024-05-07T02:59:21,694 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - PyTorch version 2.1.0+cu118 available.
2024-05-06T22:59:22.179-04:00 2024-05-07T02:59:21,955 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Load pretrained SentenceTransformer: /opt/ml/model
2024-05-06T22:59:23.697-04:00 2024-05-07T02:59:22,156 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Use pytorch device_name: cuda
2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,557 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 18
2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,818 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - /opt/conda/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator LogisticRegression from version 1.3.2 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,819 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,819 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - warnings.warn(
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,953 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - device=cuda
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,954 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model loaded!!!
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,955 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Current device - cuda
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,976 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch.compile for model SetFitModel(model_body=SentenceTransformer(
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,977 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,978 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,978 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (2): Normalize()
2024-05-06T22:59:24.458-04:00 2024-05-07T02:59:23,980 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ), model_head=LogisticRegression(), multi_target_strategy=None, normalize_embeddings=False, labels=['DOCUMENT', 'POLICY'], model_card_data=SetFitModelCardData(language=None, license=None, tags=['setfit', 'sentence-transformers', 'text-classification', 'generated_from_setfit_trainer'], model_name='SetFit with sentence-transformers/all-MiniLM-L6-v2', model_id=None, dataset_name=None, dataset_id=None, dataset_revision=None, task_name=None, st_id='sentence-transformers/all-MiniLM-L6-v2', hyperparameters={}, eval_results_dict={}, eval_lines_list=[], metric_lines=[], widget=[], predict_example=None, label_example_list=[], tokenizer_warning=False, train_set_metrics_list=[], train_set_sentences_per_label_list=[], code_carbon_callback=None, num_classes=None, best_model_step=None, metrics=['accuracy'], pipeline_tag='text-classification', library_name='setfit', version={'python': '3.10.9', 'setfit': '1.0.3', 'sentence_transformers': '2.7.0', 'transformers': '4.37.0', 'torch': '2.1.0+cu118', 'datasets': '2.19.1', 'tokenizers': '0.15.1'}))
2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,347 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model compiled successfully...
2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,348 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Model model loaded io_fd=de239efffefa3af1-00000029-00000000-1db2d9548cfdc087-9fd56c90
2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,349 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 5989
2024-05-06T22:59:28.210-04:00 2024-05-07T02:59:24,350 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-model-1
2024-05-06T22:59:32.953-04:00 2024-05-07T02:59:28,186 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 1
2024-05-06T22:59:38.015-04:00 2024-05-07T02:59:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T22:59:42.953-04:00 2024-05-07T02:59:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T22:59:47.953-04:00 2024-05-07T02:59:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T22:59:52.953-04:00 2024-05-07T02:59:48,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 1
2024-05-06T22:59:57.953-04:00 2024-05-07T02:59:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:02.992-04:00 2024-05-07T02:59:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:07.952-04:00 2024-05-07T03:00:03,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:12.953-04:00 2024-05-07T03:00:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,009 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input_data: {"inputs": ["How to create policy?"]}, content_type: application/json
2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,012 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - decoded input: {'inputs': ['How to create policy?']}, content_type: application/json
2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,013 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - data: {'inputs': ['How to create policy?']}, data_type: <class 'dict'>
2024-05-06T23:00:13.342-04:00 2024-05-07T03:00:13,020 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:17.953-04:00 2024-05-07T03:00:13,178 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:22.953-04:00 2024-05-07T03:00:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:25.975-04:00 2024-05-07T03:00:23,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,734 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:00<?, ?it/s]
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,734 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - predict_fn error: backend='inductor' raised:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,735 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,735 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:12<?, ?it/s]
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,735 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,736 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,736 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,737 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,737 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,737 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,738 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,738 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,739 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,739 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,739 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 258, in handle
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,740 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 12726
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,740 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - response = self.transform_fn(*([self.model, input_data, content_type, accept] + self.transform_extra_arg))
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,740 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 214, in transform_fn
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - predictions = self.predict(*([processed_data, model] + self.predict_extra_arg))
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/ml/model/code/inference.py", line 111, in predict_fn
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise e
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/ml/model/code/inference.py", line 108, in predict_fn
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,741 [INFO ] W-9000-model ACCESS_LOG - /169.254.178.2:42770 "POST /invocations HTTP/1.1" 400 12737
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,742 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return model.predict(inputs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,742 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/setfit/modeling.py", line 560, in predict
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,742 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - embeddings = self.encode(inputs, batch_size=batch_size, show_progress_bar=show_progress_bar)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,743 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/setfit/modeling.py", line 452, in encode
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,743 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.model_body.encode(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,743 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 371, in encode
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,744 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - out_features = self.forward(features)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,744 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,744 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input = module(input)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,744 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,745 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,745 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,745 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,746 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 98, in forward
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,746 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - output_states = self.auto_model(**trans_features, return_dict=False)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,746 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,747 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,747 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,747 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,747 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,748 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,748 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,749 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,749 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,749 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,750 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 490, in catch_errors
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,750 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return callback(frame, cache_entry, hooks, frame_state)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,750 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 641, in _convert_frame
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,751 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - result = inner_convert(frame, cache_size, hooks, frame_state)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,751 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 133, in _fn
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,751 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,752 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 389, in _convert_frame_assert
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,752 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return _compile(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,752 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 569, in _compile
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,752 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - guarded_code = compile_inner(code, one_graph, hooks, transform)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,753 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,753 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,753 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 491, in compile_inner
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,753 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - out_code = transform_code_object(code, transform)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,754 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,754 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - transformations(instructions, code_options)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,754 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 458, in transform
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,755 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - tracer.run()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,755 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2074, in run
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,755 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - super().run()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,755 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,756 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - and self.step()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,756 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,756 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - getattr(self, inst.opname)(inst)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,756 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2162, in RETURN_VALUE
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,757 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.output.compile_subgraph(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,757 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 857, in compile_subgraph
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,757 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,757 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 957, in compile_and_call_fx_graph
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = self.call_user_compiler(gm)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,758 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,759 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1024, in call_user_compiler
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,759 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,759 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1009, in call_user_compiler
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,759 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = compiler_fn(gm, self.example_inputs())
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_gm = compiler_fn(gm, example_inputs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/__init__.py", line 1568, in __call__
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compile_fx(model_, inputs_, config_patches=self.config)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,760 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 961, in compile_fx
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compile_fx(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1150, in compile_fx
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return aot_autograd(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,761 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - cg = aot_module_simplified(gm, example_inputs, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,762 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,762 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = create_aot_dispatcher_function(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,762 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,763 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,763 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,763 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,763 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,764 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fw = compiler(fw_module, flat_args)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,765 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,765 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,765 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1092, in fw_compiler_base
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,765 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return inner_compile(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - inner_compiled_fn = compiler_fn(gm, example_inputs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,766 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 54, in newFunction
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,767 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return old_func(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,768 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 341, in compile_fx_inner
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,768 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,768 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 565, in fx_codegen_and_compile
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,768 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = graph.compile_to_fn()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/graph.py", line 970, in compile_to_fn
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.compile_to_module().call
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/graph.py", line 941, in compile_to_module
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1139, in load_by_key_path
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,769 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - exec(code, mod.__dict__, mod.__dict__)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/home/model-server/tmp/torchinductor_root/av/cavozcyohmg3erjtmk3oyovim5verzsympqf2nkp4bzglt2x6rwi.py", line 424, in <module>
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - async_compile.wait(globals())
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1418, in wait
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - scope[key] = result.result()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1278, in result
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel = self.kernel = _load_kernel(self.kernel_name, self.source_code)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1261, in _load_kernel
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel.precompile()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 168, in precompile
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.launchers = [
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 169, in <listcomp>
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self._precompile_config(c, warm_cache_only_with_cc)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 205, in _precompile_config
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - binary._init_handles()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/triton/compiler/compiler.py", line 683, in _init_handles
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/mms/service.py", line 108, in predict
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self._entry_point(input_batch, self.context)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 267, in handle
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:00:28.232-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - : 400
2024-05-06T23:00:32.953-04:00 2024-05-07T03:00:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:37.953-04:00 2024-05-07T03:00:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:42.953-04:00 2024-05-07T03:00:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:47.953-04:00 2024-05-07T03:00:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:53.000-04:00 2024-05-07T03:00:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:57.953-04:00 2024-05-07T03:00:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:02.953-04:00 2024-05-07T03:00:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:07.986-04:00 2024-05-07T03:01:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:12.953-04:00 2024-05-07T03:01:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:17.953-04:00 2024-05-07T03:01:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:22.952-04:00 2024-05-07T03:01:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:27.952-04:00 2024-05-07T03:01:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:32.987-04:00 2024-05-07T03:01:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:37.953-04:00 2024-05-07T03:01:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:42.953-04:00 2024-05-07T03:01:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:47.990-04:00 2024-05-07T03:01:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:52.953-04:00 2024-05-07T03:01:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:57.952-04:00 2024-05-07T03:01:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:02.985-04:00 2024-05-07T03:01:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:07.953-04:00 2024-05-07T03:02:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:12.953-04:00 2024-05-07T03:02:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:17.953-04:00 2024-05-07T03:02:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:22.953-04:00 2024-05-07T03:02:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:27.990-04:00 2024-05-07T03:02:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:32.953-04:00 2024-05-07T03:02:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:37.953-04:00 2024-05-07T03:02:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:42.990-04:00 2024-05-07T03:02:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:47.953-04:00 2024-05-07T03:02:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:52.953-04:00 2024-05-07T03:02:48,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:02:57.952-04:00 2024-05-07T03:02:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:02.953-04:00 2024-05-07T03:02:58,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:07.953-04:00 2024-05-07T03:03:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:12.952-04:00 2024-05-07T03:03:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:17.952-04:00 2024-05-07T03:03:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:22.953-04:00 2024-05-07T03:03:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:27.990-04:00 2024-05-07T03:03:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:32.953-04:00 2024-05-07T03:03:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:37.952-04:00 2024-05-07T03:03:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:42.989-04:00 2024-05-07T03:03:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:47.952-04:00 2024-05-07T03:03:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:52.952-04:00 2024-05-07T03:03:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:57.979-04:00 2024-05-07T03:03:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:02.953-04:00 2024-05-07T03:03:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:07.953-04:00 2024-05-07T03:04:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:12.953-04:00 2024-05-07T03:04:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:17.953-04:00 2024-05-07T03:04:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:22.990-04:00 2024-05-07T03:04:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:27.952-04:00 2024-05-07T03:04:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:32.953-04:00 2024-05-07T03:04:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:37.953-04:00 2024-05-07T03:04:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:42.952-04:00 2024-05-07T03:04:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:47.952-04:00 2024-05-07T03:04:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:52.952-04:00 2024-05-07T03:04:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:57.988-04:00 2024-05-07T03:04:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:02.952-04:00 2024-05-07T03:04:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:07.953-04:00 2024-05-07T03:05:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:13.052-04:00 2024-05-07T03:05:08,174 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:17.952-04:00 2024-05-07T03:05:13,174 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:05:22.952-04:00 2024-05-07T03:05:18,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:27.985-04:00 2024-05-07T03:05:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:05:32.953-04:00 2024-05-07T03:05:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:37.953-04:00 2024-05-07T03:05:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:42.953-04:00 2024-05-07T03:05:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:47.953-04:00 2024-05-07T03:05:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:52.953-04:00 2024-05-07T03:05:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:57.953-04:00 2024-05-07T03:05:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:02.952-04:00 2024-05-07T03:05:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:07.952-04:00 2024-05-07T03:06:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:12.984-04:00 2024-05-07T03:06:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:17.952-04:00 2024-05-07T03:06:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:22.952-04:00 2024-05-07T03:06:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:27.989-04:00 2024-05-07T03:06:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:32.953-04:00 2024-05-07T03:06:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:37.953-04:00 2024-05-07T03:06:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:06:42.953-04:00 2024-05-07T03:06:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:06:47.953-04:00 2024-05-07T03:06:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:52.991-04:00 2024-05-07T03:06:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:57.953-04:00 2024-05-07T03:06:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:02.953-04:00 2024-05-07T03:06:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:07.953-04:00 2024-05-07T03:07:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:12.952-04:00 2024-05-07T03:07:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:17.952-04:00 2024-05-07T03:07:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:22.953-04:00 2024-05-07T03:07:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:27.953-04:00 2024-05-07T03:07:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:32.953-04:00 2024-05-07T03:07:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:38.016-04:00 2024-05-07T03:07:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:42.953-04:00 2024-05-07T03:07:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:47.952-04:00 2024-05-07T03:07:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:52.994-04:00 2024-05-07T03:07:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:57.953-04:00 2024-05-07T03:07:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:02.953-04:00 2024-05-07T03:07:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:07.987-04:00 2024-05-07T03:08:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:12.953-04:00 2024-05-07T03:08:08,175 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:17.953-04:00 2024-05-07T03:08:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:22.953-04:00 2024-05-07T03:08:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:27.953-04:00 2024-05-07T03:08:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:32.987-04:00 2024-05-07T03:08:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:37.953-04:00 2024-05-07T03:08:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:42.953-04:00 2024-05-07T03:08:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:47.952-04:00 2024-05-07T03:08:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:52.953-04:00 2024-05-07T03:08:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:57.953-04:00 2024-05-07T03:08:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:02.953-04:00 2024-05-07T03:08:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:07.953-04:00 2024-05-07T03:09:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:12.953-04:00 2024-05-07T03:09:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:17.953-04:00 2024-05-07T03:09:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:22.952-04:00 2024-05-07T03:09:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:27.986-04:00 2024-05-07T03:09:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:32.953-04:00 2024-05-07T03:09:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:37.953-04:00 2024-05-07T03:09:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:42.988-04:00 2024-05-07T03:09:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:47.952-04:00 2024-05-07T03:09:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:52.952-04:00 2024-05-07T03:09:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:57.952-04:00 2024-05-07T03:09:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:10:02.953-04:00 2024-05-07T03:09:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:10:07.953-04:00 2024-05-07T03:10:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:10:12.953-04:00 2024-05-07T03:10:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:17.953-04:00 2024-05-07T03:10:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:22.953-04:00 2024-05-07T03:10:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:27.953-04:00 2024-05-07T03:10:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:32.953-04:00 2024-05-07T03:10:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:37.984-04:00 2024-05-07T03:10:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:42.952-04:00 2024-05-07T03:10:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:47.953-04:00 2024-05-07T03:10:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:52.952-04:00 2024-05-07T03:10:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:57.953-04:00 2024-05-07T03:10:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:02.993-04:00 2024-05-07T03:10:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:07.952-04:00 2024-05-07T03:11:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:12.953-04:00 2024-05-07T03:11:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:17.953-04:00 2024-05-07T03:11:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:22.953-04:00 2024-05-07T03:11:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:27.953-04:00 2024-05-07T03:11:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:32.953-04:00 2024-05-07T03:11:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:38.019-04:00 2024-05-07T03:11:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:11:42.952-04:00 2024-05-07T03:11:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:47.953-04:00 2024-05-07T03:11:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:11:52.983-04:00 2024-05-07T03:11:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:57.953-04:00 2024-05-07T03:11:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:02.952-04:00 2024-05-07T03:11:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:07.987-04:00 2024-05-07T03:12:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:12.953-04:00 2024-05-07T03:12:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:17.952-04:00 2024-05-07T03:12:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:22.952-04:00 2024-05-07T03:12:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:27.953-04:00 2024-05-07T03:12:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:32.988-04:00 2024-05-07T03:12:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:37.953-04:00 2024-05-07T03:12:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:42.953-04:00 2024-05-07T03:12:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:47.953-04:00 2024-05-07T03:12:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:52.953-04:00 2024-05-07T03:12:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:57.988-04:00 2024-05-07T03:12:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:02.953-04:00 2024-05-07T03:12:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:07.953-04:00 2024-05-07T03:13:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:12.988-04:00 2024-05-07T03:13:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:17.953-04:00 2024-05-07T03:13:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:22.952-04:00 2024-05-07T03:13:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:27.952-04:00 2024-05-07T03:13:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:32.952-04:00 2024-05-07T03:13:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:13:37.988-04:00 2024-05-07T03:13:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:13:42.952-04:00 2024-05-07T03:13:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:47.953-04:00 2024-05-07T03:13:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:52.991-04:00 2024-05-07T03:13:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:57.953-04:00 2024-05-07T03:13:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:02.953-04:00 2024-05-07T03:13:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:07.953-04:00 2024-05-07T03:14:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:12.953-04:00 2024-05-07T03:14:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:17.953-04:00 2024-05-07T03:14:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:22.953-04:00 2024-05-07T03:14:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:27.983-04:00 2024-05-07T03:14:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:32.953-04:00 2024-05-07T03:14:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:37.953-04:00 2024-05-07T03:14:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:42.952-04:00 2024-05-07T03:14:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:47.952-04:00 2024-05-07T03:14:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:52.989-04:00 2024-05-07T03:14:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:57.953-04:00 2024-05-07T03:14:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:15:02.952-04:00 2024-05-07T03:14:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,343 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input_data: {"inputs": ["How to create policy?"]}, content_type: application/json
2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,343 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - decoded input: {'inputs': ['How to create policy?']}, content_type: application/json
2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,344 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - data: {'inputs': ['How to create policy?']}, data_type: <class 'dict'>
2024-05-06T23:15:08.404-04:00 2024-05-07T03:15:06,346 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:08,177 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:48522 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,253 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:00<?, ?it/s]
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,253 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - predict_fn error: backend='inductor' raised:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:05<?, ?it/s]
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,254 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 258, in handle
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - response = self.transform_fn(*([self.model, input_data, content_type, accept] + self.transform_extra_arg))
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 214, in transform_fn
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - predictions = self.predict(*([processed_data, model] + self.predict_extra_arg))
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/ml/model/code/inference.py", line 111, in predict_fn
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise e
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/ml/model/code/inference.py", line 108, in predict_fn
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return model.predict(inputs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/setfit/modeling.py", line 560, in predict
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,255 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - embeddings = self.encode(inputs, batch_size=batch_size, show_progress_bar=show_progress_bar)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/setfit/modeling.py", line 452, in encode
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.model_body.encode(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 371, in encode
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - out_features = self.forward(features)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input = module(input)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,256 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 5914
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,257 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,257 [INFO ] W-9000-model ACCESS_LOG - /169.254.178.2:58184 "POST /invocations HTTP/1.1" 400 5915
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,257 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 98, in forward
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - output_states = self.auto_model(**trans_features, return_dict=False)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,258 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self._call_impl(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return forward_call(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 490, in catch_errors
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return callback(frame, cache_entry, hooks, frame_state)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,259 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 641, in _convert_frame
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - result = inner_convert(frame, cache_size, hooks, frame_state)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 133, in _fn
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 389, in _convert_frame_assert
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return _compile(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 569, in _compile
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - guarded_code = compile_inner(code, one_graph, hooks, transform)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,260 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 491, in compile_inner
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - out_code = transform_code_object(code, transform)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - transformations(instructions, code_options)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 458, in transform
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - tracer.run()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,261 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2074, in run
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - super().run()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - and self.step()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - getattr(self, inst.opname)(inst)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2162, in RETURN_VALUE
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,262 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.output.compile_subgraph(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 857, in compile_subgraph
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 957, in compile_and_call_fx_graph
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = self.call_user_compiler(gm)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,263 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1024, in call_user_compiler
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1009, in call_user_compiler
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = compiler_fn(gm, self.example_inputs())
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_gm = compiler_fn(gm, example_inputs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/__init__.py", line 1568, in __call__
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compile_fx(model_, inputs_, config_patches=self.config)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 961, in compile_fx
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compile_fx(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,264 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1150, in compile_fx
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return aot_autograd(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - cg = aot_module_simplified(gm, example_inputs, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = create_aot_dispatcher_function(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,265 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fw = compiler(fw_module, flat_args)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1092, in fw_compiler_base
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return inner_compile(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,266 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - inner_compiled_fn = compiler_fn(gm, example_inputs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return fn(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return func(*args, **kwds)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 54, in newFunction
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return old_func(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 341, in compile_fx_inner
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 565, in fx_codegen_and_compile
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - compiled_fn = graph.compile_to_fn()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/graph.py", line 970, in compile_to_fn
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - return self.compile_to_module().call
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,267 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - r = func(*args, **kwargs)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/graph.py", line 941, in compile_to_module
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1139, in load_by_key_path
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - exec(code, mod.__dict__, mod.__dict__)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/home/model-server/tmp/torchinductor_root/av/cavozcyohmg3erjtmk3oyovim5verzsympqf2nkp4bzglt2x6rwi.py", line 424, in <module>
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - async_compile.wait(globals())
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1418, in wait
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - scope[key] = result.result()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1278, in result
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel = self.kernel = _load_kernel(self.kernel_name, self.source_code)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1261, in _load_kernel
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel.precompile()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 168, in precompile
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.launchers = [
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 169, in <listcomp>
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self._precompile_config(c, warm_cache_only_with_cc)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 205, in _precompile_config
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - binary._init_handles()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/triton/compiler/compiler.py", line 683, in _init_handles
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/mms/service.py", line 108, in predict
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self._entry_point(input_batch, self.context)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 267, in handle
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:15:13.420-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - : 400
any idea how this can be fixed? cc @chen3933