Caused by: com.amazonaws.services.sagemakerruntime.model.AmazonSageMakerRuntimeException: null (Service: AmazonSageMakerRuntime; Status Code: 400; Error Code: ThrottlingException)
gem-aniketyadav opened this issue · 1 comments
Getting these errors in my ECS container logs which is running the parser application which fetches I-frames from Kinesis video stream to Sagemaker endpoint:
ERROR com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager - SageMakerTaskException:
java.util.concurrent.ExecutionException: com.amazonaws.kinesisvideo.sagemaker.task.exception.SageMakerOrchestrateTaskException: SageMaker exception for fragment: FragmentMetadata(fragmentNumberString=91343852333181496770515865655203103050129251029, serverSideTimestampMillis=1700140824049, producerSideTimestampMillis=1700140821761, fragmentNumber=91343852333181496770515865655203103050129251029, success=true, errorId=0, errorCode=null, millisBehindNow=OptionalLong.empty, continuationToken=Optional.empty)
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_201]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_201]
at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.getOutput(OrchestrateTaskManager.java:199) [amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.handlePublishPhaseForTask(OrchestrateTaskManager.java:239) [amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.run(OrchestrateTaskManager.java:277) [amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_201]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: com.amazonaws.kinesisvideo.sagemaker.task.exception.SageMakerOrchestrateTaskException: SageMaker exception for fragment: FragmentMetadata(fragmentNumberString=91343852333181496770515865655203103050129251029, serverSideTimestampMillis=1700140824049, producerSideTimestampMillis=1700140821761, fragmentNumber=91343852333181496770515865655203103050129251029, success=true, errorId=0, errorCode=null, millisBehindNow=OptionalLong.empty, continuationToken=Optional.empty)
at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateImageTask.execute(OrchestrateImageTask.java:92) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.lambda$submitTask$0(OrchestrateTaskManager.java:168) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201]
... 3 more
Caused by: com.amazonaws.services.sagemakerruntime.model.AmazonSageMakerRuntimeException: null (Service: AmazonSageMakerRuntime; Status Code: 400; Error Code: ThrottlingException; Request ID: ab15df4a-0095-4694-902c-9265ac0dc9d0)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1630) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1302) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.services.sagemakerruntime.AmazonSageMakerRuntimeClient.doInvoke(AmazonSageMakerRuntimeClient.java:236) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.services.sagemakerruntime.AmazonSageMakerRuntimeClient.invoke(AmazonSageMakerRuntimeClient.java:212) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.services.sagemakerruntime.AmazonSageMakerRuntimeClient.executeInvokeEndpoint(AmazonSageMakerRuntimeClient.java:176) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.services.sagemakerruntime.AmazonSageMakerRuntimeClient.invokeEndpoint(AmazonSageMakerRuntimeClient.java:151) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateImageTask.execute(OrchestrateImageTask.java:88) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager.lambda$submitTask$0(OrchestrateTaskManager.java:168) ~[amazon-kinesis-video-streams-sagemaker-integration-driver-1.0.1-SNAPSHOT-shaded.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_201]
... 3 more
13:21:08.335 [new-stream-publisher-0] ERROR com.amazonaws.kinesisvideo.sagemaker.task.OrchestrateTaskManager - no output to publish
I am streaming my video using gstreamer c++ command to send the video to Kinesis video stream:
gst-launch-1.0 -v filesrc location=demo-video.mp4 ! decodebin ! videoconvert ! videorate ! video/x-raw,format=I420,width=1920,height=1080,framerate=60/1 ! timeoverlay ! identity sync=TRUE ! x264enc intra-refresh=true bitrate=7500 bframes=0 key-int-max=1 tune=zerolatency ! h264parse ! video/x-h264,stream-format=avc,alignment=au ! queue ! kvssink name=sink stream-name=demo-stream framerate=60 key-frame-fragmentation=false access-key= secret-key=
Sagemaker Endpoint is ml.g4dn.xlarge which should be quite strong as my video is only 28sec long.
The application is working fine for around 200 I-frame and are getting send to the sagemaker endpoint but then I start getting the above error. Can anyone help!!
The issue was resolved by deleting the ECS container and recreating it again