Jcodec error while reading streams
varun-tangoit opened this issue · 24 comments
Hi,
I try to read the streams with stream_id with startselectorType as Earliest, while i got error jcodec; Please have a look that this stacktrack of this issue; But i can able to view streams in aws console, while in parse library i'm facing issue, please help me out from this issue
java.lang.RuntimeException: long term
at org.jcodec.codecs.h264.decode.RefListManager.reorder(RefListManager.java:156)
at org.jcodec.codecs.h264.decode.RefListManager.buildRefListP(RefListManager.java:79)
at org.jcodec.codecs.h264.decode.RefListManager.getRefList(RefListManager.java:42)
at org.jcodec.codecs.h264.decode.SliceDecoder.decodeFromReader(SliceDecoder.java:71)
at org.jcodec.codecs.h264.H264Decoder$FrameDecoder.decodeFrame(H264Decoder.java:153)
at org.jcodec.codecs.h264.H264Decoder.decodeFrameFromNals(H264Decoder.java:104)
at com.amazonaws.kinesisvideo.parser.utilities.H264FrameDecoder.decodeH264Frame(H264FrameDecoder.java:112)
at com.amazonaws.kinesisvideo.parser.utilities.H264FrameRenderer.process(H264FrameRenderer.java:44)
at com.amazonaws.kinesisvideo.parser.utilities.FrameVisitor$FrameVisitorInternal.visit(FrameVisitor.java:105)
at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:128)
at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:67)
at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:54)
at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:128)
at com.amazonaws.kinesisvideo.parser.mkv.StreamingMkvReader.apply(StreamingMkvReader.java:132)
at com.amazonaws.kinesisvideo.parser.examples.GetMediaWorker.run(GetMediaWorker.java:86)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exiting GetMediaWorker for stream
Hi @varun-tangoit ,
The issue is caused by our dependency Jcodec, it is a bug in it: jcodec/jcodec#379. The bug is fixed in master branch, but not yet released, so you need to build from the master branch of Jcodec and use the jar file as dependency of amazon-kinesis-video-streams-parser-library.
Here are a rough set of steps:
>> Clone the jcodec’s master branch from GitHub, I.e. git clone https://github.com/jcodec/jcodec.git
>> Now I navigated to the jcodec and ran “mvn clean install” to successfully build the project and create and install the JAR file inside the local repository of maven I.e. by default in ~/.m2/respository. The JAR file which was created was named as 0.2.4-SNAPSHOT.jar
>> Then cloned the KVS parser library to my local system
>> Now opened the pom.xml file of the KVS library and instead of 0.2.3, gave the version as 0.2.4-SNAPSHOT for org.jcodec.jcodec
>> Next used, the latest version of org.jcodec.jcodec-javase i.e. 0.2.5
>> Next, since 0.2.4-SNAPSHOT.jar of org.jcodec.jcodec doesn’t contain org.jcodec.common.Assert class, the import statement in src/test/java/com/amazonaws/kinesisvideo/parser/examples/KinesisVideoGStreamerPiperExampleTest.java should be changed from import org.jcodec.common.Assert; to ==> import org.junit.Assert;
>> Once the above changes are done, you may use “mvn clean install” inside the kms parser library’s root directory and wait till the project gets built.
Once, the project gets built, a request would be to run the src/test/java/com/amazonaws/kinesisvideo/parser/examples/KinesisVideoRendererExampleTest.java’s “testexample” and check whether you’re able to render the videos. After that you may use your own stream’s video and check whether it is able to render the frames.
I hope the above information helps. Please let me know if you have any further query, I’ll be happy to help.
Thank so much @zhiyua-git this jcodec issue has been resolved. I'm facing one more issue, when I try to run the parser library in aws EC2 instance, I got error like java awt.Headless exception. and i need to run background continuously. Please give me suggestions on that usecae.
Hi @varun-tangoit ,
It is not clear to me what issue exactly are you hitting, but this post might help (https://stackoverflow.com/questions/5891123/specifying-jvm-arguments-when-calling-a-jar-file) based on your description.
@zhiyua-git I have shared printStackTrace, I'm facing those issue ;
-
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at com.amazonaws.kinesisvideo.parser.utilities.H264FrameDecoder.decodeH264Frame(H264FrameDecoder.java:102)
at com.amazonaws.kinesisvideo.parser.utilities.H264FrameRenderer.process(H264FrameRenderer.java:44)
at com.amazonaws.kinesisvideo.parser.utilities.FrameVisitor$FrameVisitorInternal.visit(FrameVisitor.java:105)
at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:128)
at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:67)
at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:54)
at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:128)
at com.amazonaws.kinesisvideo.parser.mkv.StreamingMkvReader.apply(StreamingMkvReader.java:132)
at com.amazonaws.kinesisvideo.parser.examples.GetMediaWorker.run(GetMediaWorker.java:89)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exiting GetMediaWorker for stream -
Exception in thread "main" java.awt.HeadlessException
at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
at java.awt.Window.(Window.java:536)
at java.awt.Frame.(Frame.java:420)
at java.awt.Frame.(Frame.java:385)
at javax.swing.JFrame.(JFrame.java:189)
at com.amazonaws.kinesisvideo.parser.examples.KinesisVideoFrameViewer.(KinesisVideoFrameViewer.java:31)
at com.amazonaws.kinesisvideo.parser.examples.KinesisVideoFrameViewer.(KinesisVideoFrameViewer.java:41)
at com.amazonaws.kinesisvideo.parser.examples.KinesisVideoRendererExample$GetMediaProcessingArguments.create(KinesisVideoRendererExample.java:157)
at com.amazonaws.kinesisvideo.parser.examples.KinesisVideoRendererExample$GetMediaProcessingArguments.access$000(KinesisVideoRendererExample.java:148)
at com.amazonaws.kinesisvideo.parser.examples.KinesisVideoRendererExample.execute(KinesisVideoRendererExample.java:108)
at com.amazonaws.kinesisvideo.parser.examples.Main.main(Main.java:84)
Hi @zhiyua-git, that "NoSuchElementException: No value present" still open. In producer side we didnt change or any modification we made. we just plug and deployed in client places. But i can view streams in kinesis_video_stream console. But i cant be consume using parser library. can you please help me out!. we are currently facing major issues and we cant move forward further.
I am guessing the sample is hitting an exception when it's attempting to get a value of an optional of width or the height
This is likely because the MKV doesn't contain the video width/height elements. If you are using latest producer library it should have been set by the MKV generator in the core of the producer if
- Your content type is "video/h264"
- You have valid CPD with SPS/PPS that's properly parsed by the PIC
Recommend checking your producer side, get the debug logs and see if the extraction of the video pixel width/height has failed for some reason. Make sure the content type is set correctly. If all those check out, capture the CPD from the producer side so we can look at it.
Alternatively, you could modify the parser code to check if the optionals are present and if not, assume some default video resolution.
Hi @MushMal. Yeah I tried with default pixel width height still im facing issue.kindly have look update stacktrace with im just printed trackmetadata.
MkvTrackMetadata(trackNumber=1, trackUID=Optional[1], trackName=kinesis_video, codecId=V_MPEG4/ISO/AVC, codecName=null, pixelWidth=Optional.empty, pixelHeight=Optional.empty)
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at com.amazonaws.kinesisvideo.parser.utilities.H264FrameDecoder.decodeH264Frame(H264FrameDecoder.java:57)
at com.amazonaws.kinesisvideo.parser.utilities.H264FrameRenderer.process(H264FrameRenderer.java:44)
at com.amazonaws.kinesisvideo.parser.utilities.FrameVisitor$FrameVisitorInternal.visit(FrameVisitor.java:104)
at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:120)
at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:66)
at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:53)
at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:120)
at com.amazonaws.kinesisvideo.parser.mkv.StreamingMkvReader.apply(StreamingMkvReader.java:131)
at com.amazonaws.kinesisvideo.parser.examples.GetMediaWorker.run(GetMediaWorker.java:65)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
12:33:25.722 [pool-2-thread-2] ERROR com.amazonaws.kinesisvideo.parser.examples.GetMediaWorker - Failure in GetMediaWorker for streamName 1c6ae46f3e020345497077dea2fdf81e java.util.NoSuchElementException: No value present
@varun-tangoit your printout shows (as I mentioned) that the value of an optional parameter pixelWidth=Optional.empty, pixelHeight=Optional.empty which means that it will raise an exception if dereferenced. Try to understand why the producer didn't set the EBML elements or modify the code decodeH264Frame function code to check whether the optional is not set and assume some default value to help to move your case further. I assume you have experience in Java development
@MushMal , Thanks for the reply. I set default value in parser library with pixelwidth, pixelheight, But i still this issue not resolve; Please have a look at the stacktrace;
java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.jcodec.codecs.h264.decode.SliceDecoder.putMacroblock(SliceDecoder.java:200)
at org.jcodec.codecs.h264.decode.SliceDecoder.decodeMacroblocks(SliceDecoder.java:104)
at org.jcodec.codecs.h264.decode.SliceDecoder.decodeFromReader(SliceDecoder.java:73)
at org.jcodec.codecs.h264.H264Decoder$FrameDecoder.decodeFrame(H264Decoder.java:153)
at org.jcodec.codecs.h264.H264Decoder.decodeFrameFromNals(H264Decoder.java:104)
at com.amazonaws.kinesisvideo.parser.utilities.H264FrameDecoder.decodeH264Frame(H264FrameDecoder.java:91)
at com.amazonaws.kinesisvideo.parser.utilities.H264FrameRenderer.process(H264FrameRenderer.java:44)
at com.amazonaws.kinesisvideo.parser.utilities.FrameVisitor$FrameVisitorInternal.visit(FrameVisitor.java:104)
at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:120)
at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:66)
at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:53)
at com.amazonaws.kinesisvideo.parser.mkv.MkvDataElement.accept(MkvDataElement.java:120)
at com.amazonaws.kinesisvideo.parser.mkv.StreamingMkvReader.apply(StreamingMkvReader.java:131)
at com.amazonaws.kinesisvideo.parser.examples.GetMediaWorker.run(GetMediaWorker.java:65)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
17:28:41.562 [pool-2-thread-2] ERROR com.amazonaws.kinesisvideo.parser.examples.GetMediaWorker - Failure in GetMediaWorker for streamName 1c6ae46f3e020345497077dea2fdf81e java.lang.ArrayIndexOutOfBoundsException
Were you able to troubleshoot the producer side to understand why the ebml elements are not being included?
This is likely because the MKV doesn't contain the video width/height elements. If you are using latest producer library it should have been set by the MKV generator in the core of the producer if
Your content type is "video/h264"
You have valid CPD with SPS/PPS that's properly parsed by the PIC
Recommend checking your producer side, get the debug logs and see if the extraction of the video pixel width/height has failed for some reason. Make sure the content type is set correctly. If all those check out, capture the CPD from the producer side so we can look at it.
I received the producer side logs offline.
Looking at the producer side logs I can see some timeout errors attempting to reach the backend but the retries have succeeded.
Your streaming is constantly running into network pressures - the bandwidth is not enough to handle the density of the stream so the buffer gradually gets filled up as it streams out and eventually it starts dropping frames. This will cause multiple frames and fragments being dropped at the producer side resulting in "holes" in producer timeline for the stream.
I also see you are using GStreamer assets for streaming though I can't tell from the logs whether it's the KVS GStreamer plugin with gst-launch-1.0 utility or the application is based on GStreamer.
Most notably, I am seeing the content type being set properly as "video/h264". However, the logs show that the CPD parsing failed
INFO - kinesisVideoStreamFormatChanged(): Stream format changed.
2020-05-12 13:25:48 [6668] WARN - mkvgenAdaptCodecPrivateData(): Failed extracting video configuration from SPS with 41000001.
This error is generated when the bitreader attempts to index outside of the range. This is a sign that your CPD is invalid. This is the reason you don't have the video width and height elements.
Recommend:
- Attempt to capture the CPD and attach it to this issue
- Describe the make/model of the device you are using - see if this is a known issue
- Describe in details the GStreamer pipeline you are using
- Contact the device manufacturer and see whether it's faulty encoder issue
There really isn't much we can do without this information
Hi @MushMal : Thanks for the reply
Describe the make/model of the device you are using - see if this is a known issue - Hikvision
Describe in details the GStreamer pipeline you are using - we are using open source gstreamer c++ sdk aws provided
Contact the device manufacturer and see whether it's faulty encoder issue - we are able to parse streams from same manufacture for different clients
Attempt to capture the CPD and attach it to this issue - we directly use rtsp gstreamer plugin what aws provides, so could you please let us know how to capture this
As discussed with support team, this particular streams doesn't have width and height keys, so parsing throwing error. can we add default width and height in gst-launch command itself. we are taking feeds using gstreamer kvs rtsp
Example cmd - $ gst-launch-1.0 rtspsrc location="rtsp://YourCameraRtspUrl" short-header=TRUE ! rtph264depay ! video/x-h264, format=avc,alignment=au ! kvssink stream-name="YourStreamName" storage-size=512 access-key="YourAccessKey" secret-key="YourSecretKey" aws-region="YourAWSRegion"
@Tangotrax I am not sure which Hikvision device you are using. Also, please clarify that you are able to see pixel width/height from the same device for which other clients?
Would you be able to set LOG_STREAMING and capture the first few seconds of the producer logs which should have the CPD captured?
insert add_definitions(-DLOG_STREAMING)
@MushMal : Yes we are able to see pixel width/height same device for other clients in consumer side and able to retrieve data
@Tangotrax I am not sure yet how that can happen - perhaps different configurations? Please capture the CPD as I described so we can check whether it's the CPD. I am not sure if you are working on the same case as @varun-tangoit as in their case the CPD looks to be incorrect as I saw this in the logs:
INFO - kinesisVideoStreamFormatChanged(): Stream format changed.
2020-05-12 13:25:48 [6668] WARN - mkvgenAdaptCodecPrivateData(): Failed extracting video configuration from SPS with 41000001.
The only other possibility might be that the stream was pre-created with a different codec id/content type? In that case, it's easy to delete and let the SDK re-create with the settings that are on the producer side.
Hi @MushMal, We currently changed our approach, we dont want spend more time and deeper on this. While we trying to upload our mkv video file into s3 using kinesisvideoExample module, From there mkv file we extract with xml format using below command;
github : https://github.com/vi/mkvparse
$ cat test3.mkv | ./mkv2xml | ./xml2mkv | mplayer - # convert to XML and back and play
From there xml data element we trying to convert into images, while converting we get NONE error. kindly help us to resolve this issue;
By using below code snippet we trying to convert;
- nparr = np.fromstring(byte, np.uint8)
- img_np = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
@varun-tangoit unfortunately I am not familiar with this codebase. Hope to have some time to look into it.
Hi @MushMal : we are working on same logic. from kvs produced mkv chunks, how to extract producer timestamp + images together using python. Apart from using stream parser render module
Not sure if I understand the question but MKV specification defines a frame timestamp (simple block) which is relative from the fragment (cluster) and the cluster timestamp can be anything. In the SDK absolute timestamp mode the cluster timestamps are absolute wall clock times and in the relative timestamp mode they start from 0. https://matroska.org/technical/specs/index.html
Basically we are retrieving feeds using producer time stamp as startselector and saving video data as mkv file from kvs. From mkv file we need to get frame + producer timestamp for analysis.
we are trying to parse mkv using python. in that got to know mkv to xml conversion from below git link
github : https://github.com/vi/mkvparse
command to convert: $ cat test3.mkv | ./mkv2xml | ./xml2mkv | mplayer - # convert to XML and back and play
In xml tag timecode+ data block as simple block(list of simple block under one cluster). do we need to merge all simple block bytes to one consolidate byte under cluster and convert to image
Couple of pointers:
- The KVS indexing has two modes - server-side and producer-side timestamps - both are Absolute timestamps. The embedded timestamps in the MKV can be either absolute or relate per my comment earlier. The default SDK mode is Absolute.
- From what I understand, you are streaming H264 frames. You need to un-package the frame (MKV parsing) to get the encoded frame. This is not an image. You need to feed the encoded frame into some sort of H264 decoder to get a decoded frame.
@MushMal Thanks for the response. We have few more question; we want to extract each simpleblock timecode value into local textfile. Im currently using kinesisVideoExample module using write mkv file locally at the same i want those timecode values also. Could you please help me out, is there any reservered keyword using can we able to get value?
> <Cluster>
> <Timecode>1589878994265</Timecode>
> <SimpleBlock>
> <track>1</track>
> **<timecode>1589878994.265</timecode>**
> <keyframe/>
> -------
> ------
> ---
@varun-tangoit please open separate issues for different question/problems so we can track it better.
MKV specification is defined here: https://matroska.org/technical/specs/index.html
Look for a section on SimpleBlock Header for more information on how to parse it. Let us know if you have questions by cutting an issue specific to it.
As the original question is answered, I am going to resolve this issue. Please feel free to cut a new one.