[BUG] Duplicate key error

[FlorianRuen]

Hi there,

I'm opened a Github issue as suggested by the AWS Premium Support, to continue an issue on Duplicate Key Error I got on my consumer.

To make a quick summary on this case :

• I'm using the parser library to extract images from Kinesis Video Stream and save it to the disk (this images will be treated later by another service)

• During the extract, I got error which say "Duplicate Key error" and fire an exception. Today this append many times a day, and I need to restart it manually (exception is catched, but it cause some bugs sometimes)

Here is a screenshot of the issue, including stacktrace : https://i.ibb.co/gg41ShR/duplicate-key-error-1.png

I'm using the latest version of the parser library
compile("com.amazonaws:amazon-kinesis-video-streams-parser-library:1.0.15")

And I'm using a Frame visitor which is based on this link : https://github.com/aws/amazon-kinesis-video-streams-parser-library/blob/master/src/main/java/com/amazonaws/kinesisvideo/parser/utilities/FrameRendererVisitor.java, I just surcharge the meethod to take a destination folder to save my frames on the disk.

There is any update or thing I can try to fix this issue ? Today I try a lot of things, but anything has changed, and it's almost a production system impaired, because I need to restart it many times a day ...

Hope I can find help here,

Kind regards,
Florian

[FlorianRuen]

Help needed here. I do some other debugs :

• With another stream, another camera, the problem is still the same (so we can say the problem isn't coming from my producer, but effectivly from the the consumer, because there is nothing between except KVS)

• The time before I restart isn't constant, sometime it's 1 hour, sometime 6 hours, and the last was alsmost 12 hours

• THe duplicate key error isn't on every frame, betcause some frames are writted on the disk (what I want), it's just cause a droping some frames (but many frame, If I check, maybe I can see some missing sequences)

[FlorianRuen]

Few more tests, after trying a lot of possibilities, I think that I run a multiple thread pool for every stream my consumer need to extract.

So what I trying to do, is to change newFixedThreadPoolExecutor(4) by newSingleThreadExecutor

var kvsSessionExpiredOrNeedToRestartConsumer = false
val getMediaProcessingArguments = FrameRendererVisitor(framesFolder, FragmentMetadataVisitor.create())
val executorService = Executors.newSingleThreadExecutor()

I'm testing now, it seems the error isn't complety dissapear, but it seems it fired less times than before.

Still help needed to understand why and go deeper to completly fix this error or find any workaround that can help me to deploy a production system that works !

[disa6302]

@FlorianRuen ,

Can you explain what this code snippet is or where it resides? Is this your own code?

[vattay]

This is really fascinating because I had a working Java Kinesis Video Consumer application that I last ran three weeks ago and was working fine, and now I'm getting a DuplicateKeyError as well.

java.lang.IllegalStateException: Duplicate key 25 (attempted merging values 91343852333181532036951704798106065785361265741 and 91343852333181532046855225112389109058398434006)
	at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor.getTagNameToValueMap(FragmentMetadataVisitor.java:295)
	at com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor.setMillisBehindLatestAndContinuationToken(FragmentMetadataVisitor.java:205)
	at com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor.access$400(FragmentMetadataVisitor.java:48)
	at com.amazonaws.kinesisvideo.parser.utilities.FragmentMetadataVisitor$StateMachineVisitor.visit(FragmentMetadataVisitor.java:167)
	at com.amazonaws.kinesisvideo.parser.mkv.MkvEndMasterElement.accept(MkvEndMasterElement.java:36)
	at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:66)
	at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:48)
	at com.amazonaws.kinesisvideo.parser.mkv.MkvEndMasterElement.accept(MkvEndMasterElement.java:36)
	at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visitAll(CompositeMkvElementVisitor.java:66)
	at com.amazonaws.kinesisvideo.parser.mkv.visitors.CompositeMkvElementVisitor.visit(CompositeMkvElementVisitor.java:48)
	at com.amazonaws.kinesisvideo.parser.mkv.MkvEndMasterElement.accept(MkvEndMasterElement.java:36)
	at com.amazonaws.kinesisvideo.parser.mkv.StreamingMkvReader.apply(StreamingMkvReader.java:131)
	at com.cuttingedgeai.ghost.GhostlyGetMediaWorker.run(GhostlyGetMediaWorker.java:80)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:831)

[disa6302]

@vattay ,

It is strange because nothing has changed within the SDK. Any changes in your application?

[FlorianRuen]

@disa6302 The frame renderer visitor is from AWS Github, I just personalize the way to not display the final image but save it on the disk. For the rest, yes it's a custom code
But i'm in the same situation than @vattay, I didn't get this error for a long time, without any changes on this part on the code, and sometime I get it many many times.

After some tries, I seems if we change the newFixedThreadPoolExecutor(4) by newSingleThreadExecutor (on my side, don't know what @vattay use ?), the problem doesn't append.

But from now I already used a fixedThreadPool and nothing goes wrong.

[vattay]

I might be having a slightly different issue. And yes, probably both our code is based on the aws parser, but with customization since you have to customize the example code to make something useful.

My root cause was actually that the video metadata no longer appears to have height and width embedded in it and I had a lot of exception handlers that caught exceptions during frame processing and attempted to resume, this led to the duplicate key error somehow.

This is the fix I made for myself, clearly just a temporary hack, copied from H264FrameProcessor, but I added default values to the pixelWidth and pixelHeight, and then it started working fine again.

    public BufferedImage decodeH264Frame(final Frame frame, final MkvTrackMetadata trackMetadata) {
        final ByteBuffer frameBuffer = frame.getFrameData();
        // TODO - Oddity: we aren't getting the image size. Needed to add these defaults. Could cause strange bugs.
        final int pixelWidth = trackMetadata.getPixelWidth().orElse(BigInteger.valueOf(352)).intValue();
        final int pixelHeight = trackMetadata.getPixelHeight().orElse(BigInteger.valueOf(240)).intValue();
        codecPrivateData = trackMetadata.getCodecPrivateData().array();
        log.debug("Decoding frames ... ");