kinesis List stream/multi-stream issue

[varun-tangoit]

Hi,

Im can able to consume frames by using parser libaray. i can be able to get images and locally i can able to save. Now i try to change multiple streams, though i try to get list streams and get the list of stream names iterate based on stream names consume records. Im refering this below documentation,
https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-list-streams.html
AmazonKinesisClient, Liststreams and ListstreamResult module integration issue, can you pleasehelp me out how to do in multiple streams

[MixMasterMitch]

Is your goal to process the frames from multiple streams in parallel across multiple parsers? Or is it to process the media from multiple stream sources in a single parser?

Are you having a specific problem getting the list of streams using ListStreams? The purpose of ListStreams is just to get the metadata (e.g. name) of all the streams in the account. Theoretically, whatever you have working already with the parser library could be wrapped in a for loop where you iterate over the results from ListStreams and execute the parser with each stream.

[varun-tangoit]

Thanks for the response @MixMasterMitch, Sure there are few more customization needs to done in exisitng parser library.

1. Multistream with multi-threading based in single parser library - This approach can you suggest me, how many streams can consumer handle with single parser library or shall go ahead with multi-parser library?
2. Liststream module i can't be able to extends inside kinesisvideorender example, shall you provide or suggest me any official documentation for java with any snippet of code examples.

[MixMasterMitch]

Multistream with multi-threading based in single parser library - This approach can you suggest me, how many streams can consumer handle with single parser library or shall go ahead with multi-parser library?

For simplicity, you might as well use a different parser instance per stream. But there is no hard limit on the number of streams that can use a single instance of the parser library.

Liststream module i can't be able to extends inside kinesisvideorender example, shall you provide or suggest me any official documentation for java with any snippet of code examples.

What are you trying to do?

[varun-tangoit]

Hi @MixMasterMitch, i want to run the parser library with multiple streams. For example, Get the list of streams_names parallel and consume streams with single parser. Please suggest me how to do with that usecase;

[bdhandap]

@varun-tangoit

https://docs.aws.amazon.com/streams/latest/dev/kinesis-using-sdk-java-list-streams.html
AmazonKinesisClient, Liststreams and ListstreamResult module integration issue, can you please help me out how to do in multiple streams

The link which you posted is for Kinesis Data Streams. Kinesis Video Streams is a different service and has different clients and APIs. https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/API_Operations.html

i want to run the parser library with multiple streams. For example, Get the list of streams_names parallel and consume streams with single parser. Please suggest me how to do with that usecase;

By Single parser do you mean a singleton instance of parser consuming from multiple streams? I don't think that's possible as parser instance is stateful and maintains the current EBML element while parsing data. Moreover can you explain why do you need single parser? You can achieve this by easily using multiple parser instances

[MushMal]

@varun-tangoit could you please resolve this issue if you got answers for your original issue

[varun-tangoit]

Yesh @MushMal this issue resoved.

[varun-tangoit]

Hi @MushMal, I need a suggestion on this, For example we have 1 million streams, needs to consume from those each stream_id based on performance and no of ec2 instance or how many servers would required?. Because we are currently increase in stream_id and scaling kinesis consumer and needs to consume in near realtime.

[MushMal]

@varun-tangoit it's hard to tell - it depends on the type of processing you do (whether decoding and doing analytics or only parsing), the frame rate, frame resolution (density). It also depends on ec2 instance types, GC, etc.. I would recommend starting off with an autoscaling enabled and monitor and tune your fleet as you go.

[varun-tangoit]

Thanks @MushMal. Yeah we didn't do any intensive loads, we just save mkv file and upload to s3 when 100 framecount reached, it continues indefinitely.