A demo that will intially deploy a Kinesis data stream with 4 shards and sample python scripts run in Cloud9 that purposly send a higher volume of messages to a single shard on Kinesis. This sets up a scenario with where a Kinesis data stream has a hot shard. Subseqently you enable enhanced monitoring on the Kinesis data stream and deploy a CloudWatch dashboard to identify the hot shard
- Deploy CloudFormation for Kinesis + Cloud9
The CloudFormation will deploy the following architecture
- Enable enhanced shard-level metrics on the Kinesis data stream
- Navigate to the Kinesis Data Stream Console
- Click on hot-shard-data-stream
- Navigate to the Configuration tab
- Enable all Enhanced (shard-level) metrics
- Send data to Kinesis via. Python scripts in Cloud9
- Navigate to the Cloud9 console
- Click on the kinesis-cloud9 enviorment
- Run each script in a seperate Cloud9 terminal
python Kinesis_Data_Stream_Monitoring/Data_Producer/2a_send_data_to_a_shard_0.pypython Kinesis_Data_Stream_Monitoring/Data_Producer/2b_send_data_to_a_shard_1.pypython Kinesis_Data_Stream_Monitoring/Data_Producer/2c_send_data_to_a_shard_2.pypython Kinesis_Data_Stream_Monitoring/Data_Producer/2d_send_data_to_a_shard_3.py
- Deploy CloudWatch dashboard
The CloudFormation will deploy a CloudWatch dashboard
The cloudwatch dashboard will produce a graph that looks something like the image below. Notice that the message count is higher for shard-0000 ... compared to the other shards
shard-0000 ... is our hot shard!