/InsightDE-No-One-Behind

Data Engineering project at Insight. Monitoring audience feedback of online streaming class/conference

Primary LanguagePython


No one behind

Monitering engagement level of an large size online classroom using OpenCV and RabbitMQ

License

Key FeaturesTechnical ArchitectureHow To UseConfigurationDemoScaling PerformanceCreditsContact

🎨 Key Features

  • Scalable - The system is designed to distribute work load over a processing cluster. The amount of nodes in the processing cluster is adjustable. No perfomance loss has been observed when scaling up horizontally via adding nodes to the cluster.

  • High throughput - Current tests have demonstrated capacity of streaming up to 400 videos spontaneously with just two (2) m5.24xlarge nodes. With ten (10) cheaper m5.large nodes, tests have been carried out at the capacity of 200 videos.

  • Low latency - Using m5.large nodes, each node can deliver 1 video with about 0.6 s delay (from frame extraction to report generation); 5 videos, 0.9 s; 10 videos, 1.4 s; 20 videos, 2.1 s. Using m5.24xlarge nodes, 1 video 0.6; 5 videos, 0.7 s; 10 videos, 0.7 s; 20 videos, 1.0 s.

  • Stability - The delay is stable over time if input streams stay steady. When there is a sudden change, such as increase in the number of streams input, the delay can re-establish stability over 1 second.

  • Modular approach - Replace Face recognition model with desired Image processing model to detect entities as per your use case.

🔨 Technical Architecture

architecture

▶️ How To Use

  • Setup RabbitMQ cluster
  • Launch receiver python3 feeder.py <number of streams>
  • Launch feeder(s) python3 main.py <threshold>
  • Launch dashboard to visualize the result python3 flaskapp.py or config apache service to server the app

⚙️ Configuration

  1. FRAME_EX_RATE

    • The ratio of frames are extracted. Default is 10, set to low for high accuracy, set to high for high performance.
  2. VID_THROUGHPUT_SIZE

    • The frames are compressed before analyzing. Large size can lead to higher accuracy and small size can lead to high performance.
  3. VID_ROLLING

    • When set as 'True'. the stream will keep rolling till interruption.

🐾 Demo

screenshot

🚀 Scaling Performance

latency

❤️ Credits

This software uses following open source packages.


✏️ Contact

Linkedin · GitHub · Kaggle