/Streaming-Data-Pipeline-for-a-Real-Time-Dashboard-with-Dataflow

You own a fleet of New York City taxi cabs and are looking to monitor how well your business is doing in real-time. You build a streaming data pipeline to capture taxi revenue, passenger count, ride status, and much more, and then visualize the results in a management dashboard.

Primary LanguageJupyter Notebook

Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow

You own a fleet of New York City taxi cabs and are looking to monitor how well your business is doing in real-time. You build a streaming data pipeline to capture taxi revenue, passenger count, ride status, and much more, and then visualize the results in a management dashboard.

👨‍🏫 Overview

You own a fleet of New York City taxi cabs and are looking to monitor how well your business is doing in real-time. You build a streaming data pipeline to capture taxi revenue, passenger count, ride status, and much more, and then visualize the results in a management dashboard.

🎯 Objectives

  • Create a Dataflow job from a template
  • Subscribe to a Pub/Sub topic
  • Stream a Dataflow pipeline into BigQuery
  • Monitor a Dataflow pipeline in BigQuery
  • Analyze results with SQL
  • Visualize key metrics in Looker Studio