Demo for an IoT Data Pipeline on Google Cloud Platform (GCP) using:
- Google Compute Engine
- Google Cloud Pub/Sub
- Google Cloud IoT Core
- Google Cloud Dataflow
- Google BigQuery
- Google Cloud Storage
gcloud projects create iot-data-pipeline-1
gcloud config set project iot-data-pipeline-1
Enable billing on the Cloud Console Dashboard
gcloud services enable cloudiot.googleapis.com --async
gcloud services enable pubsub.googleapis.com --async
gcloud services enable dataflow.googleapis.com --async
gcloud services list
May take a few minutes for each API to enable and show up
gcloud pubsub topics create fitbyte
- Create new dataset
- Name it fitbyte
- Location: US
- Create empty table
- Table name: metrics
- "Edit as Text" under Schema, paste:
user_id:STRING,calories_burned:FLOAT,calories_consumed:FLOAT,sleep_hours:FLOAT,water_consumed:FLOAT,steps:FLOAT,distance:FLOAT,bmi:FLOAT,heart_rate:FLOAT,weight:FLOAT,timestamp:TIMESTAMP,device:STRING
gsutil mb -p iot-data-pipeline-1 -c multi_regional -l US gs://fitbyte/
gcloud iot registries create iot-devices \
--project=iot-data-pipeline-1 \
--region=us-central1 \
--event-notification-topic=projects/iot-data-pipeline-1/topics/fitbyte
- Create job from template
- Name: fitbyte-data
- Region: us-central1
- Cloud Dataflow template: PubSub to BigQuery
- Cloud Pub/Sub input topic: projects/iot-data-pipeline-1/topics/fitbyte
- BigQuery output table: iot-data-pipeline-1:fitbyte.metrics
- Temporary Location: gs://fitbyte/tmp/
- Max-workers: 2
- Machine-type: n1-standard-1
- Run Job
gcloud compute instances create iot-simulator
sudo apt-get update
sudo apt-get install python-pip openssl git google-cloud-sdk -y
sudo pip install pyjwt paho-mqtt cryptography
git clone https://github.com/mattgartner/fitbyte.git
export PROJECT_ID=iot-data-pipeline-1
export MY_REGION=us-central1
export REGISTRY=iot-devices
cd $HOME/fitbyte/
openssl req -x509 -newkey rsa:2048 -keyout rsa_private.pem \
-nodes -out rsa_cert.pem -subj "/CN=unused"
gcloud init
gcloud iot devices create fitbit-1 \
--project=$PROJECT_ID \
--region=$MY_REGION \
--registry=$REGISTRY \
--public-key path=rsa_cert.pem,type=rs256
gcloud iot devices create fitbit-2 \
--project=$PROJECT_ID \
--region=$MY_REGION \
--registry=$REGISTRY \
--public-key path=rsa_cert.pem,type=rs256
gcloud iot devices create fitbit-3 \
--project=$PROJECT_ID \
--region=$MY_REGION \
--registry=$REGISTRY \
--public-key path=rsa_cert.pem,type=rs256
gcloud iot devices create apple-1 \
--project=$PROJECT_ID \
--region=$MY_REGION \
--registry=$REGISTRY \
--public-key path=rsa_cert.pem,type=rs256
gcloud iot devices create apple-2 \
--project=$PROJECT_ID \
--region=$MY_REGION \
--registry=$REGISTRY \
--public-key path=rsa_cert.pem,type=rs256
gcloud iot devices create apple-3 \
--project=$PROJECT_ID \
--region=$MY_REGION \
--registry=$REGISTRY \
--public-key path=rsa_cert.pem,type=rs256
cd $HOME/fitbyte/
wget https://pki.google.com/roots.pem
python cloudiot_mqtt_example_json.py \
--project_id=$PROJECT_ID \
--registry_id=$REGISTRY \
--device_id=fitbit-1 \
--private_key_file=rsa_private.pem \
--message_type=event \
--algorithm=RS256 > fitbit-1-log.txt 2>&1 &
python cloudiot_mqtt_example_json.py \
--project_id=$PROJECT_ID \
--registry_id=$REGISTRY \
--device_id=fitbit-4 \
--private_key_file=rsa_private.pem \
--message_type=event \
--algorithm=RS256
- Dataflow template name: storage-test
- Cloud Dataflow Template: GCS Text to Cloud PubSub
- Input Cloud Storage Files: gs://fitbyte/upload/*.csv
- Output Pub/Sub Topic: projects/iot-data-pipeline-1/topics/fitbyte
- Temporary Location: gs://fitbyte/tmp/