-
-
The API mimics the water quality sensor data similar to the one shared here.
-
The implementation is done in flask web framework and the response is as follows:
‘2020-02-17T11:12:58.765969 26.04 540.1 13.12 Montrose_Beach 758028’
-
-
-
The structured data is pushed to MongoDB collection with the following schema:
| Keys | Data Type | |------------------|-----------| | _id | Object Id | | Beach | String | | MeasurementID | long | | BatteryLife | Double | | RawData | String | | WaterTemperature | Double | | Turbidity | Double | | TimeStamp | timestamp |
-
-
python sensor.py
-
bash /opt/zookeeper-3.4.14/bin/zkServer.sh start
-
bin/kafka-server-start.sh config/server.properties
-
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic RawSensorData
-
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic CleanSensorData
-
python push_data_to_kafka.py
-
./bin/spark-submit structure_validate_store.py
-
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic RawSensorData --from-beginning
-
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic CleanSensorData --from-beginning
-
bokeh serve --show dashboard.py