The internet of things (IoT) is one of the hottest buzzwords in 2015 and for good reason. The promise of leveraging data generated by machines that are interconnected can provide vast operational efficiencies in virtually every industry. At Pivotal's SpringOne2GX conference 2014, a team from Pivotal demonstrated how components of Pivotal's open source portfolio can be combined to increase speed to market of IoT solutions.
Pivotal has developed a sample application to illustrate it's open source IoT platform via the use case of a connected car. In this demo, data from a moving vehicle is streamed real time to a server running Spring XD. Spring XD ingests the data and makes predictions about where the vehicle is going and how far it can go. An HTML/AngularJS based dashboard provides real time analytics illustrating the current position of the car, some of the basic telemetry available from the car as well as the predictions provided by Pivotal's Data Science team.
This code repository consists of all of the source code used in the demo application's server deployment. A separate repository for the iOS application used will be open sourced at a future date.
You can view the video of the SpringOne2GX demo online here: SpringOne2GX IoT-Realized: The Connected Car
There are three main prongs to the architecture:
- Ingestion
- Batch training
- Realtime dashboard
The source of the data for this project is the OBD II data from a moving vehicle. In the live demo, the iOS application Herbie was used to send live data to the server. In offline demonstrations as well as testing, recorded data was replayed via a Spring Batch job in the IoT-CarSimulator module.
The foundation of the ingestion is Spring XD. Using Spring XD, an ingestion stream was developed that accepted the inbound input, performed minor transformations, routed the data through the realtime predictions module, and then persisted the output to HDFS.
The stream definition itself is as follows (as seen in the IoT-Scripts/stream-create.xd script):
http | filter --script=file:DataFilter.groovy | acmeenrich | shell --inputType=application/json --command='python ./StreamPredict.py' --workingDir=IoT-ConnectedCar/IoT-Data-Science/PythonModel/ | hdfs
The above stream accepts JSON POSTed to the http source listening on the default port (9000). It then routes the data through a filter that determines if all the required data is available. If required data is missing, the record is dropped and the record (and reason) are logged. If all the required data is available, the message is passed onto an enrichment transformer that performs some basic enrichment and translation for data consistency. The enriched data is then passed to the shell module that is running a python module developed by Pivotal's Data Science team. That python module looks at the current journey and using historical data for that particular vehicle's VIN, it creates a prediction of where it's going and it's expected MPG for the journey. The results of the shell processor are stored in HDFS.
The previously mentioned stream is tapped via the following definition:
tap:stream:IoT-HTTP.shell > typeconversiontransformer | gemfire-server --regionName=car-position --keyExpression=payload.vin
This tap copies everything coming out of the shell processor and sends it to a transformer. That transformer change the type from a simple String containing JSON to a POJO to be persisted into GemFire. The transformed object is then sent to the gemfire-server sink for persistence into GemFire so that it can be made available for the dashboard.
The modules involved in the ingestion of data in this project are the IoT-DataScience, IoT-DataFilter, IoT-EnrichmentTransformer, IoT-GemFireCommons, and IoT-GemfireTransformer modules.
Before the data science module can make a prediction with the real time data, a model must be generated off of the historical data of previous journeys. The batch training occurs offline with the resulting model used as input to the predictive module in the ingest phase.
The real time dashboard is an HTML/AngularJS based application that is powered by the data in GemFire. The IoT-GemFireREST module provides a set of REST APIs that are called by the IoT-Dashboard module to provide data about the selected vehicle's current location, telemetry, as well as the current state of the predictions.
This is intended to illustrate a complete solution and as such has a few dependencies for it's environment. Specifically:
- Java 8 or higher
- Spring XD 1.1.0.RELEASE or higher
- Hadoop install compatible with Spring XD release
- Spark 1.2 or higher
- GemFire 8 or higher
- Anaconda distribution of Python 2.1.0 or higher
- Grunt CLI v0.1.13 or higher
There are two main pieces needed to build this project from source:
- The Dashboard
- Everything else
The HTML/AngularJS based dashboard was originally created via a Yeoman generator and therefore uses it's standard stack to perform builds. Given the rate of change with these tools, we can only assure that the build works on the following versions at the time of the writing of this document:
- node - v0.10.31
- npm - 1.4.24
- grunt - v0.4.5
- bower - 1.4.1
The best place to get the build setup is via the Yeoman website (referenced in the dashboard's documentation). With the various build tools installed installed, you can build the dashboard via the following from the root of the IoT-Dashboard module:
$ grunt clean build
The output of this process will end up in IoT-Dashboard/src/main/resources/public for
Spring Boot packaging.
Once you've built the dashboard, building the rest of the code is executed simply by
executing $ ./gradlew clean build from the root of this project. That will build all
modules and package them up apropriately for deployment (Spring Boot jars for the most
part).
Here are some ways for you to get involved in the community:
- Create Github tickets for bugs and new features and comment and vote on the ones that you are interested in.
- Github is for social coding: if you want to write code, we encourage contributions
through pull requests from forks of this repository.
If you want to contribute code this way, please familiarize yourself with the process outlined for contributing to Pivotal projects here: Contributor Guidelines. - Watch for upcoming articles on Pivotal by subscribing to spring.io