OSMstream
The repository provides a docker container which collects the Augmented diffs (change sets) from OpenStreetMap. The diffs are provided in XML format and can be accessed over the overpass API. For further use we convert them into JSON format and publish them with apache kafka. After that the data is ready for a streaming database of your choice.
Installation
Pull the repository, build the docker container and run the container.
Docker
cd OSMstream
docker build -t osmstream .
docker run -d --name osmstream -p 2181:2181 -p 9092:9092 osmstream
If you'd like to verify if there is something going on, use the following command.
docker exec -it osmstream bash
/opt/kafka_2.11-0.10.0.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic osm --from-beginning --consumer.config /opt/kafka_2.11-0.10.0.1/config/consumer.properties
Usage
To test your streaming database with the OSM diff data provided by kafka. This diffs could be relatively large messages for kafka, thus we recommend to set the "fetchSizeBytes" of your stream processing technology to something about 100'000'000.
Kafka settings
- Host: localhost, or docker0 ip of container
- Port: 2181, 9092
- Topic: osm, benchmark
Connect to kafka
If you connect to the osmstream docker container outside of the container you should use the docker0 IP of this container. How to get the IP? Use the following command.
docker exec osmstream /bin/sh -c "ip add show eth0 | grep -Eo 'inet \S+'"
This IP Address is also automatically added to the kafka server.properties as advertised.host.name
Test Data
Because there were several problems with the Augmented diffs, we recorded ten of them. They can published to kafka with the "test_diff.py" script. Usage (inside the docker container and better stop the cron service, which fetches the diffs):
service cron stop
python3 /opt/OSMstream/test_diff.py
Benchmak
Benchmarking is a very difficult topic and strongly depends on various parameters like the underlying hardware. Thus we decide to make this as hardware independent as possible.
The idea is to produce a lot of small kafka message, like IoT does, and to count all the words of them. Then repeat this process and relate the number of processed messages with the time spent to count the words of them.
Setup
The "benchmark.py" script produces the messages as a kafka producer and can be used like in the next box. (The parameter IP and Port are optional)
usage: benchmark.py [-h] -n NUMBER_OF_MESSAGES [-i IP] [-p PORT]
Produce input for stream benchmarks.
optional arguments:
-h, --help show this help message and exit
-n NUMBER_OF_MESSAGES, --number_of_messages NUMBER_OF_MESSAGES
Number of messages to produce.
-i IP, --ip IP IP address of Kafka. (Default is localhost)
-p PORT, --port PORT Port of Kafka. (Default is 9092)
Example usage:
docker exec -it osmstream bash
python3 /opt/OSMstream/benchmark.py -n 1000
Small Messages
The small messages are inspired by MQTT.
This messages are only generated samples for testing and look like the following.
packetId TopicName qos retainFlag payload false
0 weather 5 False 83 True
1 computer 2 True 82 False
2 weather 9 False 6 False
3 icebox 3 True 85 True
4 door 6 False 48 True
5 door 5 True 7 False