/Kafka_basic

TU Delft Wifi Access point Data extractor(part of - Responsible IOT Data Research)

  • A python application which extracts data from the Cisco Prime infrastructure of TU Delft and streams the data to Apache Kafka to be used by researchers. The following queries can be fetched by the producers -
  • Client Stats
  • Client Sessions
  • Client Details
  • Some more to be added soon

There is a producer program for each query which pushes data to different topics(Daily,Monthly,Vendor) and a consumer program which collects the data and writes it to a csv file.

Prerequisites

Docker is required to run the system and can be installed by -

Getting Started

You can clone or download the repository to run it on your local machine for development and testing purposes. The docker file will run the environment and also install the necessary libraries like -

Installing

The following command builds the images and the containers for the Dockerfile-

- docker build .

Execute this command to run the dockerfile

-  docker-compose up

The status of the docker containers can be checked by the following command

-  docker ps

After this you can run the producer and consumer on different terminals using -

- python producer.py
- python consumer.py

The consumer will produce the csv fle which can be distributed on authorization levels.

Paging for large amount of data

The data can be queried in a paged format like -

- Devices?.full=true&.firstResult=0&.maxResults=4    (First four results(0-3))
- Devices?.full=true&.maxResults=4&.firstResult=4    (four results (4-7))

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Authors

merge

  • git checkout master
  • git pull origin master
  • git merge test
  • git push origin master