Implement Kafka Producer & PoC of Monitoring

[mekya]

Implement Kafka Producer & PoC of Monitoring

[mekya]

• Zookeeper
  • Download https://zookeeper.apache.org/releases.html(3.4.14) & Unzip
  • Create a conf file at conf/zoo.cfg

tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181

 * Start zookeeper ```bin/zkServer.sh start```

*  Apache Kafka
 * Download https://kafka.apache.org/quickstart
 * Start Kafka 

bin/kafka-server-start.sh -daemon config/server.properties

  * Create topics 

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 100 --topic ams-webrtc-stats
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 100 --topic ams-instance-stats

* Logstash
  * Download 6.7.1 https://artifacts.elastic.co/downloads/logstash/logstash-6.7.1.deb
  * sudo dpkg -i logstash.*deb
  * Save below conf as `kafka-instance-stats-pipeline.conf` to /etc/logstash/conf.d

input {
kafka {
topics => ["ams-instance-stats"]
}
}
filter {
json {
source => "message"
}
}
output {
stdout { codec => rubydebug }

elasticsearch {
   hosts => "localhost:9200"
   index => "ams-instance-stats"

}
}

* Save below conf as `kafka-webrtc-stats-pipeline.conf` to  /etc/logstash/conf.d

input {
kafka {
topics => ["ams-webrtc-stats"]

}

}
filter {
json {
source => "message"
}
}
output {
stdout { codec => rubydebug }

elasticsearch {
   hosts => "localhost:9200"
   index => "ams-webrtc-stats"

}
}

* Start logstash `sudo service logstash start`
* Elastic Search
* Download 6.7.1 https://www.elastic.co/downloads/past-releases/elasticsearch-6-7-1
* sudo `dpkg -i elastic...deb`
*  Start Elastic search `sudo service elasticsearch start`

* Grafana 
* https://grafana.com/grafana/download
* wget https://dl.grafana.com/oss/release/grafana_6.1.6_amd64.deb 
* sudo dpkg -i grafana_6.1.6_amd64.deb 
* sudo /bin/systemctl daemon-reload
* sudo /bin/systemctl enable grafana-server
* sudo /bin/systemctl start grafana-server