/kafka-minimal-topology

Primary LanguageJava

Kafka Tasks

  • Kafka Consumer
  • Confirm if parallelism hint is the same as consuming from different partitions (checked messages' partition no. within Storm bolts - they're distributed)
  • Confirm if parallelism hint is the same as producing to different partitions (subscribed to output topic and checked messages' partition no. - they're distributed)
  • Starting offset configuration (UNCOMMITTED_LATEST)
  • Kafka Producer

Parsers

  • Bro http parser
  • Bro dns parser
  • yaf parser
  • snort parser
  • nio-flow parser
  • nio-dns parser
  • nio-http parser

Bolts

  • Hbase connector

    • Insert records into hbase externally
    • Read the values from database
    • Update hbase database, from bolt
    • Record caching to reduce number of reads
    • Convert threat intel aggregator to work with new setup

  • GeoIP bolt

  • threatintel bolt

  • housekeeping bolts (CRUD hbase table)

General Tasks

  • CSV manipulator class for inter-bolts communication
  • FailSafe Mechanism: A channel to display error messages, kafka topic
  • Config inferface: Flux or alternatives (Readup metron mgmt UI's way of loading parser configs)
  • Config interface: Schema reader, data structure, etc.