YOU PROBABLY DO NOT WANT TO USE THIS.
This was an open-source thought experiment I had a few years ago, when there wasn't a lot of support for Kafka within the existing syslog daemons. I had written a closed source implementation on top of Netty at the time for my employer. These days the major syslog daemons support publishing to Kafka, so I would recommend using one of the following:
Version: 0.1-SNAPSHOT
This code is built with the following assumptions. You may get mixed results if you deviate from these versions.
- Syslog4j 0.9.30
- Kafka 0.7.2
- Protocol Buffers 2.4.1
- Protocol Buffers
- Zookeeper (for Kafka)
- Kafka
You will need to setup Google's Protocol Buffers in order for this project to compile and run. Since this is fairly well documented and straightforward I'll leave that up to the user. Most platforms that have a descent packaging system will probably have a protobuf package already. If you're on a Mac using Homebrew this is as easy as:
brew install protobuf
You will also need Apache Kafka. As of this writing the latest is Kafka 0.7.2 but the project is still in the incubation stage. Unfortunately this means they can't push an official Apache Kafka jar to Maven repositories. So you'll need to add it to your environment manually. This can be done in the following steps:
curl https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2-incubating-src.tgz -o kafka-0.7.2-incubating-src.tgz
tar xvzf kafka-0.7.2-incubating-src.tgz
cd kafka-0.7.2-incubating-src
./sbt update
./sbt package
cd ./core/target/scala_2.8.0/
mvn install:install-file -Dfile=kafka-0.7.2.jar -DgroupId=org.apache.kafka -DartifactId=kafka-core -Dversion=0.7.2-incubating -Dpackaging=jar
If everything went okay you should now have a kafka-0.7.2.jar in your local Maven repository.
To make a jar you can do:
mvn package
The jar file is then located under target.
Make sure your Kafka and Zookeeper servers are running first (see Kafka documentation)
In order to run syslog-kafka on another machine you will probably want to use the dist assembly like so:
mvn assembly:assembly
The zip file now under the target directory should be deployed to SYSLOGKAFKA_HOME on the remote server.
** MORE DOCUMENTATION TO FOLLOW. THIS IS A WORK IN PROGRESS IN MY SPARE TIME. **
# comma delimited list of ZK servers
zk.connect=127.0.0.1:2181
# use syslog message encoder
serializer.class=kafka.serializer.SyslogMessageEncoder
# asynchronous producer
producer.type=async
# compression.code (0=uncompressed,1=gzip,2=snappy)
compression.codec=2
# batch size (one of many knobs to turn in kafka depending on expected data size and request rate)
batch.size=100
All aspects of this software are distributed under Apache Software License 2.0. See LICENSE file for full license text.
- Xavier Stevens (@xstevens)