Amazon CloudFront Log Parser Changelog --------- 1.4 * Fixed Issue #11 - IllegalArgumentException while parsing newer CF log format. * Fixed Issue #12 - Supporting new CF log fields. 1.3 * Improved performance by using new feature in tbm-common-parser-lib that allows a tokenizer to re-use the same IToken for every token event instead of creating a new one. 1.2 * More secure exception handling inside of Parse ensures cleanup of any temporary input streams. * Improved the exception semantics of the parse method. #2 * Parser is more verbal with MalformedContentException errors when the content of the log file doesn't match the Amazon-defined DOWNLOAD or STREAMING CloudFront log formats. #2 * Fixed incorrect bounds check for insertion of fields beyond what ILogEntry will allow. #5 * Fixed a handful of "tightening" bugs that didn't break the parser previously but could lead to leaks or bugs down the road. * Addition Javadoc to the source was added to make it more clear what certain constructs are for. * Added a new file to the benchmark that matches a worst-case-scenario log file from CloudFront (50MB uncompressed, ~174k entries) * Added library to The Buzz Media's Maven repo. 1.1 * Initial public release. License ------- This library is released under the Apache 2 License. See LICENSE. Description ----------- CloudFront Log Parser is a Java library offering a low-complexity, high-performance, adaptive CloudFront log parser for both the Download and Streaming CloudFront log formats. The library is "adaptive" in the sense that it determines the format of the log files at parse time, you don't need to tell it, and it is additionally resilient in that it can skip unknown Field names and values while parsing instead of throwing an exception. This is important if the library is ever deployed in a production setting and Amazon changes the CloudFront log format before a new version of the library is read. CloudFront Log Parser's API was designed with the existing AWS Java JDK in mind and ensuring that integration would fit naturally. More specifically, the API can directly consume raw InputStream for the stored .gz log files on S3 directly from the S3Object.getObjectContent() method. That being said, the API is coded to accept ANY InputStream; whether you are processing local copies and passing FileInputStreams or processing remote copies; it just happened to be particularly easy to integrate it with the AWS Java SDK. CloudFront Log Parser is intended to be used in any deployment scenario from a desktop analysis app to a long-running server process environment. Design ----------- CloudFront Log Parser was designed, first and foremost, to be as fast as possible with as little memory allocation as possible so as to work smoothly in a long-running server log-processing usage scenario. Object creation is kept to a minimum during parsing (no matter how big the parse job) by re-using a single ILogEntry instance, per LogParser, to wrap parsed line values and report those to the given ILogParserCallback. The ILogEntry instance received by the callbacks is ephemeral; in that the ILogEntry instance is only valid for the scope of the callback's method. Once returned from that method, the ILogEntry instance is reused and the values it holds are swapped. *** CALLBACKS SHOULD NEVER HOLD ONTO ILOGENTRY INSTANCES! *** However, the values reported by the callbacks are copies and can be safely stored. This design was chosen because over large parse jobs for busy sites, where millions of log entries would not be uncommon, the heap memory savings and performance improvement because of this design would be noticeable. The VM would not be thrown into longer GC cycles as it attempted to clean up the millions of useless objects that were so short lived. Performance ----------- Benchmarks can be found in the /src/test/java folder and can be run directly from the command line (no need to setup JUnit). [Platform] * Java 1.6.0_24 on Windows 7 64-bit * Dual Core Intel E6850 processor * 8 GB of ram [Benchmark Results] Parsed 100 Log Entries in 27ms (3703 entries / sec) Parsed 100,000 Log Entries in 864ms (115740 entries / sec) Parsed 174,200 Log Entries in 1341ms (129903 entries / sec) Parsed 1,000,000 Log Entries in 7520ms (132978 entries / sec) The Amazon CloudFront docs say log files are truncated at a maximum size of 50MB (uncompressed) before they are written out to the log directory. The 3rd test, parsing the 174k log entries is exactly 50MB uncompressed and matches this worst-case-scenario. This means that using CloudFront Log Parser to parse the largest log files that CloudFront will write out to your S3 bucket, you can parse that file in a little over a second on equivalent hardware. If you are running on beefier server hardware, you can increase that rate and if you are parsing logs in a multi-threaded environment (1 thread per LogParser) you can increase that rate my magnitudes. CloudFront Log Parser is fast. Runtime Requirements -------------------- 1. The Buzz Media common-lib (tbm-common-lib-<VER>.jar) 2. The Buzz Media common-pars-erlib (tbm-common-parser-lib-<VER>.jar) History ------- After deploying apps that utilized CloudFront for content delivery, I had the need to parse the resulting access logs to get an idea of what kind of traffic, bandwidth and access patterns the content was receiving. Amazon promotes the use of their Map/Reduce Hadoop-based log parser multiple times on their site, but that requires additional EC2 charges to run. After about a week of prototyping and engineering I had initial versions of the CloudFront Log Parser written and running. Initial "does it work" prototypes took an afternoon, but I toyed with a multitude of different API designs and approaches trying to best balance an easy-to-use API with runtime performance. Eventually settling on what was by far the cleanest and simplest API, I used HPROF to tighten up the runtime performance and minimize object creation down to the bare minimum. I hope this helps folks out there.