apurvis/s3-logs-parser

A service to quickly analyze and query AWS S3 access logs.

AWS S3 Logs Parser

AWS S3 Logs Parser is a simple PHP package to parse Amazon Simple Storage Service (Amazon S3) logs into a readable JSON format. The detailed usage report will show you how much times a file is downloaded and how much bytes are transferred.

• Getting Started
• Usage
• Contributing
  • Requirements
  • Installation steps
• License

Getting Started

1. Sign up for AWS – Before you begin, you need to sign up for an AWS account and retrieve your AWS credentials.
2. Create your own bucket – Now that you've signed up for Amazon S3, you're ready to create a bucket using the AWS Management Console.
3. Enable server access logging – When you enable logging, Amazon S3 delivers access logs for a source bucket to a target bucket that you choose.
4. Install the service – Using Composer is the recommended way to install it. The service is available via Packagist under the mtrdesign/s3-logs-parser package.

$ composer require mtrdesign/s3-logs-parser

Usage

Read Log Files From Local Storage

Extracts statistics much more quickly if you have already downloaded the logs to local storage with something like AWS CLI aws s3 sync.

<?php

$S3LogsParser->setConfigs([
    'version' => 'latest',
    'local_log_dir' => 'path/to/logs/that_i_already_downloaded/'
]);

?>

Read Log Files Directly From S3 Bucket

Via direct instantiation of S3LogsParser object:

<?php

use S3LogsParser\S3LogsParser;

$S3LogsParser = new S3LogsParser([
    'version' => 'latest',
    'region' => $awsBucketRegion,
    'access_key' => $awsAccessKey,
    'secret_key' => $awsSecretKey,
]);

?>

By Setting configuration parameters:

<?php

$S3LogsParser->setConfigs([
    'version' => 'latest',
    'region' => $awsBucketRegion,
    'access_key' => $awsAccessKey,
    'secret_key' => $awsSecretKey,
]);

?>

Extracting Statistics

Things like download, bandwidth, etc.

$date is an optional param. Pass a Carbon formatted date string. S3 logs tend to have filenames that look like 2022-05-02-19-18-32-91D293838329CB5E6. If $date is provided the %Y-%m-%d formatted date string will be used as a prefix to match log filenames.

$S3LogsParser->getStats($awsBucketName, $awsBucketPrefix, $date);

getStats() response should be something like this:

{
    "success":true,
    "statistics":{
        "bucket":"bn-test",
        "prefix":"bp-2018-10-31",
        "data":{
            "test.png":{
                "downloads":4,
                "bandwidth":4096
            },
            "test2.png":{
                "downloads":2,
                "bandwidth":2048
            }
        }
    }
}

Contributing

Ensure all the guides are followed and style/test checkers pass before pushing your code.

Requirements

• Git
• Docker
• Docker Compose
• GNU Make

Installation steps

1. Build the required services and Docker container with $ make docker-build
2. SSH into the container with $ make docker-bash
3. Confirm code style checker passes with $ make run-phpcs
4. Confirm code quality checker passes with $ make run-phpstan
5. Confirm code texts checker passes with $ make run-phpunit

Debugging

You can set the optional configuration parameter debug_mode to see a more verbose output.

$S3LogsParser = new S3LogsParser\S3LogsParser([
    'local_log_dir' => '/path/to/logs',
    'debug_mode' => 'true',
]);

License

AWS S3 Logs Parser is open source and available under the MIT License.