fluent-bit s3 output plugin

Windows binaries are available in release pages.

This plugin works with fluent-bit's go plugin interface. You can use fluent-bit-go-s3 to ship logs into AWS S3.

The configuration typically looks like:

fluent-bit --> AWS S3

Usage

$ fluent-bit -e /path/to/built/out_s3.so -c fluent-bit.conf

Or,

$ make build-image

and then, specify configuration parameters as environment variables:

$ docker run -it -e="FLUENT_BIT_ACCESS_KEY_ID=yourawsaccesskey" \
                 -e="FLUENT_BIT_SECRET_ACCESS_KEY=yourawsaccesssecret" \
                 -e="FLUENT_BIT_BUCKET_NAME=yourbucketname" \
                 -e="FLUENT_BIT_S3_PREFIX=yours3prefix" \
                 -e="FLUENT_BIT_REGION=awsregion" \
                 -e="FLUENT_BIT_SUFFIX_ALGORITHM=algorithm" \
                 cosmo0920/fluent-bit-go-s3[:built-image-version]

Using docker image from docker hub.

$ docker pull cosmo0920/fluent-bit-go-s3[:latest or other tag]

Other released images are available in DockerHub's fluent-bit-go-s3 image tags.

Or, using helm:

helm install [YOURRELEASENAME] ./helm/fluent-bit

Prerequisites

Go 1.11+
gcc (for cgo)
make

Building

$ make

Configuration Options

Key	Description	Default value	Note
Credential	URI of AWS shared credential	`""`	(See Credentials)
AccessKeyID	Access key ID of AWS	`""`	(See Credentials)
SecretAccessKey	Secret access key ID of AWS	`""`	(See Credentials)
Bucket	Bucket name of S3 storage	`-`	Mandatory parameter
S3Prefix	S3Prefix of S3 key	`-`	Mandatory parameter
SuffixAlgorithm	Algorithm for naming S3 object suffix	`""`	sha256 or no suffix(`""`)
Region	Region of S3	`-`	Mandatory parameter
Compress	Choose Compress method	`""`	gzip or plainText(`""`)
Endpoint	Specify the endpoint URL	`""`	URL with port or empty string
AutoCreateBucket	Create bucket automatically	`false`	true/false
LogLevel	Specify Log Level	`"info"`	trace/debug/info/warning/error/fatal/panic
TimeFormat	Time format to add to the S3 path	`"20060102/15"`	Specify in Go's Time Format
TimeZone	Specify TimeZone	`""`	Specify TZInfo based region. e.g.) Asia/Tokyo

Example:

Add this section to fluent-bit.conf:

[Output]
    Name s3
    Match *
    # Credential    /path/to/sharedcredentialfile
    AccessKeyID     yourawsaccesskeyid
    SecretAccessKey yourawssecretaccesskey
    Bucket          yourbucketname
    S3Prefix        yours3prefixname
    SuffixAlgorithm sha256
    Region          us-east-1
    Compress        gzip
    # Endpoint parameter is mainly used for minio.
    # Endpoint      http://localhost:9000
    # TimeFormat    20060102/15
    # TimeZone      Asia/Tokyo

Credentials

By default AWS credentials are loaded from their usual providers. See AWS CLI Configuration and Credential File Settings.

If the default credentials are not desired, this plugin supports the following credential providers.

Shared Credentials

Create the following file which includes credentials:

[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

Then, specify the following parameter in fluent-bit configuration:

Credential    /path/to/sharedcredentialfile

Static Credentials

Specify the following parameters in fluent-bit configuration:

AccessKeyID     yourawsaccesskeyid
SecretAccessKey yourawssecretaccesskey

Useful links

fluent-bit-go