trivago/gollum

Missing Root CA Certificates in Docker image breaks AwsS3 Producer

jabinb opened this issue · 4 comments

jabinb commented

Issue description

Uploads fail because the AWS SDK can't verify the https certificate from s3 as there's no root CA's in the alpine image:

$ docker-compose up
Recreating gollum_gollum_1
Attaching to gollum_gollum_1
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="GOLLUM STARTING"
gollum_1  | time="2018-08-24T02:09:38Z" level=info msg="Plugin ConsoleOut has been disabled"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Creating fallback stream for *"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Found producer S3Out"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Instantiating S3Out"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Creating fallback stream for ingress_logs"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Found consumer SyslogdUDPSocketConsumer"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Found consumer SyslogdTCPSocketConsumer"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Instantiating SyslogdUDPSocketConsumer"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Instantiating SyslogdTCPSocketConsumer"
gollum_1  | time="2018-08-24T02:09:38Z" level=warning msg="RFC3164 demands UDP" PluginID=SyslogdTCPSocketConsumer PluginType=consumer.Syslogd
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Starting *producer.AwsS3"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Starting *core.LogConsumer"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Starting *consumer.Syslogd"
gollum_1  | time="2018-08-24T02:09:38Z" level=debug msg="Starting *consumer.Syslogd"
gollum_1  | time="2018-08-24T02:09:38Z" level=info msg="We be nice to them, if they be nice to us. (startup)"
gollum_1  | time="2018-08-24T02:09:38Z" level=error msg="Failed to open udp://0.0.0.0:1514" PluginID=SyslogdUDPSocketConsumer PluginType=consumer.Syslogd
gollum_1  | time="2018-08-24T02:09:41Z" level=error msg="Can't create multipart upload" PluginID=S3Out PluginType=producer.AwsS3 error="RequestError: send request failed\ncaused by: Post https://<redacted>.s3-us-west-1.amazonaws.com/gollum_ingress_logs_2018-08-24T02%3A09%3A40.453467756Z.log?uploads=: x509: failed to load system roots and no roots provided" file="gollum_ingress_logs_2018-08-24T02:09:40.453467756Z.log"

Context

I'm running Gollum via docker-compose and providing my AWS creds via the environment.

Possible Solution

I found this related issue in another Go project with direction to install the certs via apk add ca-certificates. As a workaround I'm overriding the entrypoint for the container and installing the certs before executing gollum.

entrypoint: [ "/bin/sh", "-c", "apk update && apk add ca-certificates && /usr/local/bin/gollum -ll 3 -c /etc/gollum/gollum.conf" ]
$ docker-compose up
Removing gollum_gollum_1
Recreating 7c62c069fcce_gollum_gollum_1
Attaching to gollum_gollum_1
gollum_1  | fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
gollum_1  | fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
gollum_1  | v3.7.0-260-gbb7e90cba8 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]
gollum_1  | v3.7.0-262-g0d853c61ba [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]
gollum_1  | OK: 9054 distinct packages available
gollum_1  | (1/1) Installing ca-certificates (20171114-r0)
gollum_1  | Executing busybox-1.27.2-r7.trigger
gollum_1  | Executing ca-certificates-20171114-r0.trigger
gollum_1  | OK: 5 MiB in 12 packages
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="GOLLUM STARTING"
gollum_1  | time="2018-08-24T02:20:02Z" level=info msg="Plugin ConsoleOut has been disabled"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Creating fallback stream for *"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Found producer S3Out"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Instantiating S3Out"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Creating fallback stream for ingress_logs"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Found consumer SyslogdTCPSocketConsumer"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Found consumer SyslogdUDPSocketConsumer"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Instantiating SyslogdTCPSocketConsumer"
gollum_1  | time="2018-08-24T02:20:02Z" level=warning msg="RFC3164 demands UDP" PluginID=SyslogdTCPSocketConsumer PluginType=consumer.Syslogd
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Instantiating SyslogdUDPSocketConsumer"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Starting *producer.AwsS3"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Starting *consumer.Syslogd"
gollum_1  | time="2018-08-24T02:20:02Z" level=info msg="We be nice to them, if they be nice to us. (startup)"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Starting *core.LogConsumer"
gollum_1  | time="2018-08-24T02:20:02Z" level=debug msg="Starting *consumer.Syslogd"
gollum_1  | time="2018-08-24T02:20:02Z" level=error msg="Failed to open udp://0.0.0.0:1514" PluginID=SyslogdUDPSocketConsumer PluginType=consumer.Syslogd
gollum_1  | time="2018-08-24T02:20:06Z" level=debug msg="successfully created multipart upload" PluginID=S3Out PluginType=producer.AwsS3 uploadId=0xc4200892e

Steps to Reproduce (for bugs)

  1. Create a gollum.conf with a consumer (I'm using syslog) and an AwsS3 producer
  2. Set the producer to have a low rotation (e.g. 1 minute)
  3. Run the docker container with your config
  4. Wait for an upload

Your Environment

Used versions

  • gollum version: 0.5.3
  • go version: golang:alpine:latest
  • Operating System and version: alpine

Config:

docker-compose.yml

version: '3.0'
services:
  gollum:
    image: trivago/gollum:0.5.3
    environment:
      AWS_ACCESS_KEY_ID: ""
      AWS_SECRET_ACCESS_KEY: ""
    volumes:
      - ./gollum.conf:/etc/gollum/gollum.conf
    command: "-ll 3 -c /etc/gollum/gollum.conf"
    ports:
      - 1514:1514/udp
      - 1514:1514/tcp

gollum.conf

SyslogdUDPSocketConsumer:
    Type: consumer.Syslogd
    Streams: "logs"
    Address: "udp://0.0.0.0:1514"
    Format: "RFC3164"
    SetMetadata: true
    Modulators:
        - filter.RegExp:
            Expression: "}$"

SyslogdTCPSocketConsumer:
    Type: consumer.Syslogd
    Streams: "logs"
    Address: "tcp://0.0.0.0:1514"
    Format: "RFC3164"
    SetMetadata: true
    Modulators:
        - filter.RegExp:
            Expression: "}$"

ConsoleOut:
    Enable: false
    Type: "producer.Console"
    Streams: "logs"
    Modulators:
        - format.Envelope:
            Postfix: "\n"

S3Out:
    Type: producer.AwsS3
    Streams: "logs"
    Credential:
        Type: environment
    Region: us-west-1
    Bucket: example-bucket
    Batch:
        TimeoutSec: 60
        MaxCount: 1000
        FlushCount: 500
        FlushTimeoutSec: 0
    Rotation:
        Timestamp: 2006-01-02T15:04:05.999999999Z07:00
        TimeoutMin: 1
        SizeMB: 20
        Compress: true
    Modulators:
        - format.Envelope:
                Postfix: "\n"
arnecls commented

Thanks - I'll add these to the 0.5.x and master branch later today.

arnecls commented

Fixed and docker image pushed as trivago/gollum:0.5.3-1.
Could you verify that this is working for you?

jabinb commented

Hey @arnecls I've removed my workaround and everything seems to be working as expected, thanks for the quick fix!

arnecls commented

👍