cosmo0920/fluent-bit-go-s3

Trouble with base64 values

chussenot opened this issue · 2 comments

chussenot commented

Problem

{
  "duration": 0.131,
  "ddsource": "nginx",
  "ddsourcecategory": "http_web_access",
  "service": "www",
  "ddtags": "name:prd-site-www-appli,type:site,project:www,component:appli,env:prd,commit_id:d9e8a2b5d4d2231e566b17b7db2d2a68af8e5cd2,branch:release",
  "hostname": "i-0510d3b24987b2405",
  "file": "/var/log/nginx/access.log.json",
  "message": "23.95.94.112 -  [04/Nov/2019:02:43:22 +0100] \"POST /p/fraise-k1008m06-5677100?model_id=4247865?model_id=4247866 HTTP/1.1\" 200 35759 \"https://www.manomano.fr/p/bahco-k1008m06-fraise-lime-k1008m06-5677100?model_id=4247865\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36\" http www.monsite.fr 0.131 \"\"",
  "http": {
    "status_code": 200,
    "method": "UE9TVA==",
    "referer": "aHR0cHM6Ly93d3cubWFub21hbm8uZnIvcC9iYWhjby1rMTAwOG0wNi1mcmFpc2UtbGltZS1rMTAwOG0wNi01Njc3MTAwP21vZGVsX2lkPTQyNDc4NjU=",
    "scheme": "aHR0cA==",
    "user": "",
    "url": "L3AvYmFoY28tazEwMDhtMDYtZnJhaXNlLWxpbWUtazEwMDhtMDYtNTY3NzEwMD9tb2RlbF9pZD00MjQ3ODY1P21vZGVsX2lkPTQyNDc4NjY=",
    "useragent": "TW96aWxsYS81LjAgKFgxMTsgTGludXggeDg2XzY0KSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvNzEuMC4zNTc4Ljk4IFNhZmFyaS81MzcuMzY=",
    "host": "d3d3Lm1hbm9tYW5vLmZy",
    "route": "",
    "forwarded_for": "MjMuOTUuOTQuMTEyLCAxNTIuMTk1LjEyNS40NiwgMTUyLjE5NS4xMTEuMjY="
  },
  "network": {
    "bytes_written": 35759,
    "client": {
      "ip": "MjMuOTUuOTQuMTEy"
    },
    "geoip": {
      "location": {
        "lat": 42.8864,
        "lon": 42.8864
      },
      "country": {
        "name": "VW5pdGVkIFN0YXRlcw==",
        "iso_code": "VVM="
      },
      "region": {
        "name": "TmV3IFlvcms="
      },
      "city": {
        "name": "QnVmZmFsbw=="
      }
    }
  }
}

Sometimes the values are still encode in Base64,
https://github.com/cosmo0920/fluent-bit-go-s3/blob/master/out_s3.go#L168

Steps to replicate

Expected Behavior or What you need to ask

...

Using Fluentd and S3 plugin versions

  • OS version - Debian
  • EC2
  • fluent-bit version - 1.2.2
    • paste result of fluent-bit --version or td-agent-bin --version
  • fluent-bit-go-s3 version (last version on master, v0.2.0)
    • paste boot log of fluent-bit or td-agent-bit
cosmo0920 commented

I've created a PR #4 to fix this issue.
Could you check this?