Graylog2/graylog-plugin-beats

Add support for parsing journalbeat

salekseev opened this issue · 4 comments

salekseev commented

Add support for journalbeat input (https://github.com/mheese/journalbeat)

Example message:

{
  "@realtime_timestamp": 1501612251261196,
  "@timestamp": "2017-08-01T18:30:51.261Z",
  "MESSAGE": "I0801 18:30:51.260699 25032 http.cpp:420] HTTP GET for /master/state from 10.129.199.60:36238 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36'",
  "PRIORITY": "6",
  "SYSLOG_FACILITY": "3",
  "SYSLOG_IDENTIFIER": "mesos-master",
  "_BOOT_ID": "ed665473a2054d85b2aab95a37b686f0",
  "_CAP_EFFECTIVE": "1fffffffff",
  "_CMDLINE": "/opt/mesosphere/packages/mesos--aaedd03eee0d57f5c0d49c74ff1e5721862cad98/bin/mesos-master",
  "_COMM": "mesos-master",
  "_EXE": "/opt/mesosphere/packages/mesos--aaedd03eee0d57f5c0d49c74ff1e5721862cad98/bin/mesos-master",
  "_GID": "0",
  "_HOSTNAME": "ip-10-129-198-152",
  "_MACHINE_ID": "a8a482f5bd664ae28475981927ae888d",
  "_PID": "25020",
  "_SELINUX_CONTEXT": "system_u:system_r:init_t:s0",
  "_SYSTEMD_CGROUP": "/system.slice/dcos-mesos-master.service",
  "_SYSTEMD_SLICE": "system.slice",
  "_SYSTEMD_UNIT": "dcos-mesos-master.service",
  "_TRANSPORT": "stdout",
  "_UID": "0",
  "beat": {
    "hostname": "ip-10-129-198-152",
    "name": "journalbeat",
    "version": "5.5.0"
  },
  "meta": {
    "cloud": {
      "availability_zone": "us-east-1c",
      "instance_id": "i-05cefc8e13714f64b",
      "machine_type": "m4.2xlarge",
      "provider": "ec2",
      "region": "us-east-1"
    }
  },
  "type": "journal"
}
joschi commented

@salekseev "Generic" beats should already work out of the box.

What doesn't work with journalbeat and the current incarnation of the Graylog Beats plugin?

jalogisch commented

@salekseev could you please elaborate what you are missing?

I have this beat running myself without any issues, the only what you need todo is rewrite some of the field names, but that depends on your setup.

salekseev commented

Guys, sorry, it does work, I just had a delay due to high msg count. Only improvement that's needed is identifying this specific beat vs using a generic beat facility and prefix for easier filtering. Thanks.

jalogisch commented

thank you for the feedback @salekseev. You could write a pipeline rule that rewrites the fields and data.