logstash-forwarder.conf not found with aufs driver
Closed this issue · 17 comments
Hello!
You've done great job to make ELK stack running under docker!
I'm trying to make use of it, got working elasticsearch and logstash, but stuck on forwarder.
It seems to scan for /etc/logstash-forwarder.conf on the following path:
/var/lib/docker/%s/subvolumes/%s/etc/logstash-forwarder.confBut it finds nothing on my machine, because the correct paths are:
/var/lib/docker# find . -name "logstash-forwarder.conf"
./aufs/mnt/80f6f8ad6feaa68c4fa5e67022a2e3eb82868d688243a4f7a0b09d1a46cff848/etc/logstash-forwarder.conf
./aufs/mnt/218207076e56f02d33d08dce9ef192d2a7ed9093557bf44b8dbd04a4eab92d82/etc/logstash-forwarder.conf
./aufs/mnt/3fc18eb4f58d9c525e68a4d4e1fcf32d7e934cfe59aaf5eaf3a86b3e99f98b89/etc/logstash-forwarder.conf
./aufs/mnt/44dc64c189c9ed8b8c522956f0d1e083e542a1bc548d0654b8b502cda490dd32/tmp/logstash-forwarder.confIt's not subvolumes, it's mnt.
I have docker version:
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070
What docker version were you targeting? Is it safe for me to simply replace subvolumes with mnt and then rebuild and use your image?
Thanks a lot for the pointer. To be honest I'm not too surprised that you run into an issue since I never tested with aufs (I only use CoreOS which uses btrfs) and there isn't a way to retrieve the volume path programmatically afaik (the Docker version shouldn't matter as long as fsouza/go-dockerclient works).
Anyways, I think I fixed it, so it should work now out of the box for aufs as well. If it doesn't please reopen and we figure it out.
I just ran into this one. I think. I'm running Debian Jessie as host with two containers logging:
root@acrobat:/var/lib/docker# find . -name "logstash-forwarder.conf"
./aufs/diff/2b15b2c6b7f4234fdb02673b08893b603f01992913d909cfbd9a3f07236fa31d/tmp/logstash-forwarder.conf
./aufs/diff/8d8f55ae33441341be93aef76c88e44de21c73f43338d07dc0862dbcab133a92/etc/logstash-forwarder.conf
Docker version 1.5.0, build a8a31ef.
In the container I want to monitor I've got this copied in:
root@da031b920758:/# cat /etc/logstash-forwarder.conf
{
"files": [
{
"paths": ["/var/log/dispatcher-*.log"],
"fields": {
"project":"some-project",
"component":"some-component",
}
}
]
}
The logstash-forwarder container generates this configuration:
[root@2b15b2c6b7f4 /]# cat /tmp/logstash-forwarder.conf
{
"network": {
"servers": [
"logs.company.com:5043"
],
"ssl certificate": "/mnt/logstash-forwarder/logstash-forwarder.crt",
"ssl key": "/mnt/logstash-forwarder/logstash-forwarder.key",
"ssl ca": "/mnt/logstash-forwarder/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [
"/var/lib/docker/containers/da031b9207583f65d950d71730c210b4fb0f148405c87df24bd7307343507e91/da031b9207583f65d950d71730c210b4fb0f148405c87df24bd7307343507e91-json.log"
],
"fields": {
"codec": "json",
"docker.hostname": "da031b920758",
"docker.id": "da031b9207583f65d950d71730c210b4fb0f148405c87df24bd7307343507e91",
"docker.image": "docker.company.com/some/image",
"docker.name": "/fooo",
"type": "docker"
}
},
....
Any ideas?
This issue was originally created to add support for the aufs driver.
Based on @dryewo's input we figured the proper subdirectory for aufs bellow /var/lib/docker would be aufs/mnt while yours now seems to be bellow aufs/diff (note the mnt vs diff).
I don't run anything using the aufs driver and hence don't know if this changed and therefore is a general issue or if your setup is somehow "special" ;P
Perhaps @dryewo can confirm this one way or the other?
The problem is that the docker folks refuse to make the path available via the API (moby/moby#7915) and therefore we can only take an educated guess based on the driver.
In short I would be happy to update the path but before that I would love to be sure that this is now the new default for aufs.
@digital-wonderland Thanks for re-opening! Could this perhaps be solved by copying the configuration file from the container using the new docker cp command?
This would help only partially - as in that way the config could be retrieved in a reliable way but it wouldn't be of much use since we still would not know bellow which directory those paths should be applied.
@digital-wonderland sorry, I'm out of the business right now, can't prove or deny it.
@digital-wonderland Is it perhaps possible to logstash-forward the container's stdout, ignoring the file paths? I'm mostly interesting in the custom field configuration. My example /var/log/dispatcher-*.log files basically contain the same as in stdout.
@dryewo no worries, thanks for the response :)
@tfnico stdout & stderr go into the normal docker log file which should always get picked up (it is also shown in your logstash-forwarder config above if I'm not mistaken).
Then you could use Logstashs conditionals and the mutate filter to do something like
filter {
if [docker.name] =~ /^\/foo-.*/ {
mutate {
add_field => {
"project" => "foo-project"
"component" => "foo-component"
}
}
}
}
Now any logs with a docker.name field, that starts with /foo- get the respective project & component fields added (dunno if regex groups are supported somehow to make it more dynamic).
Regardless of the above I just tested Debian 8 and Ubuntu 15.04 and both use aufs/diff, so I'm gonna update the code. This might screw over people with older installs but I don't see how this can be avoided :/
@digital-wonderland Thanks for the fix! I just pulled the latest image, but seems no custom configuration is being picked up yet.
thomas.nicolaisen@acrobat:~$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
digitalwonderland/logstash-forwarder latest 1b7f69d2f911 12 hours ago 246.4 MB
...
thomas.nicolaisen@acrobat:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ddc8a47d9b8a digitalwonderland/logstash-forwarder:latest "/usr/local/bin/dock 2 hours ago Up 2 hours logstash-forwarder
....
thomas.nicolaisen@acrobat:~$ docker history digitalwonderland/logstash-forwarder │2015/05/11 10:04:40.174911 INFO [TriggerRefresh] Starting logstash-forwarder...
IMAGE CREATED CREATED BY SIZE │2015/05/11 10:04:40.175015 INFO [RegisterDockerEventListener] Listening to docker events...
1b7f69d2f911 10 hours ago /bin/sh -c #(nop) ENTRYPOINT ["/usr/local/bin 0 B │
7205a33e2701 10 hours ago /bin/sh -c curl -Lo /usr/local/bin/docker-log 13.53 MB │
Container conf:
thomas.nicolaisen@acrobat:~$ docker inspect logstash-forwarder
[{
"AppArmorProfile": "",
"Args": [
"-quiet"
],
"Config": {
"AttachStderr": false,
"AttachStdin": false,
"AttachStdout": false,
"Cmd": [
"-quiet"
],
"CpuShares": 0,
"Cpuset": "",
"Domainname": "",
"Entrypoint": [
"/usr/local/bin/docker-logstash-forwarder"
],
"Env": [
"LOGSTASH_HOST=logs.company.com:5043",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"ExposedPorts": null,
"Hostname": "ddc8a47d9b8a",
"Image": "digitalwonderland/logstash-forwarder",
"MacAddress": "",
"Memory": 0,
"MemorySwap": 0,
"NetworkDisabled": false,
"OnBuild": null,
"OpenStdin": false,
"PortSpecs": null,
"StdinOnce": false,
"Tty": false,
"User": "",
"Volumes": null,
"WorkingDir": ""
},
"Created": "2015-05-11T08:04:40.038815305Z",
"Driver": "aufs",
"ExecDriver": "native-0.2",
"ExecIDs": [
"fe177723c27fa43d8f02bb4f8873d92ee311c680eba74c398391d401e06fc56b"
],
"HostConfig": {
"Binds": [
"/etc/localtime:/etc/localtime:ro",
"/var/run/docker.sock:/var/run/docker.sock",
"/mnt/logstash-forwarder:/mnt/logstash-forwarder:ro",
"/var/lib/docker:/var/lib/docker:ro"
],
"CapAdd": null,
"CapDrop": null,
"ContainerIDFile": "",
"Devices": [],
"Dns": null,
"DnsSearch": null,
"ExtraHosts": null,
"IpcMode": "",
"Links": null,
"LxcConf": [],
"NetworkMode": "bridge",
"PidMode": "",
"PortBindings": {},
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"RestartPolicy": {
"MaximumRetryCount": 0,
"Name": ""
},
"SecurityOpt": null,
"VolumesFrom": null
},
"HostnamePath": "/var/lib/docker/containers/ddc8a47d9b8aa75ffe17d31e166471f7381ba56b0d5a722c95a606a13f72ad18/hostname",
"HostsPath": "/var/lib/docker/containers/ddc8a47d9b8aa75ffe17d31e166471f7381ba56b0d5a722c95a606a13f72ad18/hosts",
"Id": "ddc8a47d9b8aa75ffe17d31e166471f7381ba56b0d5a722c95a606a13f72ad18",
"Image": "1b7f69d2f911afafc4795d567930dfb31ac56112de8f488c5f4164a0a122161d",
"MountLabel": "",
"Name": "/logstash-forwarder",
"NetworkSettings": {
"Bridge": "docker0",
"Gateway": "172.17.42.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.30",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"LinkLocalIPv6Address": "fe80::42:acff:fe11:1e",
"LinkLocalIPv6PrefixLen": 64,
"MacAddress": "02:42:ac:11:00:1e",
"PortMapping": null,
"Ports": {}
},
"Path": "/usr/local/bin/docker-logstash-forwarder",
"ProcessLabel": "",
"ResolvConfPath": "/var/lib/docker/containers/ddc8a47d9b8aa75ffe17d31e166471f7381ba56b0d5a722c95a606a13f72ad18/resolv.conf",
"RestartCount": 0,
"State": {
"Error": "",
"ExitCode": 0,
"FinishedAt": "0001-01-01T00:00:00Z",
"OOMKilled": false,
"Paused": false,
"Pid": 29874,
"Restarting": false,
"Running": true,
"StartedAt": "2015-05-11T08:04:40.166481902Z"
},
"Volumes": {
"/etc/localtime": "/etc/localtime",
"/mnt/logstash-forwarder": "/mnt/logstash-forwarder",
"/var/lib/docker": "/var/lib/docker",
"/var/run/docker.sock": "/run/docker.sock"
},
"VolumesRW": {
"/etc/localtime": false,
"/mnt/logstash-forwarder": false,
"/var/lib/docker": false,
"/var/run/docker.sock": true
}
}
]
The generated /tmp/logstash-forwarder.conf looks the same as before (in my comment at #1 (comment) ). Any ideas on what I'm doing wrong?
@tfnico I just tried with Ubuntu 15.04 and for a logstash-forwarder.conf containing
"files":[
{
"paths":[
"/var/log/foo.log"
],
"fields":{
"type":"foo"
}
}
]
I end up with the following in the generated logstash-forwarder.conf:
{
"paths": [
"/var/lib/docker/aufs/diff/003d528d87c80cf348f70e9e0b50b71adce82fe6a54dcfad95fd07ac9a263b85/var/log/foo.log"
],
"fields": {
"host": "003d528d87c8",
"type": "foo"
}
}
So, as far as I can see, everything is working as expected.
@digital-wonderland Ah, I had a typo in my container's conf file: one comma too many in the json. Now it works! Thanks!
@digital-wonderland On a related note, would you consider having a way to customize the fields for the stdout json.log entry by configuration, instead of mutating them in the logstash-server as you described earlier?
@tfnico shiny, glad you got it working. Re customizable fields I can't think of a feasible way to do this from the top of my head and neither have the time nor the use case currently so I most likely wont spend any time on it.
Hi, I'm not quite sure what's happening, but I have both diff and mnt for aufs, but my /etc/logstash-forwarder.conf is only in mnt... I'm thinking if we're not sure if it's going to be in diff or mnt, why not check both?
I tried to fork your project and build it myself, but having never done anything with Go, I felt it was a bit of a pain. Could you maybe update the Readme with a walkthrough? I've tried to look at how to compile go projects, but nothing I've tried seems to work.
Hi, I found the error in my setup. I'm starting the ELK stack and logstash-forwarder before my application with the /etc/logstash-forwarder.conf, and my mounted /var/lib/docker/aufs/mnt/* directories aren't updated with the new containers in log. Fix was to run the ELK stack after my applications.
@nuth It shouldn't matter if you start your app first or not (this way it would be much to fragile). I'm happy to update the code but for this we have to know which directories should take precedence. Can you look that up in some aufs docs/spec and link it here so that stuff can finally get fixed for all?
Yes, I agree that it should work in either case.
What mount shows from the running logstash-forwarder container is this:
...
/dev/disk/by-uuid/765e78c9-092e-41fa-993b-a2256263096e on /var/lib/docker/aufs type ext4 (rw,relatime,data=ordered)
none on /var/lib/docker/aufs/mnt/6dda5d10b306550d4e9a2ea060f7927a51fc070a44b0495300a10865eb4eec7c type aufs (rw,relatime,si=90ba2622dc836224,dio)
none on /var/lib/docker/aufs/mnt/d184c2a06f6f95fbca8294197037388d2e39970997a1af96b59668ab352b05e8 type aufs (rw,relatime,si=90ba2622de86e224,dio)
none on /var/lib/docker/aufs/mnt/637df51bb48244a6f9ef30fc3f63d07e8cef0fc998fae44a125bb851c156b774 type aufs (rw,relatime,si=90ba2622df610224,dio)
none on /var/lib/docker/aufs/mnt/8a905d3963639dfbc130fec5da88f0f766ccddcb58eb942fb6e48fa297c37ee8 type aufs (rw,relatime,si=90ba2622de8e3224,dio)
none on /var/lib/docker/aufs/mnt/09c9648e400cc6735507d06df85fe151c54ab7279b95b05a1247e43fe088ff4b type aufs (rw,relatime,si=90ba2622dcef8224,dio)
none on /var/lib/docker/aufs/mnt/b827737c640221d10d8af99f470e87fae75b92394f811281af54aaeab0236135 type aufs (rw,relatime,si=90ba2622de5e7224,dio)
none on /var/lib/docker/aufs/mnt/230fc537b119b1daa57c4aab8826ca2e2a1b87ac0a657ffbf28fee8481960822 type aufs (rw,relatime,si=90ba2622ddba5224,dio)
none on /var/lib/docker/aufs/mnt/c91d9616127560f469203f267cefb0f4acfb85408ef02a7fe5617697efa8635b type aufs (rw,relatime,si=90ba2622dceb2224,dio)
none on /var/lib/docker/aufs/mnt/c91d9616127560f469203f267cefb0f4acfb85408ef02a7fe5617697efa8635b type aufs (rw,relatime,si=90ba2622dceb2224,dio)
...
The issue seems to be that the directories in /var/lib/docker/aufs are detected as aufs filesystems on boot(or something) and when /var/lib/docker/aufs is updated on the host, those new aufs directories are not made available through the /var/lib/docker/aufs mount. I also tried remounting /var/lib/docker/aufs to see if that'd force the subtree to update, but no.
I'm going to take a closer look this weekend and read the docs as you say and try find a solution.