Configuration Error
aardelean opened this issue · 13 comments
aardelean commented
Hi, I am trying to use the detect-exceptions in a docker environment configured with fluentd driver, but my configuration throws some errors in the fluentd output:
2018-06-01 15:33:22 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/conf/docker.conf"
2018-06-01 15:33:22 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/conf/docker.conf"
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in 'require': cannot load such file -- fluent/plugin/exception_detector (LoadError)
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in 'rescue in require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in 'require'
from /var/lib/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/out_detect_exceptions.rb:16:in '<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in 'require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in 'require'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/registry.rb:102:in 'block in search'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/registry.rb:99:in 'each'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/registry.rb:99:in 'search'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/registry.rb:44:in 'lookup'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/plugin.rb:146:in 'new_impl'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/plugin.rb:104:in 'new_output'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/agent.rb:128:in 'add_match'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/agent.rb:72:in 'block in configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/agent.rb:64:in 'each'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/agent.rb:64:in 'configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/root_agent.rb:112:in 'configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/engine.rb:131:in 'configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/engine.rb:96:in 'run_configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/supervisor.rb:795:in 'run_configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/supervisor.rb:579:in 'dry_run'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/supervisor.rb:597:in 'supervise'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/supervisor.rb:502:in 'run_supervisor'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/command/fluentd.rb:310:in '<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in 'require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in 'require'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/bin/fluentd:8:in'<top (required)>'
from /usr/local/bin/fluentd:22:in 'load'
from /usr/local/bin/fluentd:22:in '<main>'
and the configuration is :
<source>
@type forward
port 24224
tag removeme
bind 0.0.0.0
</source>
<filter **>
@type record_transformer
enable_ruby
<record>
timestamp ${ require 'time'; Time.now.utc.iso8601(3) }
</record>
remove_keys source container_id
</filter>
<match removeme.*>
@type detect_exceptions
message log
languages java
remove_tag_prefix removeme
multiline_flush_interval 0.5
</match>
<match *>
@type elasticsearch
host elasticsearch
port 9200
time_key timestamp
index_name fluentb
type_name fluentb
</match>
igorpeshansky commented
I strongly suspect it's because of #32 (comment). We will investigate.
aardelean commented
@igorpeshansky thanks for your fast response. How can I somehow fix it or test if this is the real issue?
igorpeshansky commented
You can find exception_detector.rb under /opt/google-fluentd/embedded and chmod a+r. If that fixes it, then it's as I suspected.
You might also need to chmod a+rx on all parent directories of that file.
aardelean commented
I changed my docker image definition and now I gave permissions 640 with
RUN chmod 640 ./var/lib/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/exception_detector.rb but still the error is occuring.
More interesting is the message saying that from /var/lib/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/out_detect_exceptions.rb:16:in '<top (required)>'
aardelean commented
I have installed the plugin with sudo gem install fluentd-plugin-detect-exceptions in the Dockerfile. Is there another way?
aardelean commented
@igorpeshansky I have tried to grant file permissions with chmod a+x -R fluent/ and still get the same error. I doubt it is the permissions. Is there another version I could try?
gotttlieb commented
I think that i am getting the same error here:
2018-06-14 11:28:53 +0000 [info]: 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in `require': cannot load such file -- fluent/plugin/exception_detector (LoadError)
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in `rescue in require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require'
from /var/lib/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/out_detect_exceptions.rb:16:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/registry.rb:102:in `block in search'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/registry.rb:99:in `each'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/registry.rb:99:in `search'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/registry.rb:44:in `lookup'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/plugin.rb:146:in `new_impl'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/plugin.rb:104:in `new_output'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/agent.rb:128:in `add_match'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/agent.rb:72:in `block in configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/agent.rb:64:in `each'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/agent.rb:64:in `configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/root_agent.rb:112:in `configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/engine.rb:131:in `configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/engine.rb:96:in `run_configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/supervisor.rb:795:in `run_configure'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/supervisor.rb:579:in `dry_run'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/supervisor.rb:597:in `supervise'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/supervisor.rb:502:in `run_supervisor'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /var/lib/gems/2.3.0/gems/fluentd-1.2.1/bin/fluentd:8:in `<top (required)>'
from /usr/local/bin/fluentd:22:in `load'
from /usr/local/bin/fluentd:22:in `<main>'```
And Fluentd conf:
```#Getting all data sended
#by the docker login
#fluentd driver
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
#Get dev.nginx.log tag
#and log to ES
<match dev.nginx** >
@type copy
<store>
@type elasticsearch
host 10.0.3.10
port 9200
logstash_format true
logstash_prefix logstash
logstash_dateformat %Y%m%d
flush_interval 5s
include_tag_key true
include_time_key true
tag_key @log_name
</store>
</match>
#Get dev.springboot.log tag
#and log to ES
<match dev.springboot** >
@type detect_exceptions
message log
languages java
multiline_flush_interval 0.1
<store>
@type elasticsearch
host 10.0.3.10
port 9200
logstash_format true
logstash_prefix logstash
logstash_dateformat %Y%m%d
flush_interval 5s
include_tag_key true
include_time_key true
tag_key @log_name
</store>
</match>
#Get dev.redis containers logs
#and log to ES
<match dev.redis** >
@type copy
<store>
@type elasticsearch
host 10.0.3.10
port 9200
logstash_format true
logstash_prefix logstash
logstash_dateformat %Y%m%d
include_tag_key true
include_time_key true
tag_key @log_name
</store>
</match>
#Match the rest to know what
#is happening to ES
<match ** >
@type copy
<store>
@type elasticsearch
host 10.0.3.10
port 9200
logstash_format true
logstash_prefix logstash
logstash_dateformat %Y%m%d
flush_interval 5s
include_tag_key true
include_time_key true
tag_key @log_name
</store>
</match>```
DaemonDude23 commented
I'm having a similar issue as well on Centos7:
Jun 14 15:42:13 centos-7 systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Jun 14 15:42:14 centos-7 fluentd[21157]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `require': cannot load such file -- fluent/plugin/exception_detector (LoadError)
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:133:in `rescue in require'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:40:in `require'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/out_detect_exceptions.rb:16:in `<top (required)>'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/registry.rb:102:in `block in search'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/registry.rb:99:in `each'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/registry.rb:99:in `search'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/registry.rb:44:in `lookup'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin.rb:146:in `new_impl'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin.rb:104:in `new_output'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/agent.rb:128:in `add_match'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/agent.rb:72:in `block in configure'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/agent.rb:64:in `each'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/agent.rb:64:in `configure'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/root_agent.rb:112:in `configure'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/engine.rb:131:in `configure'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/engine.rb:96:in `run_configure'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/supervisor.rb:770:in `run_configure'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/supervisor.rb:553:in `dry_run'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/supervisor.rb:571:in `supervise'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/supervisor.rb:476:in `run_supervisor'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
Jun 14 15:42:14 centos-7 fluentd[21157]: from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/bin/fluentd:8:in `<top (required)>'
Config block:
<match docker.**>
@type detect_exceptions # https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions
remove_tag_prefix foo
message log
languages all
multiline_flush_interval 0.1
</match>
Commands used to install:
yum install -y make gcc ruby-devel rubygems-integration
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
td-agent-gem install fluent-plugin-logzio fluent-plugin-concat fluent-plugin-parser fluent-plugin-rename-key
DaemonDude23 commented
As others have noted in this issue and others, I've resolved the error after plugin installation with the following commands:
[root@work-centos-7 ~]# ls -l /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/exception_detector.rb
-rw-r-----. 1 root root 11514 Jun 14 15:41 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/exception_detector.rb
[root@work-centos-7 ~]# chmod 644 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/exception_detector.rb
gotttlieb commented
I have tried this solution in the official docker image for the versions 1.2 and 0.14 and is not working (changing the path to the correct one /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-detect-exceptions*)
# DO NOT EDIT THIS FILE DIRECTLY, USE /Dockerfile.template.erb
FROM debian:stretch-slim
LABEL maintainer "TAGOMORI Satoshi <tagomoris@gmail.com>"
LABEL Description="Fluentd docker image" Vendor="Fluent Organization" Version="1.1"
ENV DUMB_INIT_VERSION=1.2.0
ENV GOSU_VERSION=1.10
ARG DEBIAN_FRONTEND=noninteractive
# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
# therefore an 'apt-get purge' has no effect
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
ca-certificates \
ruby \
curl \
&& buildDeps=" \
make gcc g++ libc-dev \
ruby-dev \
wget bzip2 gnupg dirmngr \
" \
&& apt-get install -y --no-install-recommends $buildDeps \
&& update-ca-certificates \
&& echo 'gem: --no-document' >> /etc/gemrc \
&& gem install oj -v 3.3.10 \
&& gem install json -v 2.1.0 \
&& gem install fluentd -v 1.2.1
RUN dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
&& curl -L -o /usr/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_$dpkgArch \
&& chmod +x /usr/bin/dumb-init \
&& curl -L -o /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
&& curl -L -o /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc" \
&& export GNUPGHOME="$(mktemp -d)" \
&& gpg --keyserver keyserver.ubuntu.com --keyserver-options http-proxy=http://proxy.wandas.sas:8080 --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
&& gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
&& rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu \
&& gosu nobody true \
&& curl -L -o /tmp/jemalloc-4.5.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/4.5.0/jemalloc-4.5.0.tar.bz2 \
&& cd /tmp && tar -xjf jemalloc-4.5.0.tar.bz2 && cd jemalloc-4.5.0/ \
&& ./configure && make \
&& mv lib/libjemalloc.so.2 /usr/lib
RUN fluent-gem install fluent-plugin-elasticsearch \
&& fluent-gem install fluent-plugin-detect-exceptions \
&& fluent-gem install fluent-plugin-parser \
&& apt-get purge -y --auto-remove \
-o APT::AutoRemove::RecommendsImportant=false \
$buildDeps \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
# for log storage (maybe shared with host)
RUN mkdir -p /fluentd/log
# configuration/plugins path (default: copied from .)
RUN mkdir -p /fluentd/etc /fluentd/plugins
COPY fluent.conf /fluentd/etc/
COPY entrypoint.sh /bin/
RUN chmod +x /bin/entrypoint.sh
RUN chmod -R 644 /var/lib/gems/2.3.0/gems/fluent-plugin-detect-exceptions*
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"
ENV LD_PRELOAD="/usr/lib/libjemalloc.so.2"
ENV DUMB_INIT_SETSID 0
EXPOSE 24224 5140
#ENTRYPOINT ["tail", "-f", "/dev/null"]
ENTRYPOINT ["/bin/entrypoint.sh"]
#CMD exec fluentd -c /fluentd/etc/${FLUENTD_CONF} -p /fluentd/plugins $FLUENTD_OPT
Markbnj commented
Original comment edited. I initially reported that setting the file permissions did not fix the issue, but I was caught by my image pull policy for the daemonset, and was execing the previous image rather than my revised image, doh :).
Setting the file permissions using the following command did fix the issue:
RUN chmod 644 /var/lib/gems/2.3.0/gems/fluent-plugin-detect-exceptions-0.0.11/lib/fluent/plugin/exception_detector.rb
Note that our container is based off the official fluentd debian image, thus the differences in where the gems are installed.
grenzr commented
Just hit the same problem whilst using the 'common' tag of a Fluentd image built from https://github.com/theasp/docker-fluentd-plugins
igorpeshansky commented
This should be fixed in v0.0.12. If not, please reopen.