Unknown output plugin 'scalyr'...
Closed this issue · 7 comments
negativeOne1 commented
Hi,
I'm trying to install scalyr-fluentd in Docker using fluentd:v0.14 any Idea what I'm doing wrong?
Dockerfile:
FROM fluent/fluentd:v0.14
ADD scalyr-fluentd /scalyr-fluentd
ADD fluent.conf /fluentd/etc/fluent.conf
RUN apk add --update --virtual .build-deps sudo build-base ruby-dev ruby-rake \
libxml2-dev git sudo\
&& cd /scalyr-fluentd \
&& fluent-gem install rails \
&& rake build \
&& fluent-gem install pkg/fluent-plugin-scalyr-0.7.4.gem \
&& fluent-gem source --clear-all \
&& apk del .build-deps git sudo\
&& rm -rf /var/cache/apk/* \
/home/fluent/.gem/ruby/2.3.0/cache/*.gem
fluent.conf:
<match scalyr.*>
@type scalyr
api_write_token <API_TOKEN>
</match>
~/p/m/fluentd_scalyr ❯❯❯ docker build -t fluent_scalyr .
Sending build context to Docker daemon 259.1kB
Step 1/7 : FROM fluent/fluentd:v0.14
---> d654e2fa1208
Step 2/7 : LABEL maintainer "Martin Kluth <martin.kluth@morressier.com>"
---> Using cache
---> 35e61c9306f0
Step 3/7 : LABEL Description "Fluentd Docker Image with Scalyr support"
---> Using cache
---> 6687e6cfc4cc
Step 4/7 : USER root
---> Using cache
---> fd0a03fa9250
Step 5/7 : ADD scalyr-fluentd /scalyr-fluentd
---> Using cache
---> 07999591dfc7
Step 6/7 : ADD fluent.conf /fluentd/etc/fluent.conf
---> Using cache
---> ecd4f6a01527
Step 7/7 : RUN apk add --update --virtual .build-deps sudo build-base ruby-dev ruby-rake libxml2-dev git sudo && cd /scalyr-fluentd && fluent-gem install rails && rake build && fluent-gem install pkg/fluent-plugin-scalyr-0.7.4.gem && fluent-gem source --clear-all && apk del .build-deps git sudo && rm -rf /var/cache/apk/* /home/fluent/.gem/ruby/2.3.0/cache/*.gem
---> Running in adc80ffc87d7
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
(1/32) Installing sudo (1.8.19_p1-r0)
(2/32) Installing binutils-libs (2.27-r1)
(3/32) Installing binutils (2.27-r1)
(4/32) Installing gmp (6.1.1-r0)
(5/32) Installing isl (0.17.1-r0)
(6/32) Installing libgomp (6.2.1-r1)
(7/32) Installing libatomic (6.2.1-r1)
(8/32) Installing pkgconf (1.0.2-r0)
(9/32) Installing libgcc (6.2.1-r1)
(10/32) Installing mpfr3 (3.1.5-r0)
(11/32) Installing mpc1 (1.0.3-r0)
(12/32) Installing libstdc++ (6.2.1-r1)
(13/32) Installing gcc (6.2.1-r1)
(14/32) Installing make (4.2.1-r0)
(15/32) Installing musl-dev (1.1.15-r8)
(16/32) Installing libc-dev (0.7-r1)
(17/32) Installing fortify-headers (0.8-r0)
(18/32) Installing g++ (6.2.1-r1)
(19/32) Installing build-base (0.4-r1)
(20/32) Installing libgmpxx (6.1.1-r0)
(21/32) Installing gmp-dev (6.1.1-r0)
(22/32) Installing ruby-dev (2.3.5-r0)
(23/32) Installing ruby-rake (2.3.5-r0)
(24/32) Installing zlib-dev (1.2.11-r0)
(25/32) Installing libxml2 (2.9.4-r3)
(26/32) Installing libxml2-dev (2.9.4-r3)
(27/32) Installing libssh2 (1.7.0-r2)
(28/32) Installing libcurl (7.56.1-r0)
(29/32) Installing expat (2.2.0-r1)
(30/32) Installing pcre (8.39-r0)
(31/32) Installing git (2.11.3-r0)
(32/32) Installing .build-deps (0)
Executing busybox-1.25.1-r1.trigger
OK: 210 MiB in 56 packages
Successfully installed concurrent-ruby-1.0.5
Successfully installed i18n-0.9.1
Successfully installed minitest-5.10.3
Successfully installed activesupport-5.1.4
Successfully installed rack-2.0.3
Successfully installed rack-test-0.8.2
Successfully installed mini_portile2-2.3.0
Building native extensions. This could take a while...
Successfully installed nokogiri-1.8.1
Successfully installed crass-1.0.3
Successfully installed loofah-2.1.1
Successfully installed rails-html-sanitizer-1.0.3
Successfully installed rails-dom-testing-2.0.3
Successfully installed builder-3.2.3
Successfully installed erubi-1.7.0
Successfully installed actionview-5.1.4
Successfully installed actionpack-5.1.4
Successfully installed activemodel-5.1.4
Successfully installed arel-8.0.0
Successfully installed activerecord-5.1.4
Successfully installed globalid-0.4.1
Successfully installed activejob-5.1.4
Successfully installed mini_mime-1.0.0
Successfully installed mail-2.7.0
Successfully installed actionmailer-5.1.4
Building native extensions. This could take a while...
Successfully installed nio4r-2.1.0
Successfully installed websocket-extensions-0.1.3
Building native extensions. This could take a while...
Successfully installed websocket-driver-0.6.5
Successfully installed actioncable-5.1.4
Successfully installed thor-0.20.0
Successfully installed method_source-0.9.0
Successfully installed railties-5.1.4
Successfully installed bundler-1.16.0
Successfully installed sprockets-3.7.1
Successfully installed sprockets-rails-3.2.1
Successfully installed rails-5.1.4
35 gems installed
fatal: Not a git repository: ../.git/modules/scalyr-fluentd
fatal: Not a git repository: ../.git/modules/scalyr-fluentd
fatal: Not a git repository: ../.git/modules/scalyr-fluentd
fluent-plugin-scalyr 0.7.4 built to pkg/fluent-plugin-scalyr-0.7.4.gem.
Building native extensions. This could take a while...
Successfully installed string-scrub-0.0.5
Successfully installed fluentd-0.12.41
Successfully installed fluent-plugin-scalyr-0.7.4
3 gems installed
*** Removed specs cache ***
(1/32) Purging .build-deps (0)
(2/32) Purging sudo (1.8.19_p1-r0)
(3/32) Purging build-base (0.4-r1)
(4/32) Purging make (4.2.1-r0)
(5/32) Purging fortify-headers (0.8-r0)
(6/32) Purging g++ (6.2.1-r1)
(7/32) Purging gcc (6.2.1-r1)
(8/32) Purging binutils (2.27-r1)
(9/32) Purging libatomic (6.2.1-r1)
(10/32) Purging libgomp (6.2.1-r1)
(11/32) Purging libc-dev (0.7-r1)
(12/32) Purging musl-dev (1.1.15-r8)
(13/32) Purging ruby-dev (2.3.5-r0)
(14/32) Purging gmp-dev (6.1.1-r0)
(15/32) Purging libgmpxx (6.1.1-r0)
(16/32) Purging ruby-rake (2.3.5-r0)
(17/32) Purging libxml2-dev (2.9.4-r3)
(18/32) Purging zlib-dev (1.2.11-r0)
(19/32) Purging libxml2 (2.9.4-r3)
(20/32) Purging git (2.11.3-r0)
(21/32) Purging binutils-libs (2.27-r1)
(22/32) Purging mpc1 (1.0.3-r0)
(23/32) Purging mpfr3 (3.1.5-r0)
(24/32) Purging isl (0.17.1-r0)
(25/32) Purging gmp (6.1.1-r0)
(26/32) Purging pkgconf (1.0.2-r0)
(27/32) Purging libstdc++ (6.2.1-r1)
(28/32) Purging libgcc (6.2.1-r1)
(29/32) Purging libcurl (7.56.1-r0)
(30/32) Purging libssh2 (1.7.0-r2)
(31/32) Purging expat (2.2.0-r1)
(32/32) Purging pcre (8.39-r0)
Executing busybox-1.25.1-r1.trigger
OK: 27 MiB in 24 packages
---> 018caa589d2e
Removing intermediate container adc80ffc87d7
Successfully built 018caa589d2e
Successfully tagged fluent_scalyr:latest
~/p/m/fluentd_scalyr ❯❯❯ docker run -it fluent_scalyr
2017-12-01 13:34:01 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2017-12-01 13:34:01 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'scalyr'. Run 'gem search -rd fluent-plugin' to find plugins"
imron commented
For the purpose of maintaining compatibility for some of our customers, the master branch of the scalyr-fluentd repository contains code compatible with fluentd-0.12, and there are breaking changes between 0.12 and 0.14.
There is a separate branch called fluent-0.14, and you need to make sure that you are using that branch if you want to use fluentd-v0.14
You also need to make sure that you specify version 0.8.0 of the fluent-scalyr gem in your Dockerfile, e.g. you'd need to update your Dockerfile to have
&& fluent-gem install pkg/fluent-plugin-scalyr-0.8.0.gem \
instead of:
&& fluent-gem install pkg/fluent-plugin-scalyr-0.7.4.gem \
If you do that, then it should work as intended. I just tested it out and using your provided Dockerfile, config and build commands, along with the changes mentioned above, this is the final output:
2017-12-01 14:22:33 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2017-12-01 14:22:33 +0000 [warn]: Pre 0.14.0 configuration file detected. Please consider updating your configuration file
2017-12-01 14:22:33 +0000 [info]: using configuration file: <ROOT>
<match scalyr.*>
@type scalyr
api_write_token <API-KEY-REMOVED>
</match>
</ROOT>
2017-12-01 14:22:33 +0000 [info]: starting fluentd-0.14.25 pid=5 ruby="2.3.5"
2017-12-01 14:22:33 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2017-12-01 14:22:34 +0000 [info]: gem 'fluent-plugin-scalyr' version '0.8.0'
2017-12-01 14:22:34 +0000 [info]: gem 'fluentd' version '0.14.25'
2017-12-01 14:22:34 +0000 [info]: adding match pattern="scalyr.*" type="scalyr"
2017-12-01 14:22:34 +0000 [warn]: #0 Pre 0.14.0 configuration file detected. Please consider updating your configuration file
2017-12-01 14:22:34 +0000 [info]: #0 starting fluentd worker pid=15 ppid=5 worker=0
2017-12-01 14:22:34 +0000 [info]: #0 fluentd worker is now running worker=0
So it looks to be working as intended.
negativeOne1 commented
still doesn't work, got stuck with the "Unknown output plugin 'scalyr'. Run 'gem search -rd fluent-plugin' to find plugins" message. I noticed that during the plugin build there is a
fatal: Not a git repository: ../.git/modules/scalyr-fluentd
fatal: Not a git repository: ../.git/modules/scalyr-fluentd
fatal: Not a git repository: ../.git/modules/scalyr-fluentd
message, do you get them to?
from within the container i ran:
fluent-gem list --local
*** LOCAL GEMS ***
actioncable (5.1.4)
actionmailer (5.1.4)
actionpack (5.1.4)
actionview (5.1.4)
activejob (5.1.4)
activemodel (5.1.4)
activerecord (5.1.4)
activesupport (5.1.4)
arel (8.0.0)
builder (3.2.3)
bundler (1.16.0)
concurrent-ruby (1.0.5)
cool.io (1.5.3)
crass (1.0.3)
did_you_mean (1.0.0)
erubi (1.7.0)
fluent-plugin-scalyr (0.8.0)
fluentd (0.14.25)
globalid (0.4.1)
http_parser.rb (0.6.0)
i18n (0.9.1)
json (2.1.0)
loofah (2.1.1)
mail (2.7.0)
method_source (0.9.0)
mini_mime (1.0.0)
mini_portile2 (2.3.0)
minitest (5.10.3)
msgpack (1.1.0)
net-telnet (0.1.1)
nio4r (2.1.0)
nokogiri (1.8.1)
oj (2.18.3)
power_assert (0.2.6)
psych (2.1.0.1)
rack (2.0.3)
rack-test (0.8.2)
rails (5.1.4)
rails-dom-testing (2.0.3)
rails-html-sanitizer (1.0.3)
railties (5.1.4)
rake (10.4.2)
ruby_dig (0.0.2)
serverengine (2.0.5)
sigdump (0.2.4)
sprockets (3.7.1)
sprockets-rails (3.2.1)
strptime (0.1.9)
test-unit (3.1.5)
thor (0.20.0)
thread_safe (0.3.6)
tzinfo (1.2.4)
tzinfo-data (1.2017.3)
websocket-driver (0.6.5)
websocket-extensions (0.1.3)
yajl-ruby (1.3.1)
so it looks like the plugin is installed and versions match
imron commented
I didn't get those errors, and I see now that it was because I had just cloned the repository in to the scalyr-fluentd for testing, which is obviously not ideal for production because you don't want to be storing the entire repository in the container.
Upon further investigation, the problem is caused because gemspec file calls out to git --ls-tree to generate the list of files to include in the gem.
I'll update the gemspec file to fix that and should have it ready to go in maybe 10-15 mins.
imron commented
Ok. That's ready to go now. Make sure you fetch the latest changes from the fluent-0.14 branch and also update your Dockerfile to use the fluent-plugin-scalyr-0.8.1.gem
negativeOne1 commented
That seems to do the trick, Thanks.
negativeOne1 commented
FYI: I also had to change the ca cert path my fluent.conf to get it to run under fluentd 0.14
ssl_ca_bundle_path /etc/ssl/certs/ca-certificates.crt
imron commented
Glad to hear it's working.