tzinfo-2.0.2 conflicts with tzinfo (~> 1.1)
toastyblast opened this issue · 8 comments
I'm having conflict issues with adding fluent-plugin-sql & pg to my project so that FluentD can start outputting to my PSQL Docker container. Prior to adding these I've had no issues, but now after getting pg to work (which was a riot of its own) the cointainer after building and starting immediately stops with a conflict error from something called activesupport-5.2.4.2 due to something called tzinfo-2.0.2 conflicting with another tzinfo of 1.1. I'll explain what I've found and tried further down in this issue.
I'm running FluentD, PostgreSQL and my applications all as Docker containers, using their respective official images (fluent/fluentd:v1.10-1 and postgres:9.6), started through a docker-compose.yml file.
fleuntd.docker:
FROM fluent/fluentd:v1.10-1
ENV PSQL_HOST "psql-server"
ENV PSQL_PORT "5432"
ENV PSQL_DB "fluentd-logs-db"
ENV PSQL_USERNAME "dadmin1"
ENV PSQL_PASSWORD "letmeinplease"
USER root
VOLUME /fluentd/log
# libpq-dev
RUN apk add --no-cache --update --virtual .build-deps postgresql-dev \
sudo build-base ruby-dev \
&& sudo gem install fluent-plugin-route fluent-plugin-record-modifier fluent-plugin-rewrite-tag-filter \
fluent-plugin-sql pg \
&& sudo gem sources --clear-all \
&& apk del .build-deps postgresql-dev \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
# && rm -rf /var/cache/apk/* /home/fluent/.gem/ruby/2.5.0/cache/*.gem
COPY docker/fluentd-conf/fluent.conf /fluentd/etc/fluent.conf
COPY docker/fluentd-conf/fluentd.sh /bin/entrypoint.sh
RUN chmod +x /bin/entrypoint.sh
CMD ["/bin/entrypoint.sh"]
Side note: These ENV credentials are not real, this personal project is used locally only and is not openly hosted anywhere.
I have already resolved issues of not being able to find the pg
gem, by adding the postgresql-dev
instead of libpq-dev
(which doesn't seem to work wherever I place it in the RUN...). I found that idea here, #65-454016821
The image finally builds now, but I get a new error within FluentD itself:
...
Creating fluentd-host ... done
Attaching to fluentd-host
fluentd-host | 2020-04-15 11:22:46 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd-host | 2020-04-15 11:22:46 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
fluentd-host | 2020-04-15 11:22:46 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
fluentd-host | 2020-04-15 11:22:46 +0000 [info]: gem 'fluent-plugin-route' version '1.0.0'
fluentd-host | 2020-04-15 11:22:46 +0000 [info]: gem 'fluent-plugin-sql' version '1.1.1'
fluentd-host | 2020-04-15 11:22:46 +0000 [info]: gem 'fluentd' version '1.10.1'
fluentd-host | /usr/lib/ruby/2.5.0/rubygems/specification.rb:2325:in `raise_if_conflicts': Unable to activate activesupport-5.2.4.2, because tzinfo-2.0.2 conflicts with tzinfo (~> 1.1) (Gem::ConflictError)
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1438:in `activate'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1472:in `block in activate_dependencies'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1458:in `each'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1458:in `activate_dependencies'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1440:in `activate'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems.rb:224:in `rescue in try_activate'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems.rb:217:in `try_activate'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:128:in `rescue in require'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:39:in `require'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:147:in `initialize'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/plugin.rb:158:in `new'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/plugin.rb:158:in `new_impl'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/plugin.rb:109:in `new_output'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/agent.rb:130:in `add_match'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/agent.rb:74:in `block in configure'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/agent.rb:64:in `each'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/agent.rb:64:in `configure'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/label.rb:31:in `configure'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/root_agent.rb:143:in `block in configure'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/root_agent.rb:143:in `each'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/root_agent.rb:143:in `configure'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/engine.rb:105:in `configure'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/engine.rb:80:in `run_configure'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/supervisor.rb:547:in `run_supervisor'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/command/fluentd.rb:330:in `<top (required)>'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
fluentd-host | from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
fluentd-host | from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/bin/fluentd:8:in `<top (required)>'
fluentd-host | from /usr/bin/fluentd:23:in `load'
fluentd-host | from /usr/bin/fluentd:23:in `<main>'
fluentd-host exited with code 1
Additionally (before the error above, but who knows it might be related in some way), I also get an error from i-18-n during the gem install procedure:
...
Successfully installed fluent-plugin-route-1.0.0
Successfully installed fluent-plugin-record-modifier-2.1.0
Successfully installed fluent-config-regexp-type-1.0.0
Successfully installed fluent-plugin-rewrite-tag-filter-2.3.0
Successfully installed thread_safe-0.3.6
Successfully installed tzinfo-1.2.7
HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.
But that may break your application.
If you are upgrading your Rails application from an older version of Rails:
Please check your Rails app for 'config.i18n.fallbacks = true'.
If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be
'config.i18n.fallbacks = [I18n.default_locale]'.
If not, fallbacks will be broken in your app by I18n 1.1.x.
If you are starting a NEW Rails application, you can ignore this notice.
For more info see:
https://github.com/svenfuchs/i18n/releases/tag/v1.1.0
Successfully installed i18n-1.8.2
Successfully installed minitest-5.14.0
Successfully installed activesupport-5.2.4.2
Successfully installed activemodel-5.2.4.2
Successfully installed arel-9.0.0
Successfully installed activerecord-5.2.4.2
Successfully installed activerecord-import-0.28.2
Successfully installed fluent-plugin-sql-1.1.1
Building native extensions. This could take a while...
Successfully installed pg-0.21.0
15 gems installed
...
I have seen mentions of this issue in places like fluent/fluentd#2726 but the fix they seem to have implemented there seems to not have it fixed for my case.
Like I mentioned, I have tried using libpq-dev
(alone or in addition to postgresql-dev
) and setting pg:0.21.0
but these would get issues earlier on about not being able to find config files.
I have put the full output log in a file attached to this issue for review.
tzinfo
is a dependency of a dependency of activerecord
. A version which supports tzinfo
2.0 or higher has not been released by Rails yet.
I found a temporary solution by using the following file I made:
FROM fluent/fluentd:v1.10.2-1.0
ENV PSQL_HOST "psql-server"
ENV PSQL_PORT "5432"
ENV PSQL_DB "fluentd-logs-db"
ENV PSQL_USERNAME "dadmin1"
ENV PSQL_PASSWORD "letmeinplease"
USER root
VOLUME /fluentd/log
RUN apk add --no-cache --update --virtual .build-deps \
sudo build-base libpq postgresql-dev ruby-dev \
&& sudo gem install fluent-plugin-route fluent-plugin-record-modifier fluent-plugin-rewrite-tag-filter \
fluent-plugin-sql pg \
&& sudo gem uninstall tzinfo -v 2.0.2 \
&& sudo gem sources --clear-all \
&& apk del sudo build-base postgresql-dev ruby-dev \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
# && rm -rf /var/cache/apk/* /home/fluent/.gem/ruby/2.5.0/cache/*.gem
COPY docker/fluentd-conf/fluent.conf /fluentd/etc/fluent.conf
COPY docker/fluentd-conf/fluentd.sh /bin/entrypoint.sh
RUN chmod +x /bin/entrypoint.sh
CMD ["/bin/entrypoint.sh"]
I'm simply removing the tzinfo-2.0.2
that fluentd
adds, so it's forced to use the one activerecord/support introduces (tzinfo-1.2.7
). However, this feels a little bit hack-y and might be problematic for my fluentd container in the long run? (no idea, haven't run into issues yet)
Also note I've added libpq
, because unless what many people suggest, there is no libpq-dev
(anymore). It is this and/or postgresql-dev
(which links to libpq
) nowadays. I import libpq
manually as well though since I delete every build-dep, including postgresql-dev
, but libpq
is needed for the adapter in fluent-plugin-sql
to work - And I'm not deleting that one! For anyone that finds this, make sure that you never delete the libpq
build-spec!
As @vbvkel mentioned, this requires Rails to be updated, so that ActiveRecord/Support gets the latest tzinfo (which is staged in their changes). However, this also means you guys still need to update your dependencies for activerecord/support to 6.0 - Since this is where these changes will be introduced (6.0.2.2 to be exact) once that is put on their stable releases branch.
However, this feels a little bit hack-y and might be problematic for fluentd in the long run? (no idea, haven't run into issues yet)
No problem. Fluentd supports tzinfo v1.x and 2.x. See runtime dependency of fluentd: https://rubygems.org/gems/fluentd
Yes, FluentD does, but your dependency on ActiveRecord/ActiveSupport doesn't, and that's what's causing our conflict. However, we (@vbvkel primarily) discovered that this can't be fixed yet, as ActiveRecord/ActiveSupport depends on Rails, which still depends on tzinfo 1.X
It's a bit of a rabbithole of dependencies fluent-plugin-sql can't do anything about yet until they update to activerecord 6.0 and rails finally pushes 6.0.2.2 to stable with their fix of changing tzinfo-1.x to tzinfo-2.x
This shouldn't be an issue for fluentd right now though indeed - It should still be kept in mind though to fix it once rails does push that update.
Okay, just because I think we were talking about different things yesterday, @repeatedly, I'm going to run down what we have discovered to cause this issue, how I solved it, and what can be done in the future.
-
Using the latest FluentD version/Docker image (1.10.2) FluentD has a dependency on
tzinfo > 1 & < 3
and as a result adds the currently latesttzinfo-2.0.2
automatically (as of right now). -
When the latest
fluent-plugin-sql
(currently 1.1.1) is added (which also requires the addition oflibpq
andpostgresql-dev
, as explained in my earlier comments), FluentD will get aGem::ConflictError
betweentzinfo-2.0.2
andtzinfo ~> 1.0
(~>
meaning 1.X, any version within the V1 major version). -
This is because
fluent-plugin-sql
itself depends on, among others, theactiverecord = 5.2.4.2
plugin from Rails. -
This
activerecord
dependency then depends onactivesupport = 5.2.4.2
(because all the active- packages depend on the same version as their parent active- dependency). -
Finally, this
activesupport-5.2.4.2
dependency then depends ontzinfo ~> 1.1
, as a result of which it installs thetzinfo-1.2.7
gem as well. This finally results in my conflict from step 1.
I have solved the conflict by simply adding the sudo gem uninstall tzinfo -v 2.0.2
as part of my Dockerfile (as seen in my earlier comments), deleting the gem that FluentD originally installed. Although FluentD can work with the tzinfo-1.2.7
that I didn't delete, I do not like this solution since I'd prefer using the latest version required by my top dependencies (as they're the ones running the show).
Upon further examination this cannot be fixed by fluent-plugin-sql
though, and this is the only available solution right now. This is due to Rails (which is currently at V6.0.2.2) not having included this change in their latest release yet - This updates their current dependency on tzinfo ~> 1.1
to tzinfo ~> 2.0
, which would fix our problem.
Since this change is not included yet, fluent-plugin-sql
can't fix this problem yet. However, as soon as Rails does include this change into one of their releases, I emplore you to update your dependency on activerecord = 5.2.4.2
to that latest version, as it would fix this whole situation and us end-users don't have to manually delete dependencies (which is hard to debug and find, especially since this is caused by a dependency of a dependency of a dependency)
What fluent-plugin-sql
can do in the meantime however is tell users they have to do this delete for the plugin to work in the README/usage manual. Additionally, you should also add that libpq
and postgresql-dev
are required for the pg
plugin as well, since you're currently not doing that and it is required. (sidenote, the postgresql-dev
build-dep can be deleted like other build-deps, but the libpq
build-dep should never be deleted, or the pg
plugin won't work! Mention that as well, please).
For the exact technical details, aka how I laid out my files and where these commands exactly go, I'll include the working version of my fluentd.docker file (which I deploy using docker-compose):
FROM fluent/fluentd:v1.10.2-1.0
USER root
VOLUME /fluentd/log
RUN apk add --no-cache --update --virtual .build-deps sudo \
build-base \
# NOTE 1
libpq \
# NOTE 2
postgresql-dev \
ruby-dev \
&& sudo gem install fluent-plugin-route \
fluent-plugin-record-modifier \
fluent-plugin-rewrite-tag-filter \
# NOTE 3
fluent-plugin-sql \
# NOTE 4
pg \
# NOTE 5
&& sudo gem uninstall tzinfo -v 2.0.2 \
&& sudo gem sources --clear-all \
&& apk del sudo \
build-base \
# NOTE 6
postgresql-dev \
ruby-dev \
&& rm -rf /tmp/* \
/var/tmp/* \
/usr/lib/ruby/gems/*/cache/*.gem
COPY docker/fluentd-conf/fluent.conf /fluentd/etc/fluent.conf
COPY docker/fluentd-conf/fluentd.sh /bin/entrypoint.sh
RUN chmod +x /bin/entrypoint.sh
CMD ["/bin/entrypoint.sh"]
Notes:
- Add the
libpq
build dependency (Other places may tell youlibqp-dev
, but this does not exist (anymore) and have been replaced by justlibpq
!!!) required by thepg
gem; - Add the
postgresql-dev
build dependency (this does link tolibpq
as well, but we want to delete this one later, but notlibpq
, hence why we are adding both build deps separately) required by thepg
gem; - Install the
fluent-plugin-sql
gem; - Install the
pg
gem (required for thepostgresql
adapter used byfluent-plugin-sql
in myfluent.conf
); - Uninstall the
tzinfo -v 2.0.2
gem automatically installed byfluentd
(this prevents the version conflicts); - Delete all build dependencies except for the
libpq
build dependency, as this is required for thepg
gem to work.
This is all added on top of the guide given on the official fluent/fluentd docker image dockerhub page, under section "How to build your own image -> 3.1 For current images -> Alpine version". However, this should work for any way you deploy the FluentD with this fluent-plugin-sql
, as long as you install the build dependencies mentioned above, and delete the tzinfo-2.X
gem installed by FluentD, keeping only the tzinfo-1.X
gem installed by fluent-plugin-sql
's current activesupport
dependency.
I hope this helps other people who run into this same issue in the future, if the developers of fluent-plugin-sql
do not update their documentation to reflect these issues. I hope this comment can too be of use for the developers to replicate the issue and document it, or use this for their documentation, until Rails finally updates their tzinfo
dependency.
If there are any other fluent plugins that also depend on Rails, they may too experience this issue, so if you see similar issues pop up on those, feel free to mention this comment, in the hopes that it helps the people there too.
@toastyblast I couldnt find libpq and postgrsql-dev where do you get these from?
@repeatedly I couldnt get my td-agent 3.8.0 to work with fluent-plugin-sql even with the workaround that @toastyblast provided. Is there any update or workaround that you can provide for the plugin to work? Really appreciate your help!
@toastyblast got it finally working with your solution. My issue was with the umask, root umask was set to 027 and plugins were being installed with not enough permission. i had already uninstalled all the plugins for finding the issue. I reinstalled them after changing the umask to 022 (002 will work as well).