tzinfo-2.0.2 conflicts with tzinfo (~> 1.1)

Question

tzinfo-2.0.2 conflicts with tzinfo (~> 1.1)

toastyblast opened this issue 5 years ago · 8 comments

I'm having conflict issues with adding fluent-plugin-sql & pg to my project so that FluentD can start outputting to my PSQL Docker container. Prior to adding these I've had no issues, but now after getting pg to work (which was a riot of its own) the cointainer after building and starting immediately stops with a conflict error from something called activesupport-5.2.4.2 due to something called tzinfo-2.0.2 conflicting with another tzinfo of 1.1. I'll explain what I've found and tried further down in this issue.

I'm running FluentD, PostgreSQL and my applications all as Docker containers, using their respective official images (fluent/fluentd:v1.10-1 and postgres:9.6), started through a docker-compose.yml file.

fleuntd.docker:

FROM fluent/fluentd:v1.10-1

ENV PSQL_HOST "psql-server"
ENV PSQL_PORT "5432"
ENV PSQL_DB "fluentd-logs-db"
ENV PSQL_USERNAME "dadmin1"
ENV PSQL_PASSWORD "letmeinplease"

USER root

VOLUME /fluentd/log

# libpq-dev

RUN apk add --no-cache --update --virtual .build-deps postgresql-dev \
        sudo build-base ruby-dev \
 && sudo gem install fluent-plugin-route fluent-plugin-record-modifier fluent-plugin-rewrite-tag-filter \
                     fluent-plugin-sql pg \
 && sudo gem sources --clear-all \
 && apk del .build-deps postgresql-dev \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
# && rm -rf /var/cache/apk/* /home/fluent/.gem/ruby/2.5.0/cache/*.gem

COPY docker/fluentd-conf/fluent.conf /fluentd/etc/fluent.conf
COPY docker/fluentd-conf/fluentd.sh /bin/entrypoint.sh

RUN chmod +x /bin/entrypoint.sh
CMD ["/bin/entrypoint.sh"]

Side note: These ENV credentials are not real, this personal project is used locally only and is not openly hosted anywhere.

I have already resolved issues of not being able to find the pg gem, by adding the postgresql-dev instead of libpq-dev (which doesn't seem to work wherever I place it in the RUN...). I found that idea here, #65-454016821
The image finally builds now, but I get a new error within FluentD itself:

...
Creating fluentd-host   ... done
Attaching to fluentd-host
fluentd-host   | 2020-04-15 11:22:46 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd-host   | 2020-04-15 11:22:46 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
fluentd-host   | 2020-04-15 11:22:46 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
fluentd-host   | 2020-04-15 11:22:46 +0000 [info]: gem 'fluent-plugin-route' version '1.0.0'
fluentd-host   | 2020-04-15 11:22:46 +0000 [info]: gem 'fluent-plugin-sql' version '1.1.1'
fluentd-host   | 2020-04-15 11:22:46 +0000 [info]: gem 'fluentd' version '1.10.1'
fluentd-host   | /usr/lib/ruby/2.5.0/rubygems/specification.rb:2325:in `raise_if_conflicts': Unable to activate activesupport-5.2.4.2, because tzinfo-2.0.2 conflicts with tzinfo (~> 1.1) (Gem::ConflictError)
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1438:in `activate'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1472:in `block in activate_dependencies'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1458:in `each'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1458:in `activate_dependencies'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/specification.rb:1440:in `activate'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems.rb:224:in `rescue in try_activate'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems.rb:217:in `try_activate'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:128:in `rescue in require'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:39:in `require'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:147:in `initialize'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/plugin.rb:158:in `new'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/plugin.rb:158:in `new_impl'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/plugin.rb:109:in `new_output'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/agent.rb:130:in `add_match'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/agent.rb:74:in `block in configure'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/agent.rb:64:in `each'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/agent.rb:64:in `configure'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/label.rb:31:in `configure'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/root_agent.rb:143:in `block in configure'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/root_agent.rb:143:in `each'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/root_agent.rb:143:in `configure'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/engine.rb:105:in `configure'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/engine.rb:80:in `run_configure'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/supervisor.rb:547:in `run_supervisor'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/lib/fluent/command/fluentd.rb:330:in `<top (required)>'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
fluentd-host   |        from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
fluentd-host   |        from /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.10.1/bin/fluentd:8:in `<top (required)>'
fluentd-host   |        from /usr/bin/fluentd:23:in `load'
fluentd-host   |        from /usr/bin/fluentd:23:in `<main>'
fluentd-host exited with code 1

Additionally (before the error above, but who knows it might be related in some way), I also get an error from i-18-n during the gem install procedure:

...
Successfully installed fluent-plugin-route-1.0.0
Successfully installed fluent-plugin-record-modifier-2.1.0
Successfully installed fluent-config-regexp-type-1.0.0
Successfully installed fluent-plugin-rewrite-tag-filter-2.3.0
Successfully installed thread_safe-0.3.6
Successfully installed tzinfo-1.2.7

HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.
But that may break your application.

If you are upgrading your Rails application from an older version of Rails:

Please check your Rails app for 'config.i18n.fallbacks = true'.
If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be
'config.i18n.fallbacks = [I18n.default_locale]'.
If not, fallbacks will be broken in your app by I18n 1.1.x.

If you are starting a NEW Rails application, you can ignore this notice.

For more info see:
https://github.com/svenfuchs/i18n/releases/tag/v1.1.0

Successfully installed i18n-1.8.2
Successfully installed minitest-5.14.0
Successfully installed activesupport-5.2.4.2
Successfully installed activemodel-5.2.4.2
Successfully installed arel-9.0.0
Successfully installed activerecord-5.2.4.2
Successfully installed activerecord-import-0.28.2
Successfully installed fluent-plugin-sql-1.1.1
Building native extensions. This could take a while...
Successfully installed pg-0.21.0
15 gems installed
...

I have seen mentions of this issue in places like fluent/fluentd#2726 but the fix they seem to have implemented there seems to not have it fixed for my case.

Like I mentioned, I have tried using libpq-dev (alone or in addition to postgresql-dev) and setting pg:0.21.0 but these would get issues earlier on about not being able to find config files.

I have put the full output log in a file attached to this issue for review.

output.txt

Answer 1 · 2020-04-15T14:51:11.000Z

tzinfo is a dependency of a dependency of activerecord. A version which supports tzinfo 2.0 or higher has not been released by Rails yet.

Answer 2 · 2020-04-15T14:56:33.000Z

I found a temporary solution by using the following file I made:

FROM fluent/fluentd:v1.10.2-1.0

ENV PSQL_HOST "psql-server"
ENV PSQL_PORT "5432"
ENV PSQL_DB "fluentd-logs-db"
ENV PSQL_USERNAME "dadmin1"
ENV PSQL_PASSWORD "letmeinplease"

USER root

VOLUME /fluentd/log

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base libpq postgresql-dev ruby-dev \
 && sudo gem install fluent-plugin-route fluent-plugin-record-modifier fluent-plugin-rewrite-tag-filter \
                     fluent-plugin-sql pg \
 && sudo gem uninstall tzinfo -v 2.0.2 \
 && sudo gem sources --clear-all \
 && apk del sudo build-base postgresql-dev ruby-dev \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
# && rm -rf /var/cache/apk/* /home/fluent/.gem/ruby/2.5.0/cache/*.gem

COPY docker/fluentd-conf/fluent.conf /fluentd/etc/fluent.conf
COPY docker/fluentd-conf/fluentd.sh /bin/entrypoint.sh

RUN chmod +x /bin/entrypoint.sh
CMD ["/bin/entrypoint.sh"]

I'm simply removing the tzinfo-2.0.2 that fluentd adds, so it's forced to use the one activerecord/support introduces (tzinfo-1.2.7). However, this feels a little bit hack-y and might be problematic for my fluentd container in the long run? (no idea, haven't run into issues yet)

Also note I've added libpq, because unless what many people suggest, there is no libpq-dev (anymore). It is this and/or postgresql-dev (which links to libpq) nowadays. I import libpq manually as well though since I delete every build-dep, including postgresql-dev, but libpq is needed for the adapter in fluent-plugin-sql to work - And I'm not deleting that one! For anyone that finds this, make sure that you never delete the libpq build-spec!

As @vbvkel mentioned, this requires Rails to be updated, so that ActiveRecord/Support gets the latest tzinfo (which is staged in their changes). However, this also means you guys still need to update your dependencies for activerecord/support to 6.0 - Since this is where these changes will be introduced (6.0.2.2 to be exact) once that is put on their stable releases branch.

Answer 3 · 2020-04-15T15:15:57.000Z

However, this feels a little bit hack-y and might be problematic for fluentd in the long run? (no idea, haven't run into issues yet)

No problem. Fluentd supports tzinfo v1.x and 2.x. See runtime dependency of fluentd: https://rubygems.org/gems/fluentd

Answer 4 · 2020-04-15T15:18:57.000Z

Yes, FluentD does, but your dependency on ActiveRecord/ActiveSupport doesn't, and that's what's causing our conflict. However, we (@vbvkel primarily) discovered that this can't be fixed yet, as ActiveRecord/ActiveSupport depends on Rails, which still depends on tzinfo 1.X

It's a bit of a rabbithole of dependencies fluent-plugin-sql can't do anything about yet until they update to activerecord 6.0 and rails finally pushes 6.0.2.2 to stable with their fix of changing tzinfo-1.x to tzinfo-2.x

This shouldn't be an issue for fluentd right now though indeed - It should still be kept in mind though to fix it once rails does push that update.

Answer 5 · 2020-04-16T10:10:37.000Z

Okay, just because I think we were talking about different things yesterday, @repeatedly, I'm going to run down what we have discovered to cause this issue, how I solved it, and what can be done in the future.

Using the latest FluentD version/Docker image (1.10.2) FluentD has a dependency on tzinfo > 1 & < 3 and as a result adds the currently latest tzinfo-2.0.2 automatically (as of right now).
When the latest fluent-plugin-sql (currently 1.1.1) is added (which also requires the addition of libpq and postgresql-dev, as explained in my earlier comments), FluentD will get a Gem::ConflictError between tzinfo-2.0.2 and tzinfo ~> 1.0 (~> meaning 1.X, any version within the V1 major version).
This is because fluent-plugin-sql itself depends on, among others, the activerecord = 5.2.4.2 plugin from Rails.
This activerecord dependency then depends on activesupport = 5.2.4.2 (because all the active- packages depend on the same version as their parent active- dependency).
Finally, this activesupport-5.2.4.2 dependency then depends on tzinfo ~> 1.1, as a result of which it installs the tzinfo-1.2.7 gem as well. This finally results in my conflict from step 1.

I have solved the conflict by simply adding the sudo gem uninstall tzinfo -v 2.0.2 as part of my Dockerfile (as seen in my earlier comments), deleting the gem that FluentD originally installed. Although FluentD can work with the tzinfo-1.2.7 that I didn't delete, I do not like this solution since I'd prefer using the latest version required by my top dependencies (as they're the ones running the show).

Upon further examination this cannot be fixed by fluent-plugin-sql though, and this is the only available solution right now. This is due to Rails (which is currently at V6.0.2.2) not having included this change in their latest release yet - This updates their current dependency on tzinfo ~> 1.1 to tzinfo ~> 2.0, which would fix our problem.

Since this change is not included yet, fluent-plugin-sql can't fix this problem yet. However, as soon as Rails does include this change into one of their releases, I emplore you to update your dependency on activerecord = 5.2.4.2 to that latest version, as it would fix this whole situation and us end-users don't have to manually delete dependencies (which is hard to debug and find, especially since this is caused by a dependency of a dependency of a dependency)

What fluent-plugin-sql can do in the meantime however is tell users they have to do this delete for the plugin to work in the README/usage manual. Additionally, you should also add that libpq and postgresql-dev are required for the pg plugin as well, since you're currently not doing that and it is required. (sidenote, the postgresql-dev build-dep can be deleted like other build-deps, but the libpq build-dep should never be deleted, or the pg plugin won't work! Mention that as well, please).

For the exact technical details, aka how I laid out my files and where these commands exactly go, I'll include the working version of my fluentd.docker file (which I deploy using docker-compose):


FROM fluent/fluentd:v1.10.2-1.0

USER root

VOLUME /fluentd/log

RUN apk add --no-cache --update --virtual .build-deps sudo \
                                                      build-base \
                                                      # NOTE 1
                                                      libpq \
                                                      # NOTE 2
                                                      postgresql-dev \
                                                      ruby-dev \
 && sudo gem install fluent-plugin-route \
                     fluent-plugin-record-modifier \
                     fluent-plugin-rewrite-tag-filter \
                     # NOTE 3
                     fluent-plugin-sql \
                     # NOTE 4
                     pg \
 # NOTE 5
 && sudo gem uninstall tzinfo -v 2.0.2 \
 && sudo gem sources --clear-all \
 && apk del sudo \
            build-base \
            # NOTE 6
            postgresql-dev \
            ruby-dev \
 && rm -rf /tmp/* \
           /var/tmp/* \
           /usr/lib/ruby/gems/*/cache/*.gem

COPY docker/fluentd-conf/fluent.conf /fluentd/etc/fluent.conf
COPY docker/fluentd-conf/fluentd.sh /bin/entrypoint.sh

RUN chmod +x /bin/entrypoint.sh
CMD ["/bin/entrypoint.sh"]

Notes:

Add the libpq build dependency (Other places may tell you libqp-dev, but this does not exist (anymore) and have been replaced by just libpq!!!) required by the pg gem;
Add the postgresql-dev build dependency (this does link to libpq as well, but we want to delete this one later, but not libpq, hence why we are adding both build deps separately) required by the pg gem;
Install the fluent-plugin-sql gem;
Install the pg gem (required for the postgresql adapter used by fluent-plugin-sql in my fluent.conf);
Uninstall the tzinfo -v 2.0.2 gem automatically installed by fluentd (this prevents the version conflicts);
Delete all build dependencies except for the libpq build dependency, as this is required for the pg gem to work.

This is all added on top of the guide given on the official fluent/fluentd docker image dockerhub page, under section "How to build your own image -> 3.1 For current images -> Alpine version". However, this should work for any way you deploy the FluentD with this fluent-plugin-sql, as long as you install the build dependencies mentioned above, and delete the tzinfo-2.X gem installed by FluentD, keeping only the tzinfo-1.X gem installed by fluent-plugin-sql's current activesupport dependency.

I hope this helps other people who run into this same issue in the future, if the developers of fluent-plugin-sql do not update their documentation to reflect these issues. I hope this comment can too be of use for the developers to replicate the issue and document it, or use this for their documentation, until Rails finally updates their tzinfo dependency.

If there are any other fluent plugins that also depend on Rails, they may too experience this issue, so if you see similar issues pop up on those, feel free to mention this comment, in the hopes that it helps the people there too.

Answer 6 · 2020-08-21T19:35:30.000Z

@toastyblast I couldnt find libpq and postgrsql-dev where do you get these from?

Answer 7 · 2020-08-22T00:07:30.000Z

@repeatedly I couldnt get my td-agent 3.8.0 to work with fluent-plugin-sql even with the workaround that @toastyblast provided. Is there any update or workaround that you can provide for the plugin to work? Really appreciate your help!

Answer 8 · 2020-08-25T23:35:09.000Z

@toastyblast got it finally working with your solution. My issue was with the umask, root umask was set to 027 and plugins were being installed with not enough permission. i had already uninstalled all the plugins for finding the issue. I reinstalled them after changing the umask to 022 (002 will work as well).