Please update to AWS SDK v3
st33v opened this issue · 4 comments
st33v commented
We're trying to use this plugin along with AWS EKS service account roles and it does not seem to work because while 'logstash-input-s3' has been updated to AWS SDK v3 here, this change does not seem to have made it into the 'logstash-output-s3' plugin.
iamcan13 commented
Could you please help with this? We also get pain from this.
WilliamDuTrendMicro commented
yes, it is required to run logstash on EKS with S3 output
jsvd commented
you can install a new integration plugin that moves all aws-related plugins to the new sdk. However you need to uninstall all individual plugins before installing logstash-integration-aws:
/tmp/logstash-8.3.2
❯ for plugin in logstash-codec-cloudfront logstash-codec-cloudtrail logstash-input-cloudwatch logstash-input-s3 logstash-input-sqs logstash-output-cloudwatch logstash-output-sns logstash-output-sqs logstash-output-s3; do bin/logstash-plugin remove $plugin;done
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
ERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
ERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
ERROR: Operation aborted, cannot remove plugin, message: This plugin has not been previously installed
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies........................................................................................................................................................................................................................................................
Resolving dependencies...............................................................
Successfully removed logstash-input-s3
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies......................................................................................................................................................................................................................................................
Resolving dependencies...................................................................................................
Successfully removed logstash-input-sqs
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies.....................................................................................................................................................................................................................................................
Resolving dependencies...............................................................
Successfully removed logstash-output-cloudwatch
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies....................................................................................................................................................................................................................................................
Resolving dependencies.......................................................................
Successfully removed logstash-output-sns
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies..................................................................................................................................................................................................................................................
Resolving dependencies..................................................................................................
Successfully removed logstash-output-sqs
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Resolving dependencies.................................................................................................................................................................................................................................................
Resolving dependencies.............................................................
Successfully removed logstash-output-s3
/tmp/logstash-8.3.2 3m 41s
❯ bin/logstash-plugin install logstash-integration-aws
Using bundled JDK: /tmp/logstash-8.3.2/jdk.app/Contents/Home
Validating logstash-integration-aws
Resolving mixin dependencies
Installing logstash-integration-aws
Installation successful
/tmp/logstash-8.3.2
❯ grep "^ aws-sdk-core" Gemfile.lock
aws-sdk-core (3.131.4)
WilliamDuTrendMicro commented
works in EKS with IRSA
RUN sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
RUN sudo apt-get install apt-transport-https
RUN sudo echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
RUN sudo apt-get update && sudo apt-get install logstash=1:7.4.2-1 -y
#RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-amazon_es
RUN sudo apt install jq cron less -y
RUN cd /usr/share/logstash/ \
&& for plugin in logstash-codec-cloudfront logstash-codec-cloudtrail logstash-input-cloudwatch logstash-input-s3 logstash-input-sqs logstash-output-cloudwatch logstash-output-sns logstash-output-sqs logstash-output-s3; do /usr/share/logstash/bin/logstash-plugin remove $plugin;done \
&& /usr/share/logstash/bin/logstash-plugin install logstash-integration-aws
#https://github.com/logstash-plugins/logstash-output-s3/issues/234#issuecomment-1197902180
#as the S3 output does not support IRSA due to aws sdk low verison , enforce unintall and install logstash-integration-aws with AWS SDK latest version.