Error while Running the Logstash Plugin

Question

Error while Running the Logstash Plugin

imewish opened this issue 9 years ago · 25 comments

bin/logstash -f logstash-dynamodb.conf
The error reported is:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

no such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)

Helps would be appreciated

Answer 1 · 2016-03-16T20:59:44.000Z

Same issue:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

no such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)

Answer 2 · 2016-03-16T21:25:46.000Z

Hi,

You can check how I install the plugins in a docker container as a reference to install them in your own environment.

Logstash 1.x: https://github.com/mantika/dockerfiles/blob/1.x/logstash/dynamo-streams/Dockerfile

Logstash 2.x: https://github.com/mantika/dockerfiles/blob/master/logstash/dynamo-streams/Dockerfile

Hope it helps.

Answer 3 · 2016-03-16T21:35:36.000Z

As far as I can tell, I just need to run:

RUN gem install logstash-input-dynamodb:'< 2' logstash-filter-dynamodb:'< 2'
RUN plugin install logstash-input-dynamodb logstash-filter-dynamodb

I can see logstash-input-dynamodb in my /opt/logstash/bin/plugin list
It says installation successful, but when I run my test conf, same error.

Answer 4 · 2016-03-16T21:37:19.000Z

@teebu keep in mind the stuff I do with the .m2 repository. Take into account that logstash usually runs with the user logstash user and if you install the gems / plugins using a different user you'll probably get the error you're seeing.

Answer 5 · 2016-03-16T22:09:41.000Z

So after cloning the GIT, compiling, installing and uninstall the GEM a few dozen times, then doing a plugin/install I finally managed to get it to work. At one point, it only ran with SUDO, (it could have been when I SUDO gem install, instead of regular gem install) but it seems if i run it from inside /opt/logstash folder, it is working.

Now, it seems I can run it without needing to be in the folder. With no errors.

I think the step where I had to compile might have installed those missing JAR files.

Answer 6 · 2016-03-17T12:08:08.000Z

@marcosnils This is Awesome ..thanks for updating the logstash gem with this dynamodb plugin. It woks fine. Installing this locally and works properly was a nightmare, it was needed to follow wired ways to make it work. Thanks again

Answer 7 · 2016-03-19T07:57:05.000Z

I am still having a problem. Even though I was able to run the plugin, i was not able to run it with the logstash service.

Getting error:
:timestamp=>"2016-03-19T07:35:37.371000+0000", :message=>"The error reported is: \n \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command\n\nno such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)"}

I tried to follow your docker example, but I dont understand what the .m2 sym link is doing. Here is my folder structure:

I am on an EC2 machine.

home: /home/ec2-user
.m2 folder is inside: /home/ec2-user/.m2
the com/aws files are listed in: /home/ec2-user/.m2/repository

Doing ln -s /var/lib/logstash/.m2 /root/.m2 doesn't work for me. I tried ln -s /var/lib/logstash/.m2 ~/.m2 looks like it creates a sym link in .m2 folder to /var/lib/logstash/.m2, so thats probably wrong.

I really need a step by step guide, because I just spend 6 hours trying to get this to work.

Right now trying to run a config, I get the error no such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError) unless I run it with sudo.

Answer 8 · 2016-03-19T12:42:49.000Z

Install jRuby

Install Git

sudo yum install git

Install Maven

Install Logstash

sudo rpm --import https https://packages.elastic.co/GPG-KEY-elasticsearch
:// https://packages.elastic.co/GPG-KEY-elasticsearchpackages.elastic.co
https://packages.elastic.co/GPG-KEY-elasticsearch/
https://packages.elastic.co/GPG-KEY-elasticsearchGPG-KEY-elasticsearch
https://packages.elastic.co/GPG-KEY-elasticsearchsudo nano
logstash.repo [logstash-2.2]

name=Logstash repository for 2.2.x packages

baseurl=http:// http://packages.elastic.co/logstash/2.2/centos
packages.elastic.co http://packages.elastic.co/logstash/2.2/centos/
http://packages.elastic.co/logstash/2.2/centoslogstash
http://packages.elastic.co/logstash/2.2/centos/2.2/
http://packages.elastic.co/logstash/2.2/centoscentos
http://packages.elastic.co/logstash/2.2/centos

gpgcheck=1

gpgkey=http:// http://packages.elastic.co/GPG-KEY-elasticsearch
packages.elastic.co http://packages.elastic.co/GPG-KEY-elasticsearch/
http://packages.elastic.co/GPG-KEY-elasticsearchGPG-KEY-elasticsearch
http://packages.elastic.co/GPG-KEY-elasticsearch

enabled=1

sudo yum install logstashexport PATH=/opt/logstash/bin:$PATH$

Install Logstash-input-dynamodb

Clonesudo git clone https
https://github.com/genesi/logstash-input-dynamodb.git://
https://github.com/genesi/logstash-input-dynamodb.gitgithub.com
https://github.com/genesi/logstash-input-dynamodb.git/
https://github.com/genesi/logstash-input-dynamodb.gitgenesi
https://github.com/genesi/logstash-input-dynamodb.git/
https://github.com/genesi/logstash-input-dynamodb.git
logstash-input-dynamodb.git
https://github.com/genesi/logstash-input-dynamodb.gitsudo chmod a+wrx
/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/sudo chmod a+wrx
/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/sudo chmod a+wrx
/etc/jruby/jruby-9.0.5.0/bin/sudo chmod a+wrx
/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/specifications/jruby -S
bundle installsudo chmod a+wrx /etc/logstash-input-dynamodbjruby -S gem
install logstash-input-dynamodb-1.0.1-java.gemsudo chmod a+wrx
/opt/logstash/Gemfilesudo chmod a+wrx
/etc/logstash-input-dynamodb/lib/logstash-input-dynamodb_jars.rbsudo chmod
a+wrx /etc/logstash-input-dynamodb/libBundle installplugin install
logstash-input-dynamodb-1.0.1-java.gemDeal with any dependenciesgem
"logstash-input-dynamodb", "1.0.1", :path =>
"vendor/local_gems/863a3d26/logstash-input-dynamodb-1.0.1-java"gem
"activesupport-json_encoder","1.1.0",:path =>
"/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-json_encoder"gem
"activesupport" , "4.1.14.1" , :path =>
"/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-4.1.14.1"

Install logstash-filter-dynamodb

Fix Logstash to run as service

Copy logstash-input-dynamodb to a new locationsudo cp -r
logstash-input-dynamodb-1.0.1-java/
/opt/logstash/vendor/bundle/jruby/1.9/gems/Update /opt/logstash/Gemfilegem
"logstash-input-dynamodb", "1.0.1", :path =>
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-java"sudo
chmod a+wrx
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-javasudo
chmod a+wrx
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-java/lib/logstash-input-dynamodb_jars.rbsudo
chmod a+wrx
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-java/lib/cd
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-javabundle
update

/opt/logstash/Gemfile bottom of file

gem "logstash-input-http_poller"

gem "logstash-input-dynamodb", "1.0.1", :path =>
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-java"

gem "activesupport-json_encoder", "1.1.0", :path =>
"/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-json_encoder"

gem "activesupport", "4.1.14.1", :path =>
"/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-4.1.14.1"

gem "logstash-filter-dynamodb", "0.0.4", :path =>
"vendor/local_gems/3182dbdf/logstash-filter-dynamodb-0.0.4"

sent from mobile
On Mar 19, 2016 4:57 AM, "teebu" notifications@github.com wrote:

I am still having a problem. Even though I was able to run the plugin, i
was not able to run it with the logstash service.

Getting error:
:timestamp=>"2016-03-19T07:35:37.371000+0000", :message=>"The error
reported is: \n \n\n\tyou might need to reinstall the gem which depends on
the missing jar or in case there is Jars.lock then resolve the jars with
lock_jars command\n\nno such file to load --
com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27
(LoadError)"}

I tried to follow your docker example, but I dont understand what the .m2
sym link is doing. Here is my folder structure:

I am on an EC2 machine.

home: /home/ec2-user
.m2 folder is inside: /home/ec2-user/.m2
the com/aws files are listed in: /home/ec2-user/.m2/repository

Doing ln -s /var/lib/logstash/.m2 /root/.m2 doesn't work for me. I tried ln
-s /var/lib/logstash/.m2 ~/.m2 looks like it creates a sym link in .m2
folder to /var/lib/logstash/.m2, so thats probably wrong.

I really need a step by step guide, because I just spend 6 hours trying to
get this to work.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#11 (comment)

Answer 9 · 2016-03-19T17:03:12.000Z

thanks, this was very hard to read with the lack of formatting. ill give it a shot. also, im using LS 1.5

I tried to follow your steps, but it feels like there are either steps missing, or some mismatch is going on. I have maven,jruby installed, with LS 1.5.

Answer 10 · 2016-03-19T19:45:57.000Z

I managed to translate whatever you wrote to this:

Clone 
sudo git clone https://github.com/genesi/logstash-input-dynamodb.git
sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/
sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/
sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/bin/
sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/specifications/
jruby -S bundle install
sudo chmod a+wrx /etc/logstash-input-dynamodb

did you miss a step here? to build the gem?

jruby -S gem install logstash-input-dynamodb-1.0.1-java.gem
sudo chmod a+wrx /opt/logstash/Gemfile
sudo chmod a+wrx /etc/logstash-input-dynamodb/lib/logstash-input-dynamodb_jars.rb
sudo chmod a+wrx /etc/logstash-input-dynamodb/lib
Bundle install
plugin install logstash-input-dynamodb-1.0.1-java.gem

Around this point, there is no "Bundle" so I fix for that, to make it "bundle"

plugin install doesn't work without sudo. I fix for that.

Doing: sudo bin/plugin install /etc/logstash-input-dynamodb/logstash-input-dynamodb-1.0.0-java.gem

Getting Error:
An error occurred while installing logstash-input-dynamodb (1.0.0), and Bundler cannot continue.
Make sure that gem install logstash-input-dynamodb -v '1.0.0' succeeds before bundling.

Trying: sudo gem install logstash-input-dynamodb -v '1.0.0' inside /opt/logstash folder.
-- Successfully installed logstash-input-dynamodb-1.0.0-java 1 gem installed

Still getting same error when doing plugin install: An error occurred while installing logstash-input-dynamodb (1.0.0), and Bundler cannot continue. Make sure that gem install logstash-input-dynamodb -v '1.0.0' succeeds before bundling.

Deal with any dependencies 
gem "logstash-input-dynamodb", "1.0.1", :path => "vendor/local_gems/863a3d26/logstash-input-dynamodb-1.0.1-java"
gem "activesupport-json_encoder","1.1.0",:path => "/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-json_encoder"
gem "activesupport" , "4.1.14.1" , :path => "/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-4.1.14.1"


Fix Logstash to run as service

Copy logstash-input-dynamodb to a new location
sudo cp -r logstash-input-dynamodb-1.0.1-java/ /opt/logstash/vendor/bundle/jruby/1.9/gems/
Update /opt/logstash/Gemfile
gem "logstash-input-dynamodb", "1.0.0", :path => "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb"
sudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb
sudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb/lib/logstash-input-dynamodb_jars.rb
sudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb/lib/
cd /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb
bundle update

One thing I noticed was changing the owner of 'opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb' to logstash seems to be doing something

One thing to note:
Using the rubygems.org version, '1.0.3' by doing just sudo bin/plugin install logstash-input-dynamodb, throws error: {:timestamp=>"2016-03-19T20:37:11.323000+0000", :message=>"The error reported is: \n \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command\n\nno such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)"} when starting logstash service.

Answer 11 · 2016-03-19T19:50:09.000Z

did you miss a step here? to build the gem?

Probably.

I didn't write this instructions, these were written by @bretd25. Maybe he can help to indicate how he solved the problem.

Keep in mind that this instructions are for logstash 2.x.. You'll have to adapt them to your version.

Marcos.

Answer 12 · 2016-03-22T03:19:08.000Z

I am getting this error

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

no such file to load -- joda-time/joda-time/2.8.2/joda-time-2.8.2 (LoadError)

No clue on how to solve.

Answer 13 · 2016-03-22T03:23:14.000Z

Which logstash version are you using and how are you installing the gem?

Answer 14 · 2016-03-22T03:26:48.000Z

Its logstash 2.2.2, and installed the gem through gem install
gem install logstash-input-dynamodb

Version for logstash-input-dynamodb is 2.0.3, got it from
https://rubygems.org/gems/logstash-input-dynamodb/versions/2.0.3-java

Earlier tried the local build option, but ended up at exact same problem.

Answer 15 · 2016-03-22T03:33:01.000Z

Also I see all the required jars are right there in my ~/.m2/repository folder, either directly or in some folders. Why is logstash still not able to find them?

Answer 16 · 2016-03-22T03:35:19.000Z

because logstash usually uses the logstash user when it starts as logstash agent. You can try to copy or symlink your .m2 folder to where logstash searches for dependencies (I believe is something like /opt/logstash)

Answer 17 · 2016-03-22T03:36:18.000Z

you probably did gem install with your user instead of logstash's. That's why deps got installed in your local .m2 folder.

Answer 18 · 2016-03-22T03:58:51.000Z

Symlinking does not seem to help. I created a symlink named .m2 in /opt/logstash where my logstash resides. Exact same problem :(
Even copying of entire .m2 folder, and changing ownership to logstash:logstash of the folder recursively does not help. Same error.

Answer 19 · 2016-03-22T04:06:30.000Z

@paragshar try creating this dir and linking .m2 there:

mkdir -p /var/lib/logstash/.m2
ln -s /var/lib/logstash/.m2 /your/user/.m2

Hope it helps.

Answer 20 · 2016-03-22T04:27:20.000Z

Not too sure what you meant by above. Should the link part not be like

ln -s  /your/user/.m2 /var/lib/logstash/.m2

and /your/user substituted by /home/ubuntu
I hope we are trying to create a softlink .m2 which points to maven repo created on /home/ubuntu

Answer 21 · 2016-03-22T04:50:35.000Z

I figured it out finally. There was no jar file for joda-time/joda-time/2.8.2/joda-time-2.8.2 in my
.m2/repository/joda-time/joda-time/2.8.2. I just had a pom file there. I downloaded a jar file (joda-time-2.8.2.jar) directly and placed in within the folder /var/lib/logstash/.m2/repository/joda-time/joda-time/2.8.2.

And it works like a breeze after that.

Many thanks. Your assistance really helped me figure out the issue.

But I still need to see why was the jar file not there in the first place :)

Answer 22 · 2016-03-22T05:30:14.000Z

Can you help me get rid of this error, whenever I try to install any plugin.

It installs the plugin I was trying to install successfully, but instead of finishing with a success, it does this with 1/10 retries.
An error occurred while installing logstash-input-dynamodb (1.0.0), and Bundler cannot continue. Make sure that gem install logstash-input-dynamodb -v '1.0.0' succeeds before bundling.

Answer 23 · 2016-03-22T08:22:57.000Z

The best idea is to run

gem install logstash-input-dynamodb

This will download and install the plugin from the gem repo.
Let me know if you encounter any problem with this.

Answer 24 · 2016-03-22T08:25:06.000Z

https://rubygems.org/gems/logstash-input-dynamodb

Answer 25 · 2016-03-22T17:19:50.000Z

i tried installing it from the gem repo before, it installs 1.0.3. but then it doesn't run in the LS service.

Using the rubygems.org version, '1.0.3' by doing just sudo bin/plugin install logstash-input-dynamodb, throws error:

{:timestamp=>"2016-03-19T20:37:11.323000+0000", :message=>"The error reported is: \n \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)"}.

when starting logstash service