Fail if `key` field not set
sovetov opened this issue · 1 comments
sovetov commented
I get scary error in Logstash logs (newlines added for clarity, it's actually a single line!):
[2018-11-22T16:09:25,272][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"4", "backtrace"=>["org.bouncycastle.crypto.digests.GeneralDigest.update(Unknown Source)
", "org.bouncycastle.crypto.digests.GeneralDigest.finish(Unknown Source)", "org.bouncycastle.crypto.digests.SHA1Digest.doFinal(Unknown Source)", "org.bouncycastle.jcajce.provider.digest.BCMessageDigest.engineDigest(Unknown Source)", "java.security.MessageDigest.digest(MessageDigest.java:365)", "org.jruby.ext.openssl.
Digest.finish(Digest.java:204)", "org.jruby.ext.openssl.Digest$INVOKER$i$0$0$finish.call(Digest$INVOKER$i$0$0$finish.gen)", "org.jruby.RubyClass.finvoke(RubyClass.java:557)", "org.jruby.runtime.Helpers.invoke(Helpers.java:399)", "org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:354)", "org.jruby.ext.digest.R
ubyDigest$DigestInstance.digest(RubyDigest.java:320)", "org.jruby.ext.digest.RubyDigest$DigestInstance.hexdigest(RubyDigest.java:339)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$method$fingerprint_opens
sl$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:161)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jrub
y.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$block$filter
$4(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logstash/filters/fingerprint.rb:140)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)"
, "org.jruby.RubyArray.each(RubyArray.java:1734)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_filter_minus_fingerprint_minus_3_dot_2_dot_0.lib.logstash.filters.fingerprint.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-fingerprint-3.2.0/lib/logs
tash/filters/fingerprint.rb:135)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-co
re/lib/logstash/filters/base.rb:162)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "usr.share.logstash.logstash_minus_core.l
ib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:44)", "org.jruby.internal.runtim
e.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby
.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIR
BlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428
)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPa
th(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)"
, "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.jav
a:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.doYield(IRBloc
kBody.java:186)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", "org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlo
ck.call(JavaMethod.java:498)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(Interpr
eterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.
call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMet
hod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpr
et(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", "org.jruby.runtime.Block.call(Block.java:124)", "org.
jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "usr.share.logstash.logstash_minus_core.l
ib.logstash.pipeline.RUBY$method$filter_batch$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$worker_loop$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:320)", "usr.share.logstash.logstash_minus_core.lib.logsta
sh.pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:286)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call
(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"], :thread=>"#<Thread:0x77bd6cd4 sleep>"}
I found this old forum topic: https://discuss.elastic.co/t/fingerprint-plugin-crashes/69468 and tried key field. Weird, but it worked. Although documentation clearly states that "key" is optional for SHA1 — default fingerprint type.
Here is issue regarding documentation: #28. But error message needs fixing as well.
robertlabrie commented
I'm able to reproduce this on an Ubuntu 14.04 box that's been "hardened" by my infosec team running LS 6.3. No issues on a 16.04 lab box with LS 6.5.