logstash-plugins/logstash-output-google_bigquery

Problems loading a plugin - Google BigQuery

Opened this issue · 0 comments

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

  • Version: 7.3.1
  • Operating System: Windows 10
  • Config File (if you have sensitive info, please remove it):
    input {
    file {
    path => "full-path/logstash-tutorial-dataset"
    sincedb_path => "nul"
    start_position => "beginning"
    }
    }

filter {
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
#'message' => '%{IP:ip} - - [%{GREEDYDATA:date}] %{GREEDYDATA:request}'
}
}
geoip {
source => "clientip"
}
}

output {
google_bigquery {
project_id => "akbm-infrastructur"
dataset => "test_logstash"
csv_schema => "clientip:STRING"
json_key_file => "full-path-to-key-file.json"
error_directory => "/tmp/bigquery-errors"
}
}

  • Sample Data:
    83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

  • Steps to Reproduce:
    Running the config file with `bin/logstash -r -f “” with the purpose of sending local data to BigQuery, using Logstash, produces a log file consisting of mainly

[2019-08-29T09:47:18,692][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"output", :name=>"google_bigquery", :path=>"logstash/outputs/google_bigquery", :error_message=>"\n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- com/google/cloud/google-cloud-bigquery/1.24.1/google-cloud-bigquery-1.24.1 (LoadError)", :error_class=>RuntimeError, :error_backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:356:in `do_require'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:265:in `block in require_jar'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:307:in `require_jar_with_block'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:264:in `require_jar'", 

And

[2019-08-29T09:47:18,697][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any output plugin named 'google_bigquery'. Are you sure this is correct? Trying to load the google_bigquery output plugin resulted in this error: Problems loading the requested plugin named google_bigquery of type output. Error: RuntimeError \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- com/google/cloud/google-cloud-bigquery/1.24.1/google-cloud-bigquery-1.24.1 (LoadError)", :backtrace=>

Please note that when running bin/logstash-plugin list it outputs logstash-output-google_bigquery