Logstash Docker Image with BigQuery plugin gives “Certificate verify failed”
Closed this issue · 7 comments
Hi,
I am trying to use Google Big Query Logstash output in a logstash docker container. After starting the job I get the following errror:
logstash_1 | [2018-03-06T19:57:58,505][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::OutputDelegator:0x5893c3c1 @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0x7a8f4807 @metric=#<LogStash::Instrument::Metric:0x772a9c6d @collector=#<LogStash::Instrument::Collector:0x104a53c0 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x568107c3 @store=#<Concurrent::Map:0x00000000000fb4 entries=3 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x37b21c13>, @fast_lookup=#<Concurrent::Map:0x00000000000fb8 entries=55 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :\"14708533f248ee18126fec935b0d7eb0e94dc0d0a50de18b101efc7435f991df\"]>, @metric=#<LogStash::Instrument::NamespacedMetric:0x4a2d78e5 @metric=#<LogStash::Instrument::Metric:0x772a9c6d @collector=#<LogStash::Instrument::Collector:0x104a53c0 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x568107c3 @store=#<Concurrent::Map:0x00000000000fb4 entries=3 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x37b21c13>, @fast_lookup=#<Concurrent::Map:0x00000000000fb8 entries=55 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @out_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @strategy=#<LogStash::OutputDelegatorStrategies::Single:0x3c841c48 @mutex=#<Mutex:0x48f26b93>, @output=<LogStash::Outputs::GoogleBigQuery project_id=>\"arched-photon-194421\", dataset=>\"msgs-2018.02.10\", json_schema=>{\"fields\"=>[{\"name\"=>\"_id\", \"type\"=>\"STRING\"}, {\"name\"=>\"_index\", \"type\"=>\"STRING\"}, {\"name\"=>\"msg.message_text\", \"type\"=>\"STRING\"}]}, key_path=>\"/usr/share/logstash/keys/my-project-fcee9ca06100.p12\", key_password=>\"notasecret\", service_account=>\"elastic-search-data@arched-photon-194421.iam.gserviceaccount.com\", date_pattern=>\"%Y-%m-%dT%H:00\", flush_interval_secs=>2, uploader_interval_secs=>60, deleter_interval_secs=>60, id=>\"14708533f248ee18126fec935b0d7eb0e94dc0d0a50de18b101efc7435f991df\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_e94431cd-7ba4-416e-b363-a10348fc9b33\", enable_metric=>true, charset=>\"UTF-8\">, workers=>1, table_prefix=>\"logstash\", table_separator=>\"_\", ignore_unknown_values=>false, temp_file_prefix=>\"logstash_bq\">>, @in_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @id=\"14708533f248ee18126fec935b0d7eb0e94dc0d0a50de18b101efc7435f991df\", @time_metric=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration_in_millis value:0, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x6939d838 @metric=#<LogStash::Instrument::Metric:0x772a9c6d @collector=#<LogStash::Instrument::Collector:0x104a53c0 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x568107c3 @store=#<Concurrent::Map:0x00000000000fb4 entries=3 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x37b21c13>, @fast_lookup=#<Concurrent::Map:0x00000000000fb8 entries=55 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :\"14708533f248ee18126fec935b0d7eb0e94dc0d0a50de18b101efc7435f991df\", :events]>, @output_class=LogStash::Outputs::GoogleBigQuery>", :error=>"certificate verify failed", :thread=>"#<Thread:0x60ea0d79 run>"}
I think this is a bug in the BigQuery Plugin, but I am not sure
I tried to update the ssl certificates in the docker image with no luck.
The docker logstash version is 6.2.2 and the bigquery plugin was installed with logstash-plugin install logstash-output-google_bigquery
It looks like this might be the same issue that's causing logstash-output-google_cloud_storage #20
@ivancruzbht I think the fix might be to use a modern Java client library rather than the ruby one. If I sent you a gemfile for the cloud storage fix would you be willing to test it out? If it works I can do the same thing with this plugin to get it going again.
As a temporary solution, dblommesteijn's (much celebrated) answer might work: googleapis/google-api-ruby-client#235
I tried to do that without luck. I downloaded the cacert.pem, saved in the folder specified and added the env variable to the container but I got the cert verification error again. Maybe I did it wrong but I didnt have time to check. I ended up disabling ssl auth by forking the repo and commenting the lines that enforcec ssl verification and rebuilding the plugin. Though I got rid of the ssl verification error, I could not manage to insert any records from elasticsearch in bigquery, so I am not quite sure if this breaks logstash. See my stackoverflow issue: https://stackoverflow.com/questions/49164102/bigquery-dataset-not-shown-after-logstash-job?noredirect=1#comment86127020_49164102
@josephlewis42 can you send us the gemfile to check it out?
@edwinallenz no problem! I've posted a gemfile here: https://github.com/josephlewis42/personal_codebase/releases/download/logstash-release/logstash-output-google_bigquery-4.0.0-java.gem
The plugin does have some breaking changes so I bumped the version number up.
New configuration options:
- json_key_file - use a downloaded JSON key rather than the P12
Optional:
- error_directory - a directory to store problem uploads to
- flush_interval_secs - max amount of time between streaming inserts
- batch_size_bytes - An approximate number of bytes to upload as part of a batch
- batch_size - The number of messages to upload at a single time
Obsolete configuration options:
- uploader_interval_secs
- deleter_interval_secs
- key_path
- key_password
- service_account
- temp_file_prefix
- temp_directory
@josephlewis42 it works well.
This is our sample configuration output
output {
google_bigquery {
project_id => "enduring-ar-xxxxx"
dataset => "test"
json_schema => { fields => [{ name => "user" type => "STRING" }, { name => "message" type => "STRING" }] }
json_key_file => "/usr/share/logstash/config/Data-xxx.json }
}
Thanks for your help
The fix for this just got merged, thank you so much @edwinallenz and @ivancruzbht for helping out! I'm going to close this ticket for now and we can open up another one if you start getting the issue again.
WARNING 4.0.0 has breaking changes so read the changelog before upgrading to understand how they'll affect you.
The plugin can be updated with the following command
bin/logstash-plugin update logstash-output-google_bigquery
Cheers!
- Joseph