logstash-plugins/logstash-input-jdbc

Paging bug: JDBC_PAGING_ENABLED and JDBC_PAGE_SIZE degraded after version 4.3.14

Closed this issue · 0 comments

bmax commented

Hello! It seems like at the same time prepared statements were introduced in version 4.3.16 is the same time paging functionality broke. I've tested this by rolling back to 4.3.14 and ensuring that jdbc_paging_enabled works. I am also working on a PR for a fix but in the mean time I thought I'd get this issue out first in case anyone else is having trouble.

The problem resides here: https://github.com/logstash-plugins/logstash-input-jdbc/blob/master/lib/logstash/plugin_mixins/jdbc/statement_handler.rb#L34

The variable @jdbc_paging_enabled is not available in that context therefore this will never evaluate as true.

Debugging information

  • Version: > 4.3.14
  • Operating System: centos (Docker image logstash:7.4.1)
  • Config File (if you have sensitive info, please remove it):
input {
  jdbc {
    jdbc_driver_library => "/usr/share/logstash/bin/postgresql-42.2.8.jar"
    jdbc_driver_class => "org.postgresql.Driver"
    jdbc_connection_string => "jdbc:postgresql://${POSTGRES_URL}/${POSTGRES_DB}"
    jdbc_user => "${POSTGRES_USER}"
    jdbc_password => "${POSTGRES_PASSWORD}"
    schedule => "* * * * * *"
    jdbc_paging_enabled => true
    jdbc_page_size => 10
    statement => "SELECT * from test"
  }
}

output {
  elasticsearch {
    document_id => "visit-%{user_id}-%{merchant_id}"
    hosts    => [ "${ELASTICSEARCH_HOST}" ]
    index => "program_users"
  }
}
  • Steps to Reproduce: Include jdbc_paging_enabled and ensure that the select query does not include limit or offset.