Error using --input-format csv
jfinzel opened this issue · 7 comments
jfinzel commented
@ankane I am getting an error trying to try csv logs. Am I doing it wrong?
foo@bar:~$ tail -F /local/spool/postgres/pg_log/foo-2017-12-26_150000.csv | dexter --input-format csv -d db_name -p 6000 --min-time 0 --log-level debug2 --log-sql --interval 5 | tee -a /tmp/dexter.log
SQL: SELECT * FROM pg_available_extensions WHERE name = 'hypopg' AND installed_version IS NOT NULL
SQL: SET lock_timeout = '5s'
Started
/opt/dexter/vendor/bundle/ruby/2.3.0/gems/pgdexter-0.3.0/lib/dexter/csv_log_parser.rb:6:in `perform': undefined method `file' for #<IO:<STDIN>> (NoMethodError)
Did you mean? fileno
from /opt/dexter/vendor/bundle/ruby/2.3.0/gems/pgdexter-0.3.0/lib/dexter/processor.rb:44:in `perform'
from /opt/dexter/vendor/bundle/ruby/2.3.0/gems/pgdexter-0.3.0/lib/dexter/client.rb:22:in `perform'
from /opt/dexter/vendor/bundle/ruby/2.3.0/gems/pgdexter-0.3.0/exe/dexter:5:in `<top (required)>'
from /opt/dexter/bin/dexter:17:in `load'
from /opt/dexter/bin/dexter:17:in `<main>'
This is related to: #13
jfinzel commented
@ankane no error, but it still does not work with this format. Try feeding this log input for your testing:
2017-12-27 10:09:49.421 CST,"jfinzel","db_name",32130,"1.1.1.24:43588",5a43c5ca.7d82,3,"SELECT",2017-12-27 10:09:46 CST,4/0,0,LOG,00000,"duration: 1533.391 ms statement: SELECT * FROM foo.bar WHERE customer_id = 3913437;",,,,,,,,,"psql"
2017-12-27 10:09:52.400 CST,"jfinzel","db_name",32130,"1.1.1.24:43588",5a43c5ca.7d82,4,"SELECT",2017-12-27 10:09:46 CST,4/0,0,LOG,00000,"duration: 1521.998 ms statement: SELECT * FROM foo.bar WHERE customer_id = 3913437;",,,,,,,,,"psql"
2017-12-27 10:09:53.384 CST,"jfinzel","db_name",32130,"1.1.1.24:43588",5a43c5ca.7d82,5,"idle",2017-12-27 10:09:46 CST,,0,LOG,00000,"disconnection: session time: 0:00:07.178 user=jfinzel database=db_name host=1.1.1.24 port=43588",,,,,,,,,"psql"
This is all that happens for me with this format:
foo@bar:~$ tail -F /local/spool/postgres/pg_log/foo-2017-12-27_100000.csv | dexter --input-format csv -d db_name -p 60000 --min-time 0 --log-level debug2 --log-sql --interval 5 | tee -a /tmp/dexter.log
SQL: SELECT * FROM pg_available_extensions WHERE name = 'hypopg' AND installed_version IS NOT NULL
SQL: SET lock_timeout = '5s'
Started
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
ankane commented
Try once more with latest package.
jfinzel commented
Nope, it's still doing the same thing as above.
Just to confirm: package: dexter_0.3.0-1514409700.d5c33602.xenial_amd64.deb
jfinzel commented
If you want to run a test on a database configured like ours, try this for postgresql.conf
:
log_destination = 'csvlog'
log_directory = '/local/spool/postgres/pg_log'
log_filename = 'foo-%Y-%m-%d_%H%M%S.log'
log_line_prefix = '%u@%h:%p %t %x '
log_min_duration_statement = '0'
log_timezone = 'America/Chicago'
logging_collector = 'on'
ankane commented
Yeah, that's the right version. Maybe try:
tail -F -n +1 ...