Error using --input-format csv

Question

Error using --input-format csv

jfinzel opened this issue 7 years ago · 7 comments

@ankane I am getting an error trying to try csv logs. Am I doing it wrong?

foo@bar:~$ tail -F /local/spool/postgres/pg_log/foo-2017-12-26_150000.csv | dexter --input-format csv -d db_name -p 6000 --min-time 0 --log-level debug2 --log-sql --interval 5 | tee -a /tmp/dexter.log
SQL: SELECT * FROM pg_available_extensions WHERE name = 'hypopg' AND installed_version IS NOT NULL
SQL: SET lock_timeout = '5s'
Started
/opt/dexter/vendor/bundle/ruby/2.3.0/gems/pgdexter-0.3.0/lib/dexter/csv_log_parser.rb:6:in `perform': undefined method `file' for #<IO:<STDIN>> (NoMethodError)
Did you mean?  fileno
	from /opt/dexter/vendor/bundle/ruby/2.3.0/gems/pgdexter-0.3.0/lib/dexter/processor.rb:44:in `perform'
	from /opt/dexter/vendor/bundle/ruby/2.3.0/gems/pgdexter-0.3.0/lib/dexter/client.rb:22:in `perform'
	from /opt/dexter/vendor/bundle/ruby/2.3.0/gems/pgdexter-0.3.0/exe/dexter:5:in `<top (required)>'
	from /opt/dexter/bin/dexter:17:in `load'
	from /opt/dexter/bin/dexter:17:in `<main>'

This is related to: #13

Answer 1 · 2017-12-27T06:32:34.000Z

@jfinzel Should be fixed on master (and latest package).

Answer 2 · 2017-12-27T16:13:23.000Z

@ankane no error, but it still does not work with this format. Try feeding this log input for your testing:

2017-12-27 10:09:49.421 CST,"jfinzel","db_name",32130,"1.1.1.24:43588",5a43c5ca.7d82,3,"SELECT",2017-12-27 10:09:46 CST,4/0,0,LOG,00000,"duration: 1533.391 ms  statement: SELECT * FROM foo.bar WHERE customer_id = 3913437;",,,,,,,,,"psql"
2017-12-27 10:09:52.400 CST,"jfinzel","db_name",32130,"1.1.1.24:43588",5a43c5ca.7d82,4,"SELECT",2017-12-27 10:09:46 CST,4/0,0,LOG,00000,"duration: 1521.998 ms  statement: SELECT * FROM foo.bar WHERE customer_id = 3913437;",,,,,,,,,"psql"
2017-12-27 10:09:53.384 CST,"jfinzel","db_name",32130,"1.1.1.24:43588",5a43c5ca.7d82,5,"idle",2017-12-27 10:09:46 CST,,0,LOG,00000,"disconnection: session time: 0:00:07.178 user=jfinzel database=db_name host=1.1.1.24 port=43588",,,,,,,,,"psql"

This is all that happens for me with this format:

foo@bar:~$ tail -F /local/spool/postgres/pg_log/foo-2017-12-27_100000.csv | dexter --input-format csv -d db_name -p 60000 --min-time 0 --log-level debug2 --log-sql --interval 5 | tee -a /tmp/dexter.log
SQL: SELECT * FROM pg_available_extensions WHERE name = 'hypopg' AND installed_version IS NOT NULL
SQL: SET lock_timeout = '5s'
Started
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints
Processing 0 new query fingerprints

Answer 3 · 2017-12-27T22:20:44.000Z

Try once more with latest package.

Answer 4 · 2017-12-28T21:18:35.000Z

Nope, it's still doing the same thing as above.

Just to confirm: package: dexter_0.3.0-1514409700.d5c33602.xenial_amd64.deb

Answer 5 · 2017-12-28T22:28:22.000Z

If you want to run a test on a database configured like ours, try this for postgresql.conf:

log_destination                                    = 'csvlog'
log_directory                                      = '/local/spool/postgres/pg_log'
log_filename                                       = 'foo-%Y-%m-%d_%H%M%S.log'
log_line_prefix                                    = '%u@%h:%p %t %x '
log_min_duration_statement                         = '0'
log_timezone                                       = 'America/Chicago'
logging_collector                                  = 'on'

Answer 6 · 2017-12-29T02:23:16.000Z

Yeah, that's the right version. Maybe try:

tail -F -n +1 ...

Answer 7 · 2017-12-29T14:47:31.000Z

@ankane it works! Sorry my last try I think was user error - tailing the logs from the wrong cluster :(. This is great - I am now going to throw some serious log output at it - let's see how it does :).