Fails on logs of any significant size
Closed this issue · 4 comments
nikess commented
With a memory error. My sample input is 250mb.
kapad commented
yep.. terrible tool performance wise.
I do need some tool that can do filtering of this kind.. no any other options @nikess ?
gforcada commented
When I developed and used it, performance was ok, maybe a bit slower but fine, for a sporadic analysis, now I'm also bitten by a way too much memory usage myself. I don't have much time to work on it unfortunately.
I guess that redesigning the whole project to use iterators would help, anyone up for it?
kapad commented
Yes. It's the memory usage that made using it so difficult for me too.
I think you're suggestion of redesigning the tool to use iterators and not
load the entire file to memory is correct. It would definitely solve the
issue but would also probably be an almost complete rewrite.
I've used halog (https://www.systutorials.com/docs/linux/man/1-halog/) and
goaccess (https://goaccess.io/) to analyze my logs.
halog is the log processor that is packaged along with haproxy, but I felt
that it has some issues when filtering the output on urls. The results
didn't match with what I got using grep/sed/awk.
goaccess is a really amazing tool and outputs a beautiful dashboard with a
lot of useful analytics data from the haproxy logs. For goaccess though,
you will need to configure the log processing string.
I used
goaccess haproxy.log --log-format='%^:%^:%^: %h:%^[%d:%t.%^] %^ %^/%v
%^/%^/%^/%^/%L %s %b %^"%r"' --date-format=%d/%b/%Y --time-format=%T -a >
report.html
My logs are in the default http format for haproxy.
…On 1 March 2018 at 21:19, Gil Forcada Codinachs ***@***.***> wrote:
When I developed and used it, performance was ok, maybe a bit slower but
fine, for a sporadic analysis, now I'm also bitten by a way too much memory
usage myself. I don't have much time to work on it unfortunately.
I guess that redesigning the whole project to use iterators would help,
anyone up for it?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#20 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB7IS1o8426I99lfmZ2NO7VzL13bQtXrks5taBiFgaJpZM4LAZ4c>
.
--
Rohan Kapadia
gforcada commented
I'm happy to report you that I finally had time during Christmas to rewrite the tool, version 4.0.0 should be much less memory hungry :)