trivago/gollum

multi-line messages

abelokonev opened this issue · 8 comments

abelokonev commented

Сan an existing version work with multiline messages? For example, a java stack trace.
I'm trying to use format.GrokToJSON:

config

"FileIn":
Type: "consumer.File"
File: /tmp/test_file
DefaultOffset: newest
OffsetFile: ""
ObserveMode: poll
PollingDelay: 1000
Streams: "file_test"
Modulators:
- format.GrokToJSON:
Patterns:
- ^(?P<timestamp>\d{2}.\d{2}.\d{4}\s+[\d:.+]+)\s+(?:%{LOGLEVEL:level})\s+(?P<trace>(?m).*$)

"rateLimiter":
Type: "router.Broadcast"
Stream: "file_test"

"fileOut":
Type: "producer.File"
Streams: "file_test"
File: /tmp/gollum.log
Batch:
MaxCount: 128
FlushCount: 64
TimeoutSec: 60
FlushTimeoutSec: 3

But only the first line is processed.

arnecls commented

If I remember correctly, @mre had the same issue and also had a fix.
I'll check with him.

arnecls commented

You have to adjust your grok pattern.
See

gollum/format/groktojson_test.go

Line 27 in 3d7deb2

func TestMultilineGrok(t *testing.T) {

arnecls commented

That (?m) is coming from the go regex engine.
See https://github.com/google/re2/wiki/Syntax

abelokonev commented

Tested various options. The expression could not be matched.

mre commented

@abelokonev can you provide a sample from the java stack trace logs you want to process? Would make debugging and writing a unit test a little easier. 😉 You could obfuscate the function names if you like.

abelokonev commented

For an example:

12.76.2058 23:05:55.365 [qtp1174829684-20166] ERROR com.dze.coreapi.controller.CoreExceptionHandler - INTERNAL ERROR org.springframework.web.bind.ServletRequestBindingException: Missing request header 'Profile' for method parameter of type UserProfile
at org.springframework.web.method.annotation.RequestHeaderMethodArgumentResolver.handleMissingValue(RequestHeaderMethodArgumentResolver.java:82) ~[spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:96) ~[spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:99) ~[spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:161) ~[spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128) ~[spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) ~[spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:832) ~[spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:743) ~[spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961) [spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895) [spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967) [spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframetork.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858) [spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843) [spring-webmvc-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:224) [websocket-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:261) [spring-boot-actuator-1.3.4.RELEASE.jar:1.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:201) [javamelody-core-1.60.0.jar:1.60.0]
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178) [javamelody-core-1.60.0.jar:1.60.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:115) [spring-boot-actuator-1.3.4.RELEASE.jar:1.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.3.4.RELEASE.jar:1.3.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.16.v20160414.jar:9.2.16.v20160414]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.16.v20160414.jar:9.2.16.v20160414]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

arnecls commented

Sorry for the late reply - vacation.
I think I found your problem.

"FileIn":
Type: "consumer.File"

This is a classic problem when reading from files.
Gollum operates on messages. When reading from a stream that is not already message based you need to define when a message actually ends. In case of a file the standard is newline.
So your multiline Grok is not working because you actually read single lines.

With files this is a quite common problem as most of the times you don't have a sufficient "end of message" separator. Because of this we normally don't use file scraping for multiline output but syslog, sockets, or similar consumers.
So I would recommend to either insert a very distinctive "end of message" separator to your output or to use another communication method.

abelokonev commented

Thanks for the consultation.