optimize: BadMessageException: 500: Response header too large
andrej-schaefer opened this issue · 9 comments
andrej-schaefer commented
Hey Camunda Team,
when starting camunda platform for local development on a local machine optimize stop working after ~5 mins after compose up.
It seems the error start occuring after first accessing the optimize UI. After that you can start timer and after ~5 mins the UI respond with http 500 and no possibility to restart optimize except to down the whole project
It is not necessary to have any processes deployed.
I have attached the log from spin up to the first error occurrence.
Tries:
- down and up the optimize container only: Fails
- Down -v the whole project: Works with complete data loss
- remove all browser cookies: Works after additional authentication
- Chrome: occur after same amount of time
Analysis:
Tried a few things to analyze the problem. After digging around it seems that the cookie contains / sends to the server the "JSESSIONID", "X-Optimize-Refresh-Token" and "X-Optimize-Authorization" but after 5 mins it fails. When the faulty request is send without the "X-Optimize-Authorization" due to it is expired
Hope i have not missed any configuration to avoid this and maybe anyone can help :)
Data:
- Windows 11
- docker for windows running in wsl2 mode (4.26.1 (131620))
- camunda version tested (to the point of reporting) with main (v8.5.0-alpha), v8.4.5, v8.3.9, v8.2.25
- Firefox developer edition 124.0b9 (64-bit)
Update: Wrote the introduction during analysis
Result is as explained in the analysis with the final result of expired token.
Kept the monologue as explanation and protocol :)
Log from start to error (Think after analyze this is not necessary anymore)
optimize | 2024-03-12T08:33:22.933054797Z
optimize | 2024-03-12T08:33:22.933219420Z Starting Camunda Optimize 3.11.7...
optimize | 2024-03-12T08:33:22.933232855Z
optimize | 2024-03-12T08:33:25.018291790Z
optimize | 2024-03-12T08:33:25.018515044Z . ____ _ __ _ _
optimize | 2024-03-12T08:33:25.018537129Z /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
optimize | 2024-03-12T08:33:25.018540559Z ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
optimize | 2024-03-12T08:33:25.018589441Z \\/ ___)| |_)| | | | | || (_| | ) ) ) )
optimize | 2024-03-12T08:33:25.018593287Z ' |____| .__|_| |_|_| |_\__, | / / / /
optimize | 2024-03-12T08:33:25.018595472Z =========|_|==============|___/=/_/_/_/
optimize | 2024-03-12T08:33:25.021100727Z :: Spring Boot :: (v3.1.9)
optimize | 2024-03-12T08:33:25.021140755Z
optimize | 2024-03-12T08:33:25.067063500Z 08:33:25.062 [background-preinit] INFO o.h.validator.internal.util.Version - HV000001: Hibernate Validator 8.0.1.Final
optimize | 2024-03-12T08:33:25.157103704Z 08:33:25.156 [main] INFO org.camunda.optimize.Main - Starting Main using Java 17.0.10 with PID 8 (/optimize/optimize-backend-3.11.7.jar started by camunda in /optimize)
... (truncated due to necessary anymore)
optimize | 2024-03-12T08:38:45.471524863Z 08:38:45.471 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeProcessDefinitionImportMediator - Records of type PROCESS from partition 1 imported in page: 0
optimize | 2024-03-12T08:38:45.476426980Z 08:38:45.476 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeProcessInstanceImportMediator - Records of type PROCESS_INSTANCE from partition 1 imported in page: 0
optimize | 2024-03-12T08:38:45.480942203Z 08:38:45.480 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeVariableImportMediator - Records of type VARIABLE from partition 1 imported in page: 0
optimize | 2024-03-12T08:38:45.483884516Z 08:38:45.483 [ZeebeImportScheduler-1] WARN o.c.o.s.i.z.f.AbstractZeebeRecordFetcher - No Zeebe index of type zeebe-record-incident found to read records from!
optimize | 2024-03-12T08:38:45.483956752Z 08:38:45.483 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeIncidentImportMediator - Records of type INCIDENT from partition 1 imported in page: 0
optimize | 2024-03-12T08:39:15.475715739Z 08:39:15.475 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeProcessDefinitionImportMediator - Records of type PROCESS from partition 1 imported in page: 0
optimize | 2024-03-12T08:39:15.481430028Z 08:39:15.480 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeProcessInstanceImportMediator - Records of type PROCESS_INSTANCE from partition 1 imported in page: 36
optimize | 2024-03-12T08:39:15.484792307Z 08:39:15.484 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeVariableImportMediator - Records of type VARIABLE from partition 1 imported in page: 0
optimize | 2024-03-12T08:39:15.486995901Z 08:39:15.486 [ZeebeImportScheduler-1] WARN o.c.o.s.i.z.f.AbstractZeebeRecordFetcher - No Zeebe index of type zeebe-record-incident found to read records from!
optimize | 2024-03-12T08:39:15.487058996Z 08:39:15.486 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeIncidentImportMediator - Records of type INCIDENT from partition 1 imported in page: 0
optimize | 2024-03-12T08:39:26.868541395Z 08:39:26.868 [qtp747827348-30] WARN o.eclipse.jetty.server.HttpChannel - handleException / org.eclipse.jetty.http.BadMessageException: 500: Response header too large
optimize | 2024-03-12T08:39:29.172048441Z 08:39:29.171 [qtp747827348-27] WARN o.eclipse.jetty.server.HttpChannel - handleException / org.eclipse.jetty.http.BadMessageException: 500: Response header too large
optimize | 2024-03-12T08:39:45.482472269Z 08:39:45.481 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeProcessDefinitionImportMediator - Records of type PROCESS from partition 1 imported in page: 2
optimize | 2024-03-12T08:39:45.485216399Z 08:39:45.484 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeProcessInstanceImportMediator - Records of type PROCESS_INSTANCE from partition 1 imported in page: 0
optimize | 2024-03-12T08:39:45.487831214Z 08:39:45.487 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeVariableImportMediator - Records of type VARIABLE from partition 1 imported in page: 0
optimize | 2024-03-12T08:39:45.489902529Z 08:39:45.489 [ZeebeImportScheduler-1] WARN o.c.o.s.i.z.f.AbstractZeebeRecordFetcher - No Zeebe index of type zeebe-record-incident found to read records from!
optimize | 2024-03-12T08:39:45.489933147Z 08:39:45.489 [ZeebeImportScheduler-1] INFO o.c.o.s.i.z.m.ZeebeIncidentImportMediator - Records of type INCIDENT from partition 1 imported in page: 0
optimize | 2024-03-12T08:40:01.358699717Z 08:40:01.358 [OnboardingSchedulerService-1] INFO o.c.o.s.o.OnboardingSchedulerService - Checking whether new data would trigger onboarding
optimize | 2024-03-12T08:40:01.371470762Z 08:40:01.371 [OnboardingSchedulerService-1] INFO o.c.o.s.o.OnboardingSchedulerService - Onboarding check completed
jessesimpson36 commented
I'm going through the steps you laid out to see if I can reproduce.
this 500 error happens often when there is an nginx proxy (or some other kind of proxy). The cookie builds up and then becomes too large. in kubernetes, we solve it by adding
nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
into the ingress.
relevant nginx link if you configure nginx yourself:
Can you confirm whether you are using a proxy or not?
also, I could not reproduce this issue using the latest docker compose file.
andrej-schaefer commented
I have retested with a fresh clone of the project and can still reproduce the error. i have no proxies or any changes on the project . After clone, just pull and up in the main branch and open optimize in the browser. After 4 when the token in the cookie expires refresh and get a white page. The console shows the 500 error
optimize | 07:50:32.476 [qtp1380332117-32] WARN o.eclipse.jetty.server.HttpChannel - handleException / org.eclipse.jetty.http.BadMessageException: 500: Response header too large
WIN + WSL2 (Docker for windows running on WSL) + FF (Also tested on Chrome)
jessesimpson36 commented
Thank you for confirming @andrej-schaefer , I tried waiting 15 minutes this time, and for me that was enough to reproduce the error. Now I think I can figure out what's wrong.
melvinkcx commented
This is happening to us as well.
We do have AD integration, and when logging in using an AD user, the problem appears to be more severe.
The workaround for us now is to create an equivalent local user. The problem still persists, but it seems to be less severe.
alessandrocavalli commented
Support case : https://jira.camunda.com/browse/SUPPORT-23177
jessesimpson36 commented
Hey all, I wanted to share some progress on this topic. I wrote a playwright script to reproduce this error for me, and I used it to produce a dataset that would show me precisely what versions have this bug and don't have this bug. Heres what I came up with:
test.sh.txt
optimize_versions.sh.txt
optimize_500.spec.ts.txt
get_dates.txt
sqlite> select version, result, optimize_versions.optimize as optimize, date from results join dates on results.version=dates.release join optimize_versions on results.version=optimize_versions.release order by date desc;
| version | result | optimize | date |
|--------------|--------|------------------|---------------------------|
| 8.5+gen8 | pass | 8.5.2 | 2024-08-13T11:42:27+01:00 |
| 8.4+gen12 | pass | 8.4.7 | 2024-08-12T16:12:50+02:00 |
| 8.6-alpha4 | pass | 8.5.2 | 2024-08-12T15:06:50+00:00 |
| 8.6-alpha3 | pass | 8.5.2 | 2024-07-01T16:43:13+02:00 |
| 8.5+gen6 | pass | 8.5.2 | 2024-07-01T14:50:51+02:00 |
| 8.4+gen10 | fail | 8.4.6 | 2024-07-01T13:08:59+00:00 |
| 8.5+gen5 | pass | 8.5.2 | 2024-06-20T11:26:44-04:00 |
| 8.6-alpha2 | pass | 8.5.2 | 2024-06-10T10:10:03-04:00 |
| 8.5.3 | pass | 8.5.2 | 2024-06-10T10:10:03-04:00 |
| 8.5.2 | pass | 8.5.2 | 2024-06-10T10:10:03-04:00 |
| 8.4.8 | pass | 8.5.2 | 2024-06-10T10:10:03-04:00 |
| 8.4+gen9 | fail | 8.4.5 | 2024-06-07T13:26:16+02:00 |
| 8.6-alpha1 | fail | 8.5.1 | 2024-05-14T16:42:55+02:00 |
| 8.5+gen2 | fail | 8.5.1 | 2024-05-14T16:42:55+02:00 |
| 8.4+gen7 | fail | 8.5.1 | 2024-05-14T16:42:55+02:00 |
| 8.5.0 | fail | 8.5.0 | 2024-04-09T10:40:21+01:00 |
| 8.4.6 | fail | 8.4.3 | 2024-04-06T02:48:00+00:00 |
| 8.4.5 | fail | 8.4.2 | 2024-03-08T18:35:09+00:00 |
| 8.5.0-alpha2 | fail | 8.5.0-alpha2-rc3 | 2024-03-08T16:14:27+00:00 |
| 8.4.4 | fail | 8.4.1 | 2024-02-28T12:04:50+00:00 |
| 8.5.0-alpha1 | fail | 8.4.1 | 2024-02-09T13:18:32+00:00 |
| 8.4.3 | fail | 8.4.1 | 2024-02-09T13:18:32+00:00 |
| 8.4.2 | fail | 8.4.1 | 2024-02-09T13:18:32+00:00 |
| 8.4.1 | fail | 8.4.0 | 2024-01-20T00:11:18+00:00 |
| 8.4.0 | fail | 8.3.5 | 2024-01-05T19:47:42+00:00 |
| 8.4.0-alpha2 | fail | 8.3.4 | 2023-12-12T10:46:23+01:00 |
| 8.4.0-alpha1 | fail | 8.3.0 | 2023-10-27T09:29:45-04:00 |
According to this table, I believe releases of optimize past june 7th or 8.4.6 do not have this bug. Let me know if you feel this table is not accurate. I think it will help me debug further.
jessesimpson36 commented
I took another look at this table:
and then pulled up the optimize commits between 8.4.6 and 8.4.7 to see what changed. And one of the commits I see is :
8f4df80b7b Maintenance/3.12 ccsmauthfilter twice (#13763)
which is a commit @matthewBearCamunda and I made to fix a similar-looking authentication issue in optimize.
Related PR: https://github.com/camunda/camunda-optimize/pull/13705
It bothers me that we're seeing the same issue but 2 different error messages. In this github issue, we see:
WARN o.eclipse.jetty.server.HttpChannel - handleException / org.eclipse.jetty.http.BadMessageException: 500: Response header too large
However the optimize issue that I created the above PR for was a really long stacktrace was this.
09:08:28.504 [qtp609946583-23] TRACE o.eclipse.jetty.http.HttpGenerator - IGNORED
java.nio.BufferOverflowException: null
at java.base/java.nio.Buffer.nextPutIndex(Buffer.java:722)
at java.base/java.nio.DirectByteBuffer.put(DirectByteBuffer.java:359)
at org.eclipse.jetty.http.HttpGenerator.putSanitisedValue(HttpGenerator.java:904)
at org.eclipse.jetty.http.HttpGenerator.putTo(HttpGenerator.java:920)
at org.eclipse.jetty.http.HttpGenerator.generateHeaders(HttpGenerator.java:672)
at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:410)
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:751)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:589)
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1046)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1123)
at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.commit(GzipHttpOutputInterceptor.java:168)
at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:96)
at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:270)
at org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:623)
at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.close(OnCommittedResponseWrapper.java:530)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:222)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:135)
at org.eclipse.jetty.server.ResourceService.sendWelcome(ResourceService.java:449)
at org.eclipse.jetty.server.ResourceService.doGet(ResourceService.java:256)
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:453)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
at org.eclipse.jetty.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:170)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:150)
at org.camunda.optimize.rest.security.ccsm.CCSMAuthenticationCookieFilter.doFilter(CCSMAuthenticationCookieFilter.java:88)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.camunda.optimize.jetty.NoCachingFilter.doFilter(NoCachingFilter.java:45)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.camunda.optimize.rest.security.SingleSignOnRequestFilter.doFilter(SingleSignOnRequestFilter.java:54)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108)
at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479)
at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340)
at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128)
at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:150)
at org.camunda.optimize.rest.security.ccsm.CCSMAuthenticationCookieFilter.doFilter(CCSMAuthenticationCookieFilter.java:88)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$3(HandlerMappingIntrospector.java:195)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74)
at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:230)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.camunda.optimize.jetty.LicenseFilter.doFilter(LicenseFilter.java:71)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
at java.base/java.lang.Thread.run(Thread.java:840)
09:08:28.507 [qtp609946583-23] DEBUG o.e.jetty.server.HttpConnection - generate: HEADER_OVERFLOW for SendCallback@319687fb[PROCESSING][i=HTTP/1.1{s=200,h=20,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@3917bed6] ([p=0,l=8192,c=8192,r=8192],[p=0,l=2230,c=32768,r=2230],true)@START
09:08:28.510 [qtp609946583-23] DEBUG o.e.jetty.server.HttpConnection - generate: NEED_HEADER for SendCallback@319687fb[PROCESSING][i=HTTP/1.1{s=500,h=1,cl=0},cb=org.eclipse.jetty.server.HttpChannel$SendCallback$1@611672f8] (null,null,true)@START
09:08:28.511 [qtp609946583-23] DEBUG o.eclipse.jetty.http.HttpGenerator - generateHeaders HTTP/1.1{s=500,h=1,cl=0} last=true content=null
09:08:28.511 [qtp609946583-23] DEBUG o.eclipse.jetty.http.HttpGenerator - Connection: close
jessesimpson36 commented
OH, THE LOG LEVEL IS TRACE!!! so they are the same error
jessesimpson36 commented
I will close this issue since it's fixed in optimize >= 8.5.4 (for 8.5.x) or >= 8.4.7 (for 8.4.x)