Allow the embedded web server to be shut down gracefully
LutzStrobel opened this issue ยท 112 comments
We are using spring-boot and spring-cloud to realize a micro service archtecture. During load tests we are facing lots of errors such as "entitymanager closed and others" if we shut the micro service down during load.
We are wondering if there is an option to configure the embedded container to shut its service connector down and waits for an empty request queue before shutting down the complete application context.
If there is no such option, I did not find any, then it would be great to extend the shutdownhook of spring to respect such requirements.
We currently stop the application context and then stop the container. We did try to reverse this ordering but it had some unwanted side effects so we're stuck with it for now at least. That's the bad news.
The good news is that you can actually get a graceful shutdown yourself if you're happy to get your hands a bit dirty. The gist is that you need to pause Tomcat's connector and then wait for its thread pool to shutdown before allowing the destruction of the application context to proceed. It looks something like this:
package com.example;
import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;
import org.apache.catalina.connector.Connector;
import org.apache.tomcat.util.threads.ThreadPoolExecutor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.embedded.ConfigurableEmbeddedServletContainer;
import org.springframework.boot.context.embedded.EmbeddedServletContainerCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatConnectorCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@SpringBootApplication
@RestController
public class Gh4657Application {
public static void main(String[] args) {
SpringApplication.run(Gh4657Application.class, args);
}
@RequestMapping("/pause")
public String pause() throws InterruptedException {
Thread.sleep(10000);
return "Pause complete";
}
@Bean
public GracefulShutdown gracefulShutdown() {
return new GracefulShutdown();
}
@Bean
public EmbeddedServletContainerCustomizer tomcatCustomizer() {
return new EmbeddedServletContainerCustomizer() {
@Override
public void customize(ConfigurableEmbeddedServletContainer container) {
if (container instanceof TomcatEmbeddedServletContainerFactory) {
((TomcatEmbeddedServletContainerFactory) container)
.addConnectorCustomizers(gracefulShutdown());
}
}
};
}
private static class GracefulShutdown implements TomcatConnectorCustomizer,
ApplicationListener<ContextClosedEvent> {
private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);
private volatile Connector connector;
@Override
public void customize(Connector connector) {
this.connector = connector;
}
@Override
public void onApplicationEvent(ContextClosedEvent event) {
this.connector.pause();
Executor executor = this.connector.getProtocolHandler().getExecutor();
if (executor instanceof ThreadPoolExecutor) {
try {
ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
threadPoolExecutor.shutdown();
if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
log.warn("Tomcat thread pool did not shut down gracefully within "
+ "30 seconds. Proceeding with forceful shutdown");
}
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
}
}
}I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it. However, that'll require a bit more work as it'll need to work with all of the embedded containers that we support (Jetty, Tomcat, and Undertow), cope with multiple connectors (HTTP and HTTPS, for example) and we'll need to think about the configuration options, if any, that we'd want to offer: switching it on or off, configuring the period that we wait for the thread pool to shutdown, etc.
Thank you for your advice.
We currently simply shut down the embedded container and the waiting a short time before closing the application context.
What do you thing, will it be possible in future to shut down a spring boot application more gracefully?
What do you thing, will it be possible in future to shut down a spring boot application more gracefully?
Yes. As I said above "I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it".
I'm going to re-open this issue as we'll use it to track the possible enhancement.
+1 on this request. We ran into a similar problem when load testing and dropping a node from the test. @wilkinsona in your example, I was thinking of using an implementation of smartlifecylce so I can insure the connector is shutdown first. You said you ran into issues shutting down tomcat first?
I think the right order is:
-
pause the io endpoint
web container just pasue(deny new request come in),and RPC framework need notify client don't send new request and wait current request be proccessed
-
wait the request to be processed and response client
-
close the io endpoint
-
close spring service
-
close log system
so we do it as follows:
@Override
public void finished(ConfigurableApplicationContext context, Throwable exception) {
if (exception == null) {
//install UncaughtExceptionHandler
UncaughtExceptionHandlerWrapper.install();
//when system startup ,register shutdown hooks to clean resouces.
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
//run all shutdown hooks before spring container avoid service dependency
ShutdownHooks.shutdownAll();
//close spring container
context.close();
shutdownLogSystem();
}
});
//log startup info
LoggerFactory.getLogger(YijiApplicationRunListener.class).info("ๅฏๅจๆๅ: http://127.0.0.1:{}",
context.getEnvironment().getProperty(Apps.HTTP_PORT));
} else {
ShutdownHooks.shutdownAll();
shutdownLogSystem();
}
}
This issue is actually a bit more complicated.
I know that SmartLifeCycle can be used to set the order in which beans are notified of life cycle events. However, there is no generalized way for a service to know it should startup/shutdown before another service.
Consider the following:
A sprint boot application is running an embedded servlet container and is also producing/consuming JMS messages.
On the close event, the servlet container really needs to pause the connector first, process its remain working (any connections that have already been establish.)
We need to insure this is the FIRST thing that happens, prior to the JMS infrastructure shutting down because the work done inside tomcat may rely on JMS.
The JMS infrastructure has a similar requirement: it must stop listening for messages and chew through any remaining messages it has already accepted.
I can certainly implement a SmartLifeCycle class that sets the phase "very high"....and I could even create two instances, one for embedded servlet container and one for JMS and insure the correct order.
But in the spirit of Spring Boot, if I add a tomcat container, its my expectation that when the application shuts down, it will gracefully stop accepting connections, process remaining work, and exit.
It would be helpful if there was a mechanism to allow ordering to be expressed relative to other services "JMS must start before tomcat", "tomcat must close before JMS". This would be similar to the AutoConfigureBefore/AutoConfigureAfter annotations that are used in Spring boot.
One way to approach this might be to create an enum for the generalized services (This is not ideal, but I can't think of another way without introducing artificial, compile time dependencies.):
EMBEDDED_CONTAINER
JMS
DISCOVERY_CLIENT
.
.
The annotations could leverage the enums to order the life cycle listeners.
For now, its up to the developer to explicitly define the shutdown order of services via "SmartLifeCycle" instances...which can get a bit messy and seems like extra work for something that should really work out of the box.
@tkvangorder You don't need to use SmartLifecycleto gracefully shut down Tomcat as shown in my comment above. It happens in response to a ContextClosedEvent which is fired at the very beginning of the context's close processing before any of the beans in the context are processed.
Beyond this, Spring Framework already has support for closing things down in the correct order. For example you can implement DisposableBean. When the container disposes of a bean, it will dispose of anything that depends on it first.
@wilkinsona is the order of shutdown anyway guaranteed ? Is spring boot always going to shutdown application context first? we are trying to implement what you said, but was not sure if this will change in the future.
Is spring boot always going to shutdown application context first?
Yes. When you're using an embedded container, it's the application context being shut down that triggers the shutdown of the container.
I want to report back on how to setup this correctly using Jetty and springboot , use this class instead of EmbeddedServletAutoconfiguration and you are set.
public class HttpConfig {
private static final Logger log = LoggerFactory.getLogger(HttpConfig.class);
private static volatile Server server;
// Jetty HTTP Server
//
// see [1] on how to implement graceful shutdown in springboot.
// Note that since use jetty, we need to use server.stop(), also StatisticsHandler must be
// configured, for jetty graceful shutdown to work.
//
// [1]: https://github.com/spring-projects/spring-boot/issues/4657
@Bean
@Autowired
public JettyEmbeddedServletContainerFactory jettyEmbeddedServletContainerFactory(HttpSetting httpSetting) {
JettyEmbeddedServletContainerFactory factory = new JettyEmbeddedServletContainerFactory();
factory.setPort(httpSetting.getPort());
log.info("Jetty configured on port: " + httpSetting.getPort());
factory.addServerCustomizers(new JettyServerCustomizer() {
@Override
public void customize(Server server1) {
server = server1;
log.info("Jetty version: " + server.getVersion());
// Configure shutdown wait time.
if (httpSetting.getShutdownWaitTime() > 0) {
// Add StatsticsHandler, in order for graceful shutdown to work.
StatisticsHandler handler = new StatisticsHandler();
handler.setHandler(server.getHandler());
server.setHandler(handler);
log.info("Shutdown wait time: " + httpSetting.getShutdownWaitTime() + "s");
server.setStopTimeout(httpSetting.getShutdownWaitTime());
// We will stop it through JettyGracefulShutdown class.
server.setStopAtShutdown(false);
}
}
});
return factory;
}
@Bean
public JettyGracefulShutdown jettyGracefulShutdown() { return new JettyGracefulShutdown(); }
// Springboot closes application context before everything.
private static class JettyGracefulShutdown implements ApplicationListener<ContextClosedEvent>{
private static final Logger log = LoggerFactory.getLogger(JettyGracefulShutdown.class);
@Override
public void onApplicationEvent(ContextClosedEvent event) {
if (server == null) {
log.error("Jetty server variable is null, this should not happen!");
return;
}
log.info("Entering shutdown for Jetty.");
if (!(server.getHandler() instanceof StatisticsHandler)) {
log.error("Root handler is not StatisticsHandler, graceful shutdown may not work at all!");
} else {
log.info("Active requests: " + ((StatisticsHandler) server.getHandler()).getRequestsActive());
}
try {
long begin = System.currentTimeMillis();
server.stop();
log.info("Shutdown took " + (System.currentTimeMillis() - begin) + " ms.");
} catch (Exception e) {
log.error("Fail to shutdown gracefully.", e);
}
}
}
}The tomcat version didn't work as-is for me because this.connector.getProtocolHandler().getExecutor() returned null. The protocolHandler was a Http11NioProtocol.
To make it work I let the customize(Connector connector) method also set an Executor on the connectors ProtocolHandler.
@clanie: Can you post the code you did for that? We're running into a similar problem and I'd like to see a working solution if you have one.
I found this article.
http://martin.podval.eu/2015/06/java-docker-spring-boot-and-signals.html
So I want to do exec java -jar even when I install my app as init.d service.
How can I pass exec to a launch.script?
or How can I use a customized launch.script than embedded launch.script?
Is there any solutions?
Hi,
forgive me if i am mistaken.
but the the question stated the request to "waits for an empty request queue before shutting "
isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?
Thanks!
isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?
No, not really. You need to stop accepting new requests, wait for any active requests to complete, and then gracefully shut everything down. If you just wait for the number of active requests to reach 0 there's then a race condition between new requests being accepted and the shutdown completing.
@wilkinsona thanks for your response ! ... i would like to regard.
You said:
- "You need to stop accepting new requests," -> My Solution below handles it.
- "wait for any active requests to complete," -> My Solution below handles it.
Solution:
public class LimitFilter implements Filter {
public static AtomicBoolean isRunning = true;
public int count;
private Object lock = new Object();
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain) throws IOException, ServletException {
try {
if (isRunning) {
// let the request through and process as usual
synchronized (lock) {
count++;
}
try{
chain.doFilter(request, response);
}finaly{
synchronized (lock) {
count--;
}
}
} else {
//return server unavailable http status 503
}
} finally {
synchronized (lock) {
if(count==0 && !isRunning)
AppSingleton.signalSafeShutdown();
}
}
}now from shutdown hook:
@PreDestroy public void cleanup(){ LimitFilter.isRunning=false; AppSingleton.waitSafeShutdown(); }
Description:
ok now the Filter's isRunning can be controlled from listening to contextCloseEvent .....
now when the contextClosedEvent is fired we will do 2 things:
1] set LimitFilter.isRunning=false in order not recieve new requests.
2] wait for the current counter to be 0 in order to wait for current requests to finish.
i would be happy if someone sees any problem with this kind of approach rather than trying complex solution of reverting the shutdown order and handling container specific complex code.... ?
thanks.
Avishay.
@AvishayHirsh Thanks for your efforts, but you are over-simplifying the problem.
Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all. Secondly, your mechanism for counting active requests means that all requests briefly become single threaded. That will have an unacceptable impact on performance.
@wilkinsona
"Thanks for your efforts, but you are over-simplifying the problem."
Well sometimes simple approach is good enough and currently i think allot of developers can live with the points you raise here which i will regard to.
"Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all. "
That's correct , but actually its fine by me to do it like this way , but thanks for point this out !
"Secondly, your mechanism for counting active requests means that all requests become single threaded. That will have an unacceptable impact on performance."
Can you explain why ? the requests processing is not singleThreaded , only the counter inc+dec is....
@wilkinsona thanks again for having this converstion :) what do you think ?
Hello,
This is a proof of concept :
spring-boot-graceful-shutdown
Support for tomcat and undertow... and works with actuator.
Hi All,
I stumbled onto this issue and saw some of the workarounds here and came up with one that seems a bit simpler than others had mentioned, so Iโm curious if there are any downsides to this approach. Thanks for the feedback.
private final List<EmbeddedServletContainer> embeddedContainers = new ArrayList<>();
@EventListener
public synchronized void onContainerInitialized(EmbeddedServletContainerInitializedEvent event) {
embeddedContainers.add(event.getEmbeddedServletContainer());
}
@EventListener
public synchronized void onApplicationContextClose(ContextClosedEvent contextClosedEvent) {
for (EmbeddedServletContainer container : embeddedContainers) {
container.stop();
}
}Regarding a proper fix for the issue, is there any update on if we should expect this to land in the Boot 2.0 release (or sooner)?
Also, I feel like it took some luck to find this thread and I was a bit surprised that clean shutdown is not being handled properly for servlet containers. I think it would be great if the Boot documentation could point out this issue and some of the workarounds.
@dfjones The doesn't give inflight requests a grace period to complete and isn't much different to what already happens when the container is stopped as part of the context closing.
Regarding a proper fix for the issue, is there any update on if we should expect this to land in the Boot 2.0 release (or sooner)?
As shown by its milestone, the issue is currently targeted at 2.0.
@wilkinsona Would a grace period for inflight requests be dependent on the servlet container impl? We are using Jetty. Taking a look at what they do when stop is called it seems like we are getting the desired graceful shutdown behavior (Jetty appears to be waiting for inflight requests to finish after closing the network connector).
For a little more detail on the issue we were seeing, without a workaround like what I posted, it appears that the container was not being stopped until after the context was already closed. It manifested in exceptions like the following resulting in HTTP 500s returned:
javax.servlet.ServletException: A MultiException has 1 exceptions. They are:
1. org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'resource': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:408)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:583)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:524)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:461)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
How should I go about gracefully shutdown a spring boot app with multiple connectors like the one implemented here?
The shutdown of the embedded tomcat works fine (the HTTPS connector), but if I set up the additional connector in this way:
@Bean public EmbeddedServletContainerFactory servletContainer() {
Connector connector = new Connector("org.apache.coyote.http11.Http11NioProtocol");
connector.setPort(httpPort);
TomcatEmbeddedServletContainerFactory tomcat = new TomcatEmbeddedServletContainerFactory();
tomcat.addAdditionalTomcatConnectors(connector);
tomcat.addConnectorCustomizers(gracefulShutdown());
return tomcat;
}
Spring never calls customize on this connector.
Is there another way of doing this?
@MarcoGnecchi You'd need to change GracefulShutdown so that it's aware of your additional connector in addition to the one that'll be passed into its customize method.
@wilkinsona Can you suggest an option for gracefulshutdown using Undertow. I see that there is a GracefulShutdownHandler but not sure how to add it to a Spring boot app.
@rdramana85 You'd need to use an UndertowBuilderCustomzer and Undertow's standard API for handler configuration to add it to a Spring Boot app.
@wilkinsona Is it possible to shutdown Tomcat gracefully in the case Spring Application run failure?
I don't think that's necessary. If the app fails to start there will be no in-flight requests as the containers connectors aren't started till the app is running.
Is this still planned for 2.0.0 as 2.0.0.M1 is out?
The undertow graceful shutdown returns 503.
Tomcat Endpoint#Pause stops to accept and process connection but the socket is not closed and the connections are stalled waiting to be accepted. Thus if the application shutdown takes some time, these connections/requests are stalled until tomcat is stopped and then refused. Actually Endpoint#Pause does not seems viable.
Returning 503 seems a common approach handled by most if not all http proxy.
Is this still planned for 2.0.0 as 2.0.0.M1 is out?
Yes. The issue is still assigned to the 2.0.0 milestone.
Actually Endpoint#Pause does not seems viable.
Why do you think that it isn't viable?
When Tomcat's StandardService is stopped it pauses the connectors, stops the underlying Engine (which will wait for requests to complete), and then stops the connectors. Tomcat and Undertow may take different approaches, but, given Tomcat's vast adoption, I think it's unlikely that its approach is not viable.
Optimal may be more appropriate than viable.
When Tomcat's StandardService is stopped it pauses the connectors
This is the main concern. For what I have understand after testing and reading the tomcat codes when the http connector is paused, new connections/requests are stalled until tomcat is stopped and then refused. It may be an issue if the application shutdown takes some time, even for 5s.
Maybe do you plan to improve the Tomcat behaviour?
Maybe do you plan to improve the Tomcat behaviour?
No, I don't think so. I think we should defer to the Tomcat team's expertise as it's highly likely that they know more about shutting Tomcat down gracefully and how that should behave than we do. If you believe Tomcat can/should be improved, then please raise the matter on the Tomcat Users mailing list.
I agree.
Could you please share how you planned to implement this ticket?
Could you please share how you planned to implement this ticket?
Nothing's been planned beyond what's been sketched out by various people in this issue.
In your opinion is it something that make sense:
- At startup a
GracefulShutdownFilterservlet filter is registered (like undertow one) which keeps track of active requests - The context shutdown is triggered
- The
GracefulShutdownFilterfilter returns http status 503 for all new requests - When all accepted requests before the shutdown are all completed, the
GracefulShutdownFilter#shutdownCompleteboolean is set to true - As late as possible and only if
GracefulShutdownFilter#shutdownCompleteis true, tomcat is stopped.
This solution has already been proposed with the following comment :
Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all.
Is it always your opinion taking into account Connector#pause does not immediatly stops accepting new requests (ie. with a connection refused) but it stalls them until the connector is closed?
For what it's worth, we are using the workaround for Jetty in our application now to get the ability to "stop" the connector accepting new requests, while allowing the existing ones to finish. If this feature is resolved with a filter that 503s back requests, we're not interested and would continue using the horrible workaround instead. Our service routing relies on the ability of our applications to cease accepting connections and having it return 503 instead would be a step backwards for us.
Maybe this isn't viable with Tomcat, but Jetty supports it out of the box, and the Tomcat-centric filter described above sounds like a strict step backwards from Jetty's behavior.
@stevenschlansker Historically, our approach with this sort of thing has been to use whatever each container's native way of handling things has been. For example, access logging uses whatever Jetty, Tomcat, and Undertow provide (with slightly different configuration options and functionality) rather than us rolling our own solution across all three containers. I am almost certain that we'll follow the same approach here.
Is it always your opinion taking into account Connector#pause does not immediatly stops accepting new requests (ie. with a connection refused) but it stalls them until the connector is closed?
@nithril As I touched on in my comment above this one, it's my opinion that the best option is for us to build upon each container's existing support. Right now, for Tomcat, that means that we would pause the connector as this is what standalone Tomcat does.
@wilkinsona Ok I understand, thanks for your feedback.
@stevenschlansker The filter I wrote about was only for Tomcat
@thefallentree solution is good, however sometimes I get this exception while testing the graceful shutdown with a pending connection:
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.resolvePreparedArguments(ConstructorResolver.java:785) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:415) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.cloud.sleuth.instrument.web.TraceHandlerInterceptor.getErrorController(TraceHandlerInterceptor.java:194) ~[spring-cloud-sleuth-core-1.2.1.RELEASE.jar:1.2.1.RELEASE]
at org.springframework.cloud.sleuth.instrument.web.TraceHandlerInterceptor.isErrorControllerRelated(TraceHandlerInterceptor.java:83) ~[spring-cloud-sleuth-core-1.2.1.RELEASE.jar:1.2.1.RELEASE]
at org.springframework.cloud.sleuth.instrument.web.TraceHandlerInterceptor.afterCompletion(TraceHandlerInterceptor.java:132) ~[spring-cloud-sleuth-core-1.2.1.RELEASE.jar:1.2.1.RELEASE]
at org.springframework.web.servlet.HandlerExecutionChain.triggerAfterCompletion(HandlerExecutionChain.java:170) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1055) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:980) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:206) [websocket-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) [spring-boot-actuator-1.5.3.RELEASE.jar:1.5.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at br.combbbbb.aaaaa.backend.filter.AuthenticationGate.doFilter(AuthenticationGate.java:40) [classes/:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at br.combbbbb.aaaaa.backend.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:41) [classes/:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at br.combbbbb.aaaaa.backend.filter.dev.SwaggerQueryStringAuthenticationFilter.doFilter(SwaggerQueryStringAuthenticationFilter.java:38) [classes/:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:153) [spring-cloud-sleuth-core-1.2.1.RELEASE.jar:1.2.1.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at br.combbbbb.aaaaa.backend.service.RequestLogging.doFilterInternal(RequestLogging.java:44) [classes/:na]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at io.prometheus.client.filter.MetricsFilter.doFilter(MetricsFilter.java:172) [simpleclient_servlet-0.0.22.jar:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at br.combbbbb.aaaaa.backend.filter.HttpStatusMetricFilter.doFilter(HttpStatusMetricFilter.java:41) [classes/:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.Server.handle(Server.java:564) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [jetty-io-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128) [jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222) [jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294) [jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) [jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) ~[jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) ~[jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.cache.config.internalCacheAdvisor' defined in class path resource [org/springframework/cache/annotation/ProxyCachingConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cache.interceptor.BeanFactoryCacheOperationSourceAdvisor]: Factory method 'cacheAdvisor' threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.autoproxy.BeanFactoryAdvisorRetrievalHelper.findAdvisorBeans(BeanFactoryAdvisorRetrievalHelper.java:92) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.autoproxy.AbstractAdvisorAutoProxyCreator.findCandidateAdvisors(AbstractAdvisorAutoProxyCreator.java:102) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.aspectj.annotation.AnnotationAwareAspectJAutoProxyCreator.findCandidateAdvisors(AnnotationAwareAspectJAutoProxyCreator.java:88) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.aspectj.autoproxy.AspectJAwareAdvisorAutoProxyCreator.shouldSkip(AspectJAwareAdvisorAutoProxyCreator.java:103) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.wrapIfNecessary(AbstractAutoProxyCreator.java:340) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.postProcessAfterInitialization(AbstractAutoProxyCreator.java:298) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:423) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1633) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
... 100 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cache.interceptor.BeanFactoryCacheOperationSourceAdvisor]: Factory method 'cacheAdvisor' threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
... 117 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.springframework.cache.annotation.ProxyCachingConfiguration.cacheAdvisor(ProxyCachingConfiguration.java:47) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.cache.annotation.ProxyCachingConfiguration$$EnhancerBySpringCGLIB$$300d9692.CGLIB$cacheAdvisor$0(<generated>) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.cache.annotation.ProxyCachingConfiguration$$EnhancerBySpringCGLIB$$300d9692$$FastClassBySpringCGLIB$$4af7b287.invoke(<generated>) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[spring-core-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at org.springframework.cache.annotation.ProxyCachingConfiguration$$EnhancerBySpringCGLIB$$300d9692.cacheAdvisor(<generated>) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
... 118 common frames omitted
2017-06-13 23:38:17.821 INFO [aaaaa_backend,,,] 6267 --- [tp1364765175-20] b.c.u.z.backend.service.RequestLogging : After request [method=GET;uri=/api/v1/sleep?UOLSER=1;status=500;client=127.0.0.1;took=5087]
2017-06-13 23:38:17.826 INFO [aaaaa_backend,,,] 6267 --- [ Thread-22] o.e.jetty.server.AbstractConnector : Stopped ServerConnector@2e446168{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-06-13 23:38:17.826 INFO [aaaaa_backend,,,] 6267 --- [ Thread-22] org.eclipse.jetty.server.session : Stopped scavenging
Disconnected from the target VM, address: '127.0.0.1:41102', transport: 'socket'
2017-06-13 23:38:17.830 INFO [aaaaa_backend,,,] 6267 --- [ Thread-22] o.e.j.s.h.ContextHandler.application : Destroying Spring FrameworkServlet 'dispatcherServlet'
2017-06-13 23:38:17.831 INFO [aaaaa_backend,,,] 6267 --- [ Thread-22] o.e.jetty.server.handler.ContextHandler : Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@1dd17c0c{/,[file:///tmp/jetty-docbase.1191171853553345825.8080/, jar:file:/home/joaop/.m2/repository/io/springfox/springfox-swagger-ui/2.6.1/springfox-swagger-ui-2.6.1.jar!/META-INF/resources],UNAVAILABLE}```
if this ever gets implemented, please consider the case for a delayed graceful shutdown with a hook linking back to the application. My usecase: i want to shutdown one node of a clustered spring-boot app but first signal to the load balancer in front of it that this particular node should no longer be used. So i would need to delay any shutdown actions until i am sure that the LB will no longer route requests to this node.
Uh..i have some ideas.
If we want shutdown the server gracefully, we should do the following:
- Return the end signal, but the server hasn't stop actually. If this can do, the clustered will think this node is CLOSED;
- Reject all requests;
- Wait active requests all completed;
- Set the maximum Expiration-Time for the server to be shutdown.
These are my understanding, may not be comprehensive. Hope these help!
@jorgheymans
Seconded.
It gets even more complicated. For example prometheus.io (monitoring) scrapes an HTTP endpoint in the application to get monitoring data. Shutting down the connector, will also prevent the monitoring system to get the metrics and statistics.
In Openshift, a service running in a container signals the infrastructure (like loadbalancer) its readyness to accept requests by returning a 200 OK response to a configurable URL. Returning something else like 500 makes the infrastructure forward requests to this instance. In this scenario it makes sense to shut down the readyness endpoint or make it return != 200, but let everything else running until the input queue is empty or a max grace period expires.
I understand that this implementation will not be standard spring boot behaviour, but there should be hooks in the shutdown machinery to allow something like this to be implemented
I would really like to see a solution in version 2.0, because it's an important production feature.
Please consider rescheduling to a 2.0 milestone release to get feedback from users in time.
We use Openshift to manage our containers and wanted a zero downtime deployment. So we implemented a Springboot Actuator Healthcheck. To Listen to the Sigterm Signal we implemented our own Springboot Starter. Once the JVM sends the Shutdownevent, the Shutdownlistener set the Healthcheck to false and some configured seconds later initiate the Springcontext Shutdown. We testet this graceful shutdown behavior hook with Openshift. Details you find here. The Jar is published in the public sonatype maven repo.
https://github.com/SchweizerischeBundesbahnen/springboot-graceful-shutdown
I had the same issues with OpenShift.
Thank you all for sharing your code examples and ideas.
I prefer the approach of @nithril to return HTTP Code 503 during shutdown process.
It's much faster than waiting for the OpenShift health check to become unhealthy.
In my oppinion OpenShift should add an option to remove the pod from service before sending the SIGTERM. But currently shutting down the HTTP Connector is the recommended solution: https://docs.openshift.com/container-platform/3.6/dev_guide/deployments/advanced_deployment_strategies.html#graceful-Termination
Considering this I created an autoconfigured SpringBoot-Application module with:
- Application-Context initializer and ShutdownHandler
- HTTPFilter to count the requests and return 503 when shutting down.
Please find the sources here: https://github.com/ramato-procon/springboot-graceful-terminator
It's not yet production ready, but I would appreciate your review/comment.
@ramato-procon i sugguested the same approach :) (under avishay username)
is it the idea same as you implemented ?
@robocide Yes it is the same idea. The filter counts each running request (but with an AtomicFieldUpdater, which is better in performance than synchronize). It returns 503, when the shutdown process has been started:
https://github.com/ramato-procon/springboot-graceful-terminator/blob/master/src/main/java/de/ra/springboot/graceful/GracefulHttpFilter.java
The shutdown process is started by the GracefulShutdownHook, which first inits the shutdown process in the Filter and waits until all running requests have been processed.
In the next step the http connector will be stopped.
Afterwards the ApplicationContext will be closed: https://github.com/ramato-procon/springboot-graceful-terminator/blob/master/src/main/java/de/ra/springboot/graceful/GracefulShutdownHook.java
This is a troublesome thing, waiting for shutdown.
Having discussed this with a couple of people at Spring IO last week, it occurred to me that this may be better handled as a Spring Framework feature or at least building upon a Spring Framework feature. There's nothing Boot-specific about allowing Spring-managed components to be shut down gracefully and the concept being part of the Framework would allow other projects in the portfolio to provide graceful shutdown integration without creating a circular dependency between those projects and Boot. @snicoll is going to raise it with the Framework team.
FWIW, I think this is a feature specific to Spring Boot, and doesn't need any Spring Framework support.
The entity which is being "gracefully shut down" here is not necessarily the entire Spring app context, but could just be the Spring Boot webapp. What is needed is a way for the Spring boot app to ask its EmbeddedServletContainer to stop accepting new requests, then to trigger a normal Spring shutdown when all in-flight requests have completed.
Since the TomcatEmbeddedServletContainer lives in the org.springframework.boot:spring-boot jar, this seems like a Spring Boot and/or a Tomcat feature request, rather than a Spring Framework feature.
I don't think there is currently any way to either a) ask Tomcat to stop accepting new requests without cancelling in-progress requests ("Pause" has problems, see above) or b) to ask Tomcat if there are any requests in progress, is there?
The thread above has gotten sidetracked into discussions of Spring-managed components dependency orders etc., which I think is not necessary for the desired feature.
I don't think there is currently any way to either a) ask Tomcat to stop accepting new requests without cancelling in-progress requests or b) to ask Tomcat if there are any requests in progress, is there?
Both already possible, as shown by code that has already been shared above.
The thread above has gotten sidetracked into discussions of Spring-managed components dependency orders etc., which I think is not necessary for the desired feature.
We disagree. If we are to introduce the concept of graceful shutdown it's important that we introduce it in the right place. This applies irrespective of exactly what it is that's being gracefully shut down. We do not want to get into the situation where we introduce something that is specific to Spring Boot and its embedded container only to realise that a more general solution should have been introduced in Spring Framework and then built upon by Boot.
Both already possible, as shown by code that has already been shared above.
As per nithril's comment that's not a graceful shutdown, as the server continues to accept requests during the shutdown period, but stalls them and fails them all at once at the end, instead of unbinding from the socket which is what I understand by "graceful shutdown", like apache httpd does
Perhaps that is a Tomcat bug, rather than a Spring Boot one, though.
This applies irrespective of exactly what it is that's being gracefully shut down.
In my opinion, this is premature generalisation. There is a real and immediate need for Spring Boot webapps to have graceful HTTP shutdown, which it is not currently possible to meet, as seen in this thread. I don't see an immediate need for a generalised Spring Component shutdown ordering framework (and as noted above, there is already such a thing to some extent via DisposableBean). It seems like it might be easier to fix the current specific case before starting a big rethink about the proposed general case.
Regardless of whether we do a generalised shutdown framework or a specific HTTP one, there is some integration work to be done in adding a graceful "unbind" command to each of the implementations of EmbeddedServletContainer. If that work were done now, the specific case would be within easy reach. I think the general case is a distraction from that.
Of course, I am just an interested bystander, so it is up to the project maintainers where to take this :-)
that's not a graceful shutdown
That's a matter of opinion and is something that's already been raised and addressed above. We're just retreading old ground here without anything new being added to the discussion which is a waste of both my time and yours.
There is a real and immediate need for Spring Boot webapps to have graceful HTTP shutdown, which it is not currently possible to meet
The comments above show that's not the case. There are multiple users who have commented above with examples of something that works for their specific needs.
It seems like it might be easier to fix the current specific case before starting a big rethink about the proposed general case.
Being easier doesn't make it the right thing to do. If we forge ahead now with something that then doesn't fit well without something that's subsequently added to Spring Framework we'll be left with a mess that will be difficult to straighten out. We need to take the time now to think things through and give us the best possible chance to get things right first time.
With Jetty, I think things can be a bit simpler than the example shared above. All you need to do is to add the StatisticsHandler:
@Bean
public WebServerFactoryCustomizer<JettyServletWebServerFactory> jettyCustomizer() {
return (factory) -> {
factory.addServerCustomizers((server) -> {
StatisticsHandler handler = new StatisticsHandler();
handler.setHandler(server.getHandler());
server.setHandler(handler);
});
};
}This is sufficient to switch on Jetty's graceful shutdown with its default timeout period of 30 seconds. To customise the timeout, a call to server.setStopTimeout(timeInMillis) can be made in the same server customizer. Closing the application context will take care of calling Server.stop().
Jetty's behaviour during graceful shutdown is as follows:
- New connections are refused
- Wait for up to the shutdown timeout period for the number of active requests to drop to zero
- Drop the connections for any requests that are still active
Here's the equivalent configuration for Undertow:
@Bean
public GracefulUndertowShutdown GracefulUndertowShutdown() {
return new GracefulUndertowShutdown();
}
@Bean
public WebServerFactoryCustomizer<UndertowServletWebServerFactory> undertowCustomizer(
GracefulUndertowShutdown gracefulUndertowShutdown) {
return (factory) -> {
factory.addDeploymentInfoCustomizers((builder) -> {
builder.addInitialHandlerChainWrapper(gracefulUndertowShutdown);
});
};
}
private static class GracefulUndertowShutdown
implements ApplicationListener<ContextClosedEvent>, HandlerWrapper {
private volatile GracefulShutdownHandler handler;
@Override
public HttpHandler wrap(HttpHandler handler) {
this.handler = new GracefulShutdownHandler(handler);
return this.handler;
}
@Override
public void onApplicationEvent(ContextClosedEvent event) {
try {
this.handler.shutdown();
this.handler.awaitShutdown(30000);
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
}Undertow's behaviour during graceful shutdown is as follows:
- Respond to new requests with a 503
- Wait for up to the shutdown timeout period for the number of active requests to drop to zero
- Drop the connections for any requests that are still active
With Jetty, I think things can be a bit simpler than the example shared above.
I take this back.
I said above that " closing the application context will take care of calling Server.stop()". That's true, but it'll get Jetty into a state where it rejects new connections after most of the context close processing has occurred. It needs to be happen earlier and when ContextRefreshedEvent is published is an ideal time to do so.
Having thought some more, I think it's actually a bit more complicated than the example shared above. We need to get Jetty into a state where it rejects new connections without actually calling Server.stop(). If we call Server.stop(), we may encounter problems that are similar to those that we encountered when we tried to invert the ordering of context close and server stop.
In my opinion there is only one logic which should work for all kind of loadbalancers:
- Stop accepting new requests (new connections or channels are refused)
- Waiting until all requests have been processed or shutdown timeout
- Shutdown Spring context
Responding with 503 only works for "real" http loadbalancers. No iptables or tcp (e.g. for ssl passthrough) loadbalancing.
@wilkinsona: does the first example works correctly with asynchronous processing? There can be pending requests and no busy executor threads. So, unless there is special logic on the executor, the shutdown could succeed even with pending requests.
@pmhsfelix I don't know yet. @snicoll and I discussed it briefly last week and it's something that I need to investigate for all our supported containers.
@wilkinsona could you update example for #4657 (comment) with code for Spring Boot 2?
it seems the relevant classes have been renamed and moved to different packages in Spring boot 2.
Jetty does not consider asynchronous requests when shutting down gracefully. I've opened jetty/jetty.project#2717.
Update : ---IGNORE THIS COMMENT---
I am not sure if this a widely faced problem or not but what I am facing is, we have a kafka stream processor running as a spring boot app(parent : spring-boot-starter-parent) inside a docker container.
As soon as we do a docker stop on container, we are observing 503s for some of the outbound calls to external REST api(using resttemplates).
Has there been any workarounds or plans to release a spring-boot version with a fix?
@buch11 Please donโt drag this issue off-topic. Outbound HTTP calls in a Docker container are unrelated to graceful shutdown of the embedded container.
@pdeva - Here's an updated snippet for Spring Boot 2:
@Bean
public WebServerFactoryCustomizer tomcatCustomizer() {
return factory -> {
if (factory instanceof TomcatServletWebServerFactory) {
((TomcatServletWebServerFactory) factory)
.addConnectorCustomizers(gracefulShutdown());
}
};
}
private static class GracefulShutdown implements TomcatConnectorCustomizer,
ApplicationListener<ContextClosedEvent> {
private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);
private volatile Connector connector;
@Override
public void customize(Connector connector) {
this.connector = connector;
}
@Override
public void onApplicationEvent(ContextClosedEvent event) {
Executor executor = this.connector.getProtocolHandler().getExecutor();
if (executor instanceof ThreadPoolExecutor) {
try {
ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
threadPoolExecutor.shutdown();
if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
log.warn("Tomcat thread pool did not shut down gracefully within "
+ "30 seconds. Proceeding with forceful shutdown");
}
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
}
}
There are unnecessary listener works for GracefulShutdown ,as written above , many application contexts can trigger the ContextClosedEvent
How to do it with netty sever
I do it with webflux server reactor-undertow ,it does not work.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-reactor-netty</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-undertow</artifactId>
</dependency> public class UndertowGracefulConfig {
/**
* https://github.com/spring-projects/spring-boot/issues/4657
*/
@Bean
public GracefulUndertowShutdown gracefulUndertowShutdown() {
return new GracefulUndertowShutdown();
}
@Bean
public WebServerFactoryCustomizer<UndertowReactiveWebServerFactory> undertowCustomizer(
GracefulUndertowShutdown gracefulUndertowShutdown) {
return (factory) -> {
factory.addDeploymentInfoCustomizers((builder) -> {
builder.addInitialHandlerChainWrapper(gracefulUndertowShutdown);
});
};
}
private static class GracefulUndertowShutdown implements ApplicationListener<ContextClosedEvent>, HandlerWrapper {
private volatile GracefulShutdownHandler handler;
@Override
public HttpHandler wrap(HttpHandler handler) {
this.handler = new GracefulShutdownHandler(handler);
return this.handler;
}
@Override
public void onApplicationEvent(ContextClosedEvent event) {
try {
this.handler.shutdown();
this.handler.awaitShutdown(30000);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
}
}when shutdown server with 'CTRL +C',exception throw
Thread-4] onfigReactiveWebServerApplicationContext : Exception thrown from ApplicationListener handling ContextClosedEvent
java.lang.NullPointerException: null
at com.example.springboot.demo.config.UndertowGracefulConfig$GracefulUndertowShutdown.onApplicationEvent(UndertowGracefulConfig.java:47) ~[classes!/:na]
at com.example.springboot.demo.config.UndertowGracefulConfig$GracefulUndertowShutdown.onApplicationEvent(UndertowGracefulConfig.java:1) ~[classes!/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:398) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:355) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:994) ~[spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:931) [spring-context-5.1.3.RELEASE.jar!/:5.1.3.RELEASE]
2018-12-23 11:39:23.724 INFO 11152 --- [ Thread-4] org.quartz.core.QuartzScheduler : Scheduler clusteredScheduler_$_bing-work1545536356420 paused.
2018-12-23 11:39:23.729 INFO 11152 --- [ Thread-4] o.s.s.quartz.SchedulerFactoryBean : Shutting down Quartz Scheduler
2018-12-23 11:39:23.731 INFO 11152 --- [ Thread-4] org.quartz.core.QuartzScheduler : Scheduler clusteredScheduler_$_bing-work1545536356420 shutting down.
2018-12-23 11:39:23.732 INFO 11152 --- [ Thread-4] org.quartz.core.QuartzScheduler : Scheduler clusteredScheduler_$_bing-work1545536356420 paused.
2018-12-23 11:39:23.733 INFO 11152 --- [ Thread-4] org.quartz.core.QuartzScheduler : Scheduler clusteredScheduler_$_bing-work1545536356420 shutdown complete.
2018-12-23 11:39:23.735 INFO 11152 --- [ Thread-4] com.zaxxer.hikari.HikariDataSource : mysql - Shutdown initiated...
2018-12-23 11:39:23.738 INFO 11152 --- [ Thread-4] com.zaxxer.hikari.HikariDataSource : mysql - Shutdown completed.
2018-12-23 11:39:41.990 ERROR 11152 --- [ XNIO-1 I/O-3] io.undertow.request : UT005071: Undertow request failed HttpServerExchange{ GET /long-process request {Postman-Token=[a6556ff1-b008-4eb9-b61a-0d6e78b730ce], Accept=[*/*], Connection=[keep-alive], cache-control=[no-cache], accept-encoding=[gzip, deflate], User-Agent=[PostmanRuntime/7.4.0], Host=[localhost:8080]} response {Connection=[keep-alive], Content-Type=[text/plain;charset=UTF-8], Content-Length=[16], Date=[Sun, 23 Dec 2018 03:39:41 GMT]}}
java.util.concurrent.RejectedExecutionException: XNIO007007: Thread is terminating
at org.xnio.nio.WorkerThread.execute(WorkerThread.java:590) [xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]
at io.undertow.server.HttpServerExchange$ReadDispatchChannel.invokeListener(HttpServerExchange.java:2166) ~[undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at io.undertow.server.HttpServerExchange$ReadDispatchChannel.runResume(HttpServerExchange.java:2387) ~[undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at io.undertow.server.HttpServerExchange.runResumeReadWrite(HttpServerExchange.java:1860) ~[undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387) ~[undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255) [undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136) [undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162) [undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100) [undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57) [undertow-core-2.0.16.Final.jar!/:2.0.16.Final]
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291) [xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:129) [xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:582) [xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:466) [xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]
@wilkinsona (or others), I'm wondering if you can answer something about the spring-boot/Tomcat case. I completely understand the need to stop accepting new connections [i.e. connector.pause()] before allowing the application context to be stopped. But what is the significance of waiting for the tomcat thread pool to shut down? How necessary is it. Assume a spring-boot/Tomcat service that does nothing to implement graceful shutdown. Ignoring new connection attempts, does a SIGTERM result in the application context being stopped even before in-flight requests are finished being handled? It surprised me a little to learn that that we need to takes steps to prevent new connections during shutdown. But it would surprise me a great deal more to think that spring-boot shutdown doesn't even deal properly with in-flight requests. Thanks for your help!
As explained above, the application context is stopped and then the embedded container is closed. This means that the context may be closed while requires are in-flight. When Tomcat is the embedded container, it will, by default, wait for 5 seconds for in-flight requests to complete before it closes the connection. If you don't want the application context to be closed until in-flight requests have completed, the example above shows roughly what needs to be done.
@wilkinsona thanks for the quick response! How unfortunate. I really look forward to a solution arriving within spring-boot. In the meantime, I hope you don't mind one more question: we have found that with async requests, the protocol handler's executor might be empty even in the face of in-flight requests. To deal with this we first wait for the protocol handler's connection count to drop to 0 and then follow with the same logic you've used. Any thoughts on this addition?
That's where things start getting tricky, and is one reason why we've yet to implement something.
Tomcat will timeout all async requests by calling timeoutAsync(-1) on the endpoint's underlying processor. You may want to consider doing something similar or you may want to wait a while for them to complete.
There's also keep-alive connections to consider. They could prevent the connection count from ever dropping to 0 and you'd end up having to close the socket to drop the connections. I'd like to investigate the possibility of causing the container to send Connection: close in the response and then close the connection once graceful shutdown has begun. Unfortunately, I've yet to have time to do so across all the containers that we support.
@wilkinsona what a tangled mess :-)
For what it's worth, I did some testing on keep-alive and it appears that after the pause(), any subsequent request on a keep-alive connection results in a close. Of course, that doesn't help if there are no subsequent requests on the keep-alive connection.
@wilkinsona could you update example for #4657 (comment) with code for Spring Boot 2?
it seems the relevant classes have been renamed and moved to different packages in Spring boot 2.
the following code work for me in spring-boot-starter-web -> 2.1.3.RELEASE with the default embedded container tomcat
@Bean
public GracefulShutdownTomcat gracefulShutdownTomcat() {
return new GracefulShutdownTomcat();
}
@Bean
public ServletWebServerFactory servletContainer(GracefulShutdownTomcat gracefulShutdownTomcat) {
TomcatServletWebServerFactory tomcat = new TomcatServletWebServerFactory();
tomcat.addConnectorCustomizers(gracefulShutdownTomcat);
return tomcat;
}
private static class GracefulShutdownTomcat implements TomcatConnectorCustomizer,
ApplicationListener<ContextClosedEvent> {
private static final Logger log = LoggerFactory.getLogger(GracefulShutdownTomcat.class);
private volatile Connector connector;
@Override
public void customize(Connector connector) {
this.connector = connector;
}
long timeWait = 30;
@Override
public void onApplicationEvent(ContextClosedEvent event) {
this.connector.pause();
Executor executor = this.connector.getProtocolHandler().getExecutor();
if (executor instanceof ThreadPoolExecutor) {
try {
ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
threadPoolExecutor.shutdown();
if (!threadPoolExecutor.awaitTermination(timeWait, TimeUnit.SECONDS)) {
log.warn("Tomcat thread pool did not shut down gracefully within " + timeWait + " seconds. Proceeding with forceful shutdown");
}
}
catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
}
}@wilkinsona I did what you said before.when connector.pause, any new request will be stocked but not fail fast.Is there any method that can make new request fail fast?
You can either leave the connector running and reject the requests in a filter by returning a 5xx response. Alternatively you could stop the connector rather than pausing it, but that will prevent in-flight requests from completing.
You can either leave the connector running and reject the requests in a filter by returning a 5xx response. Alternatively you could stop the connector rather than pausing it, but that will prevent in-flight requests from completing.
Thanks for your answer!I think returning a 5xx response isn't a good choice,it's just a feak pause.But i asked my PHP worker, swoole can really graceful shutdown progress with new request fail fast.But we still don't know how does swoole works.
How to do it with netty sever
In spring boot 2.1.5 with reactor-netty 0.8.8, I found the netty graceful shutdown failed because of the LoopResource just subscibe the disposeLater instead of block it.
//reactor.netty.resources.LoopResources#dispose
@Override
default void dispose() {
//noop default
disposeLater().subscribe();
}- so, i use reflect to call block manually, it works
@Component
public class GracefulShutdown {
@Autowired
ReactorResourceFactory reactorResourceFactory;
LoopResources loopResources;
@PostConstruct
void init() throws Exception {
Field field = TcpResources.class.getDeclaredField("defaultLoops");
field.setAccessible(true);
loopResources = (LoopResources) field.get(reactorResourceFactory.getLoopResources());
field.setAccessible(false);
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
System.out.println("Graceful: block long to 20s before real shutdown!");
loopResources.disposeLater().block(Duration.ofSeconds(20));
}));
}
}Is there any method that can make new request fail fast?
@codingman1990 You can try closing the ServerSocket after pause the connector.
connector.getProtocolHandler().closeServerSocketGraceful();
@wilkinsona I did what you said before.when connector.pause, any new request will be stocked but not fail fast.Is there any method that can make new request fail fast?
You can either leave the connector running and reject the requests in a filter by returning a 5xx response. Alternatively you could stop the connector rather than pausing it, but that will prevent in-flight requests from completing.
Hello, this post has been extremely helpful, I just have one question. I'm following your walk through, but how/ where do I return a 5xx response? I tried adding a ResponseStatus annotation with a 5xx HttpStatus to the onApplicationEvent method, although I can still make requests, hence its not properly returning a 5xx status
Hello everyone. I'm not good at English so I hope you understand it.
When using tomcat on Spring Boot, connector.pause() didn't seem to work as I wanted.
What I wanted was a situation where kill would start throwing an error immediately when a request came to the server.
So I directly touched protocolHandler like below. When I did this I confirmed that it worked as I wanted. But I'd like to ask for your opinion that this code will cause side effects.
Please comment.
public class GracefulShutdown implements TomcatConnectorCustomizer, ApplicationListener<ContextClosedEvent> {
private volatile Connector connector;
private static final String protocolHandlerClassName = "org.apache.coyote.http11.Http11NioProtocol";
@Override
public void customize(Connector connector) {
this.connector = connector;
}
@Override
public void onApplicationEvent(ContextClosedEvent contextClosedEvent) {
// this.connector.pause();
this.protocolHandlerClose();
Executor executor = this.connector.getProtocolHandler().getExecutor();
if (executor instanceof ThreadPoolExecutor) {
ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
threadPoolExecutor.shutdown();
try {
if (!threadPoolExecutor.awaitTermination(60, TimeUnit.SECONDS)) {
threadPoolExecutor.shutdownNow();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
private void protocolHandlerClose() {
try {
Class<?> clazz = Class.forName(protocolHandlerClassName);
ProtocolHandler p = (ProtocolHandler) clazz.getConstructor().newInstance();
p.pause();
} catch (Exception e) {
e.printStackTrace();
}
}
}Hi,
Thanks for the code references. I was able to make my application stop accepting new requests and wait for threads to complete processing.
However, we deploy our services on Kubernetes, and I also want that the HTTP-requests on management port are also rejected, so the the readiness probes from Kubernetes to my service fails and it stops routing traffic to the pod on which my service is running.
I tried going through the Class ManagementWebServerFactoryCustomizer and make a object out of it. But my service shuts down with the error that the management port is already in use.
Any hints on how to go about getting this requirement fulfilled ?
@pradeepanchan if the goal is to prevent the application from getting new requests, then k8s stopping routing traffic to the application is a good thing? The alternative would be that k8s routes traffic to the instance but it is refusing connections. Maybe I misunderstood the actual requirement?
@bclozel , the problem is that the pausing server.port or shutting down the ThreadPoolExecutor only ensures that requests going to the main port is paused or rejected. Requests sent to the management port (management.server.port) at actuator/health keep getting an "UP" status, until the service shuts down completely. Because of this, k8s will keep routing requests to this instance which is shutting down and the clients will keep noticing a failure (which i feel is unnecessary). I think the better option would be that the k8s will route the requests to the alternative pod which is still running.
The only way to get this done is that the instance shutting down starts failing the requests arriving at the management.server.port.
One workaround is to combine the management.server.port and server.port to be the same, but I don't think the architects in my organization would agree to that.
That's another example of the complexity of graceful shutdown. The behaviour that you want typically depends on the intelligence of the routing mechanism that's sitting in front of the app.
When using Kubernetes and HTTP endpoints for liveness and readiness checks, the readiness check needs to start failing when you initiate the graceful shutdown but the liveness check should continue to succeed, either until the app has stopped itself or until you're reading for Kubernetes to kill the pod.
If you are running the application and the management server on the same port, you don't want to stop accepting requests when graceful shutdown is initiated as the liveness check would fail. Instead, you want the readiness probe to start failing so that Kubernetes no longer routes traffic to the pod. Once the graceful shutdown has completed or a timeout period has elapsed, the liveness check can then start to fail so that the pod is killed. This is something that we hope to support out of the box in the future. At the moment I would recommend using a custom Actuator endpoint to achieve this behaviour.
If your application is using separate ports for the management server and the application then you could stop the application from handling requests and allow the management server to continue to do so. You would still need a custom health indicator to cause the application to appear alive but not ready until the graceful shutdown has completed or taken too long.
That's another example of the complexity of graceful shutdown. The behaviour that you want typically depends on the intelligence of the routing mechanism that's sitting in front of the app.
When using Kubernetes and HTTP endpoints for liveness and readiness checks, the readiness check needs to start failing when you initiate the graceful shutdown but the liveness check should continue to succeed, either until the app has stopped itself or until you're reading for Kubernetes to kill the pod.
You are right about this. But we can take care of it, by ensuring that the frequency and failure threshold of the liveness probe is larger than that of the shutdown wait time. And those of the readiness probe is shorter than that of the shutdown wait time.
there are 2 project that can help with that:
- https://github.com/timpeeters/spring-boot-graceful-shutdown
- https://github.com/SchweizerischeBundesbahnen/springboot-graceful-shutdown
there are not ideal but with appropriate configuration of health check groups (spring boot 2.2) it should cover most cases
there are 2 project that can help with that:
* https://github.com/timpeeters/spring-boot-graceful-shutdown * https://github.com/SchweizerischeBundesbahnen/springboot-graceful-shutdownthere are not ideal but with appropriate configuration of health check groups (spring boot 2.2) it should cover most cases
Thanks @kamkie , i got the hint :-)
When using Kubernetes and HTTP endpoints for liveness and readiness checks, the readiness check needs to start failing when you initiate the graceful shutdown but the liveness check should continue to succeed, either until the app has stopped itself or until you're reading for Kubernetes to kill the pod.
I now believe this is incorrect.
When Kubernetes is taking a pod out of service it will send the SIGTERM signal to it. This will cause a Spring Boot application to start shutting down. At this point, the liveness and readiness probes are no longer called.
In parallel, Kubernetes will also initiate the sequence of events that results in requests no longer being routed to the application. The parallel and eventual nature of this means that there is a period of time when the application has received SIGTERM but requests are still routed to it. Ideally, these requests should be handled so that clients do not become aware of the application being shut down.
The recommended way to minimise or perhaps even eliminate the window where requests are still received after SIGTERM is to sleep in a pre-stop hook. In the event of the window only being minimised, continuing to accept new requests during graceful shutdown should reduce the number of errors that are seen by clients. Such errors cannot be eliminated entirely as, if there are no active requests, the graceful shutdown may complete and stop accepting new connections while Kubernetes contains to route requests to the application.
We may need to provide a configurable strategy for graceful shutdown. One that rejects new connections as soon as graceful shutdown begins and one that accepts them. The former could be used in environments where the routing change is guaranteed to happen before graceful shutdown begins and the latter in environments like Kubernetes where the routing change and graceful shutdown run in parallel.
We discussed the need for different strategies for handling requests during graceful shutdown, and decided not to do anything at this time. As explained above, it is impossible to completely close the window on the application's side so it is hard to justify adding complexity that attempts to do so, particularly for a problem that is particular to Kubernetes. In that deployment environment, our recommendation will be to sleep for a period in a pre-stop hook. The length of that period will vary from deployment to deployment, depending on the needs of the application and the time it takes for Kubernetes to propagate the routing change.
308e1d3 adds built-in support for gracefully shutting down Jetty, Tomcat, Reactor Netty, and Undertow. Support is provided for both reactive and Servlet-based web applications and can be enabled by configuring the server.shutdown.grace-period property. The exact behaviour during graceful shutdown varies a little from container to container. If you are interested in this feature, we'd love to hear your feedback before 2.3 is released. The functionality will be available shortly in 2.3 snapshots and will also be in this week's 2.3.0.M3 release. Please take it for a spin and let us know if it meets your needs.
A snapshot containing the new functionality is now available from https://repo.spring.io/snapshot.
Thanks for working on this @wilkinsona
I find the documentation a bit confusing, specifically "Jetty, Reactor Netty, and Tomcat will stop accepting requests at the network layer."
- Do you know what happens for new requests that arrive on existing idle connections, during graceful shutdown?
- What about new requests that arrive on existing connections after the current in-flight request completes?
- Would it be possible to also document this?
- Do any/all of the implementations send
Connection:closeafter processing an in-flight request, such that the connection is gracefully terminated? - Assuming that they do not, is this new functionality exposed in such a way that applications could implement this easily themselves (e.g. with a servlet Filter that checks
if (iAmGracefullyShuttingDown) { response.setHeader("Connection", "close"); }
Most of our applications run on AWS Elastic Container Service (ECS). They are primarily behind two different types of load-balancers:
AWS Application Load Balancer (ALB):
With ALB, we can set a Deregistration Delay on the load-balancer.
As soon as the deregistration process starts for a container, the load-balancer will not send any new requests to the app - even on existing connections - because ALB is protocol-aware.
ECS will wait Deregistration Delay seconds before sending SIGTERM to the application.
This new graceful shutdown process in Spring does not really buy us anything new here, because we have already stopped receiving new requests/connections, and have been given time to handle any outstanding in-flight requests even before the application shutdown process begins.
AWS Network Load Balancer (NLB)
With NLB, we can set a Deregistration Delay on the load-balancer.
As soon as the deregistration process starts for a container, the load-balancer will not send any new connections to the app, but it will continue to send requests on existing connections.
ECS will wait Deregistration Delay seconds before sending SIGTERM to the application.
At this point, application shutdown begins. ECS gives the application a configurable amount of time to shut down, which defaults to 30sec, at which point it will send SIGKILL.
The app will continue to receive new requests on existing connections until it terminates or closes them.
The new graceful shutdown process in Spring could help us here, but only if it gives us the ability to gracefully close these existing connections after processing outstanding in-flight requests.
Right now, we use custom application logic (in a Filter) to discover when the LB deregistration process begins, and we start sending Connection:close after any in-flight requests complete. This gives us the ability to gracefully close connections before the application shutdown process even begins, but is cumbersome to implement, and adds significant delays to the shutdown process for our applications (which, in turn, adds delays to the overall release process).
Framework support for gracefully closing existing connections after in-flight requests complete (once the app receives SIGTERM) would be very helpful in this scenario.
Thanks for taking a look, @joedj.
Do you know what happens for new requests that arrive on existing idle connections, during graceful shutdown?
We're making as much use as possible of each container's built-in capabilities, which often aren't documented or consider the exact behaviour to be an implementation detail and this is one area where the containers vary. For example, Tomcat will close idle connections such that a new request cannot be made on it.
What about new requests that arrive on existing connections after the current in-flight request completes?
I don't believe that any of the containers distinguish between this scenario and the scenario above. As above, the exact behaviour varies from container to container.
Would it be possible to also document this?
There's an awkward balance for us to strike here. The currently described difference (rejected requests at the network layer versus them being accepted and then responded to with a 503) is within our control but, beyond that, the subtleties are container-specific and are often undocumented implementation details. I can only assume that's because the users of each container haven't needed to know the precise details to make use of the feature. If we documented things more precisely we'd be running the risk of our documentation becoming out of date as a container made a change to its behaviour. We could, perhaps, document what we do in terms of API calls as that is within our control.
Do any/all of the implementations send Connection:close after processing an in-flight request, such that the connection is gracefully terminated?
At the time of writing, Tomcat and Undertow do not. When an in-flight request completes successfully after graceful shutdown has begun, both respond with a Connection: keep-alive header. At this point, Tomcat won't deal with any more requests at the HTTP level. Undertow will deal with requests at the HTTP level and will respond with a 503. This response also has a Connection: keep-alive header. I haven't tested Jetty or Reactor Netty.
is this new functionality exposed in such a way that applications could implement this easily themselves
The container will be shutting down gracefully if you've configured server.shutdown.grace-period and the context closed event has been published. You could use this as a signal to set Connection: Close in any responses. That said, you may want to explore pushing this down into the container that you're using by requesting a change in behaviour or to understand why the container does not do this by default before doing so.
This new graceful shutdown process in Spring does not really buy us anything new here, because we have already stopped receiving new requests/connections, and have been given time to handle any outstanding in-flight requests even before the application shutdown process begins.
It sounds like the time that you've been given to handle outstanding in-flight requests is of fixed duration. For example, if you're being given 30 seconds to finish handling in-flight requests, shutdown will take at least 30 seconds even if those requests complete after 1 second. With graceful shutdown, you should be able to configure a deregistration delay of 0 seconds so that SIGTERM is sent as soon as the load-balancer has stopped sending requests to the app. SIGTERM results in the application context being closed. With server.shutdown.grace-period configured, this close processing will wait for up to the grace period for active requests to complete, before proceeding to shut down the app. In the scenario above where the active requests complete after 1 second, this would allow the shutdown to complete in a little over 1 second rather than a little over 30 seconds.
The new graceful shutdown process in Spring could help us here, but only if it gives us the ability to gracefully close these existing connections after processing outstanding in-flight requests.
While Tomcat will not send Connection: close, I believe that it will behave in a way that will work for you out of the box. Undertow, however, will not. If you have an opportunity to try things out in your environment with the container that you chose to use, it'd be great to know whether or not this is indeed the case.
Right now, we use custom application logic (in a Filter) to discover when the LB deregistration process begins, and we start sending Connection:close after any in-flight requests complete. This gives us the ability to gracefully close connections before the application shutdown process even begins, but is cumbersome to implement, and adds significant delays to the shutdown process for our applications (which, in turn, adds delays to the overall release process).
I'd like to understand why sending Connection:close adds significant delays to the shutdown process. Can you expand a bit on that please?
Hey @wilkinsona, thanks for the response.
I'll just answer your last question, for now, until I hopefully get a chance to play around:
I'd like to understand why sending Connection:close adds significant delays to the shutdown process. Can you expand a bit on that please?
(This is specifically in relation to shutting down gracefully while using NLB, which will continue to send requests on existing connections up until they are closed)
Because we don't currently have a nice way to gracefully shut down the container upon receiving SIGTERM (and some other internal legacy reasons that mean we currently only have a max of 30sec after SIGTERM), we instead perform the graceful connection draining before the actual app shutdown process begins.
To do this, we use the LB Deregistration Delay functionality. For NLB, this controls the duration between when the app will stop receiving new TCP connections, and when the app will receive SIGTERM.
For our app to discover that it has changed state in the LB from "healthy" to "draining" (signalling that the Deregistration Delay period has started), it polls the AWS APIs. This happens once per minute - any more frequent than that, and we often hit API rate limits. So, the Deregistration Delay needs to be at least 1min or so, for us to be sure we have even seen the state change.
Once we notice the state change, there are additional delays:
- wait up to the maximum idle timeout for any new requests to arrive on the existing connections (say, 60sec), then
- wait up to the maximum request timeout for those new in-flight requests to complete (say, another 60sec)
...but those additional delays also exist if we start draining in response to SIGTERM, rather than in response to an LB state change.
So really, using the Spring support for graceful shutdown in response to SIGTERM would only allow us to save that initial 60sec delay while polling the LB state (edit: and also, as you accurately pointed out in your earlier response, it would also mean that shutdown completes when all the in-flight requests are finished, rather than waiting for the fixed Deregistration Delay period - in practice, this could reduce the container shutdown time from 3min+ to mere seconds (or 60sec, i.e. the idle timeout we need to wait for any new requests on existing connections, assuming the web container we're using supports this and doesn't close those idle connections immediately when graceful shutdown begins).
...and it would also allow us to delete all that supporting code and get rid of the unreliable scheduled polling.
On kubernetes we have a different requirement when shutting down gracefully when doing rolling updates or scaling a deployment down.
According to this blog post and to our own experience you need to wait for a couple of seconds after receiving the SIGTERM from kubernetes before stopping to accept requests to actually still be able to handle stray connections that are coming in.
This is because the kubernetes endpoint object of this app has not yet been removed from the kuberenetes service via the endpoint-controller and kube-proxy although the SIGTERM already has been sent by the Kubelet, so requests are still routed to this endpoint. When we stop accepting connections immediately after receiving the SIGTERM (which we already do) we see "connection refused" errors from the clients. So another configuration option we'd like to see is something like server.shutdown.wait-for-seconds that would wait for this amount of seconds before actually pausing any connector and continuing with the shutdown.
Thanks, @ractive. What you've described matches our experience too. Rather than implementing the wait in Spring Boot, our recommendation is to use a pre-stop hook configured to run something like sh -c sleep 10. We have some documentation updates planned that will cover this and more from a Kubernetes perspective.
When Kubernetes is taking a pod out of service it will send the
SIGTERMsignal to it. This will cause a Spring Boot application to start shutting down. At this point, the liveness and readiness probes are no longer called.In parallel, Kubernetes will also initiate the sequence of events that results in requests no longer being routed to the application. The parallel and eventual nature of this means that there is a period of time when the application has received
SIGTERMbut requests are still routed to it. Ideally, these requests should be handled so that clients do not become aware of the application being shut down.The recommended way to minimise or perhaps even eliminate the window where requests are still received after
SIGTERMis to sleep in a pre-stop hook. In the event of the window only being minimised, continuing to accept new requests during graceful shutdown should reduce the number of errors that are seen by clients. Such errors cannot be eliminated entirely as, if there are no active requests, the graceful shutdown may complete and stop accepting new connections while Kubernetes contains to route requests to the application.
This is correct. Its also documented in the book : Kubernetes in Action (ISBN: 9781617293726), under Chapter 17. Best practices for developing apps.
I think the sleep based configuration would be good enough in our case (infact we did the same thing in our application). I expect that the configuration server.shutdown.grace-period will be well-documented so that users will know what exactly will happen when its configured.
Thanks for taking this up.
Hi, I have a spring-boot app with HTTP endpoints and Kafka consumers. I have tested this graceful shutdown implementation and it looks like the whole shutdown is delayed waiting for HTTP with Thread.sleep(), so no other components are notified. This means I need 2*n graceful period if I wanted to give HTTP and Kafka n seconds to finish their thing.
@wilkinsona I saw your comment about not needing SmartLifecycle just for tomcat, but have you considered other components needing graceful shutdown as well? In my case, I would like to leave some time for both @controllers and Kafka Listeners to be able to finish already started processing -- which in some cases can make external calls with exponential backoff, so I would also like to be able to check in retry loops if shutdown was initiated.
I'm thinking about such logic:
- shutdown initiated
- components notified about shutdown, should not block, clock starts
- HTTP connections no longer accepted
- kafka consumers stop polling new messages
- retry loops can early exit, throwing "retry-able" exception
- when time runs out, interrupt processing threads
- finish shutdown, cleanup
Would you consider changing the above logic, or recommend a custom implementation instead?