Asynchronous Handlers
The-EDev opened this issue ยท 30 comments
This is a comment made by u/aninteger on reddit
Since you are using asio I think you really need to be writing async handlers. You can't just set multi-threaded mode on and write handlers that do blocking calls because Crow doesn't appear to be a http server library setup with a large thread pool.
And he/she is correct. Crow does not have non blocking handlers. It would also be a nice feature to have, I'll have to look into it however since I don't even know where to begin working on this.
Update: Looking into the issue further, it seems that the individual threads can be blocked indefinitely. This means that Crow could be frozen for more time than timeout would normally allow, while keeping all connections assigned to the blocked thread on hold.
Extra context
Currently Crow's threading structure is as follows:
An Acceptor thread and n worker threads are initialized when the server starts. Each having its own io_context (io_service in code).
The Worker threads are constantly running io_context.run() and checking if it returns 0 (which would mean no more work to do).
The Acceptor thread creates a new connection object, and provides it with a selected io_context which it will then use to create a socket object to communicate with a potential client. Once a client is connected to the server, The Acceptor thread will move it over to the created connection and repeat the first step.
The bit most relevant to this issue
The Acceptor thread relies on io_context.post() to make sure the connection code is run through the worker thread. It is my assumption that when io_context.post() is called multiple times, one connection would need to finish executing before the next connection would start.
Considering the fact that connection code execution is not asynchronous (Instead going through a loop of reading, processing, and writing. Which doesn't finish until the object destroys itself when the client disconnects), A worker thread is by all accounts can be and is in some cases blocked by a connection's code.
This becomes a problem when a handler (AKA Route) has to run any code that doesn't return instantly -due to being asynchronous- has to block the entire worker thread (and by extension the other connections).
It is also a problem when considering the fact that any heavy operations will also need to run on (and block) the worker thread when they can be delegated to a separate thread and handled asynchronously.
The proposed solution
My suggestion would be for a worker to manage its connections.
- Each
connectionshould have a state (start,run,suspend, orready). - instead of immediately running
connection.start(), theconnectionshould be added to a queue managed by the worker thread. - The worker thread dequeues a
connectionand starts it - The
connectionis either run until finished, or is suspended by thehandler - Once a
connectionis suspended the thread needs to be re-enqueued and the rest of the queue needs to be handled - The
connectionneeds to be re-enqueued every time it is dequeued until its status is changed toready - Once ready, the
connection's resume function can be called to pick up where the handler left off (most likely done through coroutines, which both Boost and ASIO have implementations of)
Commenting here instead of creating a new issue as I think my question is related to this, but I'm fairly new to all of these terms so sorry if I don't have the proper terminology or understanding
Successfully got Crow to work in my program but I had to created an std::thread (which I had never done, I hadn't tried anything multithreading until this point). It's working great now! But I was wondering, do I have to use multithreading with Crow?? Originally I tried to putting the route/handle in my main loop, but as soon as my program hits the Crow part it stops everything and just waits for a new request. I suppose that's what's referred to as "blocking" vs "non-blocking", not sure though, but is it possible to have Crow be "non-blocking"?
For example would it possible right now to just insert some sort of "check" each time my program loops to see if Crow received a new request since the last loop, and if so return the proper value? Or for now the only way is to have it "wait" for a request to come in (thus doing it in another thread is the best way to have it not block the main thread)?
Just trying to understand more, Thank you!
No worries @philhzss, I understood your point.
Nice, glad to hear that you got it working, even though you had to deal with something new. Crow currently manages its own threads for accepting and handling messages and will block when app.run() is called, so no, you currently have to create a new thread (using the thread class or std::async in combination with std::launch::async) on your own in order to make it non-blocking.
You got it, a blocking function call will not return until it's done, meanwhile, a non-blocking function call will instantly return (this likely involves blocking function calls inside their own thread).
Maybe that's a feature worth introducing @The-EDev? I mean, it's a pretty common use case. Maybe something like run_async().
@luca-schlecker I was just writing my own explanation but you explained all the points I was going to (EVEN THE run_async SUGGESTION!! XD).
The one thing I would add is an example on how to run Crow asynchronously from our own unit tests: auto _ = async(launch::async, [&] { app.port(45451).run(); });
Thank you very much @luca-schlecker and @The-EDev for the explanations! I have never used async either so that's interesting, I'll read about it. For now thought my 2-threated option is working fine so I'll keep it that way. Thanks for the info though, always great to learn!!
The current discussion has only touched internals, but not what the actual goal is and what a handler will look like.
Drogon uses a simple callback function and has support using c++20 coroutines with its own set of functions.
Because using async requires some kind of runtime/executor/library and gluing two together is quite hard, I'd suggest also exposing the io_context, so that users can shedule their own asio operations on them.
well to put the issue simply, if you ever have to spend 1 or 2 minutes spinning up a tape drive to get an obscure database record for 1 request, all the other requests in that pipeline will have to wait with no way for them to be executed while this 1 record is being retrieved.
The general goal is to make it so that worker threads process requests as efficiently as possible, either by offloading the time consuming process and having the worker thread process other requests until it is complete. Or some other method (such as offloading the requests onto another worker thread that is free).
I'm not exactly sure how having users schedule ASIO operations helps this since those operations still have to wait until the connection is deleted.. (I'm not saying this as a slight @dranikpg, I genuinely don't know and am asking)
I surely get what the point of async is ๐
Let me be more specific. Most programming languages have some kind of executor that is used all over the ecosystem and that allows different libraries (webframeworks, database clients, etc.) to be used together. For Python - its asnycio, for Rust - Tokio, for JS - the Node runtime.
C++ does not really have a single runtime and most frameworks invent something on their own. That is why drogon and oatpp have their own utilities for dealing with databases and redis. They can be used, for example, even with C++ 20 coroutines.
Just because the handlers support asynchronous finishing, this doesn't mean any library/coroutine can be seamlessly glued on top of it.
So the question is: what will asynchronous handlers look like?
[the connection] is suspended by the handler
How? A regular function cannot just suspend itself. How will the handler wake up again?
The most viable and basic option is having a callback function. Like I mentioned, Drogon does it that way.
CROW_ROUTE(app, "")
[](const crow::request, std::function<crow::response> readyFunc) {
// make async request to some other api
asio::async_read(some_stream, some_buffer, [](){
readyFunc(crow::response(buffer)); // tell crow the handler has a result ready
})
// the handler simply finished without a result - it will be provided via the callback
// so there is no blocking
}That will work with libraries that provide just callbacks, but is a bit more difficult with asio for example. The user needs its own io_context for all kinds of handler operations.
How? A regular function cannot just suspend itself. How will the handler wake up again?
The idea (at least in my head) was to use some form of coroutine to get back to where the function left off. The actual suspension and waking up were to be done by calling the async code through a wrapper function that changes the connection status and notifies the server to move on for the time being, once the code is done, a callback is run to change the connection status back.
The most viable and basic option is having a callback function.
Well the concern here would be how this approach affects everything that runs after the route handler returns. Since the current behavior is to simply start returning the response. Another point to look into would be how this would work with keepalive / timeout timers and if it's possible to keep the connection alive (without memory leaks) and still have the io_context handle different requests.
I would have thought about tying the crow::request object to the connection.
The handler function would have to use that crow::request object it was given and pass it on to its workload to later invoke a function on the app object (for example app.finish(request, response) or finalize or something). This could then invoke the missing calls to complete the action which would have been called directly after the handler finished.
I hope this was somewhat understandable. ๐
It should be possible to make that finish-function static as the crow::request could store a pointer to its creator app or something.
Example:
// ...
CROW_ROUTE("/")([](crow::request req) {
auto t = std::thread([req]() -> void {
std::this_thread::sleep_for(std::chrono::seconds(2));
crow::finish(req, crow::response{"Work Done"});
});
t.detach();
});Return-type void would indicate that this request shouldn't be completed after the handler was called.
But I guess this approach isn't all too far from just making every callback async, which would make it even easier to use as one can just block in the handler...
This could work in a similar way.
Changes deep within crow seem to be necessary either way.
The idea (at least in my head) was to use some form of coroutine to get back to where the function left off
Sure, but that requires your end-user to use a specific coroutine in his handler. You can't do anything as soon as you hand control over to the handler. Boost and C++20 coroutines allow suspending, but you'd enforce them to be used. This wouldn't fit someone who wants to use a just custom library with callbacks.
It should be possible to make that finish-function static as the crow::request could store a pointer to its creator app or something.
Definitely better than passing an additional app around. Any thoughts on a completion function?
One more option would be returning some kind of std::future, but one that would allow crow to listen on multiple of them. That'd make void handlers not that ambiguous and should maybe look more familiar.
CROW_ROUTE("/")([](crow::request req) {
auto rsp = crow::Future{};
auto t = std::thread([req]() -> void {
std::this_thread::sleep_for(std::chrono::seconds(2));
fut = crow::response{"Work Done"}; // call assign operator that actually notifies
});
t.detach();
return rsp;
});But I guess this approach isn't all too far from just making every callback async, which would make it even easier to use as one can just block in the handler...
If you block, you're back to normal mode again ๐ Async handlers have to exit immediately (like in our examples). This requires two separate thread pools for blocking and non-blocking handlers.
I'm being curios and asking this many questions, because I think about taking this up. So it'd be great if you'd share any more ideas/thoughs you have.
Just to throw it in there: What about std::promise?
One thread would be needed to check on the futures though.
I think I'd prefer a mechanism similar to the crow::finish function I talked about earlier though.
This gives the user the choice wether or not to make use of async handlers whilst not making it particularly difficult to use.
Just to throw it in there: What about std::promise?
std::promise is just the sender side of a future.
One thread would be needed to check on the futures though.
Thats the problem with them :/ We cant listen on multiple of them effectively. You can only block on one or loop over all of them. Calling crow::finish allows to directly handle the response.
This gives the user the choice wether or not to make use of async handlers whilst not making it particularly difficult to use.
That's true. It's just that someone who forgets to return a response from a handler unknowingly makes it an async one, which could be annoying to debug (like if you forget calling response.end). And there is nothing (no second argument, no return type) to remind or show via the signature, that this is an asynchronous handler.
Besides this, most frameworks threat a void handler as an emptry response, that still has to be sent, so we won't be able to introduce in anytime later.
(currently crow throws compilation errors on void handlers without a second response argument)
Well, how about returning a specialized value like crow::response::async analogous to std::nullopt?
If crow::finish is then called on a request object where the handler didn't return that special value it could just throw...
Well, how about returning a specialized value like crow::response::async analogous to std::nullopt?
That'll work. This ensures:
A. Nobody makes a handler asynchronous by mistake
B. Calls crow::finish on a wrong response
Can actual http response logic be contained inside of crow::response::end() function?
Like,
crow::response res;
res.response_handler_ = [this, ...] (crow::response&& self) {
/* Send HTTP response using 'self' */
// Move crow::App's current crow::response handling logic inside of crow::response::response_handler_ callback.
// Since this may cause race condition between primary handler thread of crow::App,
// (Though I don't have any idea about internal implementations that well...)
// handling logic may be wrapped again with dispatch inside of provided end() handler.
// (asio::dispatch is used to prevent unnecessary event posting on synchronous invocation.)
// **example**.
asio::dispatch(this->worker_, [this, res = move(self)] { this->handle_response_(res); });
};
// Then when user or crow::App finishes response, it'll invalidate itself.
void crow::response::end() && {
/* do some end() work */
this->response_handler_(move(*this));
}Then user can simply move crow::response object to another thread, and make async response by calling 'end()' on it.
asio::thread_pool async_worker;
// In this way, app does not need to care whether response is async or not,
// thus user can route synchronous/asynchronous handlers in identical way.
CROW_ROUTE(app, "/some/test/api")(&handler);
void handler(crow::request const& req, crow::response& rep) {
asio::post(async_worker, [req, rep = move(rep)] {
/* do some time-consuming operation with 'rep' */
rep.end();
});
// Application instance becomes ready to accept next HTTP request immediately.
...
}Just curiosity :|
+1 for the above. I think this suits most people and it's simple enough to understand, and it works well with threadpools.
This wouldn't be too far from the solution @dranikpg and I worked out. I haven't yet looked at any code, including that of crow::response.
The actual response logic is already contained in res.end(). Actually, you can make Crow asynchronous now, but very easily shoot youself in the foot.
Like this, for example ๐
simply move crow::response object to another thread
The connection instance is alive until the request finished, you should not steal its data. Capturing by reference is fine. Maybe make response immovable ๐ค
In case you want to return a crow::returnable (like json), you'll have to call rsp.write(ret.dump()) which seems unnecessary low-level. You can't re-assign the response, or else you'll lose its connection reference.
So it seems there are a few issues with the end() design. Further proposals are welcomed ๐
@dranikpg It seems capturing reference and invoking 'end()' in another thread also blocks receiving next request. Is there any reference to implement receiving next request before current response is made?
What is the progress of this feature?
Is there already a way to offload work to a separate thread and return it's result later? (without blocking the current thread)
Does this feature is ready?
on version 1.0+5, capture Crow::response in another thread and use response::end() to send http response to client then the crow threads will not block
on version 1.0+5, this feature work well, but Aborted on version 1.1, any suggestions?
on version 1.0+5, capture Crow::response in another thread and use response::end() to send http response to client then the crow threads will not block
Does this feature is ready?
Dear @portsip i found the following info in gitter: Ex-maintainer The-EDev tried to implement it but could not make it work
on version 1.0+5, this feature work well, but Aborted on version 1.1, any suggestions?
on version 1.0+5, capture Crow::response in another thread and use response::end() to send http response to client then the crow threads will not block
I found asynchronous handler crash in crow::response::end, because crow::response free in callback complete_request_handler_, so I modified crow_all.h L7457 like
void end()
{
if (!completed_)
{
completed_ = true;
if (skip_body)
{
set_header("Content-Length", std::to_string(body.size()));
body = "";
manual_length_header = true;
}
if (auto handler = complete_request_handler_)
{
complete_request_handler_();
}
}
}This make asynchronous handler work again, so pls just try it.@InternalHigh๏ผ@witcherofthorns
@shihzh hi, oouh, okay, i'll try this in the next free time and look at the source code in crow_all.h, thanks
Hi folks, (read the thread again, and understand a little bit more, but...)
I've been trying to understand these async discussions together with the generic documentation, but I am unable to understand just "to what level" supported asynchonous calls are at the moment (in master). I apologize if this is some form of ticket-hijacking, but I dont want to write a bugreport if I am simply using crow in an unsupported fashion.
My CROW_ROUTE will often not finish with a crow::response.end(), but instead trigger another thread that will (eventually, using another async library call) do that.... and it works fine (afaik) with boost. If I've understood the above correctly, that should work?
But now, I am in the process of moving to boost-less, using only the asio library, but if I do that, I will get 100% crashes in all my asynchronous calls.
But since main crow page says that async is not (fully?) supported, I am wondering whether there is something more I have to do...
Or is this a bug, when just using "pure" asio?
08:50:47.766 - NW_LOG[Ecmi/Info ] - EcmiServer :Response: 0x5220000ff110 /api/v1/conferences 200 0: { }
=================================================================
==12140==ERROR: AddressSanitizer: attempting double-free on 0x502000017450 in thread T3:
#0 0x7f8ad32fd0d8 in operator delete(void*, unsigned long) (/lib64/libasan.so.8+0xfd0d8) (BuildId: 1827a4c72065a9f25ba519b25166029eebbf519f)
#1 0x8148e3 in std::_Function_base::_Base_manager<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) /usr/include/c++/14/bits/std_function.h:175
#2 0x81ead7 in std::_Function_base::_Base_manager<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/include/c++/14/bits/std_function.h:203
#3 0x81eb44 in std::_Function_handler<void (), crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/include/c++/14/bits/std_function.h:282
#4 0x7f5ebd in std::function<void ()>::operator=(decltype(nullptr)) /usr/include/c++/14/bits/std_function.h:505
#5 0x831444 in crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::prepare_buffers() 3pp/CrowCpp/include/crow/http_connection.h:279
#6 0x834891 in crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::complete_request() 3pp/CrowCpp/include/crow/http_connection.h:264
#7 0x836468 in crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}::operator()() const 3pp/CrowCpp/include/crow/http_connection.h:188
#8 0x836468 in void std::__invoke_impl<void, crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&>(std::__invoke_other, crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&) /usr/include/c++/14/bits/invoke.h:61
#9 0x836468 in std::enable_if<is_invocable_r_v<void, crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&>, void>::type std::__invoke_r<void, crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&>(crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&) /usr/include/c++/14/bits/invoke.h:111
#10 0x836468 in std::_Function_handler<void (), crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_invoke(std::_Any_data const&) /usr/include/c++/14/bits/std_function.h:290
#11 0x763469 in std::function<void ()>::operator()() const /usr/include/c++/14/bits/std_function.h:591
#12 0x765c6b in crow::response::end() 3pp/CrowCpp/include/crow/http_response.h:250
#13 0x766773 in WebServer::returnCrowResponse(crow::response&, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) include/ecmi/WebServer.hpp:133
#14 0x756e2f in ImmiHandler::gotImmiRsp(std::shared_ptr<ApiResponse>&) src/ecmi/ImmiHandler.cpp:244
#15 0x8765f7 in RttMixer::responseQueueLoop() src/immi/RttMixer.cpp:675
#16 0x897cf0 in void std::__invoke_impl<void, void (RttMixer::*)(), RttMixer*>(std::__invoke_memfun_deref, void (RttMixer::*&&)(), RttMixer*&&) /usr/include/c++/14/bits/invoke.h:74
#17 0x897d38 in std::__invoke_result<void (RttMixer::*)(), RttMixer*>::type std::__invoke<void (RttMixer::*)(), RttMixer*>(void (RttMixer::*&&)(), RttMixer*&&) /usr/include/c++/14/bits/invoke.h:96
#18 0x897d38 in void std::thread::_Invoker<std::tuple<void (RttMixer::*)(), RttMixer*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/14/bits/std_thread.h:292
#19 0x897d38 in std::thread::_Invoker<std::tuple<void (RttMixer::*)(), RttMixer*> >::operator()() /usr/include/c++/14/bits/std_thread.h:299
#20 0x897d38 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (RttMixer::*)(), RttMixer*> > >::_M_run() /usr/include/c++/14/bits/std_thread.h:244
#21 0x7f8ad28ec3c3 in execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:104
#22 0x7f8ad3262975 (/lib64/libasan.so.8+0x62975) (BuildId: 1827a4c72065a9f25ba519b25166029eebbf519f)
#23 0x7f8ad2492ba1 in start_thread /usr/src/debug/glibc-2.39/nptl/pthread_create.c:447
#24 0x7f8ad251400b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
0x502000017450 is located 0 bytes inside of 16-byte region [0x502000017450,0x502000017460)
freed by thread T3 here:
#0 0x7f8ad32fd0d8 in operator delete(void*, unsigned long) (/lib64/libasan.so.8+0xfd0d8) (BuildId: 1827a4c72065a9f25ba519b25166029eebbf519f)
#1 0x8148e3 in std::_Function_base::_Base_manager<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) /usr/include/c++/14/bits/std_function.h:175
#2 0x81ead7 in std::_Function_base::_Base_manager<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/include/c++/14/bits/std_function.h:203
#3 0x81eb44 in std::_Function_handler<void (), crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/include/c++/14/bits/std_function.h:282
#4 0x52c0d4 in std::_Function_base::~_Function_base() /usr/include/c++/14/bits/std_function.h:244
#5 0x7bdc12 in std::function<void ()>::~function() /usr/include/c++/14/bits/std_function.h:334
#6 0x7bdc12 in crow::response::~response() 3pp/CrowCpp/include/crow/http_response.h:33
#7 0x83c525 in crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::~Connection() (/home/taisto/repos/rtt-focus/bin/ccm+0x83c525) (BuildId: 4e6ce2619f029853620d73718fc405348fde4748)
#8 0x83c584 in std::_Sp_counted_ptr_inplace<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (/home/taisto/repos/rtt-focus/bin/ccm+0x83c584) (BuildId: 4e6ce2619f029853620d73718fc405348fde4748)
#9 0x544536 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/include/c++/14/bits/shared_ptr_base.h:175
#10 0x54462c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() /usr/include/c++/14/bits/shared_ptr_base.h:199
#11 0x5447b4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/14/bits/shared_ptr_base.h:353
#12 0x8148d6 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/14/bits/shared_ptr_base.h:1069
#13 0x8148d6 in std::__shared_ptr<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/14/bits/shared_ptr_base.h:1525
#14 0x8148d6 in std::shared_ptr<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler> >::~shared_ptr() /usr/include/c++/14/bits/shared_ptr.h:175
#15 0x8148d6 in crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}::~handle() 3pp/CrowCpp/include/crow/http_connection.h:187
#16 0x8148d6 in std::_Function_base::_Base_manager<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) /usr/include/c++/14/bits/std_function.h:175
#17 0x81ead7 in std::_Function_base::_Base_manager<crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/include/c++/14/bits/std_function.h:203
#18 0x81eb44 in std::_Function_handler<void (), crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /usr/include/c++/14/bits/std_function.h:282
#19 0x7f5ebd in std::function<void ()>::operator=(decltype(nullptr)) /usr/include/c++/14/bits/std_function.h:505
#20 0x831444 in crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::prepare_buffers() 3pp/CrowCpp/include/crow/http_connection.h:279
#21 0x834891 in crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::complete_request() 3pp/CrowCpp/include/crow/http_connection.h:264
#22 0x836468 in crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}::operator()() const 3pp/CrowCpp/include/crow/http_connection.h:188
#23 0x836468 in void std::__invoke_impl<void, crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&>(std::__invoke_other, crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&) /usr/include/c++/14/bits/invoke.h:61
#24 0x836468 in std::enable_if<is_invocable_r_v<void, crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&>, void>::type std::__invoke_r<void, crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&>(crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}&) /usr/include/c++/14/bits/invoke.h:111
#25 0x836468 in std::_Function_handler<void (), crow::Connection<crow::SocketAdaptor, crow::Crow<crow::CORSHandler>, crow::CORSHandler>::handle()::{lambda()#2}>::_M_invoke(std::_Any_data const&) /usr/include/c++/14/bits/std_function.h:290
#26 0x763469 in std::function<void ()>::operator()() const /usr/include/c++/14/bits/std_function.h:591
#27 0x765c6b in crow::response::end() 3pp/CrowCpp/include/crow/http_response.h:250
#28 0x766773 in WebServer::returnCrowResponse(crow::response&, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) include/ecmi/WebServer.hpp:133
#29 0x756e2f in ImmiHandler::gotImmiRsp(std::shared_ptr<ApiResponse>&) src/ecmi/ImmiHandler.cpp:244
#30 0x8765f7 in RttMixer::responseQueueLoop() src/immi/RttMixer.cpp:675
#31 0x897cf0 in void std::__invoke_impl<void, void (RttMixer::*)(), RttMixer*>(std::__invoke_memfun_deref, void (RttMixer::*&&)(), RttMixer*&&) /usr/include/c++/14/bits/invoke.h:74
#32 0x897d38 in std::__invoke_result<void (RttMixer::*)(), RttMixer*>::type std::__invoke<void (RttMixer::*)(), RttMixer*>(void (RttMixer::*&&)(), RttMixer*&&) /usr/include/c++/14/bits/invoke.h:96
#33 0x897d38 in void std::thread::_Invoker<std::tuple<void (RttMixer::*)(), RttMixer*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/14/bits/std_thread.h:292
#34 0x897d38 in std::thread::_Invoker<std::tuple<void (RttMixer::*)(), RttMixer*> >::operator()() /usr/include/c++/14/bits/std_thread.h:299
#35 0x897d38 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (RttMixer::*)(), RttMixer*> > >::_M_run() /usr/include/c++/14/bits/std_thread.h:244
#36 0x7f8ad28ec3c3 in execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:104