corrupted double-linked list
shenfu1991 opened this issue · 12 comments
Hello everyone, I encountered an error while running Vapor on Ubuntu 20.04 and I'm not sure where the error is coming from. The only useful information I have is this sentence: 'corrupted double-linked list'. This has been causing me a headache because the error doesn't always occur and there are no tools on Ubuntu that can specifically point out where the problem is.
OS：ubuntu20.04
vapor version:
framework: 4.74.0
toolbox: main (6000cc5)
[ INFO ] GET /users [request-id: 0E43675F-BB13-4F00-9226-DEDBF3EC114A] [ INFO ] GET /users [request-id: DE56A723-C73C-4768-99EA-2E4B3F6E2BDE] [ INFO ] GET /users [request-id: 6C7D5E86-6510-4AE0-BAB3-6E49094762B9] [ INFO ] GET /users [request-id: 79857547-0FFC-41D3-8931-1F15A4F4FE81] corrupted double-linked list
@shenfu1991 what does your code look like? This is a very strange error and unless you're calling C shouldn't ever happen (and I haven't seen one for years). What version of Swift are you using as well?
@0xTim Hello, thank you for your reply. I am working on a cryptocurrency quantitative trading system that involves functions such as obtaining historical data of currencies, querying user positions, balances and contract trading logic. It is a complete trading system. I do not know how to call C language. Due to the large amount of code in the system and sensitive algorithms involved, I suspect there may be problems with network requests which often cause crashes in this part. My current Swift version is:
Swift version： 5.7.3 (swift-5.7.3-RELEASE) Target: x86_64-unknown-linux-gnu.
By the way, if I use the production environment instead of dev mode currently being used, will it reduce the chance of crashes?
import Vapor
import AsyncHTTPClient
class CoreNetwokingManager {
static let sharedInstance = CoreNetwokingManager()
var API_Key: String = ""
typealias SuccessType = (_ result: Any?,_ data: Data?) -> Void
typealias CompleteType = (Any) -> Void
func doAsyncWork(method: HTTPMethod,URLString: String,parameters: [String:Any]?, success: SuccessType?, complete: CompleteType?) async {
let httpClient = HTTPClient(eventLoopGroupProvider: .createNew)
var request = HTTPClientRequest(url: URLString)
request.method = method
request.headers.add(name: "X-MBX-APIKEY", value: API_Key)
do {
let response = try await httpClient.execute(request, timeout: .seconds(45))
let body = try await response.body.collect(upTo: 1024 * 1024 * 100) // 100 MB
let json = try JSONSerialization.jsonObject(with: body)
if let ss = success {
ss(json,Data(buffer:body))
if let cb = complete {
cb(json)
}
}else{
if let cb = complete {
cb("json error")
}
}
}catch(let err){
if let cb = complete {
cb("response error=\(err)")
}
}
try? await httpClient.shutdown()
}
func request(method: HTTPMethod,URLString: String,parameters: [String:Any]?, success: SuccessType?, complete: CompleteType?){
Task {
await doAsyncWork(method: method, URLString: URLString, parameters: parameters, success: success, complete: complete)
}
}
}
let httpClient = HTTPClient(eventLoopGroupProvider: .createNew)
That line is definitely not right, as it will spawn a new HTTPClient and a new EventLoopGroup for every single outgoing request. That's not only a massive waste of time and resources, but since it does call down to C, it might even contribute to the flakiness with all the abrupt shutdowns.
You also seem to have an explicit singleton going on, which is... fine I guess, but your singleton should create and contain one single long-lived HTTPClient, and preferably instead of .createNew, it should use the same EventLoopGroup as the Vapor Application.
Hello, thank you for your reply. This is how I wrote it before:
let httpClient = HTTPClient(eventLoopGroupProvider: .shared(kApp.eventLoopGroup))
However, there were no problems when running it in Xcode, but it crashed on Ubuntu with the error message：
ERROR: Cannot schedule tasks on an EventLoop that has already shut down. This will be upgraded to a forced crash in future SwiftNIO versions.
But when I made the changes, there was no crash on Ubuntu, so that's why I wrote it that way：
let httpClient = HTTPClient(eventLoopGroupProvider: .createNew)
Can you provide me with a correct and complete network request? Thank you very much.
@vzsg Just now when I ran it again, there was no error surprisingly. This is the modified code.I don't know what happened either 😂
let httpClient = HTTPClient(eventLoopGroupProvider: .shared(kApp.eventLoopGroup))
There's a lot going on here. You're mixing concurrency and callbacks, creating new event loops and just background requests in a Task. Is this a Vapor project? If so why are you doing all of this (including setting up your own http client) instead of just doing req.client.send()
@0xTim yes,it is a Vapor project. Because I need to request third-party data.It seems that the data obtained by req.client.send() request cannot be obtained directly from json, and sometimes the request to obtain data exceptions cannot be handled conveniently.
@shenfu1991 that's simply not true. req.client.send() is just HTTPClient under the hood
@0xTim If the data returned by the third-party server is like this：
{
"error": "xxxxx",
"code": "xxx"
}
instead of like:
{
"msg": "xxxx",
"status": 200
}
, can the model be serialized correctly in this case?
return req.client.get("https://httpbin.org/json").flatMapThrowing { res in
try res.content.decode(MyJSONResponse.self)
}.flatMap { json in
// Use JSON here
}
struct MyJSONResponse : Content {
var msg : String?
var status : Int?
}
Yes you can wrap the decode in a do/catch and then handle the decoding error and then try decoding to the error content. Due to the use of futures you'll have to jump through some hoops and you'll have a much better time just using async/await and can write stuff linearly. I'd also check the status code to work out which type to try and decode
@0xTim i see, I still have a lot to learn, I need a comprehensive understanding of the documentation, thank you very much for your help.
No problem! I'll close this out, but feel free to ask any other questions on Discord