vapor/vapor

corrupted size vs. prev_size while consolidating

shenfu1991 opened this issue · 7 comments

shenfu1991 commented

Describe the bug

Hello everyone, the program crashes during use, there is no more useful log information, too many interface calls, or concurrent calls may cause this problem?

Environment

  • Vapor Framework version: 4.74.0
  • Vapor Toolbox version: (6000cc5)
  • OS version: ubuntu20.04

Additional context

Add any other context about the problem here.

[ INFO ] GET /allrealtrade [request-id: C8114B6E-544A-4E46-AB01-A151FF744375]
[ INFO ] GET /allrealtrade [request-id: 947E4CA2-0281-4DBF-8E9A-7ECCAD7C7453]
[ INFO ] GET /allrealtrade [request-id: 200935B5-B665-438D-A862-0225C754B1F5]
[ INFO ] GET /allrealtrade [request-id: 7A589ED6-5463-4FC5-94EF-5453D047B72C]
[ INFO ] GET /allrealtrade [request-id: 9E31711D-3BF0-4717-89B9-4ED9767BC034]
[ INFO ] GET /allrealtrade [request-id: 105BE4CB-2581-4E75-A337-F87CED006590]
[ INFO ] GET /allrealtrade [request-id: CE5394B1-DBE0-432E-AFA0-B2475F500170]
[ INFO ] GET /allrealtrade [request-id: E014A41D-0979-41A2-8985-FFC24159A422]
[ INFO ] GET /allrealtrade [request-id: 2BEC0CD2-1B7C-467F-8D63-7B0535132E32]
The operation could not be completed. The data is not in the correct format.
The operation could not be completed. The data is not in the correct format.
[ INFO ] GET /getIndicator [request-id: 644520C3-61E0-4DAA-BA6A-2884040A0318]
[ INFO ] GET /setML/BTCUSDT/0 [request-id: 3470709F-CCCC-4B41-957C-53D9755B3EAA]
[ INFO ] GET /setML/ETHUSDT/0 [request-id: 98C49A14-6278-4B48-BD3A-F3E3D0F79198]
[ INFO ] GET /setML/LINAUSDT/0 [request-id: B94D10EE-588E-46E8-97BA-83CE1300EA9A]
[ INFO ] GET /setML/AMBUSDT/2 [request-id: B9842872-0FCC-486D-9E64-BA6D65B258DD]
[ INFO ] GET /setML/INJUSDT/0 [request-id: 579691E4-B423-420B-897D-7403B7F6C048]
[ INFO ] GET /setML/OCEANUSDT/0 [request-id: 0B499D8B-63F7-4D19-8648-CAAD7218F1D3]
[ INFO ] GET /setML/JOEUSDT/0 [request-id: C169456D-A5F8-4FF5-8C08-826F8F003477]
[ INFO ] GET /setML/RDNTUSDT/0 [request-id: 7598800B-3147-4B74-A7BF-059443F07B26]
[ INFO ] GET /setML/ALPHAUSDT/0 [request-id: 366E0D9C-3E7F-481D-8A56-9E78096F9656]
[ INFO ] GET /setML/PHBUSDT/0 [request-id: 096521A6-1B03-4A8C-822A-5EE188CC9EA0]
[ INFO ] GET /setML/APTUSDT/0 [request-id: F17228A6-CF05-4991-91B6-17C0244E68C3]
[ INFO ] GET /setML/NKNUSDT/2 [request-id: 2B9E8391-1B52-4EB9-8B95-7926EE66E4B8]
[ INFO ] GET /setML/FILUSDT/0 [request-id: E6FFFF88-F9AE-41A4-A004-90979C32C381]
[ INFO ] GET /setML/CELOUSDT/0 [request-id: 19556812-04B0-4F1F-8F86-B6AA7985D31F]
[ INFO ] GET /setML/LDOUSDT/0 [request-id: 30D97B2B-CFF1-4775-808F-FA64D123D522]
[ INFO ] GET /setML/SANDUSDT/0 [request-id: 3C7D50D0-8226-4587-AB90-B210C8C2B1DC]
[ INFO ] GET /setML/WOOUSDT/0 [request-id: 09B209BD-00BD-4B9C-882B-A92B6456703E]
[ INFO ] GET /setML/KEYUSDT/0 [request-id: 8458C559-4C3D-4574-BE5B-239F536D9E73]
[ INFO ] GET /setML/FETUSDT/0 [request-id: 5FF864C5-F0F4-4110-BE1C-76387D5ED4F6]
[ INFO ] GET /setML/FOOTBALLUSDT/0 [request-id: 5875A1C2-D1DD-4D8B-B420-83635B3FFE87]
[ INFO ] GET /setML/MKRUSDT/0 [request-id: 23BEE8CC-5E42-45FB-A593-701492FB4C46]
[ INFO ] GET /setML/SKLUSDT/0 [request-id: 147833C8-0323-4217-9630-06044AD69258]
[ INFO ] GET /setML/BLZUSDT/0 [request-id: 62ABD44E-77C4-4792-B5A7-3453721B2C7D]
[ INFO ] GET /setML/ATAUSDT/0 [request-id: 94769023-7D1C-49DD-B402-19CE0A421D5F]
[ INFO ] GET /setML/ARBUSDT/0 [request-id: 38D80B27-9ABB-4C2F-AC22-BCDB36F71CFE]
[ INFO ] GET /setML/MAGICUSDT/2 [request-id: 0EC4666F-8B80-4598-86CE-83E7FABAAF31]
[ INFO ] GET /setML/HFTUSDT/0 [request-id: CA9EFF6C-A9EB-478F-A2B8-22978688F522]
[ INFO ] GET /setML/GRTUSDT/0 [request-id: BC66D7BD-AABE-4D1D-83BE-4F4FD5626557]
[ INFO ] GET /setML/COTIUSDT/0 [request-id: D0B88775-445F-43C1-9B34-5AA11E80A556]
[ INFO ] GET /setML/IOTAUSDT/0 [request-id: 86E4FBEF-8882-4418-AD8D-B1769C7E7E26]
The operation could not be completed. The data is not in the correct format.
[ INFO ] GET /getIndicator [request-id: B38EB303-2827-4CC7-8E89-17941EBDF634]
corrupted size vs. prev_size while consolidating

api ：

app.get("getIndicator") { req ->MLInfoModel in
        DispatchQueue.global().sync {
            let fdu = getGlobalInfo()
            return MLInfoModel(sbs: symbolArr,values: fdu)
        }
    }
    
    app.get("setML",":sb",":v") { req -> AddUserModel in
        let sb = req.parameters.get("sb") ?? ""
        let v = req.parameters.get("v") ?? ""
        let num = Int64(v) ?? -1
        setML(symbol: sb,value: num)
        return AddUserModel(msg: "success", success: true)
    }
vzsg commented

What does setML do, and is it threadsafe?
Because if not, parallel modifications could easily cause memory corruption and the error you got.

shenfu1991 commented

@vzsg

it is threadsafe?

var MLInfo: [Int64] = Array(repeating: -1, count: 100)

func setML(symbol: String,value: Int64){
    DispatchQueue.global().sync {
        if let index = symbolArr.firstIndex(where: { (str) -> Bool in
            str == symbol
        }){
           MLInfo[index] = value
        }
    }
}
mkll commented

@shenfu1991 No, the code is not threadsafe. You seem to assume that this code — DispatchQueue.global().sync { } — provides thread safety for the code inside the parentheses, but it doesn't. It makes sense for you to learn how DispatchQueue works and what this code do.

Also, you should avoid use DispatchQueue directly in Vapor. You should use Swift Concurrency (async/await) or the "old" way (EventLoopFuture). Here are the details: https://docs.vapor.codes/basics/async/

P.S. This code also does not provide thread safety:

app.get("getIndicator") { req ->MLInfoModel in
        DispatchQueue.global().sync {
            let fdu = getGlobalInfo()
            return MLInfoModel(sbs: symbolArr,values: fdu)
    }
}

Even more, all this code does is slow down your Vapor application. Nothing else. It doesn't do anything useful.

Synchronization of data access in Vapor is provided by a built-in mechanism, described here:
https://docs.vapor.codes/advanced/services/?h=lock#locks

  • 👍1
mkll commented

@shenfu1991 In general, I strongly recommend that you read the entire Vapor documentation from cover to cover. It is small in size, but it is of an overview nature and covers all the main points of using the Vapor.

vzsg commented

Yeah, as mkll noted, that code does not provide any kind of thread safety. The global concurrent queue is by definition incapable of that; a serial queue could work, but you would have to make sure the same queue is used for all read and write accesses to the same arrays...

This whole thing seems like a natural fit to be stored in an actor. The Vapor docs are much older than this language feature.

  • 👍1
mkll commented

@vzsg I would add that those methods of ensuring thread safety that are applicable in iOS and / or macOS applications, in particular, based on DispatchQueue and GCD in general, are far from always relevant and applicable on the server side.

0xTim commented

Going to close this and it's not an issue with Vapor. Discord is probably the best place for continuing the discussion!

  • 👍1