transactions (starting from second inside the same session) are not working

Question

transactions (starting from second inside the same session) are not working

WT2 opened this issue 2 years ago · 13 comments

Hello

Running with: softinstigate/restheart:7.0 + mongo:6.0

It seems we have a problem starting from the second transaction inside a given session

We create a session "S1"
We create a first transaction "T1" in "S1" -> We commit -> OK -> T1 is ended.
We create a second transaction "T2" in "S1" -> the POST for the creation is OK, but the GET of status is KO.

{
"http status code": 500,
"http status description": "Internal Server Error",
"message": "Error handling the request, see log for more information",
"exception": "java.lang.NumberFormatException",
"exception message": "For input string: 'with txnNumberAndRetryCounter { txnNumber: 2, txnRetryCounter: 0'"
}

com.mongodb.MongoCommandException: Command failed with error 251 (NoSuchTransaction): 'Given transaction number 1 does not match any in-progress transactions. The active transaction number is -1' on server vdemodev147dsy.ux.dsone.3ds.com:27017. The full response is {"errorLabels": ["TransientTransactionError"], "ok": 0.0, "errmsg": "Given transaction number 1 does not match any in-progress transactions. The active transaction number is -1", "code": 251, "codeName": "NoSuchTransaction", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1678193831, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "ZmPCD7m2giekDZPHdgvWs6qa/jc=", "subType": "00"}}, "keyId": 7181198839469047813}}, "operationTime": {"$timestamp": {"t": 1678193831, "i": 1}}}
at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:198)
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:416)
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:340)
at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:116)
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:643)
at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:71)
at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:206)
at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:119)
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:85)
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:75)
at com.mongodb.internal.connection.DefaultServer$OperationCountTrackingConnection.command(DefaultServer.java:293)
at com.mongodb.internal.operation.CommandOperationHelper.createReadCommandAndExecute(CommandOperationHelper.java:233)
at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:332)
at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$0(OperationHelper.java:356)
at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:381)
at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$1(OperationHelper.java:355)
at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:381)
at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:354)
at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:329)
at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:67)
at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:340)
at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:79)
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:191)
at com.mongodb.client.internal.FindIterableImpl.first(FindIterableImpl.java:213)
at org.restheart.mongodb.db.sessions.TxnsUtils.propagateSession(TxnsUtils.java:104)
at org.restheart.mongodb.db.sessions.TxnsUtils.getTxnServerStatus(TxnsUtils.java:72)
at org.restheart.mongodb.handlers.sessions.GetTxnHandler.handleRequest(GetTxnHandler.java:72)
at org.restheart.mongodb.handlers.RequestDispatcherHandler.handleRequest(RequestDispatcherHandler.java:134)
at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
at org.restheart.mongodb.handlers.injectors.ETagPolicyInjector.handleRequest(ETagPolicyInjector.java:73)
at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
at org.restheart.mongodb.handlers.injectors.ClientSessionInjector.handleRequest(ClientSessionInjector.java:99)
at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
at org.restheart.mongodb.handlers.OptionsHandler.handleRequest(OptionsHandler.java:83)
at org.restheart.mongodb.handlers.ErrorHandler.handleRequest(ErrorHandler.java:74)
at org.restheart.handlers.PipelinedWrappingHandler.handleRequest(PipelinedWrappingHandler.java:121)
at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:104)
at org.restheart.handlers.PipelinedWrappingHandler.handleRequest(PipelinedWrappingHandler.java:121)
at org.restheart.mongodb.MongoService.handle(MongoService.java:126)
at org.restheart.mongodb.MongoService.handle(MongoService.java:72)
.
.
.

    at org.restheart.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:79)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.security.handlers.AuthenticatorMechanismsHandler.handleRequest(AuthenticatorMechanismsHandler.java:59)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:74)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.security.handlers.SecurityHandler.handleRequest(SecurityHandler.java:58)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.QueryStringRebuilder.handleRequest(QueryStringRebuilder.java:92)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.RequestInterceptorsExecutor.handleRequest(RequestInterceptorsExecutor.java:146)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.injectors.XPoweredByInjector.handleRequest(XPoweredByInjector.java:63)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.CORSHandler.handleRequest(CORSHandler.java:90)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.ServiceExchangeInitializer.handleRequest(ServiceExchangeInitializer.java:103)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.BeforeExchangeInitInterceptorsExecutor.handleRequest(BeforeExchangeInitInterceptorsExecutor.java:84)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.RequestLogger.handleRequest(RequestLogger.java:86)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.TracingInstrumentationHandler.handleRequest(TracingInstrumentationHandler.java:62)
    at org.restheart.handlers.PipelinedHandler.next(PipelinedHandler.java:80)
    at org.restheart.handlers.WorkingThreadsPoolDispatcher.handleRequest(WorkingThreadsPoolDispatcher.java:70)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
    at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
    at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
    at java.base/java.lang.Thread.run(Unknown Source)

12:57:12.233 [XNIO-1 task-2] ^[[39mDEBUG^[[0;39m org.mongodb.driver.operation - Unable to retry the operation find due to the error "com.mongodb.MongoQueryException: Command failed with error 251 (NoSuchTransaction): 'Given transaction number 1 does not match any in-progress transactions. The active transaction number is -1' on server vdemodev147dsy.ux.dsone.3ds.com:27017. The full response is {"errorLabels": ["TransientTransactionError"], "ok": 0.0, "errmsg": "Given transaction number 1 does not match any in-progress transactions. The active transaction number is -1", "code": 251, "codeName": "NoSuchTransaction", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1678193831, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "ZmPCD7m2giekDZPHdgvWs6qa/jc=", "subType": "00"}}, "keyId": 7181198839469047813}}, "operationTime": {"$timestamp": {"t": 1678193831, "i": 1}}}"

Answer 1 · 2023-03-07T13:22:44.000Z

Can you please help understanding this problem?
Thanks

Answer 2 · 2023-03-07T14:23:11.000Z

I replicated the problem that occurs only with MongoDb 6.

Should be easy to fix, I'll update you soon....

Answer 3 · 2023-03-07T14:24:15.000Z

WHAAATT --- ALREADY !! MAN YOU ARE AWESOME!!!!!!!!!!!!!!!!!!!!!!!!!!

Answer 4 · 2023-03-07T17:56:50.000Z

I commited a fix

You can try it with docker snapshot image

$ docker pull softinstigate/restheart-snapshot:a80fddf

Please try, if it works for you we'll release 7.3.1 with it

Answer 5 · 2023-03-07T19:05:32.000Z

I confirm it works now :)

So the next step you deliver a new minor version?

I had also one question about sessions. Is there any way to delete a created session?
Nothing on the documentation and i tried a DELETE on a create one, but no luck. So i feel it is not possible. But it will be a good thing to do it to remove ended sessions and avoid stale sessions on mongodb. It is doable via the standard mongodb drivers.
What do you think?

Thanks a lot again

Answer 6 · 2023-03-08T09:04:41.000Z

yes 7.3.1 will be releases soon with the fix

it is not possible to delete a session.

there is a way to delete a session, e.g. with mongosh:

db.runCommand( { killSessions: [ { "id": UUID("278829b0-513b-4c9a-b054-5af7ff0b3f92") } ] } )

however we tried it and to be honest we didn't find it actually removing the session: it is still visible and usable:

db.runCommand( { killSessions: [ { "id": UUID("278829b0-513b-4c9a-b054-5af7ff0b3f92") } ] } )
{
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1678266123, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1678266123, i: 1 })
}

db.system.sessions.aggregate([{ $listSessions: { allUsers: true } }, { $match: { "_id.id": new UUID("278829b0-513b-4c9a-b054-5af7ff0b3f92") }}])
[
  {
    _id: {
      id: new UUID("278829b0-513b-4c9a-b054-5af7ff0b3f92"),
      uid: Binary(Buffer.from("e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "hex"), 0)
    },
    lastUse: ISODate("2023-03-08T08:59:03.199Z")
  }
]

and from RESTHeart

$ http -a admin:secret get :8080/coll\?sid=278829b0-513b-4c9a-b054-5af7ff0b3f92
HTTP/1.1 200 OK

....

idle sessions are anyway automatically removed by MongoDb.

Answer 7 · 2023-03-08T09:43:46.000Z

RESTHeart v 7.3.1 released.
https://github.com/SoftInstigate/restheart/releases/tag/7.3.1

Answer 8 · 2023-03-08T22:49:09.000Z

OK thanks guys for the quick fix delivery!

For the sessions, i do not reproduce on mongo6. If i do exactly what you did.

I list sessions
i select one of the sessions
i delete this session
i try to get again that session --> no longer available

dbrs [primary] config> db.system.sessions.aggregate([{ $listSessions: { allUsers: true } }, { $match: { "_id.id": new UUID("b39b6246-bcb7-4ab9-b24f-6aad6b38ff7f") }}])
[
{
_id: {
id: new UUID("b39b6246-bcb7-4ab9-b24f-6aad6b38ff7f"),
uid: Binary(Buffer.from("6399ab0dac62f20bfc466753b10fb58fb7e692bec952c69b84d997021794d1f8", "hex"), 0)
},
lastUse: ISODate("2023-03-08T22:15:47.771Z"),
user: { name: 'root@admin' }
}
]

dbrs [primary] config> db.runCommand( { killSessions: [ { "id": UUID("b39b6246-bcb7-4ab9-b24f-6aad6b38ff7f") } ] } )
{
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1678315540, i: 1 }),
signature: {
hash: Binary(Buffer.from("3579a2bc201162f71e916b4ab48d093390e1b57b", "hex"), 0),
keyId: Long("7205183324124348422")
}
},
operationTime: Timestamp({ t: 1678315540, i: 1 })
}

dbrs [primary] config> db.system.sessions.aggregate([{ $listSessions: { allUsers: true } }, { $match: { "_id.id": new UUID("b39b6246-bcb7-4ab9-b24f-6aad6b38ff7f") }}])

dbrs [primary] config>

Answer 9 · 2023-03-10T09:24:42.000Z

I have added DELETE /_sessions/sid to kill a session

You can try it with snapshot release:

$ docker pull softinstigate/restheart-snapshot:67928f6

Please test in and let me know if it works as you expect.

I did the following test to check it:

create a session:

POST /_sessions

HTTP/1.1 201 Created
Location: http://localhost:8080/_sessions/57c329fa-373a-4bde-b2dc-611ffe4bb9d4

start a transaction in the session

POST :8080/_sessions/0e4a51c8-73fb-4d20-b0b1-e3e0fcc1af53/_txns/

HTTP/1.1 201 Created
Location: /_sessions/0e4a51c8-73fb-4d20-b0b1-e3e0fcc1af53/_txns/1

the transaction is in progress:

GET :8080/_sessions/0e4a51c8-73fb-4d20-b0b1-e3e0fcc1af53/_txns/

HTTP/1.1 201 Created
{
    "currentTxn": {
        "id": 1,
        "status": "IN"
    }
}

kill the session with new endpoint

DELETE :8080/_sessions/0e4a51c8-73fb-4d20-b0b1-e3e0fcc1af53

HTTP/1.1 200 Ok

killing the session the transaction aborts

GET :8080/_sessions/0e4a51c8-73fb-4d20-b0b1-e3e0fcc1af53/_txns/

HTTP/1.1 201 Created
{
    "currentTxn": {
        "id": 1,
        "status": "ABORTED"
    }
}

Answer 10 · 2023-03-10T10:07:18.000Z

Hello

Thanks :)

Ok so if i understand, here the DELETE of SESSION aborts the current IN transaction.
But does not clear it from mongodb right? still visible in mongodb?

this is what i have on my setup now with the new snapshot + mongodb6

Answer 11 · 2023-03-10T11:26:13.000Z

as I said, even killing the transaction with mongosh, I can still see it being listed (with mongodb 6 as well) either with:

db.aggregate( [  { $listLocalSessions: { allUsers: true } } ] )

or:

use config

db.system.sessions.aggregate( [  { $listSessions: { allUsers: true } } ] )

I checked the transaction to be aborted, because I think that killing a session should kill any ongoing operations associated with it.

Answer 12 · 2023-03-10T13:17:21.000Z

ok thanks a lot :)

Answer 13 · 2023-03-13T13:41:35.000Z

thanks guys