Infinite loop in RefreshTokenManager
MasseGuillaume opened this issue · 3 comments
MasseGuillaume commented
I'm using akka-http-session 0.4.0 in scastie https://github.com/scalacenter/scastie/blob/master/server/src/main/scala/com.olegych.scastie.web/oauth2/GithubUserSession.scala
I have two threads at 100% cpu usage. This the result of jstack:
"Web-akka.actor.default-dispatcher-214" #273 prio=5 os_prio=0 tid=0x00007f69fc024000 nid=0x33ba runnable [0x00007f69bfbf8000]
java.lang.Thread.State: RUNNABLE
at scala.collection.mutable.HashTable.removeEntry(HashTable.scala:193)
at scala.collection.mutable.HashTable.removeEntry$(HashTable.scala:181)
at scala.collection.mutable.HashMap.removeEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.remove(HashMap.scala:123)
at com.softwaremill.session.InMemoryRefreshTokenStorage.remove(RefreshTokenStorage.scala:59)
at com.softwaremill.session.InMemoryRefreshTokenStorage.remove$(RefreshTokenStorage.scala:57)
at com.olegych.scastie.web.oauth2.GithubUserSession$$anon$1.remove(GithubUserSession.scala:52)
at com.softwaremill.session.RefreshTokenManager.$anonfun$rotateToken$4(SessionManager.scala:130)
at com.softwaremill.session.RefreshTokenManager$$Lambda$1909/515478046.apply(Unknown Source)
at com.softwaremill.session.InMemoryRefreshTokenStorage.schedule(RefreshTokenStorage.scala:64)
at com.softwaremill.session.InMemoryRefreshTokenStorage.schedule$(RefreshTokenStorage.scala:62)
at com.olegych.scastie.web.oauth2.GithubUserSession$$anon$1.schedule(GithubUserSession.scala:52)
at com.softwaremill.session.RefreshTokenManager.$anonfun$rotateToken$3(SessionManager.scala:130)
at com.softwaremill.session.RefreshTokenManager.$anonfun$rotateToken$3$adapted(SessionManager.scala:127)
at com.softwaremill.session.RefreshTokenManager$$Lambda$1736/616177409.apply(Unknown Source)
at scala.Option.foreach(Option.scala:257)
at com.softwaremill.session.RefreshTokenManager.rotateToken(SessionManager.scala:127)
at com.softwaremill.session.RefreshTokenManager.rotateToken$(SessionManager.scala:115)
at com.softwaremill.session.SessionManager$$anon$3.rotateToken(SessionManager.scala:24)
at com.softwaremill.session.RefreshableSessionDirectives.$anonfun$setRefreshToken$1(SessionDirectives.scala:202)
at com.softwaremill.session.RefreshableSessionDirectives$$Lambda$1731/488540775.apply(Unknown Source)
at akka.http.scaladsl.server.Directive$SingleValueModifiers.$anonfun$flatMap$1(Directive.scala:141)
at akka.http.scaladsl.server.Directive$SingleValueModifiers$$Lambda$616/1043639521.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$tflatMap$2(Directive.scala:69)
at akka.http.scaladsl.server.Directive$$Lambda$625/913242382.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$tmap$2(Directive.scala:63)
at akka.http.scaladsl.server.Directive$$Lambda$624/2117642238.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:61)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1028/41947542.apply(Unknown Source)
at akka.http.scaladsl.server.directives.OnSuccessMagnet$$anon$1.$anonfun$directive$3(FutureDirectives.scala:97)
at akka.http.scaladsl.server.directives.OnSuccessMagnet$$anon$1$$Lambda$1583/151472226.apply(Unknown Source)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:55)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.directives.OnSuccessMagnet$$anon$1.$anonfun$directive$2(FutureDirectives.scala:97)
at akka.http.scaladsl.server.directives.OnSuccessMagnet$$anon$1$$Lambda$1580/1652741769.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$5(Directive.scala:94)
at akka.http.scaladsl.server.Directive$$Lambda$1135/82863731.apply(Unknown Source)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$4(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1100/1660614532.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
at akka.http.scaladsl.server.Directive.$anonfun$recover$2(Directive.scala:93)
at akka.http.scaladsl.server.Directive$$Lambda$1098/542507242.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$1016/771814596.apply(Unknown Source)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1070/2000999693.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$1016/771814596.apply(Unknown Source)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$648/726212590.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:67)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$1015/218875704.apply(Unknown Source)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:154)
at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$626/870322840.apply(Unknown Source)
at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$878/485475507.apply(Unknown Source)
at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:79)
at akka.http.scaladsl.server.Route$$$Lambda$879/1869247587.apply(Unknown Source)
at akka.stream.impl.fusing.MapAsync$$anon$23.onPush(Ops.scala:1172)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:499)
at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:462)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:368)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:571)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:457)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:546)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:728)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:743)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:653)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
at akka.actor.ActorCell.invoke(ActorCell.scala:496)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
adamw commented
Is this on production? I see that you are using InMemoryRefreshTokenStorage, which is rather meant for tests, especially that it can cause race conditions with multiple concurrent HTTP requests. Plus after server restart all tokens are lost :). So my first guess would be to try using a database-backed refresh token storage (or remove refresh tokens altogether, if you don't need them).
And possibly better documenting InMemoryRefreshTokenStorage :)
MasseGuillaume commented
@adamw yep in production!
I back them up in a file for persistence.
adamw commented
Ah :) Would be good to know if the map maybe grows too big, causing pressure on the GC? Maybe you can see how much memory the app has and how much is free?
If you are using InMemoryRefreshTokenStorage, you should also properly implement schedule to avoid the race conditions :)