High CPU Load
Opened this issue · 16 comments
Hello,
our Monitoring + htop shows extremely High CPU usage when using the latest standalone.
The drop off u see here was caused by a restart of the standalone.
Java Version:
openjdk version "17.0.12" 2024-07-16
OpenJDK Runtime Environment (build 17.0.12+7-Ubuntu-1ubuntu222.04)
OpenJDK 64-Bit Server VM (build 17.0.12+7-Ubuntu-1ubuntu222.04, mixed mode, sharing)
Used Standalone:
What are your startup flags?
Can you enable debug and share a log file when this happens?
Can you also provide a heap dump of when the issue occurs?
With no flags... just java -jar ...jar
Just keeps claiming until I kill it (10-15% is our base load...).
I am having the exact same issue. I am running the standalone on Linux (Debian). What I am noticing is that MCXboxBroadcast starts out using very little CPU and the usage climbs until it reaches 100% or if it is restarted. (I use a cron job to restart it every 8 hours). One thing to note is that I am running MCXboxBroadcast on a quad core system, and the CPU usage will only climb to 100% of a single core. (25% of the total CPU). I run the Minecraft Bedrock server on the exact same machine, but I have not really noticed any performance dips when playing on the server. My biggest concern is the excess power consumption and the fact that the physical server runs a bit hotter than normal even when no one is actively playing on the server.
I am not using any startup flags when started MCXboxBroadcast, I am just using "java -jar MCXboxBroadcastStandalone.jar"
@TylerJohnson177 My current workaround is that I restart it every 15 min using crontab....
Maybe that helps for u too until it gets fixed.
Try build 60 I've added better handling of thread cleanup so there should be less threads that sit in the background hogging resources.
I am also experiencing this issue on build 60. CPU usage is extremely high, reaches 100+% of 1 core pretty much instantly.
https://paste.md-5.net/qujikowuco.rb
https://paste.md-5.net/awejuhonir.bash
openjdk version "21.0.4" 2024-07-16
OpenJDK Runtime Environment (build 21.0.4+7-Ubuntu-1ubuntu224.04)
OpenJDK 64-Bit Server VM (build 21.0.4+7-Ubuntu-1ubuntu224.04, mixed mode, sharing)
Using extension build #60
Startup flags:
java
-Xms4000M
-Xmx4000M
-XX:+UseG1GC
-XX:G1HeapRegionSize=4M
-XX:+UnlockExperimentalVMOptions
-XX:+ParallelRefProcEnabled
-XX:+AlwaysPreTouch
-XX:MaxInlineLevel=15
-jar velocity.jar
@rtm516 Jea, we also still habe the same issue with 60, we're currently still work around it with scheduled restarts.
@rtm516 Jea, we also still habe the same issue with 60, we're currently still work around it with scheduled restarts.
How often are you restarting? I am seeing CPU usage spike significantly < 5m after restarting the bot. Are you restarting your entire proxy on a schedule?
If someone can provide a debug log along side a thread dump that would be much appriciated as it might help me work out what exactally is going wrong
If someone can provide a debug log along side a thread dump that would be much appriciated as it might help me work out what exactally is going wrong
What type of debug log do you want and thread dump from spark?
If someone can provide a debug log along side a thread dump that would be much appriciated as it might help me work out what exactally is going wrong
What type of debug log do you want and thread dump from spark?
A debug log from the server, enable it in the geyser config and then when the issue occurs send over that and the spark thread dump
If someone can provide a debug log along side a thread dump that would be much appriciated as it might help me work out what exactally is going wrong
What type of debug log do you want and thread dump from spark?
A debug log from the server, enable it in the geyser config and then when the issue occurs send over that and the spark thread dump
Hmmm that is bit difficult for me, geyser-debug is preeettty annoying as it generates a lot of logs when players move around item frames etc, I may hit PID limit, I will get you the spark dump soon. - Feel free to send me the exact command you want me to use.
[18:56:20 INFO] [geyser]: [mcxboxbroadcast] [Primary Session] Successfully authenticated as Utopiverse (2535421049843580)
....[27821.487s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[27821.487s][warning][os,thread] Failed to start the native thread for java.lang.Thread "HttpClient-95-Worker-1"
[18:56:20 ERROR] [geyser]: [mcxboxbroadcast] [Primary Session] Failed to check profile settings
java.io.IOException: unable to create native thread: possibly out of memory or process/resource limits reached
at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:966) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133) ~[java.net.http:?]
at com.github.mizosoft.methanol.Methanol$InterceptorChain.forward(Methanol.java:828) ~[?:?]
at com.github.mizosoft.methanol.Methanol$AutoDecompressingInterceptor.intercept(Methanol.java:937) ~[?:?]
at com.github.mizosoft.methanol.Methanol$InterceptorChain.forward(Methanol.java:833) ~[?:?]
at com.github.mizosoft.methanol.Methanol$RequestRewritingInterceptor.intercept(Methanol.java:881) ~[?:?]
at com.github.mizosoft.methanol.Methanol$InterceptorChain.forward(Methanol.java:833) ~[?:?]
at com.github.mizosoft.methanol.Methanol.send(Methanol.java:316) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.SessionManagerCore.checkGamertagUpdate(SessionManagerCore.java:492) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.SessionManagerCore.init(SessionManagerCore.java:148) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.SessionManager.init(SessionManager.java:79) ~[?:?]
at com.rtm516.mcxboxbroadcast.bootstrap.geyser.MCXboxBroadcastExtension.createSession(MCXboxBroadcastExtension.java:213) ~[?:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method) ~[?:?]
at java.base/java.lang.Thread.start(Thread.java:1553) [?:?]
at java.base/java.lang.System$2.start(System.java:2577) ~[?:?]
at java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1375) ~[?:?]
at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.ensureExecutedAsync(HttpClientImpl.java:183) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:175) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.SocketTube$InternalWriteSubscriber.startSubscription(SocketTube.java:396) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.AsyncTriggerEvent.handle(AsyncTriggerEvent.java:54) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:1314) ~[java.net.http:?]
....[27821.488s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[18:56:20 INFO] [geyser]: [mcxboxbroadcast] [Primary Session] Creating Xbox LIVE session...
....[27821.488s][warning][os,thread] Failed to start the native thread for java.lang.Thread "HttpClient-95-Worker-2"
[27821.570s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[27821.570s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-98-thread-2"
[18:56:20 ERROR] [geyser]: [mcxboxbroadcast] [Auth] Failed to get/refresh auth token
java.io.IOException: unable to create native thread: possibly out of memory or process/resource limits reached
at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:966) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133) ~[java.net.http:?]
at net.lenni0451.commons.httpclient.executor.HttpClientExecutor.execute(HttpClientExecutor.java:48) ~[?:?]
at net.lenni0451.commons.httpclient.HttpClient.execute(HttpClient.java:232) ~[?:?]
at net.lenni0451.commons.httpclient.HttpClient.execute(HttpClient.java:213) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.AuthManager.fetchPlayfabSessionTicket(AuthManager.java:142) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.AuthManager.initialise(AuthManager.java:129) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.AuthManager.getPlayfabSessionTicket(AuthManager.java:169) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.SessionManagerCore.setupSession(SessionManagerCore.java:375) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.SessionManagerCore.createSession(SessionManagerCore.java:196) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.SessionManagerCore.init(SessionManagerCore.java:162) ~[?:?]
at com.rtm516.mcxboxbroadcast.core.SessionManager.init(SessionManager.java:79) ~[?:?]
at com.rtm516.mcxboxbroadcast.bootstrap.geyser.MCXboxBroadcastExtension.createSession(MCXboxBroadcastExtension.java:213) ~[?:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method) ~[?:?]
at java.base/java.lang.Thread.start(Thread.java:1553) [?:?]
at java.base/java.lang.System$2.start(System.java:2577) ~[?:?]
at java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1375) ~[?:?]
at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.ensureExecutedAsync(HttpClientImpl.java:183) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:175) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.SocketTube$InternalWriteSubscriber.startSubscription(SocketTube.java:396) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.AsyncTriggerEvent.handle(AsyncTriggerEvent.java:54) ~[java.net.http:?]
at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:1314) ~[java.net.http:?]
[18:56:26 INFO] [geyser]: [mcxboxbroadcast] [Primary Session] Starting SessionManager...
[18:56:27 INFO] [geyser]: [mcxboxbroadcast] [Primary Session] Successfully authenticated as Utopiverse (2535421049843580)
[18:56:28 INFO] [geyser]: [mcxboxbroadcast] [Primary Session] Creating Xbox LIVE session...
....[27829.615s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[27829.615s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-102-thread-1"
[27829.619s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[27829.620s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Thread-116"
[18:56:28 ERROR]: Exception in thread "WebSocketConnectReadThread-3313" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
[18:56:28 ERROR]: at java.base/java.lang.Thread.start0(Native Method)
[18:56:28 ERROR]: at java.base/java.lang.Thread.start(Thread.java:1526)
[18:56:28 ERROR]: at org.java_websocket.client.WebSocketClient.run(WebSocketClient.java:499)
[18:56:28 ERROR]: at java.base/java.lang.Thread.run(Thread.java:1583)
@rtm516 had to remove the plugin, it caused major issues, A lot of issues, Constant 1000% CPU Usage, Randomly hitting maximum process limit/memory limit causing some of the database plugins to stop working properly.