storage: PCU hangs

Question

storage: PCU hangs

benjaminp opened this issue 6 months ago · 9 comments

Environment details

Specify the API at the beginning of the title. For example, "BigQuery: ...").
General, Core, and Other are also allowed as types
OS type and version: Linux
Java version: 17
version(s): 2.37.0

Steps to reproduce

I wanted to try out the new parallel composite support, but my example below simply hangs. The program completes with .setAllowParallelCompositeUpload(false).

I see it created many 16mib intermediate objects in the bucket, though. (That also seems wrong since 4gib/16mib ≫32, the maximum number of composite object components.)

Code example

import com.google.cloud.storage.Storage.BlobWriteOption;
import com.google.cloud.storage.StorageOptions;
import com.google.cloud.storage.transfermanager.ParallelUploadConfig;
import com.google.cloud.storage.transfermanager.TransferManagerConfig;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.List;

class X {
  private static final String BUCKET_NAME = "SOME BUCKET";
  private static final String PREFIX = "prefix";

  /** Will create a large local file at this path. */
  private static final String TMP_PATH = "/tmp/x";

  public static void main(String[] args) throws Exception {
    var path = Path.of(TMP_PATH);
    try (var ch = Files.newByteChannel(path, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {
      ch.position(4L * 1024L * 1024L * 1024L - 1);
      ch.write(ByteBuffer.wrap(new byte[1]));
    }
    var options = StorageOptions.newBuilder().build();
    var tmc =
        TransferManagerConfig.newBuilder()
            .setStorageOptions(options)
            .setAllowParallelCompositeUpload(true)
            .build();
    try (var tm = tmc.getService()) {
      System.out.println(
          tm.uploadFiles(
                  List.of(path),
                  ParallelUploadConfig.newBuilder()
                      .setBucketName(BUCKET_NAME)
                      .setPrefix(PREFIX)
                      .setWriteOptsPerRequest(List.of(BlobWriteOption.disableGzipContent()))
                      .build())
              .getUploadResults());
    }
  }
}

Answer 1 · 2024-04-30T21:39:35.000Z

is this being run from a Google Compute Engine instance?

Answer 2 · 2024-04-30T22:28:50.000Z

Yes, I ran the test from a GCE VM.

Answer 3 · 2024-05-01T19:42:32.000Z

I have tried your same code with smaller file sizes and it has completed just fine for me. The 4GB file is "hanging" but I think this is just file generation + upload time coming from my laptop. Working on running it from GCE.

Have you experienced this for other file sizes?

Answer 4 · 2024-05-02T17:25:08.000Z

as for the 16mib intermediate objects that is correct as the per worker buffer size is set by transfer manager (of which 16 mib is the default) so that is to be expected. There will be intermediate composes to reach the final object.

Answer 5 · 2024-05-02T21:19:06.000Z

I ran the following on my personal GCE instance with success.

d7f6094

Answer 6 · 2024-05-03T17:43:04.000Z

perhaps if we got a thread dump we could assist you better jstack -F <pid> for a dump of the process.

Answer 7 · 2024-05-03T19:05:35.000Z

I will try publishing my reproducer more completely: https://github.com/benjaminp/java-storage-puc-hang

I put a bucket name in the BUCKET_NAME constant. Then I do bazel build //:X_deploy.jar. Then, I push bazel-bin/X_deploy.jar to a n1-standard-4 VM running Debian 12 in the region with the bucket. On that VM, I execute:

$ java --version
openjdk 17.0.11 2024-04-16
OpenJDK Runtime Environment (build 17.0.11+9-Debian-1deb12u1)
OpenJDK 64-Bit Server VM (build 17.0.11+9-Debian-1deb12u1, mixed mode, sharing)
$ java -Xmx1g -jar X_deploy.jar

This last command hangs.

The stacks are not too illuminating to me anyway:

2024-05-03 18:54:58
Full thread dump OpenJDK 64-Bit Server VM (17.0.11+9-Debian-1deb12u1 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007ff064001890, length=20, elements={
0x00007ff0f00138d0, 0x00007ff0f01366d0, 0x00007ff0f0137ac0, 0x00007ff0f013e6a0,
0x00007ff0f013fa60, 0x00007ff0f0140e80, 0x00007ff0f0142840, 0x00007ff0f0143d80,
0x00007ff0f014d1f0, 0x00007ff0f0154920, 0x00007ff0f01581b0, 0x00007ff0f03abdb0,
0x00007ff02c05bb90, 0x00007ff02c051980, 0x00007ff02c04ef00, 0x00007ff02c060d10,
0x00007ff02c061cd0, 0x00007ff02c062e20, 0x00007ff02c063ef0, 0x00007ff064000ea0
}

"main" #1 prio=5 os_prio=0 cpu=446.29ms elapsed=39.94s tid=0x00007ff0f00138d0 nid=0x48c2e waiting on condition  [0x00007ff0f5dfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c015ffb0> (a com.google.common.util.concurrent.CollectionFuture$ListFuture)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:211)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:563)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:110)
	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:67)
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:247)
	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1378)
	at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:53)
	at com.google.cloud.storage.transfermanager.UploadJob.getUploadResults(UploadJob.java:60)
	at X.main(X.java:39)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=2.87ms elapsed=39.91s tid=0x00007ff0f01366d0 nid=0x48c35 waiting on condition  [0x00007ff0f42c8000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@17.0.11/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@17.0.11/Reference.java:253)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@17.0.11/Reference.java:215)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=1.61ms elapsed=39.91s tid=0x00007ff0f0137ac0 nid=0x48c36 in Object.wait()  [0x00007ff0f41c7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@17.0.11/Native Method)
	- waiting on <0x00000000c0000ae8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@17.0.11/ReferenceQueue.java:155)
	- locked <0x00000000c0000ae8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@17.0.11/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@17.0.11/Finalizer.java:172)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.44ms elapsed=39.91s tid=0x00007ff0f013e6a0 nid=0x48c37 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=5.40ms elapsed=39.91s tid=0x00007ff0f013fa60 nid=0x48c38 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #6 daemon prio=9 os_prio=0 cpu=4.99ms elapsed=39.91s tid=0x00007ff0f0140e80 nid=0x48c39 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=2910.82ms elapsed=39.91s tid=0x00007ff0f0142840 nid=0x48c3a waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #9 daemon prio=9 os_prio=0 cpu=1179.88ms elapsed=39.91s tid=0x00007ff0f0143d80 nid=0x48c3b waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #10 daemon prio=9 os_prio=0 cpu=0.15ms elapsed=39.91s tid=0x00007ff0f014d1f0 nid=0x48c3c runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Notification Thread" #11 daemon prio=9 os_prio=0 cpu=0.15ms elapsed=39.91s tid=0x00007ff0f0154920 nid=0x48c3d runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #12 daemon prio=8 os_prio=0 cpu=1.79ms elapsed=39.91s tid=0x00007ff0f01581b0 nid=0x48c3f in Object.wait()  [0x00007ff0d4cf7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@17.0.11/Native Method)
	- waiting on <0x00000000c0000cf8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@17.0.11/ReferenceQueue.java:155)
	- locked <0x00000000c0000cf8> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@17.0.11/CleanerImpl.java:140)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)
	at jdk.internal.misc.InnocuousThread.run(java.base@17.0.11/InnocuousThread.java:162)

"pool-1-thread-1" #15 prio=5 os_prio=0 cpu=2070.69ms elapsed=39.50s tid=0x00007ff0f03abdb0 nid=0x48c48 waiting on condition  [0x00007ff0d46f1000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c0049ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.11/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.11/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.11/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.11/AbstractQueuedSynchronizer.java:1625)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.11/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.11/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.11/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.11/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)

"pool-1-thread-2" #16 prio=5 os_prio=0 cpu=1489.22ms elapsed=39.38s tid=0x00007ff02c05bb90 nid=0x48c49 waiting on condition  [0x00007ff0d45f0000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c0049ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.11/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.11/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.11/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.11/AbstractQueuedSynchronizer.java:1625)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.11/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.11/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.11/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.11/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)

"pool-1-thread-3" #17 prio=5 os_prio=0 cpu=1211.94ms elapsed=39.34s tid=0x00007ff02c051980 nid=0x48c4a waiting on condition  [0x00007ff0d44ef000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c0049ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.11/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.11/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.11/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.11/AbstractQueuedSynchronizer.java:1625)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.11/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.11/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.11/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.11/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)

"pool-1-thread-4" #18 prio=5 os_prio=0 cpu=1288.16ms elapsed=39.32s tid=0x00007ff02c04ef00 nid=0x48c4b waiting on condition  [0x00007ff0d43ee000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c0049ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.11/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.11/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.11/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.11/AbstractQueuedSynchronizer.java:1625)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.11/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.11/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.11/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.11/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)

"pool-1-thread-5" #19 prio=5 os_prio=0 cpu=1108.59ms elapsed=39.29s tid=0x00007ff02c060d10 nid=0x48c4c waiting on condition  [0x00007ff0d42ed000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c0049ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.11/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.11/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.11/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.11/AbstractQueuedSynchronizer.java:1625)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.11/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.11/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.11/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.11/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)

"pool-1-thread-6" #20 prio=5 os_prio=0 cpu=1077.57ms elapsed=39.27s tid=0x00007ff02c061cd0 nid=0x48c4d waiting on condition  [0x00007ff0d41ec000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c0049ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.11/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.11/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.11/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.11/AbstractQueuedSynchronizer.java:1625)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.11/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.11/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.11/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.11/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)

"pool-1-thread-7" #21 prio=5 os_prio=0 cpu=1403.73ms elapsed=39.23s tid=0x00007ff02c062e20 nid=0x48c4e waiting on condition  [0x00007ff07affe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c0049ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.11/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.11/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.11/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.11/AbstractQueuedSynchronizer.java:1625)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.11/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.11/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.11/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.11/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)

"pool-1-thread-8" #22 prio=5 os_prio=0 cpu=1212.10ms elapsed=39.17s tid=0x00007ff02c063ef0 nid=0x48c51 waiting on condition  [0x00007ff07aefd000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.11/Native Method)
	- parking to wait for  <0x00000000c0049ec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.11/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.11/AbstractQueuedSynchronizer.java:506)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.11/ForkJoinPool.java:3465)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.11/ForkJoinPool.java:3436)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.11/AbstractQueuedSynchronizer.java:1625)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.11/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.11/ThreadPoolExecutor.java:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.11/ThreadPoolExecutor.java:1122)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.11/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(java.base@17.0.11/Thread.java:840)

"Attach Listener" #38 daemon prio=9 os_prio=0 cpu=1.23ms elapsed=23.58s tid=0x00007ff064000ea0 nid=0x48d23 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=101.33ms elapsed=39.92s tid=0x00007ff0f0132740 nid=0x48c34 runnable  

"GC Thread#0" os_prio=0 cpu=199.72ms elapsed=39.94s tid=0x00007ff0f0050bc0 nid=0x48c2f runnable  

"GC Thread#1" os_prio=0 cpu=199.45ms elapsed=39.54s tid=0x00007ff0740050d0 nid=0x48c45 runnable  

"GC Thread#2" os_prio=0 cpu=203.19ms elapsed=39.54s tid=0x00007ff074005b20 nid=0x48c46 runnable  

"GC Thread#3" os_prio=0 cpu=196.49ms elapsed=39.54s tid=0x00007ff074006570 nid=0x48c47 runnable  

"G1 Main Marker" os_prio=0 cpu=11.15ms elapsed=39.94s tid=0x00007ff0f005a1b0 nid=0x48c30 runnable  

"G1 Conc#0" os_prio=0 cpu=759.31ms elapsed=39.94s tid=0x00007ff0f005b120 nid=0x48c31 runnable  

"G1 Refine#0" os_prio=0 cpu=0.13ms elapsed=39.93s tid=0x00007ff0f0109e20 nid=0x48c32 runnable  

"G1 Service" os_prio=0 cpu=8.70ms elapsed=39.93s tid=0x00007ff0f010ad20 nid=0x48c33 runnable  

"VM Periodic Task Thread" os_prio=0 cpu=29.73ms elapsed=39.91s tid=0x00007ff0f0156270 nid=0x48c3e waiting on condition  

JNI global refs: 16, weak refs: 0

Answer 8 · 2024-05-06T19:06:30.000Z

We were able to reproduce this finally. We don't run with heaps that small typically which is why we were unable to reproduce it initially. Turns out we were suppressing an OOM causing the operation to hang.

Answer 9 · 2024-05-09T15:27:23.000Z

fix went out in #2533 closing.