`java.lang.OutOfMemoryError: Direct buffer memory` during uploads
phlax opened this issue · 4 comments
phlax commented
We are seeing the above error quite frequently during uploads
fuller trace:
WARNING: Remote Cache: Error while uploading artifact with digest 'dd662f55f250fdc85f7d649f5deb2a38995973e85edc0b36a23b12bec6a0e71c/3594364'
com.google.devtools.build.lib.remote.common.BulkTransferException: Error while uploading artifact with digest 'dd662f55f250fdc85f7d649f5deb2a38995973e85edc0b36a23b12bec6a0e71c/3594364'
at com.google.devtools.build.lib.remote.util.RxUtils$BulkTransferExceptionCollector.onResult(RxUtils.java:91)
at io.reactivex.rxjava3.internal.operators.flowable.FlowableCollectSingle$CollectSubscriber.onNext(FlowableCollectSingle.java:94)
at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.innerSuccess(FlowableFlatMapSingle.java:173)
at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber$InnerObserver.onSuccess(FlowableFlatMapSingle.java:342)
at io.reactivex.rxjava3.internal.operators.single.SingleDoFinally$DoFinallyObserver.onSuccess(SingleDoFinally.java:73)
at io.reactivex.rxjava3.internal.observers.ResumeSingleObserver.onSuccess(ResumeSingleObserver.java:46)
at io.reactivex.rxjava3.internal.operators.single.SingleJust.subscribeActual(SingleJust.java:30)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleResumeNext$ResumeMainSingleObserver.onError(SingleResumeNext.java:80)
at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.tryOnError(CompletableCreate.java:91)
at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.onError(CompletableCreate.java:77)
at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe$1.onFailure(RxFutures.java:102)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1124)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at io.grpc.Status.asRuntimeException(Status.java:535)
... 10 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Unknown Source)
phlax commented
following guidance from the interwebs i set:
startup --host_jvm_args=-XX:MaxDirectMemorySize=512m
altho that doesnt seep to have helped (fwiw i noticed an errant "
so will remove and see if that does help after all)
phlax commented
seems like this may be more of a bazel issue - i switched to using google RBE and got a similar result
mostynb commented
I think you're right- this is happening on the client/bazel side.
phlax commented
thanks - i raised it upstream so will close here