Provisioning of new volumes is not working if remote is not available
kvaps opened this issue · 1 comments
kvaps commented
logs:
W0109 08:32:52.666743 1 controller.go:934] Retrying syncing claim "484e23e7-f6a5-4d16-a46e-41620f8c4a2d", failure 6
E0109 08:32:52.666771 1 controller.go:957] error syncing claim "484e23e7-f6a5-4d16-a46e-41620f8c4a2d": failed to provision volume with StorageClass "linstor-thindata-r2": rpc error: code = Internal desc = failed to check on potential left-over source snapshot: failed to check remote 'minio' for backups of id 'for-pvc-484e23e7-f6a5-4d16-a46e-41620f8c4a2d': Message: 'Exception thrown.'; Details: 'Client execution did not complete before the specified timeout configuration.'; Reports: '[63BBCE26-00000-000006]'
I0109 08:32:52.666785 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"disk-mydata", UID:"484e23e7-f6a5-4d16-a46e-41620f8c4a2d", APIVersion:"v1", ResourceVersion:"110412648", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "linstor-thindata-r2": rpc error: code = Internal desc = failed to check on potential left-over source snapshot: failed to check remote 'minio' for backups of id 'for-pvc-484e23e7-f6a5-4d16-a46e-41620f8c4a2d': Message: 'Exception thrown.'; Details: 'Client execution did not complete before the specified timeout configuration.'; Reports: '[63BBCE26-00000-000006]'
report:
ERROR REPORT 63BBCE26-00000-000006
============================================================
Application: LINBIT�� LINSTOR
Module: Controller
Version: 1.20.0
Build ID: 9c6f7fad48521899f7a99c564b1d33aeacfdbfa8
Build time: 2022-11-07T16:37:38+00:00
Error time: 2023-01-09 08:32:52
Node: linstor-controller-587ccf8cdd-vj2x4
============================================================
Reported error:
===============
Category: RuntimeException
Class name: ClientExecutionTimeoutException
Class canonical name: com.amazonaws.http.timers.client.ClientExecutionTimeoutException
Generated at: Method 'handleInterruptedException', Source file 'AmazonHttpClient.java', Line #872
Error message: Client execution did not complete before the specified timeout configuration.
Call backtrace:
Method Native Class:Line number
handleInterruptedException N com.amazonaws.http.AmazonHttpClient$RequestExecutor:872
execute N com.amazonaws.http.AmazonHttpClient$RequestExecutor:755
access$500 N com.amazonaws.http.AmazonHttpClient$RequestExecutor:713
execute N com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl:695
execute N com.amazonaws.http.AmazonHttpClient:559
execute N com.amazonaws.http.AmazonHttpClient:539
invoke N com.amazonaws.services.s3.AmazonS3Client:5453
invoke N com.amazonaws.services.s3.AmazonS3Client:5400
invoke N com.amazonaws.services.s3.AmazonS3Client:5394
listObjectsV2 N com.amazonaws.services.s3.AmazonS3Client:971
listObjects N com.linbit.linstor.api.BackupToS3:388
getAllS3Keys N com.linbit.linstor.core.apicallhandler.controller.backup.CtrlBackupApiHelper:259
listBackups N com.linbit.linstor.core.apicallhandler.controller.backup.CtrlBackupApiCallHandler:661
lambda$listBackups$0 N com.linbit.linstor.api.rest.v1.Backups:305
doInScope N com.linbit.linstor.api.rest.v1.RequestHelper:231
doInScope N com.linbit.linstor.api.rest.v1.RequestHelper:203
listBackups N com.linbit.linstor.api.rest.v1.Backups:299
invoke0 Y jdk.internal.reflect.NativeMethodAccessorImpl:unknown
invoke N jdk.internal.reflect.NativeMethodAccessorImpl:62
invoke N jdk.internal.reflect.DelegatingMethodAccessorImpl:43
invoke N java.lang.reflect.Method:566
lambda$static$0 N org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
run N org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:124
invoke N org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:167
doDispatch N org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker:176
dispatch N org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:79
invoke N org.glassfish.jersey.server.model.ResourceMethodInvoker:469
apply N org.glassfish.jersey.server.model.ResourceMethodInvoker:391
apply N org.glassfish.jersey.server.model.ResourceMethodInvoker:80
run N org.glassfish.jersey.server.ServerRuntime$1:253
call N org.glassfish.jersey.internal.Errors$1:248
call N org.glassfish.jersey.internal.Errors$1:244
process N org.glassfish.jersey.internal.Errors:292
process N org.glassfish.jersey.internal.Errors:274
process N org.glassfish.jersey.internal.Errors:244
runInScope N org.glassfish.jersey.process.internal.RequestScope:265
process N org.glassfish.jersey.server.ServerRuntime:232
handle N org.glassfish.jersey.server.ApplicationHandler:680
service N org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
run N org.glassfish.grizzly.http.server.HttpHandler$1:200
doWork N org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:569
run N org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:549
run N java.lang.Thread:829
END OF ERROR REPORT.
WanzenBug commented
I can probably fix this specific case for new volumes, but a lot of other calls involving snapshots probably won't work until the remote is available.