Failed to obtain input file
ling2yt opened this issue · 2 comments
I'm seeking help actually.
I run the remote execution with docker-compose referring to the deployment repo:
It runs two shared storage. One of them down unexpected. It works well after I restart the storage service.
But I always get "Failed to obtain input file "external/xxx": Blob not found" error. And the error is gone after I re-run the "bazel build xxx" command.
I'm wondering whether there is any problem with one of the storage services. How can I figure out which the bad storage service is? How can I fix this problem? Clean all the disk storage of the bad one?
Are you running the example build or something else?
Hey there!
It could be the case that the size of your storage is too small, causing objects to be evicted during the build. You may run Prometheus queries like the following to get an estimation on the retention duration of your storage setup:
time() - buildbarn_blobstore_old_new_current_location_blob_map_last_removed_old_block_insertion_time_seconds