Timeout when submitting request to deploy job

[fwilliams]

I'm running caliban to deploy a model to Google Cloud as follows:

caliban cloud --project 'taglia' --region 'us-central1' --machine_type 'n1-standa
rd-32' --gpu_spec '4xV100' train_panoptic.py -- <TRAIN SCRIPT ARGS>

I am getting stuck with the following error:

I0708 16:33:54.308651 139689314522944 core.py:324] Submitting request!                                                   
W0708 16:33:54.516765 139689314522944 http.py:171] Sleeping 1.11 seconds before retry 1 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:33:55.784710 139689314522944 http.py:171] Sleeping 2.46 seconds before retry 2 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:33:58.420140 139689314522944 http.py:171] Sleeping 5.17 seconds before retry 3 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:34:03.754786 139689314522944 http.py:171] Sleeping 1.37 seconds before retry 4 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:34:05.324873 139689314522944 http.py:171] Sleeping 5.63 seconds before retry 5 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:34:11.127667 139689314522944 http.py:171] Sleeping 13.70 seconds before retry 6 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:34:25.041211 139689314522944 http.py:171] Sleeping 124.27 seconds before retry 7 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:36:29.560709 139689314522944 http.py:171] Sleeping 79.67 seconds before retry 8 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:37:49.466338 139689314522944 http.py:171] Sleeping 351.15 seconds before retry 9 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:43:40.860112 139689314522944 http.py:171] Sleeping 656.62 seconds before retry 10 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
Submitting caliban_franciswi_google_com_1:   0%|                                             | 0/1 [09:46<?, ?requests/s]

I'm not sure what to do to resolve this.

Thanks!

[sritchie]

Hey @fwilliams ! I suspect that this is a case of not having a service account key set up yet. I'll see what I can do replicate on my end and see if we can get a better error, vs passing the timeout 429 back directly.

I sent some notes offline and managed to get a job submitted to your project. let me know if this guess is correct. The info for service account key setup lives here: https://caliban.readthedocs.io/en/latest/getting_started/cloud.html

Good luck, and let me know!

[sritchie]

We tracked this down internally at Google, and it looks like was a case of V100 GPUs being unavailable on this particular project. The 429 response is overloaded to mean "no quota" AND to mean that our submissions are being rate limited. I've submitted a bug internally to see if we can get these cases distinguished.

@fwilliams , closing this for now. Thanks again!