Retrying idempotent requests that return 503 backendError

[tomcham]

Is your feature request related to a problem? Please describe.

Idempotent request that return 503 with reasonbackEndError is not being retried.

Sample stacktrace using v2.22.6:

Caused by: c.g.a.c.g.j.GoogleJsonResponseException: 503 Service Unavailable
POST https://storage.googleapis.com/storage/v1/b/<redacted>/<redacted>/rewriteTo/b/<redacted>?ifSourceGenerationMatch=1690520663979255&projection=full
{
  "code" : 503,
  "errors" : [ {
    "domain" : "global",
    "message" : "We encountered an internal error. Please try again.",
    "reason" : "backendError"
  } ],
  "message" : "We encountered an internal error. Please try again."
}
	at c.g.a.c.g.j.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at c.g.a.c.g.s.j.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at c.g.a.c.g.s.j.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at c.g.a.c.g.s.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
	at c.g.a.c.h.HttpRequest.execute(HttpRequest.java:1111)
	at c.g.a.c.g.s.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
	at c.g.a.c.g.s.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
	at c.g.a.c.g.s.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
	at c.g.c.s.s.v.HttpStorageRpc.rewrite(HttpStorageRpc.java:1137)
	at c.g.c.s.s.v.HttpStorageRpc.openRewrite(HttpStorageRpc.java:1077)

Creating this issue as a feature request because I'm not sure if this is intentional or a bug.

Describe the solution you'd like

Make 503 with reason backendError as part of the retryable errors.

[tomcham]

Apologies, this is a misconfiguration on our end.