rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout)

[kaykhan]

Terraform Version, ArgoCD Provider Version and ArgoCD Version

Terraform version:Terraform v1.8.2
ArgoCD provider version:
ArgoCD version:2.11.1

Terraform configuration

resource "argocd_application" "multiple_sources" {
  metadata {
    name      = "examples-hello-world"
    namespace = "argocd"
  }

  spec {
    project = "examples"

    source {
      repo_url        = "https://charts.bitnami.com/bitnami"
      chart           = "wordpress"
      target_revision = "22.3.0"
      helm {
        value_files = ["$values/example/values.yaml"]
      }
    }

    source {
      repo_url        = "git@github.com:<redcated>/test-temp-repo1.git"
      target_revision = "HEAD"
      ref             = "values"
    }

    sync_policy {
      automated {
        prune       = true
        self_heal   = true
        allow_empty = true
      }
      sync_options = ["Validate=false"]
      retry {
        limit = "5"
        backoff {
          duration     = "30s"
          max_duration = "2m"
          factor       = "2"
        }
      }
    }

    destination {
      server    = "https://kubernetes.default.svc"
      namespace = "examples"
    }
  }

  depends_on = [kubernetes_namespace.examples, argocd_project.examples]

}

│ Error: failed to create application examples-hello-world
│
│ with argocd_application.multiple_sources,
│ on main.tf line 64, in resource "argocd_application" "multiple_sources":
│ 64: resource "argocd_application" "multiple_sources" {
│
│ rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout)

Question

When trying to deploy an argocd_application i intermittent encounter this 504 Gateway Timeout error. If i rerun a few times it works. Can somoene help me understand why this is occuring. ( i can rule out my internet )

tail of argocd log

time="2024-05-24T10:11:30Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2024-05-24T10:11:30Z" grpc.time_ms=1.234 span.kind=server system=grpc
time="2024-05-24T10:11:31Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.claims="{\"exp\":1716631890,\"iat\":1716545490,\"iss\":\"argocd\",\"jti\":\"1356f272-dc2b-4331-ac85-0c3f1a8f3062\",\"nbf\":1716545490,\"sub\":\"admin\"}" grpc.request.content= grpc.request.deadline="2024-05-24T10:16:29Z" grpc.service=version.VersionService grpc.start_time="2024-05-24T10:11:31Z" span.kind=server system=grpc
time="2024-05-24T10:11:31Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.request.deadline="2024-05-24T10:16:29Z" grpc.service=version.VersionService grpc.start_time="2024-05-24T10:11:31Z" grpc.time_ms=0.314 span.kind=server system=grpc
time="2024-05-24T10:11:31Z" level=info msg="received unary call /application.ApplicationService/List" grpc.method=List grpc.request.claims="{\"exp\":1716631890,\"iat\":1716545490,\"iss\":\"argocd\",\"jti\":\"1356f272-dc2b-4331-ac85-0c3f1a8f3062\",\"nbf\":1716545490,\"sub\":\"admin\"}" grpc.request.content="name:\"examples-hello-world\" appNamespace:\"argocd\" " grpc.request.deadline="2024-05-24T10:16:29Z" grpc.service=application.ApplicationService grpc.start_time="2024-05-24T10:11:31Z" span.kind=server system=grpc
time="2024-05-24T10:11:31Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=List grpc.request.deadline="2024-05-24T10:16:29Z" grpc.service=application.ApplicationService grpc.start_time="2024-05-24T10:11:31Z" grpc.time_ms=0.265 span.kind=server system=grpc
time="2024-05-24T10:11:31Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"exp\":1716631890,\"iat\":1716545490,\"iss\":\"argocd\",\"jti\":\"1356f272-dc2b-4331-ac85-0c3f1a8f3062\",\"nbf\":1716545490,\"sub\":\"admin\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.request.deadline="2024-05-24T10:16:29Z" grpc.service=application.ApplicationService grpc.start_time="2024-05-24T10:11:31Z" span.kind=server system=grpc
time="2024-05-24T10:12:31Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for examples-hello-world is invalid: InvalidSpecError: Unable to generate manifests in .: rpc error: code = Canceled desc = context canceled;InvalidSpecError: Unable to generate manifests in .: rpc error: code = Canceled desc = context canceled" grpc.code=InvalidArgument grpc.method=Create grpc.request.deadline="2024-05-24T10:16:29Z" grpc.service=application.ApplicationService grpc.start_time="2024-05-24T10:11:31Z" grpc.time_ms=60002.19 span.kind=server system=grpc