Access denied to GCP storage from Germany location (Hetzner cloud provider)

Question

Access denied to GCP storage from Germany location (Hetzner cloud provider)

Closed this issue a month ago · 6 comments

/kind bug

I was trying to create a kubernetes cluster using kops cli command. And observed that the worker node doesn't join the cluster.
Investigating at the syslogs of the worker node, it is observed that the GCP storage is returning 403 Access Denied error.

1. What kops version are you running? The command kops version, will display
this information.

v1.29.0-beta.1

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

v1.29.0

3. What cloud provider are you using?

hetzner

4. What commands did you run? What is the simplest way to reproduce this issue?

kops create cluster --name=demo2.hcloud.k8s.local --cloud=hetzner --zones=nbg1 --kubernetes-version=v1.29.0 --node-count=1 --node-size=cax11 --control-plane-count=1 --control-plane-size=cax11 --image=ubuntu-22.04 --network-cidr=10.10.0.0/24 --ssh-public-key=/Users/demouser/.ssh/demo2_id_rsa_170424172233.pub --yes

5. What happened after the commands executed?

The kubernetes worker node never joins the cluster.

6. What did you expect to happen?

The worker node joining the kubernetes cluster and cluster should be heathy

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

Unfortunately, the manifest information is not collected.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

root@nodes-nbg1-1ff5a5747705b634:/var/log# curl https://storage.googleapis.com/k8s-artifacts-cni/release/v1.2.0/cni-plugins-linux-arm64-v1.2.0.tgz
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>We're sorry, but this service is not available in your location</Details></Error>

Answer 1 · 2024-04-17T18:25:46.000Z

@purushred This is a common issue with Hetzner. Google is blocking IPs that were used in the past for DDOS or similar things.
To fix this, you need to delete the instance, update the cluster and hope the new server has a non-blocked address.

Answer 2 · 2024-04-17T20:22:07.000Z

Thank you for quick response.
Unfortunately this is happening consistently.
When choose the Helsinki region it works fine.
I am wondering if there is there a workaround to change to a custom path for the package download instead of GCS location?

Answer 3 · 2024-04-18T01:31:07.000Z

You could try using a proxy or a mirror the artifacts.

Answer 4 · 2024-04-18T10:06:19.000Z

You could try using a proxy or a mirror the artifacts.

Any references on how to configure proxy/mirror for the artifacts?
Or maybe I could create a custom image for nodes with the required pkgs pre-installed so that it doesn't have to download again.
Any clues will be really helpful. 🙇

Answer 5 · 2024-04-18T10:36:05.000Z

@prachetasp Please check:

for proxy - https://kops.sigs.k8s.io/http_proxy/
for artifacts mirror - https://kops.sigs.k8s.io/operations/asset-repository/
custom image - get the assets with kops get assets and bake them into the image, nodeup logs should tell you where the files should be copied

Answer 6 · 2024-04-22T10:39:53.000Z

Closing this ticket as the issue is not in the kops as the IPs are blocked by google while accessing the assets.