Creation error with Hetzer cloud in combination with a AWS S3 bucket for the state
tkoeck opened this issue · 6 comments
/kind bug
1. What kops version are you running? The command kops version, will display
this information.
Client version: 1.28.4 (git-v1.28.4)
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
No one yet, it just stops at the beginning
3. What cloud provider are you using?
Hetzner
4. What commands did you run? What is the simplest way to reproduce this issue?
Export those values
export HCLOUD_TOKEN=some-value
// export S3_ENDPOINT=s3.bucket.amazonaws.com (variable not included)
export S3_ACCESS_KEY_ID=somevalue
export S3_SECRET_ACCESS_KEY=somevalue
export KOPS_STATE_STORE=s3://somebucket-k8s
kops create cluster --name=my-cluster.example.k8s.local --ssh-public-key=~/.ssh/id_rsa.pub --cloud=hetzner --zones=fsn1 --image=ubuntu-20.04 --networking=calico --network-cidr=10.10.0.0/16 -v 10
5. What happened after the commands executed?
tkoeck@tron-pc:/gitlab/k8s-test$ kops create cluster --name=my-cluster.example.k8s.local --ssh-public-key=/.ssh/id_rsa.pub --cloud=hetzner --zones=fsn1 --image=ubuntu-20.04 --networking=calico --network-cidr=10.10.0.0/16 -v 4
I0414 20:01:16.560595 25058 create_cluster.go:881] Using SSH public key: /home/tkoeck/.ssh/id_rsa.pub
I0414 20:01:16.560626 25058 factory.go:82] state store s3://somebucket-k8s
I0414 20:01:16.560675 25058 s3context.go:329] unable to read /sys/devices/virtual/dmi/id/product_uuid, assuming not running on EC2: open /sys/devices/virtual/dmi/id/product_uuid: permission denied
I0414 20:01:20.121650 25058 s3context.go:165] unable to get region from metadata:unable to get region from metadata: EC2MetadataRequestError: failed to get EC2 instance identity document
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/dynamic/instance-identity/document": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I0414 20:01:20.121670 25058 s3context.go:175] defaulting region to "us-east-1"
I0414 20:01:32.586256 25058 s3context.go:192] unable to get bucket location from region "us-east-1"; scanning all regions: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Error: error reading cluster configuration "my-cluster.example.k8s.local": error reading s3://somebucket-k8s/my-cluster.example.k8s.local/config: Unable to list AWS regions: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
6. What did you expect to happen?
I wanted to use AWS just for the S3 bucket kOps state, but for everything else Hetzner cloud. For some reason it wants to do a lot of different things in AWS.
7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.
It's not available because it failed before.
8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
Just tried your example for Hetzner (https://kops.sigs.k8s.io/getting_started/hetzner/ )
tkoeck@tron-pc:/gitlab/k8s-test$ kops create cluster --name=my-cluster.example.k8s.local --ssh-public-key=/.ssh/id_rsa.pub --cloud=hetzner --zones=fsn1 --image=ubuntu-20.04 --networking=calico --network-cidr=10.10.0.0/16 -v 10
I0414 20:12:19.800951 25455 create_cluster.go:881] Using SSH public key: /home/tkoeck/.ssh/id_rsa.pub
I0414 20:12:19.800997 25455 factory.go:82] state store s3://somebucket-k8s
I0414 20:12:19.801058 25455 s3context.go:329] unable to read /sys/devices/virtual/dmi/id/product_uuid, assuming not running on EC2: open /sys/devices/virtual/dmi/id/product_uuid: permission denied
I0414 20:12:23.469841 25455 s3context.go:165] unable to get region from metadata:unable to get region from metadata: EC2MetadataRequestError: failed to get EC2 instance identity document
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/dynamic/instance-identity/document": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I0414 20:12:23.469861 25455 s3context.go:175] defaulting region to "us-east-1"
I0414 20:12:35.996157 25455 s3context.go:192] unable to get bucket location from region "us-east-1"; scanning all regions: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Error: error reading cluster configuration "my-cluster.example.k8s.local": error reading s3://somebucket-k8s/my-cluster.example.k8s.local/config: Unable to list AWS regions: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
9. Anything else do we need to know?
@johngmyers Maybe you have an idea?
Same problem/output. I am wondering why it assumes to be on an EC2 instance. I am running it on my local machine.
Not sure what you have in env vars, in general. kOps uses AWS SDK to connect to S3. For me it works like this:
export S3_REGION=eu-central-1
export S3_ENDPOINT=https://s3.eu-central-1.amazonaws.com
export S3_ACCESS_KEY_ID=...
export S3_SECRET_ACCESS_KEY=...
Do you have AWS_REGION in you env vars?
Yeah, thanks. Now it works.
I get another error
W0416 19:16:20.915358 9998 executor.go:139] error running task "SSHKey/my-cluster.example.k8s.local-e5:f8:64:37:f4:8f:07:f1:1e:85:fe:1b:28:60:d6:e8" (9m41s remaining to succeed): Field cannot be changed: PublicKey
but that's something else.