SSH to other VMs doesnt work

Question

SSH to other VMs doesnt work

meirhazon2 opened this issue 5 years ago · 21 comments

Good morning,

SSH from the bastion to other VMs in the same project doesnt work though the permissions are set in the following way,

service_account_roles = [
"roles/logging.logWriter",
"roles/monitoring.metricWriter",
"roles/monitoring.viewer",
"roles/compute.osLogin",
"roles/container.clusterAdmin",
"roles/container.viewer",
"roles/container.developer",
]

Meirs-MacBook-Pro:terraform meirhazon$ gcloud compute ssh bastion-vm
External IP address was not found; defaulting to using IAP tunneling.
Last login: Fri Dec 6 12:05:58 2019 from 35.235.240.192
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
[meir_hazon_evbox_io@bastion-vm ~]$ ssh ftp
The authenticity of host 'ftp (10.0.0.4)' can't be established.
ECDSA key fingerprint is SHA256:XTFunA0g2m749ItL10dnyvo9KDPADXiOLKa3BVYOY3I.
ECDSA key fingerprint is MD5:50:27:35:cf:38:23:20:d6:0b:24:7b:9b:6d:6f:2b:62.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ftp,10.0.0.4' (ECDSA) to the list of known hosts.
Permission denied (publickey).
[meir_hazon_evbox_io@bastion-vm ~]$ sudo su -
Last login: Fri Dec 6 12:06:01 UTC 2019 on pts/0
[root@bastion-vm ~]# ssh ftp
Permission denied (publickey).
[root@bastion-vm ~]#

How should I change the bastion roles to make this work?

Thanks so much,

Meir

Answer 1 · 2019-12-11T14:54:15.000Z

I suspect the sudo command might be what's causing this to fail. Note that you will also need os login access on all VMs you want to SSH to.

Please follow/try the example: https://github.com/terraform-google-modules/terraform-google-bastion-host/tree/master/examples/two_service_example

Answer 2 · 2019-12-11T20:09:45.000Z

Hi,
I have added the role in the VM.

still, ssh from the bastion doesnt work,

[meir_hazon_evbox_io@bastion-vm ~]$ gcloud beta compute --project "everon-eu" ssh --zone "europe-west1-d" "ftp"
Warning: Permanently added 'compute.5850356010427702296' (ECDSA) to the list of known hosts.
Permission denied (publickey).
ERROR: (gcloud.beta.compute.ssh) [/usr/bin/ssh] exited with return code [255].
[meir_hazon_evbox_io@bastion-vm ~]$

Should adding a role with the following attribute to the bastion should solve this?
'compute.projects.setCommonInstanceMetadata' permission for 'projects/'

Please help.

Answer 3 · 2019-12-17T09:06:16.000Z

Hi, Please see the following,

Meirs-MacBook-Pro:terraform meirhazon$ cd ~/.ssh # change working directory to ssh directory

Meirs-MacBook-Pro:.ssh meirhazon$ mv google_compute_engine.pub google_compute_engine_backup.pub # backup public key

Meirs-MacBook-Pro:.ssh meirhazon$ mv google_compute_engine google_compute_engine_backup_backup # backup private key

Meirs-MacBook-Pro:.ssh meirhazon$ gcloud config set project everon-eu
Updated property [core/project].

Meirs-MacBook-Pro:.ssh meirhazon$ gcloud config set compute/region europe-west1-d
Updated property [compute/region].

Meirs-MacBook-Pro:.ssh meirhazon$ gcloud compute ssh bastion-vm
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/meirhazon/.ssh/google_compute_engine.
Your public key has been saved in /Users/meirhazon/.ssh/google_compute_engine.pub.
The key fingerprint is:
SHA256:wjPODUI8r651WHmBYErnsTU2jML/s1v0GK+dVIfDJvw meirhazon@Meirs-MacBook-Pro.local
The key's randomart image is:
+---[RSA 2048]----+
| .. =o= |
| .o*..+ |
| .o . . |
| ..+ . o . . |
| ..X S o * . |
| BoO = = o |
| + +o+ + E |
| o ... + . |
| ... ... o |
+----[SHA256]-----+
External IP address was not found; defaulting to using IAP tunneling.
Last login: Tue Dec 17 09:01:50 2019 from 35.235.240.194
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
[meir_hazon_evbox_io@bastion-vm ~]$ exit
logout
Connection to compute.1143489831642057891 closed.

Meirs-MacBook-Pro:.ssh meirhazon$ eval "$(ssh-agent -s)"
Agent pid 83158
Meirs-MacBook-Pro:.ssh meirhazon$ ssh-add ~/.ssh/google_compute_engine

Identity added: /Users/meirhazon/.ssh/google_compute_engine (meirhazon@Meirs-MacBook-Pro.local)
Meirs-MacBook-Pro:.ssh meirhazon$ gcloud compute ssh bastion-vm --ssh-flag="-A" --command "ssh ftp" -- -t

External IP address was not found; defaulting to using IAP tunneling.
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
Permission denied (publickey).
Connection to compute.1143489831642057891 closed.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Thanks so much for your help.

Meir

Answer 4 · 2019-12-17T19:04:12.000Z

You might want to try adding the SSH key to your account before running gcloud compute ssh commands: https://cloud.google.com/compute/docs/instances/managing-instance-access#add_oslogin_keys

Answer 5 · 2019-12-18T07:51:06.000Z

Hi,
I was using ~/.ssh/google_compute_engine as suggested at the wiki and it didnt work (output is attached in previous messages)

Nonetheless, I have created a new set of keys, added them to Gcloud and tried.

Still, It doesn't work. output is following. I have changed the keys in the attached print screen.

I would appreciate your help.

Thank you

Meirs-MacBook-Pro:.ssh meirhazon$ gcloud compute os-login ssh-keys add --key-file id_rsa.pub
loginProfile:
name: '111650669127142506719'
posixAccounts:

gid: '186263490'
homeDirectory: /home/meir_hazon_evbox_io
operatingSystemType: LINUX
primary: true
uid: '1000000'
username: meir_hazon_evbox_io
sshPublicKeys:
21a7ede24d815a491c4e26c9703c:
fingerprint: 21a7ede24d65ff6f53501227a6245b6f68c
key: |
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIVMTVdZjS6IaIDiCPNrXAiBaQKZyyPTfmI3mC0u0HJVwbRwwSLT6Mz8VR0yb4xJoXqkTF0gpZjrguQPlZt10cEyE2GMxU1noLW8MnH4SQfIPbAqmxmipFRaz+Z/g13QoU/SBa3WWbf7etQZVn0z9 meirhazon@Meirs-MacBook-Pro.local
name: users/meir.hazon@evbox.io/sshPublicKeys/21a7ede24d815a491c4e26c97033ea6c7844b3d65ff6f53501227a6245b6f68c
c72458923b80b76829cd7e712fe59297e419782b80772976668bf4ce5fa521ec:
fingerprint: c72458923b80b76829cd7e712fe59297e419782b80772976668bf4ce5fa521ec
key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvJuz4HKsMMECstFuZloiYfPT+dHMbcn2IDlUli/8tzVMnOpVqIRSQ4qtv62eLA7yKLnX8lXDpwJBzx0Z0kckU/69wHgR7zNIeNt67yIDP2J74h9vu0Naq0JqsCUYef4lsqZ6I1rKhNR9LCvkNQROm
meirhazon@Meirs-MacBook-Pro.local
name: users/meir.hazon@evbox.io/sshPublicKeys/c72458923b80b76829cd7e712fe59297e419782b80772976668bf4ce5fa521ec
c8122cc1cde5e4f1d7a70b2b05277fa6f7fe506fa:
fingerprint: c8122cc1cde5277fa6f7fe506fa
key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGj0Umz9HyaZxxOt7GVkYfjqJ7qj/lNHOiy2a6fCsU7XSjX9ws/7K6IK1YP/wxqaE1k8+aTM//yuhgjpQhcqLuL/wpEfQFGt0WsTFb4760Euym1I3TNccN5vKE2f/Zt/mlt8/stTXfbTkXZ1pzhl
meirhazon@Meirs-MacBook-Pro.local
name: users/meir.hazon@evbox.io/sshPublicKeys/c8122cc1973a667b505277fa6f7fe506fa
Meirs-MacBook-Pro:.ssh meirhazon$ gcloud compute ssh bastion-vm
External IP address was not found; defaulting to using IAP tunneling.
Last login: Tue Dec 17 09:03:40 2019 from 35.235.240.130
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
[meir_hazon_evbox_io@bastion-vm ~]$ ls -ltr .ssh
total 16
-rw-r--r--. 1 meir_hazon_evbox_io meir_hazon_evbox_io 412 Dec 5 13:42 google_compute_engine.pub
-rw-------. 1 meir_hazon_evbox_io meir_hazon_evbox_io 1675 Dec 5 13:42 google_compute_engine
-rw-r--r--. 1 meir_hazon_evbox_io meir_hazon_evbox_io 174 Dec 11 07:47 known_hosts
-rw-r--r--. 1 meir_hazon_evbox_io meir_hazon_evbox_io 189 Dec 11 20:07 google_compute_known_hosts
[meir_hazon_evbox_io@bastion-vm ~]$ exit
logout
Connection to compute.11434057891 closed.
Meirs-MacBook-Pro:.ssh meirhazon$ eval "$(ssh-agent -s)"
Agent pid 99848
Meirs-MacBook-Pro:.ssh meirhazon$ pwd
/Users/meirhazon/.ssh
Meirs-MacBook-Pro:.ssh meirhazon$ ssh-add id_rsa
Identity added: id_rsa (meirhazon@Meirs-MacBook-Pro.local)
Meirs-MacBook-Pro:.ssh meirhazon$ gcloud compute ssh bastion-vm --ssh-flag="-A" --command "ssh ftp" -- -t
External IP address was not found; defaulting to using IAP tunneling.
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
Permission denied (publickey).
Connection to compute.1143489831642057891 closed.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Answer 6 · 2019-12-18T08:02:44.000Z

@onetwopunch any ideas?

Ultimately since your Terraform deployment succeeded I'm not sure how much we can help here. I would suggest filing a support case for more detailed troubleshooting assistance.

Answer 7 · 2019-12-18T08:09:51.000Z

Hi, I was thinking that this is what I was doing, can you suggest how and where? Thank you

Answer 8 · 2019-12-18T08:13:17.000Z

@meirhazon2 https://cloud.google.com/support/

Answer 9 · 2019-12-18T08:17:03.000Z

OK, Thanks

…

On Wed, Dec 18, 2019 at 9:13 AM Morgante Pell ***@***.***> wrote: @meirhazon2 <https://github.com/meirhazon2> https://cloud.google.com/support/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25?email_source=notifications&email_token=ANFOA3KUHUMCIGQWPIDDJ5DQZHLR5A5CNFSM4JZKWPW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHFJMJA#issuecomment-566924836>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANFOA3IUG32YNMKOKMWJ5GLQZHLR5ANCNFSM4JZKWPWQ> .

Answer 10 · 2019-12-18T14:36:06.000Z

I opened a case with gcloud support. will update this case with their findings

Answer 11 · 2019-12-18T14:53:34.000Z

@meirhazon2 Could you post your config for The bastion module please? Specifically if the user you are trying to SSH with is part of the members variable?

Also the SSH public key will be uploaded via the API as part of the gcloud command so you should be have to upload it beforehand, though it won’t hurt anything if you do.

Answer 12 · 2019-12-18T15:07:08.000Z

Hi,
Yes, My user is a member of the oncall group.

Meirs-MacBook-Pro:terraform meirhazon$ cat common/bastion.tf
module "iap_bastion" {
source = "terraform-google-modules/bastion-host/google"
version = "1.0.0"
project = "everon-${terraform.workspace}"
zone = local.is_production ? "europe-west1-d" : "europe-west1-d"
network = "everon-network"
subnet = local.is_production ? "everon-k8s" : "everon"
startup_script = <<-EOB
#!/usr/bin/env bash
yum install -y git kubectl vim telnet tcpdump traceroute
EOB
members = [
"group:oncall@evbox.io",
]
service_account_roles = [
"roles/logging.logWriter",
"roles/monitoring.metricWriter",
"roles/monitoring.viewer",
"roles/compute.osLogin",
"roles/container.clusterAdmin",
"roles/container.viewer",
"roles/container.developer",
]
}

I think that I understand the second part, Its informative only, right?

Thanks so much,

Meir

Answer 13 · 2019-12-18T15:28:24.000Z

Interesting. I’m not seeing anything particularly wrong with your config. I might suggest using the service_account_roles_supplemental variable instead, but that wouldn’t have anything to do with this. Have you been able to successfully run the simple_example or two_services_example in an isolated environment?

Answer 14 · 2019-12-18T15:28:39.000Z

I would like to try something, can I create an additional svc acct for the bastion?
Thanks

Answer 15 · 2019-12-18T15:43:27.000Z

You can add different permissions to the service account but in GCP, there is only one service account allowed per instance.

Answer 16 · 2019-12-18T15:44:52.000Z

OK, Thanks

Answer 17 · 2019-12-19T06:29:25.000Z

Hello,
I have used the simple example to begin with, I have added more roles and packages afterwards. It worked exactly the same as today. it is an isolated environment.
I hope that my answers help and I really appreciate your inputs and help.
Thank you,
Meir

Answer 18 · 2019-12-20T09:19:18.000Z

Hi,
OK, It is solved, I had to add a key to the bastion svc account and activate it.
Would you automate this?
Thanks so much

Answer 19 · 2019-12-27T14:47:46.000Z

The bastion service account is already attached to the bastion vm and therefore doesn’t need a key or to be activated. Possibly are you talking about the service account you are using to SSH from your computer into the bastion? In that case, yes you would need to activate that with gcloud before using it: https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account

Answer 20 · 2019-12-27T16:12:22.000Z

Hi. No. This is not the case. I was referring to the bastion SVC Acct that was created by the bastion tf. Thanks

…

Sent from my iPhone

On 27 Dec 2019, at 15:47, Ryan Canty ***@***.***> wrote: The bastion service account is already attached to the bastion vm and therefore doesn’t need a key or to be activated. Possibly are you talking about the service account you are using to SSH from your computer into the bastion? In that case, yes you would need to activate that with gcloud before using it: https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 21 · 2019-12-28T01:03:29.000Z

I'm glad you were able to solve this issue. However, creating a key and activating a service account on the bastion would not have had any effect on your ability to use OS Login to SSH onto the bastion. In fact this is highly discouraged for security reasons since the JSON key is plaintext and could be exfiltrated from the machine. Instead, service acccounts should be attached to the VM when the VM is created. The VM can then fetch short-lived OAuth tokens using the GCE metadata server, not a private key from the filesystem.

This seems like an issue in your environment that was fixed and it happened to correspond to activating the service account. The simple_example and two_services_example work as written in a clean environment if you follow the README. If you are unable to get those to work, it is probably an issue with your environment. Since there doesn't seem to be an issue with the bastion module, I'm going to go ahead and close this issue for now. I would recommend reading up on how service accounts and IAM permissions work in GCP and try these examples again in a clean environment. Here are some links to get you started:

Hope this helps