Vault Audit log only shows cluster IP address?
MaxDiOrio opened this issue · 9 comments
I have audit logging turned on in Vault and it's logging to StackDriver perfectly fine.
However, for each entry, the remote_address is always the Cluster Node IP's address, not the source address.
I'm guessing the Google Load Balancer doesn't pass the originating IP address? Is there a way to see the actual source IP address?
My guess is that we need to set x_forwarded_for_hop_skips to 2? Can you try that?
How would I go about changing that without disrupting Vault? I added it to the Terraform vault.yaml file but it doesn't appear to change when I run terraform apply and it lists the actions it's going to perform.
Still new to Terraform.
I got it. But setting x_forwarded_for_hop_skips to 2 didn't change anything. 172.22.3.2 is the GCP vm IP. Wondering if the load balancer passes it. It looks like you have to use a Target Proxy in the Load Balancer which injects the X-Forwarded-For header.
{
"time":"2019-01-04T14:25:35.592258976Z",
"type":"response",
"auth":{
"client_token":"hmac-sha256:cde5bea9add01084c13986c2fe89a0ce30c8d3a8097d10678d4401607c41b755",
"accessor":"hmac-sha256:e5616a0070cf9186d1ed81886e73dcc971b81c440d4c8a692ba75ea0337e229b",
"display_name":"ldap-mdiorio",
"policies":[
"admin",
"default"
],
"token_policies":[
"admin",
"default"
],
"metadata":{
"username":"mdiorio"
},
"entity_id":"6e8bfb5a-a191-ee11-3731-ad112c9620af",
"token_type":"service"
},
"request":{
"id":"baf7d3fe-8d52-a567-45fb-86ce68f4c841",
"operation":"update",
"client_token":"hmac-sha256:77fe97032d47aa76d1c4a31030ba2b43293c52423b48a8f03c59caa2a2935622",
"client_token_accessor":"hmac-sha256:62013b5807095b9452134b2adf416cb033437eb339b537ba7dcf57160c692d0a",
"namespace":{
"id":"root",
"path":""
},
"path":"auth/ldap/login/mdiorio",
"data":{
"password":"hmac-sha256:85b75e596da708573351e57cbc0397a6ecf44a8fa18097cf69391770c307237c"
},
"policy_override":false,
"remote_address":"172.22.3.2",
"wrap_ttl":0,
"headers":{
}
},
"response":{
"auth":{
"client_token":"hmac-sha256:cde5bea9add01084c13986c2fe89a0ce30c8d3a8097d10678d4401607c41b755",
"accessor":"hmac-sha256:e5616a0070cf9186d1ed81886e73dcc971b81c440d4c8a692ba75ea0337e229b",
"display_name":"ldap-mdiorio",
"policies":[
"admin",
"default"
],
"token_policies":[
"admin",
"default"
],
"metadata":{
"username":"mdiorio"
},
"entity_id":"6e8bfb5a-a191-ee11-3731-ad112c9620af",
"token_type":"service"
}
},
"error":""
}Can you share how you enabled the audit log device and which device you're using?
$ vault audit enable file file_path=stdout
I also saw that there are no headers being logged, so I did this too:
$ vault read sys/config/auditing/request-headers
Key Value
--- -----
headers map[x-forwarded-for:map[hmac:false]]
Which is supposed to add the header to the audit logs, but it still doesn't show up - even after disabling and re-enabling audit logging.
{
"time":"2019-01-04T15:41:13.83762549Z",
"type":"response",
"auth":{
"client_token":"hmac-sha256:75bcb41b7e9dca088a870732d1838dda9d379cfe29ec7303a93eef470ecca1ee",
"accessor":"hmac-sha256:75a521bef0b698982e1cbee91ec1784637b59ddeb03e42e2eca69c26cd44d344",
"display_name":"root",
"policies":[
"root"
],
"token_policies":[
"root"
],
"metadata":null,
"entity_id":"",
"token_type":"service"
},
"request":{
"id":"0b0892ce-0cab-d56a-c0e6-a64f257313f1",
"operation":"update",
"client_token":"hmac-sha256:75bcb41b7e9dca088a870732d1838dda9d379cfe29ec7303a93eef470ecca1ee",
"client_token_accessor":"hmac-sha256:75a521bef0b698982e1cbee91ec1784637b59ddeb03e42e2eca69c26cd44d344",
"namespace":{
"id":"root",
"path":""
},
"path":"sys/audit/file",
"data":{
"description":"hmac-sha256:bb2f3c41481a5d8c69eb385818d3ce889231f31a26df06f4d2dcedb6af513284",
"local":false,
"options":{
"file_path":"hmac-sha256:17d533775ea8022c09bfbd2891e60ba9f2b1758bb833a56ab2f80d64bd5a68e2"
},
"type":"hmac-sha256:4f1e67473eca6bdfed2f938e741b1388ddb70e9643a6f242ebff63269ac62875"
},
"policy_override":false,
"remote_address":"172.22.3.4",
"wrap_ttl":0,
"headers":{
}
},
"response":{
},
"error":""
}Okay - try removing all the x_forwarded_for things in the Vault config and add this to the service:
spec:
type: LoadBalancer
loadBalancerIP: ${load_balancer_ip}
+ externalTrafficPolicy: Local
selector:
app: vaultPerfect! Getting our HQ's public IP as the remote_address now. Thank you.
Awesome! I'm working on some other changes (tightened permissions, etc) that will hopefully get PRed today/tomorrow. I'll include this in those changes.