proxy-agent crashing because it is relying on GCP metadata server
isavcic opened this issue · 6 comments
$ kubectl -n kubeflow logs pod/proxy-agent-7bd656ff7d-b6qlm
+++ dirname /opt/proxy/attempt-register-vm-on-proxy.sh
++ cd /opt/proxy
++ pwd
+ DIR=/opt/proxy
+ kubectl get configmap inverse-proxy-config
Error from server (NotFound): configmaps "inverse-proxy-config" not found
+ [[ ! -z '' ]]
++ curl http://metadata.google.internal/computeMetadata/v1/instance/zone -H 'Metadata-Flavor: Google'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: metadata.google.internal
+ INSTANCE_ZONE=/
Proxy agent shld not be important. If I am not wrong, it proxy MySQL requests to cloud SQL.
But let me have a look again.
Ok I had a look. Proxy agent is an agent of an inverting proxy.
https://github.com/google/inverting-proxy
I need to have a closer look to where they are configuring to. But in most use cases, should not be relevant.
i.e. they are using this agent to expose their API service.
For my own use case, I have an API gateway to manage access to the service, hence I do not need this proxy agent.
I need to drill in to see the configs more. But it shld not affect much if u have ur own way to expose the pipeline UI endpoint.
Btw, I will be updating this manifest soon. The current manifest has a slight bug - it AWS IAM does not work with the API service (access key still works thou).
My PR just went it. 0.2.23 will make the entire deployment compatible with AWS.
Ok. I verified that the proxy-agent is deployed to expose kubeflow pipelines to data lab. So it shld not be important.
I will fix this in the my upcoming release - i.e. remove it from the manifest.
@isavcic
I just updated to v0.2.3, the current release of kubeflow pipeline. I have removed the proxy agent from the manifest. Made some significant changes. U might need to delete and start a new mysql server.
Thank you!
Fixed by the latest PRs.