Enable access to a Seldon deployed model from a user's Kubeflow Pipeline step

[ca-scribner]

When users deploy models via Seldon into their namespace, they want to be able to access these models from a step in a Kubeflow Pipeline¹. This currently does not work - if you try to access the model from a pipeline step, you get the message "403 RBAC: access denied". This may be due to pipeline steps not having istio sidecars

For this task to be complete, we need to:

• debug the access of seldon models from Kubeflow pipelines
• write a guide that shows how to access models from a Kubeflow pipeline
• [if possible] write tests that assert this works <-- This might be a full bundle test. Or maybe we just need KFP deployed.

Related is canonical/bundle-kubeflow#557, which describes some challenges in access.
This solves part of #109.

Footnotes

1. if at all possible, this should also keep traffic inside the kubernetes cluster. Going outside and authenticating back through the front door is a last resort ↩

[Barteus]

Example: https://github.com/canonical/ai-ml-demos/blob/main/talking-jellyfish/models/chatbot/deploy-pipeline.ipynb

[ca-scribner]

Some testing results using Kubeflow 1.7 edge (with a patch to add SeldonDeployment access to users)

• Calling curl http://SELDONDEPLOYMENT_SERVICE_IP:8000/api/v0.1/predictions -X POST -H 'Content-Type: application/json' -d '{"data": {"ndarray": ["x", "Hi"]}}' from different places gives different results:
  • from my local terminal: RBAC: access denied
  • from a terminal in a Kubeflow Notebook: works
  • from a pod deployed to the user's namespace (kubectl run curl -n user1 --rm -i --tty --image radial/busyboxplus:curl -- sh): works
  • from a pod deployed to the kubeflow namespace (kubectl run curl -n kubeflow --rm -i --tty --image radial/busyboxplus:curl -- sh): RBAC: access denied

As noted by @Barteus, the difference appears to be the presence of an istio sidecar. Pods in the user's namespace get sidecar injection by default, whereas pods outside user namespaces do not.

This is because when seldon creates a deployment, it also creates an istio AuthorizationPolicy that controls access to the SeldonDeployment:

kubectl get authorizationpolicy -A 

NAMESPACE   NAME                    AGE
user1       ns-owner-access-istio   21m

kubectl get authorizationpolicy ns-owner-access-istio -o yaml

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 annotations:
   role: admin
   user: user1
 creationTimestamp: "2023-03-17T13:57:24Z"
 generation: 1
 name: ns-owner-access-istio
 namespace: user1
 ownerReferences:
 - apiVersion: kubeflow.org/v1
   blockOwnerDeletion: true
   controller: true
   kind: Profile
   name: user1
   uid: 5beb08c7-2e0e-4c4e-a417-9661203a4ace
 resourceVersion: "15512"
 uid: 77ad6252-c51e-4bff-ab78-d1d20b745ea2
spec:
 rules:
 - when:
   - key: request.headers[kubeflow-userid]
     values:
     - user1
 - when:
   - key: source.namespace
     values:
     - user1
 - to:
   - operation:
       paths:
       - /healthz
       - /metrics
       - /wait-for-drain
 - from:
   - source:
       principals:
       - cluster.local/ns/kubeflow/sa/notebook-controller-service-account
   to:
   - operation:
       methods:
       - GET
       paths:
       - '*/api/kernels'

We can spot this using instructions from step 5 here, running istioctl x authz check on the running seldon deployment's pod:

istioctl x authz check chatbot-default-0-classifier-5f8b495999-7r789.user1

ACTION   AuthorizationPolicy           RULES
ALLOW    ns-owner-access-istio.user1   4

Because the AuthorizationPolicy mentions "request.headers[kubeflow-userid]", I think it has to be kubeflow-aware somehow and because of that is adding this policy? Not sure, still need to investigate more

[ca-scribner]

I believe the AuthorizationPolicy is being created here by the profile controller

[ca-scribner]

@Barteus I think the workaround here is instruct seldon to add the sidecar.istio.io/inject: "false" label to the SeldonDeployment pods like at the bottom of this:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: chatbot
spec:
  name: chatbot
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - name: classifier
          image: bponieckiklotz/jellyfish.chatbot:dev@sha256:a1ce5fcdc31e3c393eb47e18245bebc789aa6879f54611471c0a57f0a440b2e4
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 0
          resources:
            requests:
              cpu: 1000m
              memory: 2000Mi
    graph:
      name: classifier
    name: default
    replicas: 1
    labels:
      sidecar.istio.io/inject: "false"

afaict, that clears things up. Does that work for you?

[Barteus]

The solution works but allows everyone (not only you) to access your model.
This is a great workaround. I used it for one of the demos and it works.
Thanks!

[ca-scribner]

ty for the feedback!

Yeah, this is not perfect. I wonder if when we revisit the seldon story we can find a more holistic solution.

Closing this issue, but adding a note to #109