couchbase/service-broker

Ability to populate credentials registry key via referenced K8s secret

Opened this issue · 3 comments

joshisa commented

Is your feature request related to a problem? Please describe.
I have a scenario whereby my broker configuration is creating template resources for a service-instance and service-binding. These work well with the newish Step and ReadyCheck functionality and provision a service well. The connection information is expectedly created in a similarly named secret. However, my problem is that my CloudFoundry end-users will not have direct access or awareness to that secret - so my broker must act as a middle-man. Therefore, my broker configuration needs a way of being configured to know that my credentials snippet is a base64 decoded string residing within a secret named "foo". This string represents the JSON payload that needs to be fed to the registry credentials key for surfacing by the broker.

Describe the solution you'd like
I realize that service catalogs could help ease my pain, but unfortunately I cannot depend on the service catalog feature being present within the k8s cluster. I would like to create a broker configuration whereby I can define a service-instance named "CoolCloudService" with a plan named "standard" that takes two parameters: service-class and service-plan. This service instance is comprised of two templates.
Template 1 may look roughly like:

  - name: some-service-instance
    template:
      apiVersion: my.coolcloud.com/v1
      kind: Service
      metadata:
        name: '{{ registry "instance-name" }}'
        namespace: '{{ registry "coolcloud-serviceinstance-namespace" }}'
        labels:
          my.coolcloud.com/service-class: '{{ parameter "/service-class" }}'
          my.coolcloud.com/service-plan: '{{ parameter "/service-plan" }}'
      spec:
        plan: '{{ parameter "/service-plan" }}' 
        serviceClass: '{{ parameter "/service-class" }}'

Template 2 may look roughly like:

  - name: some-service-binding
    template:
      apiVersion: my.coolcloud.com/v1
      kind: Binding
      metadata:
        name: '{{ registry "instance-name" }}'
        namespace: '{{ registry "coolcloud-serviceinstance-namespace" }}'
        labels:
          my.coolcloud.com/service-class: '{{ parameter "/service-class" }}'
          my.coolcloud.com/service-plan: '{{ parameter "/service-plan" }}'
      spec:
        serviceName: '{{ registry "instance-name" }}'

Using steps and readiness gates, the ultimate result of this service instance provisioning is really creation of a :

  • A service instance resource named "foo" + service binding resource named "foo" + a resulting secret named "foo" which contains the binding connection information.

I would like the service broker to propagate the base64 decoded content (which is a JSON payload) from the secret named "foo" into the credentials registry key during the brokers "binding" process.

Describe alternatives you've considered
I was unable to think of a way to use the current built-in accessors (e.g. to read a named secret) and mutators (e.g. to decode a base64 encoded string). I'd like to propose adding a new mutator to decode Base64 strings and also have a new accessor that can read data from a named secret. I believe if these two elements existed, I could provide them within the credentials registry key and be able to propagate the contents of the dynamically generated secret into the servicebroker API credential set that's passed during its binding process.

Additional context
I realize this is a meta use-case. Since I'm using the concept of the broker's instance to actually encompass the creation of 3 things: a service-instance + service-binding + service-instance secret containing creds. Then, during the broker's binding process - I'm wanting to propagate the service-instance secret creds as binding credentials. @spjmurray I welcome your thoughts on whether adding a couple of built-ins is the best way to go ... or if there's some existing mechanics for connecting secrets content into the broker config that I may have overlooked.

joshisa commented

@spjmurray I've made great progress on a prototype approach. Welcome your thoughts ...

I'm proposing two enhancements for consideration:

  1. Creation of a new built-in called readSecret which can query a secret named foo within the namespace ipsum where ipsum must be co-located within the owning service instance registry secret namespace location (to limit security exposure) in order to decrypt a base64 encoding of a property named bar and provide its string representation as a JSON object for injection within the credentials portion of VCAP_SERVICES. It's calling syntax might look similar to:
'{{ readSecret "ipsum" "foo" "bar" }}'

and here's a sample implementation code snippet within the provisioners template.go file ...

// templateFunctionReadSecret returns the data payload for the named secret.
func templateFunctionReadSecret(secretnamespace, secretname, secretproperty string) (string, error) {
   secret, err := config.Clients().Kubernetes().CoreV1().Secrets(secretnamespace).Get(secretname, metav1.GetOptions{})
   if err != nil {
   	if !k8s_errors.IsNotFound(err) {
   		return "", err
   	}
   }

   return string(secret.Data[secretproperty]), nil
}

Open Questions/Challenges

  • Can ipsum be omitted entirely as user-input and calculated instead. I think so, but haven't looked at how.
  • I've been struggling mightily with double-quote escaping pains on the resulting JSON string. Any returned value from this prototype built-in is resulting in a VCAP_SERVICES entry with undesirable escaped quotes. On one level, I understand why since my function is returning a string and having double quotes around a JSON string with double-quotes will necessitate some type of escaping.
{
"VCAP_SERVICES": {
"service-name": [
 {
  "binding_name": null,
  "credentials": {
   "connection": "{\"somekey\":\"somevalue\",\"anotherkey\":\"anothervalue\"}"
[....]

I have not figured out how to inject a JSON object rather than stringified JSON within the VCAP_SERVICES JSON object in the credentials section. No matter what approach I've taken so far, I cannot seem to pass from the Go code a return value that is then rendered in the template as a JSON object rather than an escaped JSON string.

Would welcome any and all advice/tips on this. This is my first attempts with Go, so it's been a fun learning journey so far.

  1. Update to the readiness provisioner to provide a case switch against a new readinessCheck field called readinessScope. By default, if it doesn't exist or is provided as empty or with explicit text of Conditions, then status-quo behavior applies. It will assume the readinessCheck is looking within the status.Conditions stanza for a Type and Status. However, if the scope is provided as TopLevel - then it will look for a user-provided named key found in the TopLevelKey field and test until that named key's value matches the user-provided expected value found in the TopLevelValue field. As a result, the required fields for readinessChecks lessens by toggling Type and Status to optional now. This would be defined for each step where a readinessCheck is defined. My prototype code has been working pretty well so far. I've written some basic tests to verify non-existent, empty and with defined enum values.

Open Questions/Challenges

  • Struggling a bit with how to define a unit test to capture the readinessCheck fail case for a bad value. The behavior of the code is to wait until timeout and it is receiving an error (since return of nil indicates success). Don't know how to write an expected fail unit test. Would welcome any and all advice/tips on this.
spjmurray commented

Sorry for the delay, I've been on holiday (yey!)

So, just to get the ball rolling, you want to have essentially nested service instances? So instance A creates instance B, and binds to it, then instance A reports the credentials from instance B?

joshisa commented

@spjmurray 👍 for holidays. I'll be heading on a 1 week sojourn myself next week.
Let me "try" to net out what I've been up to ...

Assume: I have a pre-installed cluster operator present within my K8s. This operator engages/generates/reacts to 3 resources: a CR ServiceInstance Foo-Instance, a CR ServiceBinding Foo-Bind and as an output after CR's flip to an "online" status ... a K8s secret Foo-Secret is generated that holds the credentials for accessing/connecting to this newly minted service (which in this case runs outside of the K8s cluster entirely).

Want: I would like to use the service-broker to provide this provisioning dance to a OSB end-user without them having to know anything about K8s or CRs or Secrets. So, I'm using the service-broker to mask k8s complexity. A dev would gesture cf create-service candy snickers snack -c '{"service-class": "candy-db", "service-plan": "standard"}' and behind the scenes, a two step template rendering sequence would occur. When the dev gestures cf bind-service myapp snack, a template for credentials would be rendered and include the "connection" details derived from the dynamically generated secret Foo-Secret.

Status: Most of this all works great today with the existing service-broker code. However, I had to address two challenges with new code:

  1. Readinesschecks assumed only a Conditions style structure. However, based on my reading here, suffice it to say that there is alot of variance in how status can be portrayed. Sadly, the CRs that I'm dealing with ... have key:value pairs directly living under status. I've now enhanced the broker code to handle the default backwards compatible expectation of Conditions with Type and Status while also now allowing a new way to define a TopLevel key and value as an alternate. This behavior is dictated through a new configuration value called readinessScope and resides at the same level as the Timeout within a given Step. This has been working well in my testing thus far.
  2. Transfer of a specifically designated secret's content into a rendered Credentials snippet. To solve this, I created a new built-in function called readSecret. This allows one to define a string result that represents the content residing within a secret. [TBD] I plan to scope it to only being able to read a secret that resides within the same namespace context. This has also been working fine as well at a basic level.

My plan is to open up two separate PRs which encompass the above two enhancements. But wanted to first get your opinion if you saw value in the what, how or why in what I'm doing.

Open Switches

  1. The best that I've been able to achieve with the readSecret is to have it populate a JSON string that is fully-escaped (e.g. "{\"somekey\":\"somevalue\"}" . I'd love to figure out a way to get the JSON string to actually be a rendered JSON object used within the credentials section rather than surfaced to the bound app in VCAP_SERVICES purely as a single escaped substring value.
  2. I've added a few positive Unit test cases for the readinessCheck enhancement, but haven't figured out how to define any negative tests (e.g. bad or errant typo value for the readinessScope as an example).

If you agree with how I've approached it, happy to open a couple of PRs and perhaps go through comment/feedback/iteration on the code from there. I'm new to go .... but I think I've gotten pretty close - inspired by the existing codebase. 😅