Ability to have environment variables in your OC components

[ricardo-devis-agullo]

Slightly related to issue #769, if it happens that you package your component beforehand, and then publish the same packaged OC to different registries, it would be nice to have a way to have env variables for each registry. Right now the only way of doing that is having to fully package/publish on each registry, changing values on the server side for each one. What I would propose is to have a new file (.env) that gets loaded side by side with the dataProvider, and passed as part of the context.

So you could define it in your package.json

{
  "oc": {
    "files": {
      "data": "server.js",
      "template": {
        "src": "index.js",
        "type": "oc-template-react"
      },
      "env": "myFolder/.myEnvFile"
    },
  }
}

The compiler will add that your .myEnvFile as .env in your _package folder, upload it to your private storage, and then load it.

There's two options of where to pass this.

1. As part of the context object env property. This means it will get mixed with the OC registry level environment variables you decide to push.

2. As part of process.env global object. That way you separate OC registry envs from OC component ones.

Changes needed for this

1. Change all storage adapters so they consider .env, along with server.js to be a private file.
2. Change oc-generic-template-compiler so it copies .env file to _package folder, given that it's defined in package.json. Update templates afterwards to use the new version of the compiler.
3. Change get-component.js in OC, so that it also checks for retreiving the .env file, and adds it to the context object before running the dataProvider code.

[pbazydlo]

I believe context.env is only used to pass environment name and there are no other things passed in it, so we could have something like:
context.env.values which would have the environment variables.

However, process.env seems to be a standard for interacting with env variables so maybe it's better to use that after all. Is process separated between different components?

[ricardo-devis-agullo]

AFAIK there's nothing stopping you code-wise to pass extra parameters in the env object (whatever is there it will be passed directly to your oc data provider context see here).

About process.env, the only thing is that right now the process object is not passed at all, so either we start passing the whole process object and we merge process.env with our env values, or we pass a fake process object that only has the env property.

[matteofigus]

In general there are a number of boundaries that are created for security. If component A relies on process.env.foo to be there, but then component B overwrites that it could break component B for instance.
At the moment that's done by making sure globals such as process are not usable on the server.

If you can figure out an elegant way to use .env files as convention and process it on build time so that you have full encapsulation on a component level, why not, that could be useful.

[ricardo-devis-agullo]

The proposed solution would not overwrite environment variables between components (I think), since each one will have it's own context object with their own .env file (stored along server.js in the storage).

Basically here, we could check after that if oc.files.env exists, and if it does, do a repository.getEnv (new method to get the env file and parse it as a JS object), then merge that new object either in contextObj.env or in the context passed to your vm.runInNewContext method (with a fake process object or the real one). At most you will override env variables from the OC registry, but never from another component, as your own will always take precedence. Creating a fake process.env object with just the .env variables avoids overriding at all levels.

[matteofigus]

Can I ask you what would you need this for? An example of what would you use a env setting here that you just couldn't embed in the server.js (or just require it).

When I used OC I remember I typically had something in my server like

const settings = require('./settings.json')[context.env.name];
...

And that was enough for connection strings or any other thing that was env specific. And being not in the oc.files webpack would bundle it it on the server for me to keep it private.

The issue with your solution is that it may affect performance a bit as the (multi-tenant) registry application will need to merge envs and I suspect that may still need to be env specific (like dev vs qa vs prod) so it wouldn't be as simple as merging two objects? But if you can give me an example perhaps we can clarify.

[ricardo-devis-agullo]

That approach works for us for a lot of environment variables that are not sensitive, but if we also want to use private keys, it would mean that the private keys will be versioned in the repo where we contain our OC, which is not ideal. With the .env approach, we can just replace the keys in the middle of packaging/publishing on the pipeline itself (we package once at the beginning, then for each environment/registry we unpack, change the .env file values, and publish those built files)

I'm not sure I understand the last part of why would you need to merge multiple envs, as the concept is similar of the
const settings = require('./settings.json')[context.env.name]; approach (same as you only have one settings.json per component/version, you also have only one .env file published with it), perhaps I don't know what a multi-tenant registry application really means?

[kmcrawford]

I also had this same issue yet resolved it by creating a plug-in that can read env/secrets from a different source.

[pbazydlo]

Did you manage to limit env/secrets to a single component? i.e. component-a only accesses secrets for component-a not for component-b

[kmcrawford]

Did you manage to limit env/secrets to a single component? i.e. component-a only accesses secrets for component-a not for component-b

Yes, I can share an example how I did it. My configs and secrets are kubernetes artifacts, but they could be stored in any other resource or remote file system.

[ricardo-devis-agullo]

If you could share how your solution works would be nice, although I think it solves different problems (living keys that can change and you always want to retreieve the freshest?)

[pbazydlo]

Hi @kmcrawford this conversation got forgotten I guess :P we're kind of coming back to it on our side, however, before we'd jump into something similar to what is proposed in that issue it would be awesome to be awesome to get a peek at how you're doing it

[kmcrawford]

@pbazydlo let me look into open sourcing the solution.

[pbazydlo]

that would be cool, if it helps the main thing I'm not sure about is how you could achieve isolation across different components with the current plugin system

[kmcrawford]

In the oc plug-in framework there isn’t the ability to know who is calling your function. Our method is to store configurations by the package name of the calling service. Which needs to be passed in to the plug-in as a parameter. If isolation is required for security, I’d assume that a different registry would be required to prevent components from accessing configurations that they don’t have access to. But if the oc plug-in framework knew the name/version of the calling service then it could by default provide isolation.