GitOps Secrets

Hello, GitOps Secrets! Goodbye environment variable storage limits.

It's been a long-standing frustration that AWS Lambda deployments have a 4KB environment variable limit. This limit also impacts other environments such as Vercel and the Serverless framework who use AWS Lambda as their infrastructure provider.

A GitOps Secrets workflow eliminates environment variable limits without insecure hacks such as storing unencrypted .env files in your builds and only takes three steps:

Install the GitOps Secrets package (currently in developer preview):

npm install gitops-secrets

Bundle encrypted secrets into your build

// ./bin/encrypt-secrets.js
const secrets = require("gitops-secrets");

async function main() {
  const payload = await secrets.providers.doppler.fetch();
  secrets.build(payload);
}

main();

// package.json
{
  "scripts: {
    ...
    "encrypt-secrets": "node ./bin/encrypt-secrets.js"
  }
}

Decrypt secrets at runtime

const { loadSecrets } = require("gitops-secrets");
const secrets = loadSecrets();

Background and Motivation

As creators of the Doppler Universal Secrets Platform who provide secrets sync integrations for Vercel and Serverless, we've helped customers individually to work around this limitation.

But long-term, we wanted a generic, flexible and open source solution that both our customers and other teams experiencing the same issue could use.

Our goal was to design a new way of accessing secrets in production that:

Allowed for a secrets payload of any size
Could be up and running in minutes
Scaled to work in any environment, including local development
Could support the most restrictive serverless platforms
Provided first class support for ES modules
Prevented unencrypted secrets from ever touching the file system
Abstracted away the complexity of secrets fetching using community contributed providers

Providers

A provider is designed to abstract away the complexities of fetching secrets from any secret manager or secrets store by exposing a single async fetch method.

A secrets provider returns a plain Key-Value Object to ensure that serializing to and from JSON during encryption and decryption produces the same object structure originally fetched from the provider.

The current list of providers are:

Doppler

We'd love to see the list of providers grow! Please see our contributing guide to get started.

Encryption and Decryption

There are two file formats available for bundling secrets into your build:

JSON: Encrypted JSON file.
JS Module: Encrypted JSON embedded in JS module.

You may be forced to use the JS module format if reading static JSON at runtime is problematic, e.g. Vercel prefers a JS module with a custom path, but otherwise, there isn't a compelling reason to use one format over another.

JSON

To encrypt secrets to a JSON file:

const secrets = require("gitops-secrets");

async function main() {
  const payload = await secrets.providers.doppler.fetch();

  // Internally managed storage
  secrets.encryptToFile(payload);

  // Custom path
  secrets.encryptToFile(payload, { path: ".secrets.enc.json" });
}

main();

To decrypt secrets from a JSON file:

const { decryptFromFile } = require("gitops-secrets");

// Internally managed storage
const secrets = decryptFromFile();

// Custom Path
const secrets = decryptFromFile(".secrets.enc.json");

// Optionally merge secrets into environment variables
secrets.populateEnv();

JS Module

The JS module format suits restricted environments where reading static files is problematic and depending upon the platform, building with a custom path may be required.

To encrypt secrets to a JS module:

const secrets = require("gitops-secrets");

async function main() {
  const payload = await secrets.providers.doppler.fetch();

  // Internally managed storage
  secrets.build(payload);

  // Custom path for restrictive environments
  secrets.build(payload, { path: "lib/secrets.js" });
}

main();

Then to decrypt secrets from a JS module, you can rely on internally managed storage:

const { loadSecrets } = require("gitops-secrets");
const secrets = loadSecrets();

// Optionally merge secrets into environment variables
secrets.populateEnv();

Or import directly from the generated JS module:

const { loadSecrets } = require("../lib/secrets");
const secrets = loadSecrets();

// Optionally merge secrets into environment variables
secrets.populateEnv();

Getting Started

We recommend checking out the Working around Vercel’s 4KB Environment Variables Limit for Node.js with GitOps Secrets article which takes you through the entire process step-by-step.

Examples

Take a look at the Vercel GitOps Secrets Next.js sample repository and deploy to Vercel to see it in action.

Contributing

As this package is still in developer preview, a huge contribution you can make is simply testing this with your preferred framework and serverless provider as we'd love your feedback!

You can get support in the Doppler community forum, find us on Twitter, and for bugs or feature requests, create an issue on this repository.

We'd also love to see the number of providers grow and you can check out our contributing guide to get started.

Louies89/gitops-secrets-nodejs