environment context variables do not override build variables

Question

environment context variables do not override build variables

naipath opened this issue 6 years ago · 46 comments

If you define environment variables in the netlify.toml only the build.environment variables are being used.

Eg:

[build.environment]
  GREETING="Hello World! I am a variable set in a .env file"

# When building on the production environment this variable will NOT be used.
[context.production.environment]
  GREETING="production greeting"

Answer 1 · 2018-10-11T21:20:15.000Z

hi Naipath, i've checked with our support and unfortunately this is a bug in our platform. :(

The fix is to set the environment variable inside the app.netlify.com UI itself.

This is totally on us. Sorry. The requisite bug report has been filed.

Answer 2 · 2018-10-12T16:38:22.000Z

@sw-yx but if I need them? I need them to separate prod and stage environment. May be you know a way to do this?

Answer 3 · 2018-10-17T18:23:19.000Z

yes @weijinnx, you set up the env variables inside the app.netlify.com UI itself. netlify also provides you extra variables you can use inside your code: https://www.netlify.com/docs/continuous-deployment/#build-environment-variables

Answer 4 · 2018-12-20T09:04:17.000Z

Hi, I see that this issue was closed, but was it resolved?
I am having similar issues when trying to use environment variables for deploy-preview in my lambda functions.

Answer 5 · 2018-12-21T04:37:19.000Z

i see this. now that im rereading this issue im not sure why our support thought this is a netlify functions issue. it looks like there is a related issue with deployed netlify functions reading environment variables, but possibly separate from this one.

if you deploy this does it work? so can we confirm that its just a netlify-lambda issue?

Answer 6 · 2018-12-21T06:06:03.000Z

Afaict this is working as intended, the context when using netlify-lambda is not production, it could potentially be development or something, but the context needs to be set somewhere..

I might be forgetting but does netlify-lambda provide a cli argument to set the context and or is any context use by default?

Answer 7 · 2018-12-21T06:09:51.000Z

yup, the branch fixes it for me 👍

Answer 8 · 2018-12-21T06:11:15.000Z

my use case is for [context.deploy-preview.environment] and [context.branch-deploy.environment]

Answer 9 · 2018-12-27T20:06:50.000Z

@8eecf0d2 the context gets set automatically by netlify;
for production is [context.production]
for pull requests/deploy previews: [context.deploy-preview]
for all branch deploys: [context.branch-deploy]
and for a specific branch: [context.name-of-my-custom-branch]

Answer 10 · 2018-12-27T20:17:57.000Z

@shmargum right, I was talking about netlify-lambda which afaict doesn't set the context.

Answer 11 · 2018-12-27T20:53:51.000Z

the process.env.CONTEXT is being set correctly when the netlify-lambda build command runs on the netlify servers, this is why the pull request works
I actually now realize that my pull request doesn't take into account the process.env.BRANCH in order to override things specific to a branch (branch-deploy affects all branches but we can also have branch specific overrides such as for the staging branch)

Answer 12 · 2019-01-10T18:29:11.000Z

we have merged #94 which should address this - please reopen/comment if any further issues

Answer 13 · 2019-03-26T22:35:35.000Z

I'm using netlify-lambda@^1.4.3 and still have this issue with lambda functions in deploy-preview.

[context.production.environment]
  CONTACT_FORM_EMAIL = "info@example.com"

[context.deploy-preview.environment]
  CONTACT_FORM_EMAIL = "developers@example.com"

When using deploy-preview-1--my-app.netlify.com, emails are still sent to info@example.com.

It is worth mentioning that we also have the build environment variables set in the UI, should I remove those?

Answer 14 · 2019-03-27T16:52:29.000Z

i dont understand what this means, "the build environment variables set in the UI".

its worth opening up netlify-lambda in node_modules, and tracing through to see what is going on with the context object so you can debug. i'm worried about that hyphen in deploy-preview. i'm not sure how we handle that.

Answer 15 · 2019-03-27T20:41:33.000Z

@sw-yx I'm working on creating a cleaned up public repo to show the issue, stay tuned.

i dont understand what this means, "the build environment variables set in the UI".

i'm worried about that hyphen in deploy-preview

That's how it is done in the TOML docs in netlify: https://www.netlify.com/docs/netlify-toml-reference/#getting-started

Answer 16 · 2019-03-27T23:21:08.000Z

Ok, here you go. Let me know if this is enough info to explain what I mean.

I created a lambda function that only returns Hello <process.env.WHO_AM_I>!. In production it should be "World", in staging (deploy previews) should be "Developer".

Production: https://netlify-context-in-previews.netlify.com/.netlify/functions/hello should return "Hello World!"

Deploy Preview 1: https://deploy-preview-1--netlify-context-in-previews.netlify.com/.netlify/functions/hello should return "Hello Developer!"

Both return "Hello Build!", as WHO_AM_I was set to "Build" when creating the app (see image below). My understanding is that it should be overwritten, correct? Am I doing something wrong?

Repo: https://github.com/pgarciacamou/netlify-context-in-previews

// functions/hello.js
exports.handler = async event => {
  const { WHO_AM_I } = process.env;

  return {
    statusCode: 200,
    body: `Hello ${WHO_AM_I}!`
  };
}

# netlify.toml
[build]
  command = "npm run build"
  functions = ".netlify/functions/"

[context.production.environment]
  WHO_AM_I = "World"

[context.deploy-preview.environment]
  WHO_AM_I = "Developer"

Netlify App: (notice the ENV variable)

Answer 17 · 2019-03-28T17:09:17.000Z

I think this issue should be reopened. Unless there is something not stated in the docs that must be done to change the context when building with netlify-lambda.

Answer 18 · 2019-03-28T23:49:37.000Z

ok thank you for the detailed repro so i can understand your issue.

so first of all, netlify-lambda is just a tool for local emulation of functions and building of them for deployment into the main Netlify service. anything main service related as you have shown above, is not at all part of netlify-lambda's responsibility. so this is not actually the right place to resolve this issue.

however, we still care that you are seeing this behavior that is unexpected. Basically, what is being set in the UI is overriding the more specific context dependent env variables set from netlify.toml. I agree that this is unintuitive. I'm referring this to our support team, who have probably seen this before. please stay tuned...

Answer 19 · 2019-03-29T00:06:32.000Z

@pgarciacamou we have a possible bug that might be worth trying a small fix for - can you try clicking the Trigger deploy button to see if it might update the new env variables from your netlify.toml?

Answer 20 · 2019-03-29T17:14:46.000Z

@sw-yx

Anything main service related as you have shown above, is not at all part of netlify-lambda's responsibility.

You are totally correct, this issue has nothing to do with netlify-lambda, my bad 😅. Would you mind pointing me to the best place to add this issue?

we have a possible bug that might be worth trying a small fix for - can you try clicking the Trigger deploy button to see if it might update the new env variables from your netlify.toml?

I tried it but the issue is still there. If you want me to, I can add you to the app so you don't have to wait for me, although it might be easier & faster (and more flexible) for you to fork the repo and deploy to netlify.

Answer 21 · 2019-04-02T04:10:24.000Z

thanks for your patience :) as with everything, the best place is Netlify Support: https://www.netlify.com/support/ - they have the right process and knowledge to help you. in this case, we suspect that there may be a known bug that is in our backlog to solve. in the mean time, Support can work with you to suggest workarounds, and/or notify you when the bug is done. best to take it there.

Answer 22 · 2019-04-12T04:10:12.000Z

@pgarciacamou did you get help from support? If yes would you mind adding your solution here: https://community.netlify.com/t/netlify-toml-context-env-variables-do-not-apply-to-functions/410

I tried what they suggested but still can't get my staging functions to have staging values for the Env Variables.

Answer 23 · 2019-04-12T04:34:05.000Z

@renschler I did get help but did not work. The response to the email I sent was:

Yup, that's a known bug. Env vars set in the netlify.toml file are not accessible to netlify functions. I've added you to the issue that is tracking that and we'll let you know when we get a fix out for it. For now, you will need to set your env var in the UI and handle the context changes via the build command.

Like so: eval MY_CONTEXT=${MY_UI_VARIABLE} && npm run build

That example is based on this: https://stackoverflow.com/questions/42107655/travis-can-you-construct-env-var-using-other-env-vars

Hope that helps.

That was Dennis from Netlify. NOTE: he mentioned he added me to the issue, but I believe this is an internal issue because he did not attach a link.

That said, I did not try using eval, I tried using env-cmd as explained in https://facebook.github.io/create-react-app/docs/deployment#customizing-environment-variables-for-arbitrary-build-environments (you can look at the PR of my attempt here pgarciacamou/netlify-context-in-previews#2). NOTE: it looked promising, but it did not work so I just closed it and moved on.

I tried what they suggested but still can't get my staging functions to have staging values for the Env Variables.

I might quickly try eval tomorrow morning (PST). If you try it before I do, please let me know of your findings.

Answer 24 · 2019-04-12T05:22:04.000Z

I tried eval and it also didn't work. I updated the post in netlify community.

Maybe the shorterm solution is just to push different versions of the functions to my staging branch. Run them through a string replace or something like that before pushing.

Answer 25 · 2019-04-12T06:28:48.000Z

Hmm, too complicated for my taste.

I would honestly hack around it by creating a second Netlify app with the same git repo (temporarily, while the bug is fixed), which would be the staging environment and attach it to a "staging" branch instead of "master". And now you can have different custom environment variables for each app in the UI.

e.g. myapp.netlify.com to master branch and preview-myapp.netlify.com to staging. Which is what I used to do before deploy-previews.

Does that make sense?

I just did it for the repo I created for this issue:

environment var: WHO_AM_I=Build
https://netlify-context-in-previews.netlify.com/.netlify/functions/hello

environment var: WHO_AM_I=Staging
https://preview-netlify-context-in-previews.netlify.com/.netlify/functions/hello

Now just do:

git push origin current-branch:staging --force

Answer 26 · 2019-04-12T06:41:36.000Z

@pgarciacamou thanks yeah that's cleaner. It won't have the same netlify identity pool but I dont think that will matter for me.

Answer 27 · 2019-04-25T18:31:08.000Z

In order to access environment variables that are present at build time (such as those you might specify in netlify.toml or the special build env variables such as DEPLOY_URL) in a function you can do the following if you're building your functions with netlify-lambda.

Create a custom webpack config file. Let's call it webpack.functions.js with the following content and simply add the env vars that you need. You could even export the entire build env with new webpack.EnvironmentPlugin(process.env) but you probably shouldn't unless you fully understand what that does.

// webpack.functions.js
const webpack = require('webpack')
module.exports = {
    plugins: [
        new webpack.EnvironmentPlugin(['VARIABLE1', 'VARIABLE2', ...])
    ]
}

Change your build command to include the config file:

netlify-lambda build --config ./webpack.functions.js functions/

Use process.env like you normally would in your functions.

What is happening? The issue is that the aforementioned env variables don't get exported to your function env - they are only present at build time. What we're basically doing is that we're replacing instances of process.env.VARIABLE in the function code at build time where the env variables we want are present. You can read about the environment plugin here.

Answer 28 · 2019-04-25T19:30:19.000Z

@EmilEriksen thanks! Can you expand how you would use that approach to have different variable values for Staging vs Master branches?

Answer 29 · 2019-04-25T19:59:40.000Z

@renschler You'd simply use contexts in netlify.toml like you normally would. You just need to add the names of the variables to new webpack.EnvironmentPlugin(['VARIABLE1', 'VARIABLE2', ...]). As I mentioned you could also simply change the line to new webpack.EnvironmentPlugin(process.env) and then it should work for all variables without you having to manually specify any names. There could potentially be conflicts with this method though if you're using some functions specific env variables that are also defined at build time but it's probably not an issue.

Answer 30 · 2019-06-24T20:07:43.000Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Answer 31 · 2019-06-25T07:56:16.000Z

never stale

Answer 32 · 2019-07-15T05:49:14.000Z

hi there any news or workarounds about this issue? It would be awesome to use different env variables in functions based on dev/branch/master 🙏🏼

Answer 33 · 2019-07-15T19:42:08.000Z

i believe this will be a better first class solution once the new buildbot @DavidWells is working on comes alive

Answer 34 · 2019-07-15T20:13:14.000Z

i believe this will be a better first class solution once the new buildbot @DavidWells is working on comes alive

Sounds promising. Anything to read more about the new buildbot?

Answer 35 · 2019-07-17T07:08:06.000Z

nothing public yet and no timeline, its more r&d work so i dont want to overpromise.

Answer 36 · 2019-08-19T18:46:16.000Z

In order to access environment variables that are present at build time (such as those you might specify in netlify.toml or the special build env variables such as DEPLOY_URL) in a function you can do the following if you're building your functions with netlify-lambda.

Create a custom webpack config file. Let's call it webpack.functions.js with the following content and simply add the env vars that you need. You could even export the entire build env with new webpack.EnvironmentPlugin(process.env) but you probably shouldn't unless you fully understand what that does.
// webpack.functions.js
const webpack = require('webpack')
module.exports = {
    plugins: [
        new webpack.EnvironmentPlugin(['VARIABLE1', 'VARIABLE2', ...])
    ]
}
Change your build command to include the config file:
netlify-lambda build --config ./webpack.functions.js functions/
Use process.env like you normally would in your functions.

What is happening? The issue is that the aforementioned env variables don't get exported to your function env - they are only present at build time. What we're basically doing is that we're replacing instances of process.env.VARIABLE in the function code at build time where the env variables we want are present. You can read about the environment plugin here.

This would be so handy to add to the official Netlify documentation. It’s far from clear otherwise how to safely use secret variables in lambdas.

Answer 37 · 2019-10-18T19:29:05.000Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Answer 38 · 2020-02-26T18:17:03.000Z

I don't think this should have been closed? @sw-yx

Answer 39 · 2020-03-25T20:20:24.000Z

i have left the company and dont maintain this anymore sorry

Answer 40 · 2020-08-07T21:39:15.000Z

Sorry about that - reopened now

Answer 41 · 2022-01-15T03:32:25.000Z

Anyone got a solution for this? I thought nextjs api would solve my problem, but no, it also doesnt pull the env vars

Answer 42 · 2022-02-04T19:11:10.000Z

I built https://www.npmjs.com/package/netlify-plugin-inline-functions-env-typescript not sure it can help

Answer 43 · 2022-03-15T18:04:59.000Z

Not sure how this is still open 3yrs later because its seems simple enough. Since Netlify is using build containers can't the tool just export the ENVs during container instantiation?

What is the point of having the CLI and netlify.toml allow for context overrides and the netlify build can specify a flag --context xyz if those contexts do not use those ENV Vars specified in the container env?

Is the work around to mess with build env within the chosen bundler?? I'm using in Vite and I need it to string replace URLs in bundled code based on environments (this is similar to webpack define plugin). Do I need to ignore Netlify configs and bake them into the bundler?

Answer 44 · 2022-04-18T14:45:42.000Z

I'm also having issues with environment variables per branch deploy. Maybe you should remove this feature from the docs if it's not working correctly.

Answer 45 · 2022-05-23T21:53:05.000Z

I built https://www.npmjs.com/package/netlify-plugin-inline-functions-env-typescript not sure it can help

This plugin worked perfectly for me. Thank you for building it an linking it @Elyx0 .

Answer 46 · 2022-11-17T17:48:04.000Z

Hi folks, this is somewhat orthogonal to the bug in netlify.toml config, but we now support contextual environment variables (including branch-based values) in the Netlify UI, CLI, and API. If you'd prefer to set your env vars with one of those methods, you can move your site over to the new experience today! Hope that helps.