`.env` file support issue tracker
anonrig opened this issue ยท 56 comments
This is a follow-up of dotenv support pr to track the development process, and answer some questions that require a general discussion for making the decision to implement them.
Todos:
- Precisely define the syntax of
.envfiles, and consider a better format not based on ini - Supporting absolute paths for .env file #51425
- Should we support multiple
--env-filethrough the CLI? (cc @GeoffreyBooth) #49542 - Support multi-line values #51289
- Follow up after @tniessen's PR has landed to main #49213
- Should we add a programmatic API? If yes, what would be the use-case? (cc @cjihrig) #51476
Questions:
-
Should we merge NODE_OPTIONS if both environment variable and .env version exist? Currently, we prefer environment variable over .env configuration.(cc @ljharb) #49217 - Should we throw if
.envfile does not exist? We currently follow the default behavior ofdotenvpackage and do not throw. (cc @MoLow)
Please feel free to edit this issue in case there are more questions that needs to be answered.
cc @nodejs/tsc
I vote "throw on non-existing env file" and "throw on multiple --env-file parameters" since those are easy to change later and lets us land without waiting for a decision on both.
IMO:
Should we merge NODE_OPTIONS if both environment variable and .env version exist?
The usual practice (of environment variable precedence ordering) is command line overriding stored setting in files. That way, the end user who runs the code has the last say on the settings.
Should we support multiple --env-file through the CLI?
Yes. This will help co-existence of application and its dependencies with their own env definitions.
Should we add a programmatic API?
Yes. This will help embedded use cases where node.js is not started through the regular launcher with regular command line parsing sequences.
Should we throw if .env file does not exist?
Yes. This will help the feature function to be deterministic.
Should we add a programmatic API?
Yes. This will help embedded use cases where node.js is not started through the regular launcher with regular command line parsing sequences.
I assume that this refers to a C++ API for embedders. This might be useful if the API can be used before any of Node.js or any of its dependencies are initialized. Otherwise, embedders will run into the same problem as we did with NODE_OPTIONS.
I assume that this refers to a C++ API for embedders.
Yes, I meant that.
This might be useful if the API can be used before any of Node.js or any of its dependencies are initialized. Otherwise, embedders will run into the same problem as we did with NODE_OPTIONS.
can you please elaborate on the problem we did with NODE_OPTIONS? sorry, I am not up-to-date with that.
What I mean is that environment variables that affect Node.js, its dependenices, the embedding application, or the embedder's dependencies must be set before the respective component attempts to use it. In many cases, that means before initialization of the respective component.
(On the other hand, reading environment variables from a file is simple enough, so if an application doesn't use the Node.js CLI, it probably might as well implement this feature itself.)
What I mean is that environment variables that affect Node.js, its dependenices, the embedding application, or the embedderโs dependencies must be set before the respective component attempts to use it. In many cases, that means before initialization of the respective component.
Yes. As in, it doesnโt do much good to write JavaScript code like process.env.NODE_OPTIONS = '...' because by the time this JavaScript code runs, Node has already loaded and configured itself and so most (all?) of what you would set by this point is too late to have any effect.
Re merging versus overwriting, I feel somewhat strongly that we should overwrite, because this .env file support is general purpose for any environment variable and not just Node-specific ones, and we canโt know how to merge variables like DATABASE_PASSWORD or whatever. I think it would be confusing UX if Node environment variables got merged but non-Node ones didnโt. I was assuming that whatever variables were set in the environment would take precedence over ones set in a loaded file, but if others feel strongly that it should be the reverse I donโt have a strong opinion on this part.
I would assume that individual variables would overwrite but that if the env has A and the file has B that the process would end up with both A and B.
I would assume that individual variables would overwrite but that if the env has A and the file has B that the process would end up with both A and B.
@ljharb This is the default behavior right now, but do not merge NODE_OPTIONS.
โ node git:(dotenv-support) cat .env
TESTING_KEY=THIS_IS_VALUE
โ node git:(dotenv-support) TESTING_KEY=OVERRIDEN ./out/Release/node -e "console.log(process.env.TESTING_KEY)"
OVERRIDENah ok, gotcha. then yeah i think #49148 (comment) makes sense because indeed it would get confusing figuring out which node options are mutually exclusive, which are single values, which are multiple values, etc.
I was wondering, do you plan to support also something like dotenv-vault that lets you encrypt your secret and decrypt it just in time.
I was wondering, do you plan to support also something like
dotenv-vaultthat lets you encrypt your secret and decrypt it just in time.
Not at the moment, but contributions are welcome.
I was just seeing how this feature worked and I think that either the documentation or the feature is wrong in the 20.6.0 release.
If the same variable is defined in the environment and
in the file, the value from the environment takes precedence.
But, I checked out the v20.6.0-proposal branch, built the latest executable and tried the following:
ฯ echo $PHIL_VAR
what
node (4e4bb9f)
ฯ ./node --version
v20.6.0
node (4e4bb9f)
ฯ ./node -e "console.log(process.env.PHIL_VAR)"
what
node (4e4bb9f)
ฯ ./node -e "console.log(process.env.PHIL_VAR)" --env-file .env
hello
node (4e4bb9f)
ฯ cat .env
PHIL_VAR=helloFrom the documentation, I would have expected the PHIL_VAR variable set in the environment to always be the result. Instead, it seems that the .env file version of the PHIL_VAR variable over-rides it when it is used.
Does this need a quick documentation fix ("the value from the .env file takes precedence") or does the feature need fixing?
does the feature need fixing?
The feature needs fixing. The environment should take precedence over the file.
The environment should take precedence over the file.
I don't think so. The whole point of env files is to override the local environment.
The environment should take precedence over the file.
I don't think so. The whole point of env files is to override the local environment.
We've discussed this quite a bit in this pull request, but every existing implementation of dotenv, including dotenv-node and bun, does not override the environment by default. They tend to have an option to override, which we should consider implementing, but to make overriding the default would go against every existing implementation.
My bad!
@anonrig Big and important use-case for programmatic API is setup hooks for testing frameworks, such as vitest or jest.
I opened a PR to address multiple --env-file declarations support: #49542
@anonrig Big and important use-case for programmatic API is setup hooks for testing frameworks, such as vitest or jest.
Thanks @kibertoad. It is on my agenda.
Big and important use-case for programmatic API is setup hooks for testing frameworks, such as vitest or jest.
@kibertoad Previously, the programmatic API was described as a C++ API for embedders. Is that what you have in mind? If not, could you clarify?
@tniessen I mean being able to put an equivalent of require('dotenv').config() inside a hook :D
In that case, this discussion has already requested two very different runtime APIs. The JS runtime API likely does not need to live in core since it won't be able to properly set NODE_OPTIONS etc.
@tniessen That would be an understandable limitation of JS API. But if native .env support can't be used from JS userspace, then users will still need to depend on dotenv for tests, and then there are no good reasons not to use dotenv for everything else as well, for consistency.
Precisely define the syntax of .env files, and consider a better format not based on ini
I think for this, we can just use existing formats like YAML or TOML, that way we don't have to redefine any conventions and just follow the standard definitions provided by those common formats. A breaking change, but certainly handles scenarios like multi-line values and other issues that are present with the INI format environment file.
If we have to change the format, I would go with TOML.
Now, this change leads to a huge breaking change from all community packages (even cross-language/runtime).
What I imagined for that TODO was that we support a JSON file as an additional config file that was enabled via a different flag, like --config=node.config.json. Node already parses JSON config files such as package.json and policies.
What I imagined for that TODO was that we support a JSON file as an additional config file that was enabled via a different flag, like
--config=node.config.json. Node already parses JSON config files such aspackage.jsonand policies.
Yes, JSON is also good and it is baked into Node, but was thinking that the other two formats has better multi-line/line-break support, which is a common occurrence in using .env files.
In any case, this TODO looks promising and I am looking forward for its feature complete implementation.
It's true that JSON is easier to support, but it's not a good format for humans to edit.
I think what should have happened is a precise specification of the file format, and perhaps a reconsideration of the format before the feature was released.
YAML, JSON, and TOML all seem rather complicated for something as trivial as a flat list of environment variables. More complicated configuration files are out of scope for this feature (and I am not convinced that they are desirable at all).
I agree with @tniessen that discussion about of an additional config file shouldn't be related to the .env feature.
ini seems pretty simple, and since npmrc uses ini, if node added a builtin ini parser/serializer, that seems like a major benefit to the ecosystem regardless.
Came here looking for "Support multi-line values" after trying to migrate off dotenv โ this is dearly needed for e.g. setting private keys, which are usually formatted over multiple lines (-----BEGIN PRIVATE KEY----- etc) and this is supported by dotenv.
Came here looking for "Support multi-line values" after trying to migrate off
dotenvโ this is dearly needed for e.g. setting private keys, which are usually formatted over multiple lines (-----BEGIN PRIVATE KEY-----etc) and this is supported bydotenv.
This is exactly why I switched back to dotenv after previously switching to --env-file. This is way too basic of a use case to not support it out of the box. Even support for escaped newline characters would have been acceptable.
Could #51147 be added here as well?
Is there any feature/behavior change request before making the dotenv functionality stable?
Is there any feature/behavior change request before making the dotenv functionality stable?
We should maybe do the โdefine the .env file formatโ one, as otherwise the supported format is โwhatever --env-file supportsโ which isnโt really a spec, and then any improvements would be breaking changes (like if we add support for multiline values). If you can find some external definition to point to, great, but if not then I guess we should define our own definition based on how the code parses the file, and in our definition we can mention potential future improvements like multiline files, substitutions, encryption, etc.
We should also decide if we want .env loaded automatically. That would be a semver-major change, but we could land such a PR now to ship with Node 22 in April. Iโm not sure if we do want that, since .env is a very common filename and so this could be a very surprising breaking change for a lot of users. If we finish the work to support a Node config file, we could have a filename like .config.node.json loaded automatically in Node 22, and that JSON file could define all the Node options including an envFile option.
Another feature we might support is the dotenv package's override option Do you know how often this is used ?
Another feature we might support is the dotenv packageโs override option
Adding that shouldnโt be a breaking change, so it could happen after marking the API as stable.
We should maybe do the โdefine the
.envfile formatโ one, as otherwise the supported format is โwhatever--env-filesupportsโ
docker compose has something spec-ish for env files:
https://docs.docker.com/compose/environment-variables/env-file/
Lines breaks have to be done using \n + quotes. Maybe it is possible to align with the same spec, so env-files could be re-used etc.
Is there a reason why only relative paths are supported? For example, when I want to load a secret in a container I am forced to count dots:
node --env-file=../../run/secrets/envs
instead of:
node --env-file=/run/secrets/envs
@IlyasShabi override option is a prerequisite for having incremental config files (e. g. .env + .env.test with test-specific overrides), which is a pretty common pattern.
You can use download stats for https://www.npmjs.com/package/dotenv-flow to see that it is quite popular.
Specs of systemd env files:
https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#EnvironmentFile=
I've read the Node.js --env-file docs, and searched the Node.js GitHub issues and PRs, and am struggling to find out if Node.js supports environment variable interpolation/expansion, like:
https://www.npmjs.com/package/dotenv-expand#what-rules-does-the-expansion-engine-follow
A lot of projects use that, so having an upfront answer if it is supported or not, or if it ever might be supported, would be useful for people migrating projects from dotenv-expand to --env-file. If Node.js plans to support interpolation in the future, we will just wait for it before dropping dotenv-expand. If not, we will think seriously about avoiding interpolation in our .env files and adopt --env-file now.
@jaydenseric I'm not opposed to adding it. For making env file stable, we don't need it, but at any time pull-requests are welcome.
I don't know whether it's a bug or if it's just that comments are not really a supported feature yet, but comments in .env files have a weird behavior. Here are some results I've gotten with Node v20.12.2 for the command node --env-file .env -e "console.log(process.env.FOO)":
| .env content | Expected | Result | OK? |
|---|---|---|---|
| undefined | undefined | โ | |
| FOO=1 | 1 | 1 | โ |
| # FOO=1 | undefined | 1 | โ |
| FOO=1 # FOO=2 |
1 | 1 | โ |
| # FOO=2 FOO=1 |
1 | 1 | โ |
| FOO=1 # FOO=2 # FOO=3 |
1 | 3 | โ |
| # FOO=2 FOO=1 # FOO=3 |
1 | 1 | โ |
| # FOO=2 # FOO=3 FOO=1 |
1 | 1 | โ |
| FOO=1 # FOO=2 # FOO=3 # FOO=4 |
1 | 3 | โ |
| # FOO=2 FOO=1 # FOO=3 # FOO=4 |
1 | 4 | โ |
| # FOO=2 # FOO=3 FOO=1 # FOO=4 |
1 | 1 | โ |
| # FOO=2 # FOO=3 # FOO=4 FOO=1 |
1 | 1 | โ |
| FOO=1 # FOO=2 # FOO=3 # FOO=4 # FOO=5 |
1 | 5 | โ |
| # FOO=2 FOO=1 # FOO=3 # FOO=4 # FOO=5 |
1 | 4 | โ |
| # FOO=2 # FOO=3 FOO=1 # FOO=4 # FOO=5 |
1 | 5 | โ |
| # FOO=2 # FOO=3 # FOO=4 FOO=1 # FOO=5 |
1 | 1 | โ |
| # FOO=2 # FOO=3 # FOO=4 # FOO=5 FOO=1 |
1 | 1 | โ |
The simple # FOO=1 doesn't seem to work, and for more complex cases, it seems the proper value is only provided for last and before-last positions, otherwise it's wrong (and the wrong yielded value also depends on the position of the non-commented declaration).
It's quite surprising and potentially dangerous, since someone might keep commented out values like this in order to be able to switch between them easily:
DB_URL=<dev URL>
# DB_URL=<staging URL>
# DB_URL=<prod URL>As I've shown in the table, the value of DB_URL here won't be <dev URL>, it will be <prod URL>.
@acidoxee, we recently addressed a PR to fix issues with parsing comments. Could you please check with Node v22?
@IlyasShabi I guess you're talking about #52406. It's not part of any release yet.
@targos yes my bad sorry
Should we throw if .env file does not exist? We currently follow the default behavior of dotenv package and do not throw. (cc @MoLow)
Yes (and that's already implemented in #50588) but I'd argue in favour of an --optional-env-file argument too. Can be extremely useful in cases where we know there might not be one and we're ok with it, such as production environments where the container already has the environment variables set up and no .env file is needed. Related Feature Request: #50993.
I'm working on it, but I haven't touched C++ since University (and even then only on a very basic level) so any help with this topic will be appreciated. Currently checking all related PRs to try and build on that ๐๐ผ
@BoscoDomingo I'll open a PR to revert the error. I agree that we shouldn't throw an error if env file is not found as a cli argument.
@BoscoDomingo I'll open a PR to revert the error. I agree that we shouldn't throw an error if env file is not found as a cli argument.
@anonrig I'm on it right now, so maybe wait a bit and review mine if you want? Currently writing tests and it should be ready. Also, I do believe the --env-file should error. What I'm proposing a completely new arg (that maybe throws a warning instead?)
Edit: here you go #53060
FWIW #53461 has some ambiguities in the parsing that might need clarification / adjustment.
I'm sure I'm missing something here, but is there a reason Node.js doesn't use a dependency for Dotenv?
I'm sure I'm missing something here, but is there a reason Node.js doesn't use a dependency for Dotenv?
Part of the goal was to support NODE_OPTIONS as a de facto config file, which can only happen if the parsing is done by core before V8 starts.