Environment Config settings add slashes

Question

Environment Config settings add slashes

sketchthat opened this issue 7 years ago · 32 comments

Running Firebase-Tools 3.9.1 on a Mac, when I store a \ it adds slashes and converts to \\.

This is an issue when trying to store the Firebase Storage PEM file in environment settings.

--- BEGIN PRIVATE KEY ---\n
...
--- END PRIVATE KEY ---\n

Converts to

--- BEGIN PRIVATE KEY ---\\n
...
--- END PRIVATE KEY ---\\n

Which causes the authentication to fail.

This issue wasn't happening last week, so it appears to be a new issue.

Answer 1 · 2017-06-16T16:27:17.000Z

Can you please add repro instructions, including the exact set command you used? Thanks.

Answer 2 · 2017-06-17T07:52:51.000Z

$ firebase functions:config:set foo.bar="Hello\World" --project my-project-0000
✔  Functions config updated.

Please deploy your functions for the change to take effect by running firebase deploy --only functions

$ firebase functions:config:get foo.bar --project my-project-0000
"Hello\\World"

Should it be adding slashes? Last week (Not sure of the version) I added a string with a slash and it didn't double up, but this week it does. So not sure how to handle it in code.

Answer 3 · 2017-06-17T07:54:49.000Z

In real-world apps I've been adding the firebase json credentials to the config so I can connect with the storage tools. Last week PEM uploaded correctly, this week it adds slashes on the \n and converts to \\n.

Answer 4 · 2017-06-20T04:54:39.000Z

Thanks for the repro instructions, I've reproduced it and will work on a fix.

Answer 5 · 2017-06-20T09:49:16.000Z

Hey @sketchthat , after some investigations, I figured out that this is an issue with one of our dependencies - Cloud Runtime Configuration. I've filed the bug with the team, and will keep you posted on their progress.

Answer 6 · 2017-07-03T06:46:08.000Z

I have the same issue.

Answer 7 · 2017-08-04T14:31:29.000Z

Hi @laurenzlong I don't know why my comment was downvoted, but is there any update on this issue?

Answer 8 · 2017-08-04T16:59:17.000Z

@janakaj Comments like "I have the same issue" and "is there any update" are spam to watchers of Github repos.

If you have the same issue as reported, upvote the original, and then please exercise patience rather than filling our inboxes with comments that don't add to the conversation.

Answer 9 · 2017-10-17T11:15:13.000Z

I found a walkthrough (or awful solution) :S I don't wanna to do that but ... It's works >,<

You can do it a .replace(/\\n/g, '\n') to the private_key

const serviceAccount = functions.config().fireenv;

admin.initializeApp({
   credential: admin.credential.cert({
      "projectId": serviceAccount.project_id,
      "private_key": serviceAccount.private_key.replace(/\\n/g, '\n'),
      "clientEmail": serviceAccount.client_email
   }),
   databaseURL: whatever,
   ...
});

@sketchthat @janakagamini @PatrickMcD I know it was 4 months ago but .. :)

Answer 10 · 2017-10-17T22:01:57.000Z

@YuniorGlez - Are you using the environment details to initialise the admin library? Cloud Functions will do this by itself.

I needed the environment details to access Google Cloud Storage. I ended up including a json file into the build.

  const storage = await import('@google-cloud/storage');
  const storageCreds = require('credentials.json');

  const gcs = storage({
    projectId: storageCreds.project_id,
    credentials: storageCreds,
  });

  const bucket = gcs.bucket(functions.config().firebase.storageBucket);

Answer 11 · 2018-08-14T18:57:09.000Z

Has this issue been fixed? I just came across a similar issue when using a dollar sign in the value-part of functions:config:set. E.g. firebase functions:config:set test.var="abc$def" deploys fine. But firebase functions:config:get test.var gives only "abc". The same issue arises when reading the environment variables with functions.config().

I am using firebase-tools v4.1.0 and OS X 10.13.6

Answer 12 · 2018-09-11T02:20:56.000Z

If you add a slash it fixes it abc\$def.

Answer 13 · 2019-01-09T00:34:59.000Z

Similar to #378 - should investigate this as well and do make a decision. Maybe documentation needs to be updated to talk about using special characters?

Answer 14 · 2020-02-04T21:11:26.000Z

in Feb 2020 this still seems to be an issue. With no mention in the documentation. Any update?
I know we are supposed to up vote issues, but a year later I feels warrants a comment here.

Answer 15 · 2020-02-04T21:16:37.000Z

@YuniorGlez Thanks for the save here. The amount of time I spent trying to figure out what was going on with my keys is too much.

This really needs to be updated.

Answer 16 · 2020-03-24T19:47:11.000Z

@laurenzlong @worknudge @LukeInTheCloud

Also faced this. Couldn't find a solution on Windows except spinning up a WSL with firebase CLI and using:

firebase functions:config:set service_account="$(cat service-account.json)"

It's way too hard to securely access Google APIs from a Dart function (without saving credentials in code). All the Dart APIs are http wrappers, but some googleapis_auth functions like clientViaMetadataServer don't work on Node, GCloud Function Env Vars aren't exposed in Firebase Functions Interop, and getting the key into Firebase env config takes silly backflips :)

Answer 17 · 2020-06-03T22:18:48.000Z

Hey @sketchthat , after some investigations, I figured out that this is an issue with one of our dependencies - Cloud Runtime Configuration. I've filed the bug with the team, and will keep you posted on their progress.

Hi @laurenzlong @kevinajian, any update on this issue? thanks.

Answer 18 · 2020-09-25T07:53:31.000Z

just found this today

I stored

  "hosting": {
    "base_url": "https://mentor-cv-staging.web.app"
  },

which adds a slash and breaks my CORS setting

app.use(cors({ origin:  functions.config().hosting.base_url}))

Access to fetch at 'https://us-central1-mentor-cv-staging.cloudfunctions.net/allFunctions/createUser' from origin 'https://mentor-cv-staging.web.app' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header has a value 'https://mentor-cv-staging.web.app/' that is not equal to the supplied origin. Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Answer 19 · 2020-09-25T08:25:14.000Z

For now I just done

app.use(cors({ origin: functions.config().hosting.base_url.replace(/\/$/, "") }))

Answer 20 · 2021-06-23T17:46:34.000Z

@taeold, passing this along to you to make sure it gets fixed in the incoming env:* commands.

Answer 21 · 2021-06-25T09:38:16.000Z

In fairness, this problem isn't specific to the Firebase CLI. Passing escape sequence to a program from shell is generally tricky. https://stackoverflow.com/questions/11433977/passing-new-lines-in-command-line-argument explains things better than I can.

To demo, here's a program that just spits out arguments it gets:

// args.js
console.log(process.argv.slice(2))

$ node args.js "a\nb"
[ 'a\\nb' ]

$ node args.js "Hello\World"
[ 'Hello\\World' ]

$ node args.js "abc$def"
[ 'abc' ]

Firebase CLI is just working with the argument it gets.

Here's a trick - surround your strings with $' (ANSI-C Quoting - your shell probably implements it):

$ node args.js $'a\nb'
[ 'a\nb' ]

$ node args.js $'Hello\World'
[ 'Hello\\World' ] # \W isn't a valid escape character so nothing is done.

$ node args.js $'abc$def'
[ 'abc$def' ]

Given that, I'm not sure how I can explain the fact that this issue was reported as a regression. But hoping that the trick is useful for folks experiencing a similar issues.

Answer 22 · 2021-07-06T01:00:36.000Z

Hey @sketchthat. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

Answer 23 · 2021-07-09T01:00:23.000Z

Since there haven't been any recent updates here, I am going to close this issue.

@sketchthat if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

Answer 24 · 2021-07-28T11:18:36.000Z

@taeold this problem is specific to the Firebase CLI.

Even if you correctly escape the arguments so that Node receives the string as intended, something in the Firebase CLI stack escapes the new line characters.

Repro instructions

As you said above we know that escaping a string with $'' doesn't cause newlines to be represented by \\n once the argument is interpreted by Node, and we can test that with your args.js script:

$ node args.js $'a\nb'
[ 'a\nb' ]

Set a functions config variable including a newline using the Firebase CLI using the same syntax, deploy it and read it out again

> firebase functions:config:set a.b=$'a\nb'
> firebase deploy --only functions
> firebase functions:config:get
{
  "a": {
    "b": "a\\nb"
  }
}

Given that, I'm not sure how I can explain the fact that this issue was reported as a regression. But hoping that the trick is useful for folks experiencing a similar issues.

It's been four years now and the closest this issue has come to a response by the maintainers is being attributed to user error and closed for inactivity. That's a bit frustrating. It's really not hard to reproduce based on @sketchthat's initial description, and the end result is everyone is committing service account credentials to their source code.

Please address the issue or document the limitation that newlines will be incorrectly encoded by the CLI.

Answer 25 · 2021-07-28T15:48:06.000Z

@wmadden Thanks for giving your feedback. I'll go ahead and re-open the issue.

I'm having trouble reproducing the issue you are reporting. Here's what I got:

$ node -v
v14.15.4
$ firebase --version
9.16.0
$ firebase functions:config:set a.b=$'a\nb'
$ firebase functions:config:set c.d="c\nd"
$ firebase functions:config:get a.b
{
  "a": {
    "b": "a\nb"
  },
  "c": {
    "d": "c\\nd"
  },
}

(I tried running firebase deploy in between the config commands, but that didn't seem to change anything).

On my Mac, using zsh, things appear to work as expected. I tried running the command on a different platform (Ubuntu) and on a different terminal (bash) with same results.

I'm unsure where the bug might be - maybe this is a platform-specific issue? Old node version? Old firebase CLI version?

Do you mind sharing your setup with us?

Answer 26 · 2021-07-28T15:51:55.000Z

No problem! I'm also on a Mac using zsh, here are my Node and Firebase versions:

$ node -v
v14.16.1
$ npx firebase --version
9.14.0

Answer 27 · 2021-07-28T16:02:02.000Z

@taeold npx might be adding another layer of escaping. If I use your commands exactly as in your comment then it records a\nb correctly, but if I use npx firebase functions config:set it stores the result with double backslashes.

I'm not sure if that would have caused the original issue, which might have predated npx, but it's worth noting now

$ npx firebase functions:config:set a.b=$'a\nb'; npx firebase functions:config:get
✔  Functions config updated.

Please deploy your functions for the change to take effect by running firebase deploy --only functions

{
  "a": {
    "b": "a\\nb"
  }
}

$ firebase functions:config:set a.b=$'a\nb'; firebase functions:config:get
✔  Functions config updated.

Please deploy your functions for the change to take effect by running firebase deploy --only functions

{
  "a": {
    "b": "a\nb"
  }
}

Answer 28 · 2021-07-28T17:35:15.000Z

😢 That's very strange. Wonder what's going on innpx that swallows the string quoting.

Here's yet another "it works on my machine" 🤦

$ npx --version
6.14.10
$ npx firebase functions:config:set a.b=$'a\nb' && npx firebase functions:config:get                                                                                                           
✔  Functions config updated.

Please deploy your functions for the change to take effect by running firebase deploy --only functions

{
  "c": {
    "d": "c\\nd"
  },
  "a": {
    "b": "a\nb"
  },
}

Newline escaping is looking to be a big PITA, and Firebase CLI could be something more to make this process better. There are some updates to how environment configurations are managed rolling out soon, and I'll make sure this problem is considered in that rollout.

@wmadden You are not the original author of this issue, but wondering if at least your problem is addressed with the $' escaping. If so, feel free to close the issue.

Answer 29 · 2021-07-28T17:49:05.000Z

@taeold I don't have adequate Github permissions to close the issue and, no offence, I'm not convinced it's adequately addressed. It took me a solid day and a half of experimentation and Googling to a) find this thread and b) give up and add .replace(/\\n/g, '\n') to my code.

People have been having issues with this since 2017 and, like the original author way back then, I'm trying to store a service account private key in a config variable. So it's probably safe to assume other people will do too and run into the same issues, if they don't just commit their service account keys to their codebase 😬

There is low-key confusion scattered around the web surrounding this issue, so it deserves a better solve than "couldn't reproduce".

Also, in case it helps:

$ npx --version
7.10.0

P.S. I appreciate you responding so quickly considering how long this ticket was open and unresolved ❤️

Answer 30 · 2021-07-28T19:09:40.000Z

@wmadden Great suggestions there. I went ahead and filed a FR on better support for multiline strings for firebase config. I can't promise that I will work on it right away, but as I mentioned before, I'll try to include this fragile UX in our discussions.

Since this issue seems to get good SEO, here is a summary of suggested workaround to avoid having extra slashes add to your config values:

Option 1: Use ANSI-C quoting

$ firebase functions:config:set some.key=$'Hello\nWorld'
$ firebase functions:config:get some.key
"Hello\nWorld"

Option 2: Write config value to a file and run the following commands:

$ firebase functions:config:set some.key="$(cat path/to/private.pem)"
$ firebase functions:config:get some.key
"-----BEGIN PRIVATE KEY-----\nMI...

Answer 31 · 2021-08-06T01:00:18.000Z

Hey @sketchthat. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

Answer 32 · 2021-08-10T01:00:20.000Z

Since there haven't been any recent updates here, I am going to close this issue.

@sketchthat if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.