As an Open Forest Admin I'd like to know I can log-in and view admin Pages

Question

As an Open Forest Admin I'd like to know I can log-in and view admin Pages

Opened this issue 4 years ago · 33 comments

Notes

It has recently come to our attention that admins for Open Forest are not able to see any admin pages once they log-in. Instead a "blank" page is displayed with only the header and footer being displayed. Additionally a 401 error is thrown in the console and it appears that users are actually logged in but not being routed correctly or not being given the proper permissions.

Tasks

Work with dev and backend team to derermine the cause of this issue
Work with someone from the eAuth team to determine if the issue could have originated on their end
Design and implement any changes or corrections necessary to resolve the issue
PO approved

Definition of Done

Admins are able to login and view all pertinent admin pages

Answer 1 · 2020-08-18T18:08:10.000Z

We discovered today that this issue is most likely a cookie-related issue on Google Chrome specifically. As you can see in the screen shot below admins are able to access the site via Firefox:

Answer 2 · 2020-08-18T18:10:34.000Z

I don't know if this is helpful, but I ran into a similar issue in January. Chrome got very picky about 3rd-party cookies, and if the user only just recently updated, it could be that they just now got hit with it. I know Chrome is also planning to restrict 3rd-party cookies even further.

Answer 3 · 2020-08-18T18:14:45.000Z

@mgwalker just as an additional note this is only affecting production. Staging is working as expected.

Answer 4 · 2020-08-18T18:47:00.000Z

So we THINK this may be related to the chrome issue that started back in Fed 2020. And it looks like the only browser that will work is Firefox.
The error we are getting in Chrome relates to Set-Cookie SameSite attribute which is not being specified.

So we are getting the 401 error because of this issue.

Per Web/Dev site https://web.dev/samesite-cookies-explained/

Changes to the default behavior without SameSite #
While the SameSite attribute is widely supported, it has unfortunately not been widely adopted by developers. The open default of sending cookies everywhere means all use cases work but leaves the user vulnerable to CSRF and unintentional information leakage. To encourage developers to state their intent and provide users with a safer experience, the IETF proposal, Incrementally Better Cookies lays out two key changes:

• Cookies without a SameSite attribute will be treated as SameSite=Lax.
• Cookies with SameSite=None must also specify Secure, meaning they require a secure context.

Chrome implements this default behavior as of version 84. Firefox has them available to test as of Firefox 69 and will make them default behaviors in the future. To test these behaviors in Firefox, open about:config and set network.cookie.sameSite.laxByDefault. Edge also plans to change its default behaviors.

So looks like WE/Open Forest need to look at how we are implementing cookies.

Answer 5 · 2020-08-18T18:50:29.000Z

FS version issued for Firefox is 78, so looks like it will not work if user has been good about updating their browsers.

Answer 6 · 2020-08-18T19:01:23.000Z

https://www.chromium.org/updates/same-site/test-debug

OK I just set my browser to disable samesite

I got my content.

Answer 7 · 2020-08-18T19:06:52.000Z

If you disable samesite in chrome you can get the content working again.
https://www.chromium.org/updates/same-site/test-debug

Answer 8 · 2020-08-18T19:12:52.000Z

@JonathanLerner54 nice find, that works for me as well. Thanks!

Answer 9 · 2020-08-18T19:13:51.000Z

Screenshot

Answer 10 · 2020-08-18T19:25:51.000Z

We still need to fix how ePermits is dealing with cookies on the server side. Eventually, the security gurus will lock out the ability to change this in the browser settings. The site https://www.chromium.org/updates/same-site/test-debug details the parameters we need to set.

Answer 11 · 2020-08-19T16:24:13.000Z

mwreiss commented 4 years ago

Answer 12 · 2020-08-19T16:37:09.000Z

New Code has been vetted

SIA documenting the securty review of this change was completed by ePermits security auditor, Jon Lerner and archived in Pinyon Folder: \Box\ePermits\SOP\ePermits Releases\FY 20 ePermits Release to Production 1420

Answer 13 · 2020-09-02T21:43:01.000Z

NOTE: @Rebekah-Hernandez and myself tested the functionality of special-uses in both Firefox and Windows Edge. Everything from the submission and acceptance can be completed. Additionally, both admin and basic user access features function as expected.

Accordingly, we sent out an email to all MBS Admins informing them of these two additional options for accessing the site, as well as provided them with a detailed description of how to switch the default "sameSite" setting to "disabled" to enable a user to use the site on Chrome as well.

Answer 14 · 2020-09-03T18:53:04.000Z

Some updates, per my understanding (@abdul-fs, @mwreiss please weigh in if any of this is inaccurate):

The reason staging worked while production did not is because in staging, the cookies are not third-party.
- In staging, the frontend and backend are on the same domain (*.apps.cloud.gov), so the cookies move through just fine.
- In production, the frontend and backend are at different domains (openforest.fs.usda.gov and fs-intake-api.app.cloud.gov, respectively), so cookies are subject to 3rd-party cookie constraints
As of the February 2020 update, Chrome requires that 3rd-party cookies contain the secure and samesite=None attributes.
The backend indirectly uses the cookies library to write the authentication cookie. That library could not set the samesite attribute until version 0.8.0. The backend directly uses cookie-session, which then pulls in the cookies library. cookie-session did not include cookies@0.8.0 until version 1.4.0
The team updated the backend's package.json to use cookie-session 1.4.0. That should likewise update the cookies library. However, when deployed to production, that did not work – cookies still did not have the samesite attribute.
- package.json is a machine-readable file where the list of libraries is stored, so that libraries can be installed by automated tools in deployment.
The package-lock.json was not updated when package.json was updated. It seems that cloud.gov uses package-lock.json when it is available, so despite the update to package.json, an older version of the cookies library was still installed and so it did not support setting the samesite attribute.
- package-lock.json is a "lock file" version of package.json. The package.json lets you specify a range of versions of dependencies, but package-lock.json lists the exact versions that were actually installed. That way, when it goes through testing and deployment, you can be certain that the same versions of dependencies are used everywhere.

The team deleted the package-lock.json file and re-deployed. That resulted in a build error:

2020-09-02T15:02:38.48-0400 [RTR/0] OUT
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR Error: Invalid signature
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at Q.fcall.then.then.certs (/home/vcap/app/node_modules/passport-saml/lib/passport-saml/saml.js:611:15)
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at _fulfilled (/home/vcap/app/node_modules/q/q.js:854:54)
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at /home/vcap/app/node_modules/q/q.js:883:30
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at Promise.promise.promiseDispatch (/home/vcap/app/node_modules/q/q.js:816:13)
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at /home/vcap/app/node_modules/q/q.js:624:44
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at runSingle (/home/vcap/app/node_modules/q/q.js:137:13)
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at flush (/home/vcap/app/node_modules/q/q.js:125:13)
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at Shim.applySegment (/home/vcap/app/node_modules/newrelic/lib/shim/shim.js:1424:20)
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at wrapper (/home/vcap/app/node_modules/newrelic/lib/shim/shim.js:2065:17)
2020-09-02T15:02:38.49-0400 [APP/PROC/WEB/1] ERR     at process._tickCallback (internal/process/next_tick.js:61:11)

The build error does not seem related to cookies but rather a SAML signature issue. In any case, reverting the change fixed the build error.

So I think that's where we are right now.

A potential next step might be to delete package-lock.json and then running npm install to rebuild the package-lock.json with the updated versions of things. That way we still have the package-lock.json file in case that's part of the build error above, and the package-lock.json itself is updated.

As an aside, in the future, rather than manually editing dependencies in package.json, it would be better to use npm update or npm install from the command line. That will ensure that the package-lock.json stays in sync, so long as it appears that package-lock.json is required.

Answer 15 · 2020-09-09T14:53:03.000Z

@mgwalker https://www.npmjs.com/package/should-send-same-site-none.

https://github.com/USDAForestService/fs-open-forest-platform/pull/1451/files

Answer 16 · 2020-09-30T14:19:47.000Z

It appears this resolved the issue! Woo!hoo!

Great work team!

Answer 17 · 2020-09-30T14:46:31.000Z

Awesome! Can you temporarily add me as an admin to take a look too? I assume the reason I still get stuck on the logged-in page is because I'm not an admin. 😛

While I was logging in, I decided I might as well capture the network traffic just to hand-check the cookie samesite attribute was getting set. I turned off caching and enabled network log preservation in the Chrome dev tools, then watched the network tab. It looks like I'm still getting a cookie without SameSite being set. This is from the response to https://fs-intake-api.app.cloud.gov/auth/usda-eauth/saml/callback, which I assume happens when eAuth is finished and sending us back to Open Forest:

Then later in the request chain, the app attempts a request to https://fs-intake-api.app.cloud.gov/auth/user, and the dev tools show that the request cookies are being filtered out because SameSite is not set:

All of this is a long-winded way of saying, it's not working for me, but since it IS working for you with default Chrome settings, then I assume the issue must be that I'm not an admin, though I don't know why that would affect the way the cookies are set.

Answer 18 · 2020-10-01T13:11:07.000Z

@mgwalker is this the email address for prod eauth for you? Michael.Walker2@usda.gov .

I did an additional test, looks like something was cached on my end so I'm thinking the change didn't work as I'm seeing the "logged-in" screen again.

Answer 19 · 2020-10-01T14:32:25.000Z

Yep, that's me. Rats. I was hoping it was just my end. Ah well...

Answer 20 · 2020-10-01T15:09:17.000Z

@mgwalker yes, you have mbs POC1 role, so you should be in good shape for testing.

Answer 21 · 2020-10-01T15:14:59.000Z

Copying this comment from the PR:

I've run the tests locally with the following set of session configurations:

As-is

app.use(
  session({
    name: 'session',
    keys: new Keygrip([vcapConstants.PERMIT_SECRET], 'sha256', 'base64'),
    maxAge: 3600000, // 1 hour
    cookie: {
      secure: true,
      httpOnly: true,
      domain
    }
  })
);

Result: 4 tests related to Christmas trees fail

Un-nested

app.use(
  session({
    name: 'session',
    keys: new Keygrip([vcapConstants.PERMIT_SECRET], 'sha256', 'base64'),
    maxAge: 3600000, // 1 hour
    secure: true,
    httpOnly: true,
    domain
  })
);

Result: 13 tests fail, including one that got a 401 status code when it expected a 404. The extra tests failing suggests that either:

the cookies were working fine with nested properties and un-nesting broke it
the cookies were never working properly, and this particular set of options causes them to fail in testing

Un-nested, without `secure` and `domain` configurations

app.use(
  session({
    name: 'session',
    keys: new Keygrip([vcapConstants.PERMIT_SECRET], 'sha256', 'base64'),
    maxAge: 3600000, // 1 hour
    httpOnly: true,
  })
);

Result: Back to the original four failing tests. This suggests that the cookie options should be un-nested and that the secure and domain options are causing the tests to fail. This makes sense: the tests are probably not running over SSL, so cookies wouldn't be set, and domain is being derived from VCAP variables, which are not present in testing.

So I suspect we need to do two things:

Un-nest the cookie properties
Set secure and domain differently if process.NODE_ENV is equal to test
- alternatively, setup SSL and a DNS name in testing, but that seems like a lot more effort

Maybe something like:

app.use(
  session({
    name: 'session',
    keys: new Keygrip([vcapConstants.PERMIT_SECRET], 'sha256', 'base64'),
    maxAge: 3600000, // 1 hour
    secure: process.env.NODE_ENV !== 'test',
    httpOnly: true,
    domain: process.env.NODE_ENV === 'test' ? 'localhost' : domain
  })
);

Answer 22 · 2020-10-01T15:43:48.000Z

@mgwalker makes sense. I'd be willing to move forward with that change to see if it works.

Answer 23 · 2020-10-01T21:49:48.000Z

I'm not seeing Greg's logon attempts in the audit logs in production. All I can see for the last 48 hours are:

This is production accesses

Answer 24 · 2020-10-01T21:54:43.000Z

I just checked staging a not seeing Greg in there either.

Answer 25 · 2020-10-14T20:28:25.000Z

@mgwalker - Hey Greg, I spoke with Mike on this earlier today and he said he thought you were the one working on finishing it up. I was wondering how we were looking. I've gotten several inquiries from the Special-Use admin team and was hoping to give them an update. Thanks for any info!

Answer 26 · 2020-10-14T22:24:29.000Z

I have PR #1471 into staging that is another attempt. Just working out with @abdul-fs the right way to setup that PR so we can move it along!

Answer 27 · 2020-10-15T14:53:41.000Z

Awesome! Thank you for the update.

Answer 28 · 2020-10-19T18:21:57.000Z

Still not complete. @mtlaney and @mgwalker will pair on this issue

Answer 29 · 2020-10-21T13:45:13.000Z

I see this in staging now after the last change made. Samesite cookie issue appears to be resolved as I'm able to login, but I dont see any permits like before.

We need to make sure this change doesn't get push to PROD before resolving the permits being displayed. @abdul-fs @mgwalker @mtlaney

@Dmac26 do you want to do a test permit in staging to see if new ones show up?

Answer 30 · 2020-10-21T13:59:24.000Z

When I try to go to that applications page, I get dumped out at https://open-forest-staging.app.cloud.gov/logged-in. I don't know if that's a permissions thing, or some weird caching issue on my end, but as long as it's just me, I'm not too worried about it. I still see the cookie getting the samesite attribute and that's the main thing I was looking for.

Answer 31 · 2020-10-21T16:22:44.000Z

I was able to submit and view applications as a user. I'm having trouble logging in as an Admin though. I think it's a problem with my PW. I'm working on resetting that now.

Answer 32 · 2020-10-23T14:43:44.000Z

This appears to be fixed by #1500. Woohoo!

Answer 33 · 2020-11-10T19:10:07.000Z

This is completed. Thanks team !!