Network requests slower in Cypress than outside of Cypress

Question

Network requests slower in Cypress than outside of Cypress

Closed this issue 2 years ago · 18 comments

Current behaviour:

I'm trying to test my Nuxt app with Cypress, but find the tests to run very slowly because of how long it takes to download the JS bundles built by Nuxt. In development mode, bundles can get very large - upwards of 10mb. If I visit my site in Chrome, the network request for these bundles is very quick, <100ms. However, in my Cypress tests it takes almost 2 seconds just to download the bundle for each test case.

Desired behaviour:

Network speeds to be more comparable to those outside of Cypress.

Test code to reproduce

Run yarn test:e2e:dev in the example repository I've created.

This is a vanilla initial Nuxt app with nothing added apart from importing the large library zxcvbn in order to add a bit of arbitrary bulk to the bundle to bring it slightly closer to the size of a real app. Then compare the network requests during the test with those when you just visit localhost:3000 separately in Chrome.

For example, comparing the time for default~app.js:

In Cypress test: 1.1mb takes 324ms
In Chrome: 2.8mb takes 75ms

In each case, the vast majority of that time is spent in 'Content Download'. The request is about 10x slower in Cypress than it normally is in Chrome. I realise the request goes through a proxy with Cypress, but is it expected to cause this much extra overhead? (I'm not quite sure why the the file is two different sizes - perhaps to do with the proxy?)

Versions

Cypress: 4.11.0
OS: OS X 10.15.5
Browser: Chrome 84.0.4147.105

Thanks in advance for any insight you can provide!

Answer 1 · 2020-08-18T07:56:59.000Z

I do see the different in timing of loading the XHR request inside and outside of Cypress.

I think you will see some performance increase if you set modifyObstructiveCode to false (if not needed), as there is some processing time done there.

Other than that, I'm not sure of how to increase it any further at this time. Everything does goes through a proxy.

in Cypress

in Chrome

Answer 2 · 2020-08-18T10:15:45.000Z

Setting modifyObstructiveCode to false does help a decent amount - it cuts the network request time in half which adds up to a lot over our entire test suite.

Other than that, I guess the best thing I can do at the moment is to try and keep the bundles being loaded as small as possible on our end.

Thanks for taking the time to look into this.

Answer 3 · 2020-10-29T10:57:36.000Z

We are also experiencing slow network requests with bigger bundles (20..25 MB). The modifyObstructiveCode=false did not help in our case.

I measured with https://test-page-speed.cypress.io/index1000.html. The results are:

setup	duration
Locally, Chrome 86	1363 ms
Cypress 5.5.0, Chrome 86, inside `cy.visit()`	8241 ms
Cypress 5.5.0, Chrome 86	6188 ms
Locally, Firefox 81	1232 ms
Cypress 5.5.0, Firefox 81, inside `cy.visit()`	8194 ms
Cypress 5.5.0, Firefox 81	6087 ms
Cypress 5.5.0, Electron 85, inside `cy.visit()`	7149 ms

System
Windows 10 2004

My colleagues measurements are around 3.3 seconds vs. 7.3 seconds in Cypress.

Answer 4 · 2020-12-02T21:08:46.000Z

I face same issue - even with video: false and modifyObstructiveCode: false, page is loaded in Chrome 86 manually 3-4 times faster than in same Chrome 86 running by Cypress

Answer 5 · 2021-01-11T11:27:16.000Z

Hello @jennifer-shehane I am facing the same issues as well with experimentalSourceRewriting set to true. It was able to do the network request after the login was done much more quickly with WebDriverIO and Selenium WebDriver.

I am not able to unset the experimentalSourceRewriting as the web application as this causes the test to stop running and produce this Type error below:

Answer 6 · 2021-03-04T09:21:19.000Z

@jennifer-shehane Any suggestions on the above?

Answer 7 · 2021-03-11T15:17:01.000Z

I am facing the same issue, what is the status of this current issue?

Answer 8 · 2021-03-12T20:22:48.000Z

It is expected that Cypress slows down page loads - it has to do some work to intercept all requests, including de/recompressing requests and responses, un/re-encrypting HTTPS requests, modifying obstructive code...

You can check https://app.circleci.com/pipelines/github/cypress-io/cypress/17887/workflows/522dbc50-bd74-46d4-a3d0-974100962549/jobs/646752 to get an idea of the existing Cypress proxy performance compared to a regular browser. cc @dintifla - we run those page loads in CI and analyze the timings, check out the link

You can see that while the difference between Chrome and Cypress (HTTP/1.1 only) is small, it can start to take up to 2-3 times as long if there is an HTTPS destination, or if there is another upstream proxy that the request has to traverse.

One thing that can contribute to slowness in Cypress is the lack of HTTP/2 support. If your app is using HTTP/2, Cypress only supports HTTP/1.1, so it will be slower: #3708

@rayhan15243 experimentalSourceRewriting will slow down your tests, since it means Cypress has to parse the AST of your application's JS and HTML and rewrite your code. The feature needs optimization, which is a big part of why it's still "experimental". See #7565, this is a known issue.

Eventually, experimentalSourceRewriting will be the new default implementation of modifyObstructiveCode, but it's an even bigger performance drop.

Answer 9 · 2021-03-30T09:59:43.000Z

@flotwig I have to enable experimentalSourceRewriting as it won't work without it. See Screenshot.

Answer 10 · 2021-03-30T17:52:46.000Z

@rayhan15243 I am only telling you that experimentalSourceRewriting is slower, I am not denying that you need it :)

Answer 11 · 2021-03-31T08:21:05.000Z

@flotwig Ah okay cool. Yes I would not like to use it however it's the only way that I can get the web application to run the tests. Is there another way without using experimentalSourceRewriting to prevent this application code error?

This web application is an off the shelf product.

Answer 12 · 2021-03-31T14:07:58.000Z

Is this issue and #3708 getting any love from the cypress devs?
Especially since http/2 and https is basically everywhere now, it's quite a severe problem that cypress is not handling http/2 at all and https very slowly.
We decided to move from using protractor to cypress mainly because of developer ergonomics, but our tests run extremely slow now, to the point that we might have to drop cypress. Of course, if significant speed ups are on the horizon it's probably better to wait for them, but since #3708 looks to have been basically dead for more than 2 years now, it's not looking good.
Is there any active work going on in this area?

Answer 13 · 2021-04-04T16:55:51.000Z

Thanks for the https://on.cypress.io/configuration#modifyObstructiveCode option. It does speed up my initial page load network Content Download timeline for 30% ~40%

Answer 14 · 2021-11-03T11:12:31.000Z

I have done some testing on later versions of Cypress:

Test page with the following script and an element with id=header:
<script>
(async() => {
let startTime = performance.now();
for (let i = 0; i < 100; i++) {
await fetch('webclient/test.jpg');
}
const nonParallellResult = performance.now() - startTime;
document.getElementById('header').innerText = 'Fetching in parallell...';
startTime = performance.now();
const createRunner = () => new Promise(async (res, _) => {
for (let i = 0; i < 100; i++) {
await fetch('webclient/test.jpg');
}
res();
});
const inParallell = 5;
const runners = [];
for (let i = 0; i < inParallell; i++) {
runners.push(createRunner());
}
await Promise.all(runners);
const parallellResult = performance.now() - startTime;
document.getElementById('header').innerText = Done, fetch 100 in sequence in ${nonParallellResult} ms and 500 in parallell in ${parallellResult} ms;
})();
</script>

The result is the following when loading the test page in various versions of cypress (test.jpg is 415 kB):
8.7.0: Done, fetch 100 in sequence in 1992.800000011921 ms and 500 in parallell in 7448.199999988079 ms
7.6.0: Done, fetch 100 in sequence in 1470.2999999523163 ms and 500 in parallell in 6233.800000011921 ms
7.5.0: Done, fetch 100 in sequence in 1301 ms and 500 in parallell in 4818.800000011921 ms
6.7.0: Done, fetch 100 in sequence in 948.8000000119209 ms and 500 in parallell in 4337.900000035763 ms

If this issue is not solved, I created #18771 to make it possible to bypass cypress processing of network requests. For our imaging application, the performance when fetching many images in a cypress test makes the tests very slow and unstable.

Answer 15 · 2022-02-10T19:55:31.000Z

I have found that is MacOS specific issue with Nodejs I have reported nodejs/node#41699
if you also have this network performance above link, than that might be the case within Cypress as well.
I already made a PR #20062 and hopefully get reviewed soon.

Answer 16 · 2023-04-18T04:10:25.000Z

I am also facing the same issue on my end. I thought I did something wrong but looks like a lot of people are facing the same problem where the tests are terribly slow. I'm using an M1 Max. I have tried on other Windows laptops as well but the results were same.

Answer 17 · 2023-04-18T04:24:42.000Z

This would be one step to tackle this: #25201

Answer 18 · 2023-04-18T13:18:49.000Z

@antick Please open a new issue with a reproducible example of your problem and we will be happy to investigate.

This ticket is around an extremely old version of Cypress. As such I am going to close this issue. If folks have reproductions of this problem in later versions of Cypress, please open a new ticket and we will investigate.