Page.setContent should wait for resources to be loaded

Question

Page.setContent should wait for resources to be loaded

aslushnikov opened this issue 7 years ago · 39 comments

aslushnikov commented 7 years ago

(as mentioned in #486 and other places)

We need a way to wait for page to load all the resources after the page.setContent.

The lifecycle events might help help.

GautierT commented 7 years ago

Nop. :/

Answer 1 · 2017-10-04T21:56:57.000Z

Meanwhile. a good workaround for page.setContent that waits for all the resources to load:

await page.goto(`data:text/html,${html}`, { waitUntil: 'networkidle0' });

Answer 2 · 2017-10-10T17:19:31.000Z

It's amazing that you posted the workaround. I'm running exactly into this issue, attempting to use puppeteer as a PDF generating service from HTML.

Thank you for filing a formal issue.

Answer 3 · 2017-11-22T22:08:53.000Z

I'm having trouble with this too on version 0.13-alpha with browser configs as:

"waitUntil": "networkidle2",
"timeout": 60000

Answer 4 · 2017-12-07T11:29:10.000Z

I confirm the same trouble.

Answer 5 · 2017-12-13T15:52:16.000Z

with the latest release this hangs too:

await page.goto(`data:text/html,${html}`, { waitUntil: 'networkidle' });

wait for networkidle0 instead

await page.goto(`data:text/html,${html}`, { waitUntil: 'networkidle0' });

Answer 6 · 2017-12-15T13:27:14.000Z

Hey @aslushnikov

You said in #1312 to wait for https://chromium-review.googlesource.com/c/chromium/src/+/747805
The patch is merged...

Everything is OK to solve this issue?

Cheers!

Answer 7 · 2018-01-10T06:19:18.000Z

@HanXHX this requires more work upstream: in order to reuse lifecycle events, page.setContent should initiate a navigation, which in turn should be plumbed through browser-side navigation aka "plznavigate".

Answer 8 · 2018-01-12T16:37:41.000Z

The workaround doesn't seem to work for me. Using 1.0.

Answer 9 · 2018-01-12T17:14:20.000Z

For reference,

      const loaded = page.waitForNavigation({
        waitUntil: 'load',
      });

      await page.setContent(html);
      await loaded;

seems to work for me.

Answer 10 · 2018-01-12T18:17:52.000Z

My use case is that background-color css isn't loading. I'll give the above a try as well.

Answer 11 · 2018-01-15T21:37:05.000Z

@LeonineKing1199 I got navigation exceeded timeout using the above method.

Answer 12 · 2018-01-15T22:17:47.000Z

Darn, it worked for me. Sorry it didn't for you :P

Did you copy-paste it verbatim in that example? You need to register the event callback before you invoke setContent

Answer 13 · 2018-01-16T15:06:46.000Z

Yeah verbatim

Answer 14 · 2018-01-18T08:57:55.000Z

@aslushnikov : the workaround does'n't work correctly when encoding is not set on the <head> of the HTML.
With setContent the accent are correctly set.
Repro :

 const html = `<html><head></head><body>A word with àccênt</body></html>`

 await page.goto(`data:text/html,${html}`, { waitUntil: 'networkidle0' });
 const pdf_buffer = await page.pdf(options)
 await page.close()

Any idea when the setContent with waitForNavigation will work again ?

Thanks.

Answer 15 · 2018-01-18T12:51:22.000Z

I´m also having some problems.

page.goto(data:text/html,${html}, { waitUntil: 'networkidle0' });

This does not load images from remote urls. Any suggestions ?

Answer 16 · 2018-01-19T06:47:48.000Z

Any idea when the setContent with waitForNavigation will work again ?

@GautierT the feature is not in our plans for this quarter so there are no estimates. We'll see if we have some time to have this covered.

Answer 17 · 2018-01-19T08:26:28.000Z

Thanks. It's a feature? Not a bug? It was working on some version of 0.13.0.

Answer 18 · 2018-01-19T08:41:40.000Z

@GautierT an experimental implementation was landed once; it got reverted later.

Answer 19 · 2018-01-19T08:54:20.000Z

@aslushnikov : ok thanks for the explanation.

Any idea how can i convert HTML string to PDF without having trouble with encoding ?

With await page.goto(data:text/html,${html}, { waitUntil: 'networkidle0' });the accent are badly handled if <meta charset="UTF-8"> is not set in the header.

But i can't put it myself (users send me HTML string to be converted to PDF).

Thanks for your help.

Answer 20 · 2018-01-19T18:18:28.000Z

@GautierT there's another trick you can try that involves request interception.

const page = await browser.newPage();
await page.setRequestInterception(true);
// Capture first request only
page.once('request', request => {
  // Fulfill request with HTML, and continue all subsequent requests
  request.respond({body: myHTML});
  page.on('request', request => request.continue());
});
await page.goto('http://example.com');

Answer 21 · 2018-02-16T23:37:31.000Z

@GautierT Did you find a way to fix the encoding issue without having to put the meta in head? Facing the same issue

Answer 22 · 2018-02-23T11:00:48.000Z

You can specify charset in the data url e.g. data:text/html;charset=UTF-8,<h1>👍</h1>.

This seems to have fixed the issue with emoji for me.

Answer 23 · 2018-06-14T17:13:47.000Z

Hey guys, I've had the same issue with setContent. Using page.goto worked, but when you have a huge html it just don't stop rendering. One possible workaround and that's what i'm using now: Save the into a file and use the file protocol on goto.

something like:

await page.goto("file://" + tempFilePath)

Answer 24 · 2018-07-31T14:47:54.000Z

The following code from above posts

await page.goto(`data:text/html,${html}`, { waitUntil: 'networkidle0' });

does not work for me anymore. Tested with Puppeteer v1.6.1.

Any workaround or news regarding this functionality?

Answer 25 · 2018-08-22T22:07:37.000Z

@jmanuelbr try with this

await page.goto("data:text/html," + html, {
      waitUntil: "networkidle2"
    });

Answer 26 · 2018-09-07T23:17:20.000Z

#728 (comment) doesn't appear to work anymore for version v1.8.0.

I tried:

await page.goto(`data:text/html,${html}`, { waitUntil: 'networkidle0' });

and

await page.goto(`data:text/html,${html}`, { waitUntil: 'networkidle2' });

So I just reverted back to v1.1.1 since that's what worked for me prior to trying v1.8.0. Didn't get a chance to test version between v1.1.1 and v1.8.0.

Answer 27 · 2018-09-07T23:53:44.000Z

#728 (comment) doesn't appear to work anymore for version v1.8.0.

@jeremypeter what do you mean by "doesn't work"? I just tried, works fine for me.

Answer 28 · 2018-09-23T11:04:36.000Z

Hi @aslushnikov

Regarding this #728 issue, i've been using the initial workaround you've proposed for a while :

await page.goto('data:text/html,${html}', { waitUntil: 'networkidle0' });

But now i'm encountering users cases producing HTML to big for that data URI trick (2Mo limitation)

So far, I've used the workaround mentioned by @filipemonteiroth the 14th of June, creating a temporary file to be serve thru page.goto, because my process need to wait for both network stabilisation with networkidle0 and for a browser side JS script completion custom event.

But this workaround necessitate to manage the lifetime of this new file ...

As I've spotted your other trick that involves request interception today, I'm wondering if this would enable me to pass HTML bigger than 2Mo to chromium while having both features of networkIdle0 and custom events working, all of this without having to manage lifecycle of a newly created temporary file ?

Is there any limitations related to this trick I should be aware of before using it ?

Answer 29 · 2018-09-23T19:48:58.000Z

As I've spotted your other trick that involves request interception today, I'm wondering if this would enable me to pass HTML bigger than 2Mo to chromium while having both features of networkIdle0 and custom events working, all of this without having to manage lifecycle of a newly created temporary file ?
Is there any limitations related to this trick I should be aware of before using it ?

@Mumeii I'm not aware about limitations; it should just work.

Answer 30 · 2018-10-08T11:40:18.000Z

As I've spotted your other trick that involves request interception today, I'm wondering if this would enable me to pass HTML bigger than 2Mo to chromium while having both features of networkIdle0 and custom events working, all of this without having to manage lifecycle of a newly created temporary file ?

I'm running the same issue that handling another newly created file. I think this is not the best practice since you might run into the race condition when concurrent requests come at once, the hard disk keeps receiving many write requests.

I'm wondering if there is any workaround such as streaming files instead of saving file and use "file//" protocol

Answer 31 · 2018-10-10T11:54:55.000Z

Hi Team!

Please advise at the WO is not working in our case. (puppeteer 1.9.0)
I am trying to convert the XHTML content. I am providing XHTML content as an excaped inlined string.
The generated document contains raw XHTML and there are no external resources requests (in this case CSS).

Example

'use strict'

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch()
    const page = await browser.newPage()
    await page.setRequestInterception(true)

    const xhtml = `&lt;!DOCTYPE html PUBLIC &quot;-//W3C//DTD XHTML 1.0 Transitional//EN&quot; &quot;http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd&quot;&gt; &lt;html xmlns=&quot;http://www.w3.org/1999/xhtml&quot;&gt; &lt;head&gt; &lt;meta charset=utf-8&quot;/&gt; &lt;meta http-equiv=&quot;Content-Type&quot; content=&quot;application/xhtml+xml; charset=utf-8&quot;/&gt; &lt;link href=&quot;css.css&quot; media=&quot;all&quot; rel=&quot;stylesheet&quot; type=&quot;text/css&quot;/&gt; &lt;/head&gt; &lt;body&gt; some content &lt;/body&gt; &lt;/html&gt;`

    console.log(xhtml)

    page.on('request', request => {
        console.log(`Intercepted request with URL: ${request.url()}`)
        request.continue()
    });

    await page.goto(`data:text/html,${xhtml}`, {
        waitUntil: 'networkidle0'
    });
    await page.pdf({
        path: 'xhtml.pdf'
    })
    await browser.close()
})()

Here is the initial document content

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
   <head>
      <meta charset=utf-8"/>
      <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8"/>
      <link href="css.css" media="all" rel="stylesheet" type="text/css"/>
   </head>
   <body>
      some content 
   </body>
</html>

Answer 32 · 2018-11-16T00:24:57.000Z

Is there a way to set the url if we end up using the data:text/html workaround? Do relative paths for resources work via this method?

Answer 33 · 2018-11-20T13:52:16.000Z

I had the exact same problem with external resources. So the workaround from @aslushnikov helped me a lot. But as @ObviouslyGreen points out it lacks the support of resolving relative paths. I investigated what puppeteer takes as "url" when using this workaround and it is the whole html (obviously).

I could solve the problem with relative paths (for me in CSS styles) with the following approach:

create a folder (let's name it dist) in which all relative resources are placed in
generate the html as needed (paths should be relative to the root of dist)
write the html file to the root of dist
use the following code to load the html with all the relative resources resolved correctly:

const pathToHtml = path.join(__dirname, 'dist', `${randomName}.html`);

const page = await browser.newPage();
await page.goto(`file:${pathToHtml}`, { waitUntil: 'networkidle0' });

Note that the html file needs to have the '.html' suffix for puppeteer to render the html properly (at least for me this was the case).

Answer 34 · 2018-11-21T07:56:31.000Z

@aslushnikov Great to see the same options for page.setContent!

Is it possible now with page.setContentto load resources with relative paths as I described in my workaround in the above comment?

Answer 35 · 2018-11-21T18:34:45.000Z

@kamekazemaster yeah, the paths should be resolved against the page's URL.

await page.goto('https://example.com');
// logo.png becomes https://example.com/logo.png
await page.setContent('<img src="/logo.png"></img>');

Answer 36 · 2018-11-22T14:17:57.000Z

@aslushnikov when can we expect next release with updated setContent?

Answer 37 · 2019-08-26T14:32:48.000Z

@kamekazemaster yeah, the paths should be resolved against the page's URL.
await page.goto('https://example.com');
// logo.png becomes https://example.com/logo.png
await page.setContent('<img src="/logo.png"></img>');

This solution is really helpful, though if described with little bit details might have saved some time.

Thank you.

Answer 38 · 2024-05-16T23:18:01.000Z

@kamekazemaster yeah, the paths should be resolved against the page's URL.
await page.goto('https://example.com');
// logo.png becomes https://example.com/logo.png
await page.setContent('<img src="/logo.png"></img>');

Can anyone confirm if adding a <base href=...> tag to the html works instead of the initial page.goto above ?