kkwiatkowski/bot-render

Renders a webpage for bot consumption (not production ready)

bot-render

A Docker container which runs headless Chrome and renders web pages on the fly, which can be set up to serve pages to search engines, social networks and link rendering bots.

Contents

• Middleware
• API
  • Render
  • Screenshot
• FAQ
  • Query parameters
  • Auto detecting loading function
  • Rendering budget timeout
  • Explicit rendering event
  • Web components
  • Status codes
• Installing & deploying

Middleware

Once you have the service up and running, you'll need to implement the differential serving layer. This checks the user agent to determine whether prerendering is required.

This is a list of middleware available to use with the bot-render service:

• Firebase functions (Community maintained)

API

Render

/render/<url>

The render endpoint will render your page and serialize your page. Available options:

• wc-inject-shadydom default false - used to correctly render Web Components v1. See Using with web components for more information.

Screenshot

/screenshot/<url>

The screenshot endpoint can be used to verify that your page is rendering correctly. Available options:

• width default 1000 - used to set the viewport width (max 2000)
• height default 1000 - used to set the viewport height (max 2000)

FAQ

Query parameters

When setting query parameters as part of your URL, ensure they are encoded correctly. In JS, this would be encodeURIComponent(myURLWithParams).

Auto detecting loading function

The service detects when a page has loaded by looking at the page load event, ensuring there are no outstanding network requests and that the page has had ample time to render.

Rendering budget timeout

There is a hard limit of 10 seconds for rendering. Ensure you don't hit this budget by ensuring your application is rendered well before the budget expires.

Explicit rendering event

In some cases, the auto loading function may be insufficient, for example if there is content being streamed on the page. To explicitly signal when the page is visually complete, fire an event as follows:

  myElement.dispatchEvent(new Event('render-complete', { bubbles: true, composed: true}));

Web components

Headless Chrome supports web components but shadow DOM is difficult to serialize effectively. As such, shady DOM is required for web components.

If you are using web components v0 (deprecated), you will need to enable Shady DOM to render correctly. In Polymer 1.x, which uses web components v0, Shady DOM is enabled by default. If you are using Shadow DOM, override this by setting the query parameter dom=shady when directing requests to the bot-render service.

If you are using web components v1 and either webcomponents-lite.js or webcomponents-loader.js, set the query parameter wc-inject-shadydom=true when directing requests to the bot-render service. This renderer service will force the necessary polyfills to be loaded and enabled.

Status codes

Status codes from the initial requested URL are preserved. If this is a 200, or 304, you can set the HTTP status returned by the rendering service by adding a meta tag.

<meta name="render:status_code" content="404" />

Installing & deploying

Dependencies

This project requires Node 7+ and Docker (installation instructions). For deployment this project uses the Google Cloud Platform SDK.

Installing

Install node dependencies using:

npm install

Install Chrome:

apt-get install google-chrome

Running locally

With a local instance of Chrome installed, you can start the server locally:

npm start

To test a rendering, send a request:

http://localhost:3000/?url=https://dynamic-meta.appspot.com

Docker

After installing docker, build the docker image:

docker build -t bot-render . --no-cache=true

Running the container

The container enables the cache to run by default, so be sure to disable the cache when running locally.

Building the container:

docker run -it -p 8080:8080 --name bot-render-container bot-render

In the case where your kernel lacks user namespace support or are receiving a ECONNREFUSED error when trying to access the service in the container (as noted in issues 2 and 3), the two recommended methods below should solve this:

1. [Recommended] - Use Jessie Frazelle' seccomp profile and -security-opt flag
2. Utilize the --cap-add SYS_ADMIN flag

[Recommended] Start a container with the built image using Jessie Frazelle' seccomp profile for Chrome:

wget https://raw.githubusercontent.com/jfrazelle/dotfiles/master/etc/docker/seccomp/chrome.json -O ~/chrome.json
docker run -it -p 8080:8080 --security-opt seccomp=$HOME/chrome.json --name bot-render-container bot-render

Start a container with the built image using SYS_ADMIN:

docker run -it -p 8080:8080 --cap-add SYS_ADMIN --name bot-render-container bot-render

Send a request to the server running inside the container:

curl http://localhost:8080/?url=https://dynamic-meta.appspot.com

Stop the container:

docker kill bot-render-container

Clear containers:

docker rm -f $(docker ps -a -q)

Deploying to Google Cloud Platform

gcloud app deploy app.yaml --project <your-project-id>