Multi-threaded generate command for Nuxt.js

Use multiple workers to generate the static files for your Nuxt.js project

Setup

Install the package

yarn add nuxt-generate-cluster

Add a generate script to your package.json

  "scripts": {
    "generate": "nuxt-generate -w 4"
  }

Nuxt config options

Configure the generate options in nuxt.config.js

  generate: {
    workers: 4,
    workerConcurrency: 500,
    concurrency: 500,
    routes (callback, params) {
      return axios.get('https://api.example.com/routes?since=' + params.lastFinished)
      .then((res) => {
        return res.data
      })
    },
    done ({ duration, errors, workerInfo }) {
      if (errors.length) {
        axios.post('https://api.example.com/routes', { generate_errors: errors })
      }
    }
  }

`workers`

Default: number of processors

The number of workers that should be started. It probably has no use to start more workers then number of processors in your system.

`workerConcurrency`

Default: 500

To even the load between workers they are sent batches of routes to generate, otherwise a worker with 'easy' routes might finish long before others. Workers will also still use the concurrency option from Nuxt.

`routes`

The default Nuxt.js routes method has been extended so you can pass additional parameters to it, see params parameter in example config under Setup. By default it will list 3 timestamps:

lastStarted The unix timestamp when the nuxt-generate command was last executed, should be just now
lastBuilt The unix timestamp when the nuxt project was last built by nuxt-generate
lastFinished The unix timestamp when nuxt-generate last finished succesfully (eg not interrupted by ctrl+c)

Timestamps are locally stored in ~/.data-store/nuxt-generate-cluster.json, see data-store

`beforeWorkers`

This method is called on the master just before the workers are started/forked. It receives the Nuxt options as first argument.

Use this method if you experience serialization issues or when your Nuxt config is too big. The Nuxt options are stringified and then passed as environment variable to the workers, on Windows there seems to be a maximum size of 32KB for env variables.

Properties which should be safe to remove are (not a complete list):

buildModules (and their options)
serverMiddleware

`done`

This method will be called when all workers are finished, it receives two arguments:

The first argument is an object with statistics:

duration The total time in seconds that the command ran
errors An array of all the errors that were encountered. Errors can have type handled or unhandled, for the latter the error message will contain the stacktrace

[ { type: 'handled',
    route: '/non-existing-link',
    error:
     { statusCode: 404,
       message: 'The message from your 404 page' } }
]

workerInfo An object with detailed information about each worker. Data passed is from the watchdog object that we use internally to monitor the worker status.

{{ '6707':                            // the process id of the worker
   { start: [ 1929158, 859524606 ],   // process.hrtime the worker was started
     duration: 109567,                // how long the worker was active
     signal: 0,                       // the signal by which the worker was killed (if any)
     code: 0,                         // the exit status of the worker
     routes: 73,                      // the number of routes generated by this worker
     errors: [] },                    // the errors this worker encountered, errors of all workers
                                      // combined is the error argument above
}

The second argument is the Nuxt instance

Command-line options

Please note that you need to explicitly indicate with -b that you want to (re-)build your project

$ ./node_modules/.bin/nuxt-generate -h

  Multi-threaded generate for nuxt using cluster

  Description
    Generate a static web application (server-rendered)
  Usage
    $ nuxt-generate <dir>
  Options
    -b, --build           Whether to (re-)build the nuxt project
    -c, --config-file     Path to Nuxt.js config file (default: nuxt.config.js)
    -h, --help            Displays this message
    -p, --params          Extra parameters which should be passed to routes method
                            (should be a JSON string or queryString)
    -q, --quiet           Decrease verbosity (repeat to decrease more)
    -v, --verbose         Increase verbosity (repeat to increase more)
    --fail-on-page-error  Immediately exit when a page throws an unhandled error
    -w, --workers [NUM]   How many workers should be started
                            (default: # cpus)
    -wc [NUM],            How many routes should be sent to
    --worker-concurrency [NUM]    a worker per iteration

If you need to have more control which routes should be generated, use the -p option to pass additional parameters to your routes method.

# nuxt.config.js
generate: {
  routes (callback, params) {
    console.log(params)
  }
}

$ nuxt-generate -w 2 -p id=1&id=2
// will print =>
{ id: [ '1', '2' ],
  lastStarted: 1508609323,
  lastBuilt: 0,
  lastFinished: 0 }

If you are using a npm script under bash use -- to pass the parameters to nuxt-generate instead of npm:

$ npm run generate -- -p '{ "id": [1,2,3] }'
// will print =>
{ id: [ 1, 2, 3 ],
  lastStarted: 1508786574,
  lastBuilt: 0,
  lastFinished: 0 }

Logging

You can use multiple -v or -q on the command-line to increase or decrease verbosity. We use consola for logging and set a default log level of 3 unless DEBUG is set, then its 5. If you want to log debug messages without setting DEBUG you can use -vv on the command-line

The difference between log levels 2, 3 and 4 are:

Level 2 (-q) Only print master messages like worker started / exited. No worker messages.
Level 3 Also print which routes the workers generated
Level 4 (-v) Also print how much time the route generation took and messages between master and workers

nuxt-community/nuxt-generate-cluster

Multi-threaded generate command for Nuxt.js

Setup

Nuxt config options

workers

workerConcurrency

routes

beforeWorkers