/lighthouse4u

LH4U provides Google Lighthouse as a service, surfaced by both a friendly UI+API, and backed by various storage clients (S3, ElasticSearch, etc) for all your query and visualization needs

Primary LanguageEJSMIT LicenseMIT

Lighthouse4u

travis-ci

LH4U provides Google Lighthouse as a service, surfaced by both a friendly UI+API, and backed by various storage and queue clients to support a wide range of architectures.

UI

Usage

Start server via CLI:

npm i -g lighthouse4u
lh4u server --config local --config-dir ./app/config --config-base defaults --secure-config ./app/config/secure --secure-file some-private.key

Or locally to this repo via npm start (you'll need to create test/config/local.json5 by copying test/config/COPY.json5).

Architectures

With the release of v1, we've decoupled compute and storage into multiple tiers to permit a diverse set of architectures. Not only are custom storage and queue clients supported, but compute can even be run serverless.

Queue Architecture

This most closely resembles that of v0.7, but without the restrictions of being locked into Elasticsearch and RabbitMQ.

{  
  reader: {
    module: 'lighthouse4u-elasticsearch', options: { /* ... */ }
  },
  writer: {
    module: 'lighthouse4u-elasticsearch', options: { /* ... */ }
  },
  queue: { // rabbit
    module: 'lighthouse4u-amqp', options: { /* ... */ }
  }
}

Note: The lighthouse4u-elasticsearch storage client is not yet available. Happy to take contributions!

Serverless Architecture

Same as above, you're not tied to Elasticsearch. Use whatever storage and queue client fits your needs. Due to the nature of AWS Lambda, your Lighthouse "runner" (triggered via SQS) is separate from the Lighthouse4u UI+API.

Serverless example via AWS Lambda

Configuration Options

Option Type Default Desc
store reader writer
->.module string required Path of client module to import. See storage clients
->.options object Collection of options supplied to storage client
queue Queue configuration
->.module string required Path of client module to import. See queue clients
->.idleDelayMs number 1000 Time (in MS) between queue checks when queue is empty
->.enabled boolean true Consuming queue may be disabled
->.options object Collection of options supplied to queue client
http HTTP(S) Server options
->.bindings Hash<HttpBinding> An object of named bindings
->.bindings.{name} HttpBinding With {name} being the name of the binding -- can be anything, ex: myMagicBinding. See value to false if you want to disable this binding.
->.bindings.{name}.port number 8994 Port to bind to
->.bindings.{name}.ssl TLS Options undefined Required to bind HTTPS/2
->.auth Zero or more authorization options
->.auth.{authName} HttpAuth A set of auth options keyed by any custom {authName}
->.auth.{authName}.type `basic custom` required
->.auth.{authName}.groups `string array` *
->.auth.{authName}.customPath string Path of the custom module
->.auth.{authName}.options Options provided to the auth module
->.authRedirect undefined URL to redirect UI to if auth is enabled and fails
->.routes Optional set of custom routes
->.routes.{name} `string HttpRoute`
->.routes.{name}.path string Path to resolve to connect middleware, relative to current working directory
->.routes.{name}.method string GET Method to bind to
->.routes.{name}.route string Uses {name} if undefined Route to map, ala /api/test
->.staticFiles Optional to bind routes to static assets
->.staticFiles.{route} Express Static Options to supply to static middleware
lighthouse Options related to Lighthouse usage
->.config LighthouseOptions Google Lighthouse configuration options
->.config.extends string lighthouse:default What options to default to
->.config.logLevel string warn Level of logging
->.config.chromeFlags array<string> [ '--headless', '--disable-gpu', '--no-sandbox' ] Array of CLI arguments
->.config.settings LighthouseSettings See Defaults Settings applied to Lighthouse
->.validate hash<string> Zero or more validators
->.validate.{groupName} string Path of validation module used to determine if the response is coming from the intended server. Useful in cases where you only want to measure results coming from an intended infrastructure
->.concurrency number 1 Number of concurrently processed tasks permitted
->.report bool true Save the full report, viewable in LH viewer
->.samples After N lighthouse samples are taken, only the best result is recorded
->.samples.default number 3 Default number of lighthouse tests performed
->.samples.range tuple<min,max> [1, 5] Minimum and maximum samples taken before returning result
->.attempts Number of attempts at running Google Lighthouse before giving up due to failures
->.attempts.default number 2 Default number of attempts before giving up
->.attempts.range tuple<min,max> [1, 10] Minimum and maximum attempts before giving up
->.attempts.delayMsPerExponent number 1000 Exponential backoff after failure
->.delay Time (in ms) before a test is executed
->.delay.default number 0 Default time to wait before test can be run
->.delay.range tuple<min,max> [0, 1000 * 60 * 60] Minimum and maximum time before test can be run
->.delay.delayMsPerExponent number 1000 * 30 Maximum time before delayed messages will be requeued

API

API - GET /api/website

Fetch zero or more website results matching the criteria.

Query String Options

Option Type Default Desc
format string json Format of results, be it json, svg, reportHtml, or reportJson
scale number 1 Scale of svg
q string optional Query by URL or document ID
top number 1 Maximum records to return. Only applicable for json format

API - GET /api/website/compare

Similar to the single website GET, but instead compares results from q1 with q2.

Query String Options

Option Type Default Desc
format string json Format of results, be it json or svg
scale number 1 Scale of svg
q1 string required First query by URL or document ID
q2 string required Second query by URL or document ID

API - POST /api/website

Submit to process website with Google Lighthouse.

JSON Body Options

Option Type Default Desc
url string required URL to process via Google Lighthouse
wait number optional Block returning until job is complete, or the wait time (in ms) has elapsed, resulting in a partial 206 response
headers hash<string> optional Collection of HTTP headers to supply in request
secureHeaders hash optional Same use as headers, but stored securely in queue, and never persisted to ElasticSearch
samples number (See options.lighthouse.samples) Number of samples to take before recording result
attempts number (See options.lighthouse.attempts) Number of failure attempts before giving up
hostOverride string optional Map host of request to an explicit IP. Not yet supported by Chrome in Headless mode
delay number (See options.lighthouse.delay) Delay (in milliseconds) before test will be performed. Useful if the intended service or domain is not expected to be available for some time
group string unknown Group to categorize result to. Useful when searching/filtering on different groups of results
report bool true If the full report should be stored, allowing for viewing in LH Viewer
auditMode `false 'simple' 'details'
cookies `array<{name=string Cookie}>` optional
cookies[idx].value string required Value to set for cookie
cookies[idx].domain string optional Domain to apply cookie to. Defaults to root of requestedUrl domain
cookies[idx].url string optional If the page required to apply cookie to differs from the requestedUrl you can set that here
cookies[idx].path string optional Path to apply to cookie
cookies[idx].secure boolean optional Is cookie secured
cookies[idx].httpOnly boolean optional HTTP only
commands array<Commmand> optional DevTool Commands to be executed prior to running Lighthouse
commands[idx].command string required DevTool Command to execute
commands[idx].options object required DevTool Command Options to supply command
meta object optional Object containing metadata that will be attached to final lighthouse results

Command

Storage Clients

  • AWS S3 - Store and basic query support via S3 buckets.
  • File System - Not recommended for production usage. Good for local testing.
  • Elasticsearch - Store and query support via Elasticsearch.

Queue Clients

  • AWS SQS - Support for Simple Queue Service.
  • AMQP - Support for RabbitMQ and other AMQP compatible queues.

Secure Configuration

Optionally you may run LH4U with the --secure-config {secureConfigPath} and --secure-file {securePrivateFile} powered by Config Shield.

npm i -g config-shield
cshield ./app/config/secure some-private.key
> help
> set queue { options: { url: 'amqp://lh4u_user:someSuperSecretPassword@rmq.on-my-domain.com:5672/lh4u' } }
> get queue
: { "options": { "url": "amqp://lh4u_user:someSuperSecretPassword@rmq.on-my-domain.com:5672/lh4u" } }
> save
> exit

The example above will allow you to merge sensitive credentials with your existing public configuration. Only options defined in the secure config will be deeply merged.