Fastly JSC8 Realtime Log Analytics using GDN

Live Demo: https://macrometacorp.github.io/demo-fastly-jsc8-realtime-logs

This tutorial shows you how to integrate Fastly real-time log analytics with Macrometa GDN to monitor the status and activity of Stream workers and Query workers. Fastly provides data about HTTP latency, response count, response size, and unique visitor traffic.

Setup

Federation	Email	Passsword
Global Data Network	demo-fastly-realtime-logs@macrometa.io	`xxxxxxxx`
Fastly Account	demo@macrometa.com	--

Overview

Dashboard:

Macrometa GDN setup

Create and publish the following Stream Workers in your federation:

fastly-log-generator
fastly-http-request-worker
fastly-http-request-stats-1m-worker

Refer to the following links to add content for each Stream worker:

Create the following Query workers in your federation:

fastlyGetTopUrl
fastlyGetStatusCodeRatio
fastlyGetStatsByCollection
fastlyGetTopErrorByUrlPath
fastlyGetUniqueVisitorsByCountry

Refer to this link to add content for each Query worker:

Query Workers

Create the following collections in your federation:

fastly_users (global)
fastly_logs (global)
fastly_http_url_stats_1m (global)
fastly_http_response_code_stats_1m (global)
fastly_http_response_latency_stats_1m (global)
fastly_http_error_response_code_stats_1m (global)
fasty_unique_visitor_by_country_stats_1m (global)

Note: If you have run this tutorial before, you might want to truncate the collections.

On the development machine, run the following commands in a console:

1. git clone git@github.com:Macrometacorp/demo-fastly-jsc8-realtime-logs.git
2. cd demo-fastly-jsc8-realtime-logs/react-app
3. npm install
4. npm run start

Fastly Compute@Edge setup

Prerequisite

Create API token by following Fastly API Token doc. The generated token will be used to configure Fastly CLI. Create Token with Global API access Scope
To install and configure Fastly CLI follow Fastly CLI Doc

Create Fastly Compute@Edge service

$ fastly service create --name="Fastly Logs Analytics" --type=wasm
SUCCESS: Created service AAAAAAAAAAA

Update fastly.toml file with service ID generated by the above command

Create Fastly Compute@Edge backend to connect to GDN

$ fastly backend create --version=latest --name="gdn_url" --address="api-gdn.paas.macrometa.io" --port=443
SUCCESS: Created backend gdn_url (service AAAAAAAAAAA version 1)

Setup Fastly Logging with HTTP for the newly created service. In below command, replace HTTPS_URL with url which hosts fastly-logs-capture-service

$ fastly logging https create --name="gdn_logging" \
  --version="latest" \
  --url="HTTPS_URL" \
  --method="POST" \
  --json-format=1 \
  --request-max-entries=100 \
  --request-max-bytes=1000000 \
  --content-type="application/json" \
  --placement=none

Create Fastly Dictionary and Dictionary items to store environment variables

$ fastly dictionary create --version=latest --name="env"
$ fastly dictionary list --version=latest

# Replace `DICTIONARY_ID` with dictionary id from above command.
$ fastly dictionary-item create --dictionary-id="DICTIONARY_ID" --key="backend_name" --value="gdn_url"
$ fastly dictionary-item create --dictionary-id="DICTIONARY_ID" --key="logging_endpoint_name" --value="gdn_logging"
$ fastly dictionary-item create --dictionary-id="DICTIONARY_ID" --key="gdn_api_key" --value="GDN_API_KEY"
$ fastly dictionary-item create --dictionary-id="DICTIONARY_ID" \
  --key="gdn_api_url" \
  --value="https://api-gdn.paas.macrometa.io/_fabric/_system/_api"

How to Run and Publish on Fastly Compute@Edge

git clone git@github.com:Macrometacorp/demo-fastly-jsc8-realtime-logs.git
cd demo-fastly-jsc8-realtime-logs/fastly-compute-service
npm install
npm run dev
npm run deploy

In the above deployment command, it will prompt you for a domain, you can either select Fastly generated domain or type your own. Once deployment is complete you will use that domain to access the service

How to Run in local

git clone git@github.com:Macrometacorp/demo-fastly-jsc8-realtime-logs.git
cd demo-fastly-jsc8-realtime-logs/fastly-compute-service
npm install
# Remove bin folder
fastly compute serve

For more information, refer to Fastly HTTP Logging Guide.

Sample Log Format:

{
  "timestamp": "2021-09-02T14:44:31+0000",
  "request_method": "POST",
  "response_status": 400,
  "url": "/collections/query",
  "response_body_size": 134,
  "time_elapsed": 16,
  "geo_country": "india",
  "client_ip": "172.105.56.148"
}