Create your own New Relic demo environment using AWS or Azure K8s cluster

This tutorial will walk you through the process on how you can fully observe a complex, multi-language microservice based E-Commerce application running in a Kubernetes cluster using https://pixielabs.ai and New Relic
This walkthought will cover the following features of New Relic One Platform:
- Create new Kubernetes cluster using either Azure AKS or AWS EKS
- Deploy a typical Microservice E-Commerce Kubernetes application to your cluster
- Start observing your kubernetes application with Pixie without instrumentation
- Enable Distributed Tracing with New Relic APM
- Monitor Real User Experience using Browser Monitoring
- Pull custom K8s metrics into New Relic Database using Pixie Script and New Relic Flex
- View application Errors in New Relic Errors Inbox
- Jump straight to the line of code in VSCode using New Relic CodeStream Integration
- Add Logs In Context

Prerequisites

Azure or AWS Cloud account
A new New Relic account sign up here
Docker desktop link
Docker Hub account sign up here
Github account sign up here
Kubectl, helm3, nodejs, git, Azure or AWS cli installed
K6 load test tool link
VScode downloa here
Basic Bash knowledge since all commands are Bash commands
Clone this repo to your local machine

Create fully managed Kubernetes cluster in AWS or Azure

Create EKS Cluster in AWS

# install aws cli at https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
# login
aws configure

# install eksctl https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html
# create new Managed nodes cluster (not the Fargate), this will create 2 nodes cluster (m5.larger)
eksctl create cluster --name pixiecluster --region us-east-1

# install kubectl cli tool to your computer https://kubernetes.io/docs/tasks/tools/
# download correct kubeconfig
eksctl utils write-kubeconfig --cluster=pixiecluster

# confirm can connect to the cluster
kubectl get nodes -o wide

Create Azure AKS cluster

# install azure cli at https://docs.microsoft.com/en-us/cli/azure/install-azure-cli
# login
az login

# if you have multiple subscriptions, select the right subscription
az account set -s SUBSCRIPTION_ID

# create resource group
az group create --name pixiedemo --location eastus

# create the K8s Cluster with 2 nodes WITHOUT monitoring. We need 2 in order to see 2 nodes in New Relic K8s Cluster Explorer
az aks create --resource-group pixiedemo --name pixiecluster --node-count 2 --enable-addons http_application_routing --generate-ssh-keys --enable-rbac

# install kubectl
az aks install-cli

# connect to the cluster via kubectl
az aks get-credentials --resource-group pixiedemo --name pixiecluster

# confirm can connect to the cluster
kubectl get nodes -o wide

Deploy simple 2-Tier application (UI + Redis) on K8s Cluster and observe it using Pixie

First, let's deploy a very simple application and monitor it using New Relic and Pixie

# create simple Voting application
kubectl create namespace simplevote

kubectl apply -f apps/simple-vote.yaml --namespace=simplevote

# check external IP for the front end app and wait until the external-ip is on
kubectl get service simple-vote-front --watch --namespace=simplevote

# test to make sure you can access the html, you can also open this URL on browser
CURL http://<EXTERNAL-IP>

# install helm3 at https://helm.sh/docs/intro/install/
# login to newrelic one and follow guided install to add new k8s integration using heml3 command

# install using helm3
kubectl apply -f https://download.newrelic.com/install/kubernetes/pixie/latest/px.dev_viziers.yaml && \
kubectl apply -f https://download.newrelic.com/install/kubernetes/pixie/latest/olm_crd.yaml && \
helm repo add newrelic https://helm-charts.newrelic.com && helm repo update && \
kubectl create namespace newrelic ; helm upgrade --install newrelic-bundle newrelic/nri-bundle \
 --set global.licenseKey=LICENSE_KEY \
 --set global.cluster=pixiedemo \
 --namespace=newrelic \
 --set newrelic-infrastructure.privileged=true \
 --set global.lowDataMode=true \
 --set ksm.enabled=true \
 --set kubeEvents.enabled=true \
 --set prometheus.enabled=true \
 --set logging.enabled=true \
 --set newrelic-pixie.enabled=true \
 --set newrelic-pixie.apiKey=PIXIE_API_KEY \
 --set pixie-chart.enabled=true \
 --set pixie-chart.deployKey=PIXIE_DEPLOY_KEY \
 --set pixie-chart.clusterName=pixiedemo

# install k6 load test from https://k6.io
# run quick load test (replace EXTERNAL-IP with the correct external IP you get above)
k6 run -e PUBLIC_IP=<EXTERNAL-IP> loadtests/azure-vote.js

# go to http://one.newrelic.com, select Kubernetes
# click on Live debugging with Pixie and select "px/http_request_stats"
# you will see traffic going into the cluster

Run Pixie CLI locally

# install pixie-cli
bash install-pixie-no-auth.sh

# login to NewRelic One UI, select Kubernetes, click on the cluster and select Live debugging with Pixie and click "Copy command"
# from the copied command, you will see your Pixie API Key
# run this command
px auth login --api_key='PIXIE_API_KEY'

# show the list of out the box script you can run
px scripts list

# run one of the scripts
# open http://<EXTERNAL-IP> and click on Dogs or Cats then run this command
px run px/redis_data

Setup E-Commerce Microservice application to your K8s Cluster

Next, let's deploy this e-commerce microservice application to the cluster

# create new namespace
kubectl create namespace sock-shop

kubectl apply -f "https://raw.githubusercontent.com/microservices-demo/microservices-demo/master/deploy/kubernetes/complete-demo.yaml" --namespace=sock-shop

# change service type from NodePort to LoadBalancer so that we can access it from outside
kubectl patch svc front-end -p '{"spec": {"type": "LoadBalancer"}}' -n sock-shop

# get the external IP of the front-end service
kubectl get service --watch --namespace=sock-shop

# make sure all containers are running properly
kubectl get pod --namespace=sock-shop

# make sure the frontend is now accessible
curl http://<EXTERNAL-IP>

# open http://<EXTERNAL-IP> and play around with the app (sign up for user, add to cart, checkout, etc..)

# run load test against this new Website
k6 run -e PUBLIC_IP=<EXTERNAL-IP> loadtests/sock-shop.js

# go to New Relic One, select Kubernetes and click on Live debugging with Pixie and select "px/http_request_stats"
# you will see traffic going into the cluster

# run a custom Pixie script to see the MongoDB connections
px run -f get_connections_to_mongodb.py

Add APM to a NodeJs Service and enable distributed tracing

Right now, we don't have Distributed Tracing for the sock-shop application. This is because Pixie can only inspect HTTP traffic but cannot modify them. For Distributed Tracing to work, we will need to inject custom traceid header to every HTTP requests
This is why under Services - APM you wouldn't see any entries belong to the sock-shop
In this step, we will add New Relic APM agent to the front end of the sock-shop

# make sure you have an account with hub.docker.com and have docker desktop installed on your machine and authenticated
# go to https://github.com/microservices-demo/front-end and fork this repo (you don't want to modify this repo directly)
mkdir sock-shop
cd sock-shop

# clone the front-end app
git clone https://github.com/<YOUR_GITHUB_USER>/front-end.git
cd front-end

# modify Dockerfile and change from 'FROM node:10-alpine' to 'FROM node:12-alpine' since our latest newrelic npm packaged does not support Node 10
# build new docker image
docker build . -t <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:v1

# if you're on Apple M1 macbook, then you will need to build amd64 version like this
# docker buildx build --platform linux/amd64 . -t <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:v1
docker push <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:v1

# update the image of the runing front-end deployment
kubectl set image deployment/front-end front-end=<YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:v1 -n sock-shop

# make sure URL still working
curl http://<EXTERNAL-IP>

# to Add Node Js APM Agent:
# from inside front-end folder, install NR nodejs APM agent
cd sock-shop/front-end
npm install newrelic

# build and push new image containing APM Agent
docker build . -t <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:apm
docker push <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:apm

# set required New Relic environment variables

kubectl set env deployment/front-end \
    NEW_RELIC_LICENSE_KEY=YOUR_NR_INGEST_API \
    NEW_RELIC_APP_NAME=sock-shop-frontend \
    NEW_RELIC_NO_CONFIG_FILE=true \
    NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true \
    --namespace=sock-shop

kubectl set image deployment/front-end front-end=YOUR_DOCKER_ACCOUNT/sock-shop-frontend:error5 -n sock-shop

# manually navigate to the app via browser, click through some pages
# go back to NR1, click on APM, you should see new app in the list

Add Browser monitoring

Now that we have APM agent installed for our backend services, let's monitor Real User experience by enabling Browser Integration

# go to New Relic One, select `Add more data` and select Browser and select 'Copy/Paste Javascript code'
# select 'sock-shop-frontend' from the list of apps

# edit /sock-shop/front-end/public/js/front.js file and paste the content of the <script type="text/javascript"> tag in it (NOT including <script type="text/javascript"> itself since this is not Javascript)

# build and push new front-end docker image
cd sock-shop/front-end
docker build . -t <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:rum
docker push <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:rum

# update the image of front-ent app
kubectl set image deployment/front-end front-end=YOUR_DOCKER_ACCOUNT/sock-shop-frontend:rum -n sock-shop

# manually navigate to the app via browser, click through some pages
# go back to NR1, click on Browsers app, you should see new app in the list

Install New Relic Infrastructure Agent + Flex on K8s and push Pixie data to NRDB

Currently, for security reasons, when Pixie is installed into your k8s cluster, New Relic only fetches and stores data that related to an application's performance. Therefore, you can only create Alert and Dashboard on a predefined subset of the data collected by Pixie read more here.
However, with New Relic Infrastructure Agent and Pixie CLI installed in your K8s cluster, you can write custom Pixie script and periodically push Pixie metrics to New Relic Database and then create dashboard and alerts on these metrics

# first, let's build and push new New Relic Infrastructure agent image with Pixie cli installed

# inspect `Dockerfile` file in the root of this repo
docker build . -t <YOUR_DOCKER_ACCOUNT>/newrelic_infrastructure_with_pixie:latest

docker push <YOUR_DOCKER_ACCOUNT>/newrelic_infrastructure_with_pixie:latest

# store NR Ingest and PIXIE API key in k8s secret
# you can get PIXIE API by click on `Copy command` link under `Live debugging with Pixie` tab on New Relic Kubernetes Cluster Explorer page
echo -n 'YOUR_PIXIE_API_KEY' > ./pixieapikey
echo -n 'YOUR_NR_INGEST_API' > ./nringestapi

# assuming your kubectl config file is at default location: $HOME/.kube/config
kubectl create secret generic pixiesecrets --from-file=./pixieapikey --from-file=./nringestapi --from-file=$HOME/.kube/config

# edit nri-flex.yml file and update line 23, updating YOUR_DOCKER_ACCOUNT
# deploy the change
kubectl apply -f nri-flex.yml

# go back to the sock shop app and click around
# After couple minutes, you can query the data in New Relic like this

Introduce some error code and see Errors in New Relic

Right now the app is working perfectly, so there is no error
To introduce some error, let's modify /front-end/api/cart/index.js file and add the following

app.post("/cart/update", function (req, res, next) {
    console.log("Attempting to update cart item: " + JSON.stringify(req.body));

    // throw an error when quantity is greater than 10
    if (parseInt(req.body.quantity) > 10) {
      throw new Error("10 items is too much");
    }

    // rest of the file..

Since our ExpressJS has Error handler middleware, all exceptions are handled and because New Relic only reports unhandled exceptions, we will need to manually report exceptions to New Relic by modifying /front-end/helpers/index.js file

// initialize newrelic
var newrelic = require('newrelic');

helpers.errorHandler = function (err, req, res, next) {
	var ret = {
		message: err.message,
		error: err,
	};

	// send error directly to NewRelic
	newrelic.noticeError(err);
	res.status(err.status || 500).send(ret);
};

// rest of the file...

Build and deploy the new image

cd front-end
docker build . -t <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:error
docker push <YOUR_DOCKER_ACCOUNT>/sock-shop-frontend:error

# use the newly built image
kubectl set image deployment/front-end front-end=YOUR_DOCKER_ACCOUNT/sock-shop-frontend:error -n sock-shop

# go to the app on browser, add an item into the cart and then update the cart
# - set the quantity to 10 => works fine
# - set the quantity to 11 => get 500 server (you can see error in the network tab in browser Dev Tools)
# go to NR, select 'sock-shop-frontend' under APM, you should see some error reported under Events > Errors

Click on error will show stack trace

Add workload and view Errors inbox

To use Errors Inbox, we will need to create a workload first
Go to New Relic One, select Workloads, then create new Workload
Select sock-shop-frontend app from APM
Go to Errors Inbox, select your new workload from the workload dropdown
You will see error in your Error inbox

Install CodeStream to VSCode and view Errors inside the IDE

If you're a developer and you have Core User or Full License, you can view Errors directly in your IDE
Install Vscode CodeStream extension and Sign up for a Code Stream account directly inside VSCode
Select "Connect to New Relic One" and connect to New Relic using User API Key, you can get your User Api key by login to New Relic One, select API Keys from your profile drop down menu
Create tag v0.0.1 and get current commit SHA from the front-end repository

# get the current commit SHA from the front-end repository
cd sock-shop/front-end
git log -1 --format="%H"

# from the root of this repo, apply changes

kubectl set env deployment/front-end \
    NEW_RELIC_METADATA_REPOSITORY_URL=https://github.com/<YOUR_GITHUB_USER>/front-end.git \
    NEW_RELIC_METADATA_COMMIT=<COMMIT_ID> \
    --namespace=sock-shop

kubectl set image deployment/front-end front-end=YOUR_DOCKER_ACCOUNT/sock-shop-frontend:error5 -n sock-shop

# go back to the sock shop and reproduce the error again (update cart to 11 items)
# go back to Errors Inbox, click on the latest error
# click on 'Open in IDE', you should see VScode open at the right location, in read-only mode

Note: even when the code is changed, because CodeStream has the git SHA, Code Stream will still be able to show you the line of code from the stack trace
Let's demonstrate this by removing the lines of code we added earlier, like this

Click on Open In IDE again, you will see that VScode is displaying the previous version

Add Logs in Context for the front-end app

Right now the front-end nodejs app is not using any structure logging, only console.log is used
Let's add winston structured logging to it

# install winston and newrelic winston enricher
npm i --save winston @newrelic/winston-enricher

Add windston logger by modifying the helpers/index.js file

'use strict';

var request = require('request');
var newrelic = require('newrelic');
var helpers = {};

const winston = require('winston');
const newrelicFormatter = require('@newrelic/winston-enricher');
helpers.logger = winston.createLogger({
	level: 'info',
	format: winston.format.combine(
		winston.format.json(),
		// combine with newrelic enricher
		newrelicFormatter()
	),
	defaultMeta: { service: 'front-end' },
	transports: [
		// just push to console this will be picked up by Newrlic logger FluentBit daemon
		new winston.transports.Console(),
	],
});

// remaining of the file

overwrite console.log globally with Winston logger.info in server.js file

const { logger } = helpers;
// overrwrite global console.log object
global.console.log = (...args) => logger.info.call(logger, ...args);

// remaining of the file

Redeploy the app again (follow above instructions) (build new image, push, update deployment etc.)
Reload the browser and go to New Relic One, you will see something like this

The reason for this is because both EKS and AKS changed container runtime from docker to contianerd and since then the log format is now changed and we're unable to process the nested structured logs correctly. When we install newrelic-bundle helm chart in step 1, default fluentBit.criEnabled configuration for newrelic-logging is false, we need to change this to true

# update LOG_PARSER env to cri
kubectl set env DaemonSet/newrelic-bundle-newrelic-logging LOG_PARSER=cri --namespace=newrelic

# force daemonset restart
kubectl rollout restart DaemonSet newrelic-bundle-newrelic-logging -n newrelic

Reload the browser and go to New Relic One, Logs In Context should now be working

Clean up your Resources

# delete the Azure AKS cluster
az aks delete --name pixiecluster --resource-group pixiedemo

# delete the AWS EKS cluster
eksctl delete cluster --name pixiecluster --region us-east-1

curl https://raw.githubusercontent.com/newrelic/helm-charts/master/charts/newrelic-logging/k8s/fluent-conf.yml > fluent-conf.yml curl https://raw.githubusercontent.com/newrelic/helm-charts/master/charts/newrelic-logging/k8s/new-relic-fluent-plugin.yml > new-relic-fluent-plugin.yml curl https://raw.githubusercontent.com/newrelic/helm-charts/master/charts/newrelic-logging/k8s/rbac.yml > rbac.yml

jay-nginx/demotron-anthony

Create your own New Relic demo environment using AWS or Azure K8s cluster

Prerequisites

Create fully managed Kubernetes cluster in AWS or Azure

Create EKS Cluster in AWS

Create Azure AKS cluster

Deploy simple 2-Tier application (UI + Redis) on K8s Cluster and observe it using Pixie

Run Pixie CLI locally

Setup E-Commerce Microservice application to your K8s Cluster

Add APM to a NodeJs Service and enable distributed tracing

Add Browser monitoring

Install New Relic Infrastructure Agent + Flex on K8s and push Pixie data to NRDB

Introduce some error code and see Errors in New Relic

Add workload and view Errors inbox

Install CodeStream to VSCode and view Errors inside the IDE

Add Logs in Context for the front-end app

Clean up your Resources