CloudWatch

[paalkr]

Hi

Thanks for this useful tool, that seems to work well. I have tested reading from Prometheus and writing to CloudWatch, and the basic functionality work.

I have two small issues though, that I would like to report.

1. Decimals are rounded to the nearest value. I use this query to get the memory pressure of my Kubernetes cluster, and would like to get the result into CloudWatch with one or two decimals. An configurable rounding options would be nice

    // queries section
    "queries":
    {
		"k8sprod2.WorkerMemoryUtilization":
		{
			// the prometheus instant-query
			"query": "sum(container_memory_working_set_bytes{id='/',kube_aws_coreos_com_role='worker'}) / sum(machine_memory_bytes{kube_aws_coreos_com_role='worker'}) * 100",

			// time=<rfc3339 | unix_timestamp>: evaluation timestamp, optional.
			"time": ""
		}
    },

2. There is no dimension assigned to the metric

[shamil]

Hi @paalkr
Thanks for reporting.

Regarding decimals. Graphout currently has a fixed size of four decimals (https://github.com/shamil/graphout/blob/master/lib/output.js#L83). So this should be already working. If it doesn't, please set debug log level and send logs

Regarding dimensions, I can add those as part of the output params (https://github.com/shamil/graphout-output-cloudwatch#configuration-params), so you will able to specify them there, will this work for you?

[paalkr]

@shamil , thanks for the quick response.

I will turn on debug logs and report back about rounding.

Regarding the dimensions, adding them to an output configuration would work for me. It should be possible to add a list, like

{
    "output": "graphout-output-cloudwatch",
    "params":
    {
        "accessKeyId": "",
        "secretAccessKey": "",
        "region": "eu-west-1",
        "namespace": "Kubernetes",
	"dimensions":[ "foo", "bar", "baz" ],
        "dot_notation": false
    }
}

Thanks!

[paalkr]

Result with debug log enabled

2018-06-06 20:49:40 severity="info" interval="60000"
2018-06-06 20:49:40 severity="info" log_level="debug"
2018-06-06 20:49:40 severity="info" query_engine="prometheus"
2018-06-06 20:49:40 severity="info" splay="false"
2018-06-06 20:49:40 message="loading query engine" severity="info" engine="prometheus"
2018-06-06 20:49:40 message="loading output" severity="info" output="logfile" module="./logoutput"
2018-06-06 20:49:40 message="outputing to file" severity="info" output="logfile" module="./logoutput" file="/tmp/logoutput.log"
2018-06-06 20:49:40 message="loading output" severity="info" output="CloudWatch" module="graphout-output-cloudwatch"
2018-06-06 20:49:40 message="executing query" severity="debug" query="k8sprod2.WorkerMemoryUtilization" request="{'_pd':null,'protocol':'http:','hostname':'prometheus','port':'9090','path':'/api/v1/query?_=123456&query=sum(container_memory_working_set_bytes%7Bid=%27/%27,kube_aws_coreos_com_role=%27worker%27%7D)%20/%20sum(machine_memory_bytes%7Bkube_aws_coreos_com_role=%27worker%27%7D)%20*%20100&time=','method':'GET','headers':{'Accept':'application/json, text/javascript'}}"
2018-06-06 20:49:40 message="got values" severity="debug" query="k8sprod2.WorkerMemoryUtilization" values="[71]" count="1"
2018-06-06 20:49:40 message="putting metric" severity="debug" output="CloudWatch" module="graphout-output-cloudwatch" namespace="Kubernetes/k8sprod2" metric="WorkerMemoryUtilization" value="71"
2018-06-06 20:49:40 message="query completed, sleeping for 60000 ms" severity="debug" query="k8sprod2.WorkerMemoryUtilization"
2018-06-06 20:49:40 message="response" severity="debug" output="CloudWatch" module="graphout-output-cloudwatch" ResponseMetadata="{'RequestId':'22124489-69cb-11e8-9331-731e5368e8a0'}"
2018-06-06 20:50:40 message="executing query" severity="debug" query="k8sprod2.WorkerMemoryUtilization" request="{'_pd':null,'protocol':'http:','hostname':'prometheus','port':'9090','path':'/api/v1/query?_=123456&query=sum(container_memory_working_set_bytes%7Bid=%27/%27,kube_aws_coreos_com_role=%27worker%27%7D)%20/%20sum(machine_memory_bytes%7Bkube_aws_coreos_com_role=%27worker%27%7D)%20*%20100&time=','method':'GET','headers':{'Accept':'application/json, text/javascript'}}"
2018-06-06 20:50:40 message="got values" severity="debug" query="k8sprod2.WorkerMemoryUtilization" values="[71]" count="1"
2018-06-06 20:50:40 message="putting metric" severity="debug" output="CloudWatch" module="graphout-output-cloudwatch" namespace="Kubernetes/k8sprod2" metric="WorkerMemoryUtilization" value="71"
2018-06-06 20:50:40 message="query completed, sleeping for 60000 ms" severity="debug" query="k8sprod2.WorkerMemoryUtilization"
2018-06-06 20:50:40 message="response" severity="debug" output="CloudWatch" module="graphout-output-cloudwatch" ResponseMetadata="{'RequestId':'45d785aa-69cb-11e8-a9e1-75f4a10a311f'}"

and this is the resulting json from Prometheus if I query the server using rest

{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1528318538.249,"71.81218291842112"]}]}}

[shamil]

Thanks a lot @paalkr
I think I found the problem. Very easy fix. I will release new version along with support for cloud-watch dimensions in about 2 days.

[paalkr]

That is just great @shamil

Thanks for looking into this, I appreciate your help!

[shamil]

@paalkr please try new version 0.5.12 and let me know how it goes.
for adding dimensions please look here: https://github.com/shamil/graphout-output-cloudwatch/blob/master/README.md#configuration-params

[paalkr]

Thanks, I will test and report back

[paalkr]

Excellent, both the rounding issue and adding dimensions works as expected!

Thanks a lot for your help and effort.