Service Utilization API for tracking resource usage and metrics reporting
Please refer to the design for more information.
The module exposes a client, named UtapiClient. Projects can use this client to push metrics directly to the underlying datastore (Redis) without the need of an extra HTTP request to Utapi.
import { UtapiClient } from 'utapi';
const config = {
redis: {
host: '127.0.0.1',
port: 6379
},
localCache: {
host: '127.0.0.1',
port: 6379
}
}
const c = new UtapiClient(config);
// The second argument to `pushMetric` is a hexadecimal string Request Unique
// Identifier used for logging.
c.pushMetric('createBucket', '3d534b1511e5630e68f0', { bucket: 'demo' });
c.pushMetric('putObject', '3d534b1511e5630e68f0', {
bucket: 'demo',
newByteLength: 1024,
oldByteLength: null,
});
c.pushMetric('putObject', '3d534b1511e5630e68f0', {
bucket: 'demo',
newByteLength: 1024,
oldByteLength: 256,
});
c.pushMetric('multiObjectDelete', '3d534b1511e5630e68f0', {
bucket: 'demo',
byteLength: 1024,
numberOfObjects: 999,
});
If an error occurs during a pushMetric
call and the client is unable to record
metrics in the underlying datastore, metric data is instead stored in a local
Redis cache. Utapi attempts to push these cached metrics (every five minutes, by
default) using a component named UtapiReplay. If the pushMetric
call initiated
by UtapiReplay fails, the metric is reinserted into the local Redis cache. The
particularities of this behavior are configurable. For further information, see
design.
To make a successful request to Utapi you would need
Note: The examples here use AWS CLI but any AWS SDK is capable of these actions.
endpoint-url: This would be https://<host>:<port>
where your Identity(IAM)
Server is running.
-
Create an IAM user
aws iam --endpoint-url <endpoint> create-user --user-name utapiuser
-
Create access key for the user
aws iam --endpoint-url <endpoint> create-access-key --user-name utapiuser
-
Create a managed IAM policy
sample utapi policy
cat - > utapipolicy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Sid": "utapiMetrics", "Action": [ "utapi:ListMetrics" ], "Effect": "Allow", "Resource": "arn:scality:utapi:::buckets/*" } ] } EOF
The user can also be restricted to only some buckets (foo bucket for example) by changing the
Resource
property to"Resource": "arn:scality:utapi:::buckets/foo"
aws iam --endpoint-url <endpoint> create-policy --policy-name utapipolicy \ --policy-document file://utapipolicy.json
A sample output of the above command would look like
{ "Policy": { "PolicyName": "utapipolicy", "CreateDate": "2017-06-01T19:31:18.620Z", "AttachmentCount": 0, "IsAttachable": true, "PolicyId": "ZXR6A36LTYANPAI7NJ5UV", "DefaultVersionId": "v1", "Path": "/", "Arn": "arn:aws:iam::0123456789012:policy/utapipolicy", "UpdateDate": "2017-06-01T19:31:18.620Z" } }
The arn property of the response, which we call
<policy arn>
, will be used in the next step to attach the policy to the user. -
Attach user to the managed policy
aws --endpoint-url <endpoint> iam attach-user-policy --user-name utapiuser --policy-arn <policy arn>
Now the user utapiuser
has access to ListMetrics request in Utapi on all
buckets.
There are two options here.
You can generate V4 signature using AWS SDKs or the node module aws4. See the following urls for reference.
- http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
- http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
- https://github.com/mhart/aws4
You may also view examples making a request with Auth V4 using various languages and AWS SDKs here.
Alternatively, you can use a nifty command line tool available in Scality's S3.
You can git clone S3 repo from here https://github.com/scality/S3.git and follow the instructions in README to install the dependencies.
If you have S3 running inside a docker container you can docker exec into the S3 container as
docker exec -it <container id> bash
and then run the command
node bin/list_bucket_metrics
It will generate the following output listing available options.
Usage: list_bucket_metrics [options]
Options:
-h, --help output usage information
-V, --version output the version number
-a, --access-key <accessKey> Access key id
-k, --secret-key <secretKey> Secret access key
-b, --buckets <buckets> Name of bucket(s)with a comma separator if
more than one
-s, --start <start> Start of time range
-e --end <end> End of time range
-h, --host <host> Host of the server
-p, --port <port> Port of the server
--ssl Enable ssl
-v, --verbose
A typical call to list metrics for a bucket demo
to Utapi in a https enabled
deployment would be
node bin/list_bucket_metrics -a myAccessKey -k mySecretKey -b demo -s
1476231300000 -h 127.0.0.1 -p 8100 --ssl
Both start and end times are time expressed as UNIX epoch timestamps expressed in milliseconds.
Keep in mind, since Utapi metrics are normalized to the nearest 15 min. interval, so start time and end time need to be in specific format as follows.
Start time needs to be normalized to the nearest 15 minute interval with seconds
and milliseconds set to 0. So valid start timestamps would look something like
09:00:00:000
, 09:15:00:000
, 09:30:00:000
and 09:45:00:000
.
For example
Date: Tue Oct 11 2016 17:35:25 GMT-0700 (PDT)
Unix timestamp (milliseconds): 1476232525320
Here's a typical JS method to get start timestamp
function getStartTimestamp(t) {
const time = new Date(t);
const minutes = time.getMinutes();
const timestamp = time.setMinutes((minutes - minutes % 15), 0, 0);
return timestamp;
},
This would format the start time timestamp to 1476231300000
End time needs to be normalized to the nearest 15 minute end interval with
seconds and milliseconds set to 59 and 999 respectively. So valid end timestamps
would look something like 09:14:59:999
, 09:29:59:999
, 09:44:59:999
and
09:59:59:999
.
Here's a typical JS method to get end timestamp
function getEndTimestamp(t) {
const time = new Date(t);
const minutes = time.getMinutes();
const timestamp = time.setMinutes((minutes - minutes % 15) + 15, 0, -1);
return timestamp;
}
This would format the end time timestamp to 1476233099999
Please read our coding and workflow guidelines at scality/Guidelines.
In order to contribute, please follow the Contributing Guidelines.