Cloud Governance

What is it?

Cloud Governance tool provides a lightweight and flexible framework for deploying cloud management policies focusing on cost optimize and security.

This tool support the following policies: policy

Real time Openshift Cluster cost, User cost
ec2_idle: idle ec2 in last 2 days, cpu < 2% & network < 5mb ec2_idle
ec2_run: running ec2 ec2_run
ebs_unattached: volumes that did not connect to instance, volume in available status ebs_unattached
ebs_in_use: in use volumes ebs_in_use
tag_resources: Update cluster and non cluster resource tags fetching from the user tags or from the mandatory tags
zombie_cluster_resource: Delete cluster's zombie resources
tag_non_cluster: tag ec2 resources (instance, volume, ami, snapshot) by instance name
tag_iam_user: update the user tags from the csv file
cost_explorer: Get data from cost explorer and upload to ElasticSearch
gitleaks: scan Github repository git leak (security scan)

** You can write your own policy using Cloud-Custodian and run it (see 'custom cloud custodian policy' in Policy workflows).

First release: Support AWS only

Reference:

The cloud-governance package is placed in PyPi
The cloud-governance container image is placed in Quay.io
The cloud-governance readthedocs link is ReadTheDocs

Table of Contents

Installation
Configuration
Run Policy Using Podman
Run Policy Using Pod
Pytest
Post Installation

Installation

Download cloud-governance image from quay.io

# Need to run it with root privileges
sudo podman pull quay.io/ebattat/cloud-governance

Environment variables description:

(mandatory)AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID

(mandatory)AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY

Policy name:

(mandatory)policy=ec2_idle / ec2_run / ebs_unattached / ebs_in_use / tag_cluster_resource / zombie_cluster_resource / tag_ec2_resource

Policy logs output

(mandatory)policy_output=s3://redhat-cloud-governance/logs

Cluster or instance name:

(mandatory policy:tag_cluster_resource)resource_name=ocs-test

Cluster or instance tags:

(mandatory policy:tag_cluster_resource)mandatory_tags="{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}"

gitleaks

(mandatory policy: gitleaks)git_access_token=$git_access_token (mandatory policy: gitleaks)git_repo=https://github.com/redhat-performance/cloud-governance (optional policy: gitleaks)several_repos=yes/no (default = no)

Choose a specific region or all for all the regions, default : us-east-2

(optional)AWS_DEFAULT_REGION=us-east-2/all (default = us-east-2)

Choose dry run or not, default yes

(optional)dry_run=yes/no (default = yes)

Choose log level, default INFO

(optional)log_level=INFO (default = INFO)

Configuration

Create a user and a bucket

Create user with IAM iam
Create a logs bucket create_bucket.sh

Run Policy Using Podman

# policy=ec2_idle
sudo podman run --rm --name cloud-governance -e policy=ec2_idle -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e dry_run=yes -e policy_output=s3://bucket/logs -e log_level=INFO quay.io/ebattat/cloud-governance

# policy=ec2_run
sudo podman run --rm --name cloud-governance -e policy=ec2_run -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e dry_run=yes -e policy_output=s3://bucket/logs -e log_level=INFO quay.io/ebattat/cloud-governance

# select policy ['ec2_stop', 'empty_buckets', 'empty_roles', 'zombie_elastic_ips', 'zombie_nat_gateways', 'zombie_snapshots']
sudo podman run --rm --name cloud-governance -e policy=policy -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e dry_run=yes  -e log_level=INFO quay.io/ebattat/cloud-governance

# policy=ebs_unattached
sudo podman run --rm --name cloud-governance -e policy=ebs_unattached -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e dry_run=yes -e policy_output=s3://bucket/logs -e log_level=INFO quay.io/ebattat/cloud-governance

# policy=ebs_in_use
sudo podman run --rm --name cloud-governance -e policy=ebs_in_use -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e dry_run=yes -e policy_output=s3://bucket/logs -e log_level=INFO quay.io/ebattat/cloud-governance

# policy=zombie_cluster_resource
sudo podman run --rm --name cloud-governance -e policy=zombie_cluster_resource -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e dry_run=yes -e resource=zombie_cluster_elastic_ip -e cluster_tag=kubernetes.io/cluster/test-pd9qq -e log_level=INFO quay.io/ebattat/cloud-governance

# policy=tag_resources
sudo podman run --rm --name cloud-governance -e policy=tag_resources -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e tag_operation=read/update/delete -e mandatory_tags="{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}" -e log_level=INFO -v /etc/localtime:/etc/localtime quay.io/ebattat/cloud-governance

# policy=tag_non_cluster
sudo podman run --rm --name cloud-governance -e policy=tag_non_cluster -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e tag_operation=read/update/delete -e mandatory_tags="{'Owner': 'Name','Email': 'name@redhat.com','Purpose': 'test'}" -e log_level=INFO -v /etc/localtime:/etc/localtime quay.io/ebattat/cloud-governance

# policy=tag_iam_user
sudo podman run --rm --name cloud-governance -e policy=tag_iam_user -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e user_tag_operation=read/update/delete -e remove_tags="['Environment', 'Test']" -e username=test_username -e file_name=tag_user.csv  -e log_level=INFO -v /home/user/tag_user.csv:/tmp/tag_user.csv --privileged quay.io/ebattat/cloud-governance

# policy=cost_explorer
sudo podman run --rm --name cloud-governance -e policy=cost_explorer -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e es_host=elasticsearch_host -e es_port=elasticsearch_port -e es_index=elasticsearch_index -e cost_metric=UnblendedCost -e start_date=start_date -e end_date=end_date -e granularity=DAILY -e cost_explorer_tags="['User', 'Budget', 'Project', 'Manager', 'Owner', 'LaunchTime', 'Name', 'Email']" -e log_level=INFO quay.io/ebattat/cloud-governance:latest
sudo podman run --rm --name cloud-governance -e policy=cost_explorer -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e es_index=elasticsearch_index -e cost_metric=UnblendedCost -e start_date=start_date -e end_date=end_date -e granularity=DAILY -e cost_explorer_tags="['User', 'Budget', 'Project', 'Manager', 'Owner', 'LaunchTime', 'Name', 'Email']" -e file_name=cost_explorer.txt -v /home/cost_explorer.txt:/tmp/cost_explorer.txt -e log_level=INFO quay.io/ebattat/cloud-governance:latest

# policy=validate_iam_user_tags
sudo podman run --rm --name cloud-governance  -e policy=validate_iam_user_tags -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e validate_type=spaces/tags -e user_tags="['Budget', 'User', 'Owner', 'Manager', 'Environment', 'Project']"   -e log_level=INFO quay.io/ebattat/cloud-governance:latest

# policy=gitleaks
sudo podman run --rm --name cloud-governance -e policy=gitleaks -e git_access_token=$git_access_token -e git_repo=https://github.com/redhat-performance/cloud-governance -e several_repos=no -e log_level=INFO quay.io/ebattat/cloud-governance

# custom cloud custodian policy (path for custom policy: -v /home/user/custodian_policy:/custodian_policy)
sudo podman run --rm --name cloud-governance -e policy=/custodian_policy/policy.yml -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION=us-east-2 -e dry_run=yes -e policy_output=s3://bucket/logs -e log_level=INFO -v /home/user/custodian_policy:/custodian_policy --privileged quay.io/ebattat/cloud-governance

Run Policy Using Pod

Run as a pod job via OpenShift

Job Pod: cloud-governance.yaml

Configmaps: cloud_governance_configmap.yaml

Quay.io Secret: quayio_secret.sh

AWS Secret: cloud_governance_secret.yaml

* Need to convert secret key to base64 [run_base64.py](pod_yaml/run_base64.py)

Pytest

Cloud-governance integration tests using pytest

python3 -m venv governance
source governance/bin/activate
(governance) $ python -m pip install --upgrade pip
(governance) $ pip install coverage
(governance) $ pip install pytest
(governance) $ git clone https://github.com/redhat-performance/cloud-governance
(governance) $ cd cloud-governance
(governance) $ coverage run -m pytest
(governance) $ deactivate
rm -rf *governance*

Post Installation

Delete cloud-governance image

sudo podman rmi quay.io/ebattat/cloud-governance

smalleni/cloud-governance