A tool for generating sample cost and usage data for testing purposes.
This is a Python project developed using Python 3.6. Make sure you have at least this version installed.
To get started developing against Nise first clone a local copy of the git repository.
git clone https://github.com/project-koku/nise
Developing inside a virtual environment is recommended. A Pipfile is provided. Pipenv is recommended for combining virtual environment (virtualenv) and dependency management (pip). To install pipenv, use pip
pip3 install pipenv
Then project dependencies and a virtual environment can be created using
pipenv install --dev
To activate the virtual environment run
pipenv shell
To build the command line tool run
python setup.py install
Nise uses tox to standardize the environment used when running tests. Essentially, tox manages its own virtual environment and a copy of required dependencies to run tests. To ensure a clean tox environment run
tox -r
This will rebuild the tox virtual env and then run all tests.
To run unit tests specifically:
tox -e py36
To lint the code base
tox -e lint
- AWS population requires prior setup of AWS Cost and Usage Report of same name to be created, as well as associated Bucket, Policy, Role, etc.
nise is a command line tool. Currently only accepting a limited number of arguments:
- --start-date YYYY-MM-dd (not supplied, if using --static-report-file yaml)
- --end-date YYYY-MM-dd (optional, defaults to today and current hour)
- (--aws | --ocp | --gcp | --azure) required provider type
- --aws-s3-bucket-name bucket_name (optional, must include --aws-s3-report-name) Note: Use local directory path to populate a "local S3 bucket".
- --aws-s3-report-name report_name (optional, must include --aws-s3-bucket-name)
- --aws-s3-report-prefix prefix_name (optional)
- --aws-finalize finalize_choice (optional, choices: ['copy', 'overwrite'])
- --ocp-cluster-id cluster-id (required when providing ocp type)
- --insights-upload UPLOAD_URL (optional) Note: Use local directory path to populate a "local upload directory".
- --static-report-file file_name (optional) Note: Static report generation based on specified yaml file. See example_aws[ocp]_static_data.yml for examples.
- --gcp-report-prefix prefix_name (optional)
- --gcp-bucket-name bucket_name (optional, see example usage below)
Note: If --aws-s3-report-name or --aws-s3-report-prefix are specified they should match what is configured in the AWS cost usage report settings.
- Note: If --aws-finalize is used the copy choice will create a local copy of the data with a -finalized suffix and invoice id populated.
- If overwrite is used, the regular data file generated will have invoice id populated
Note: If --insights-upload is and pointing to a URL endpoint you must have INSIGHTS_USER and INSIGHTS_PASSWORD set in your environment.
Note: If --static-report-file is used start_date will default to first day of current month. start_date: last_month will be first day of previous month. start_date: today will start at the first hour of current day. end_date can support relative days from the start_date. i.e end_date: 2 is two days after start date.
Note: --static-report-file usage dates has a special full_period key value which will specify a usage for the entire start_date - end_date range.
Below is an example usage of nise for AWS data:
nise --start-date 2018-06-03 --aws nise --start-date 2018-06-20 --aws --aws-s3-bucket-name testbucket --aws-s3-report-name cur nise --start-date 2018-06-20 --aws --aws-s3-bucket-name /local/path/testbucket --aws-s3-report-name cur nise --start-date 2018-06-20 --aws --aws-s3-bucket-name /local/path/testbucket --aws-s3-report-name cur --aws-s3-report-prefix my-prefix nise --start-date 2018-06-20 --aws --aws-finalize copy nise --aws --static-report-file aws_static_data.yml
Generated reports will be generated in monthly .csv files with the file format <Month>-<Year>-<Report Name>.csv.
Below is an example usage of nise for OCP data:
nise --start-date 2018-06-03 --ocp --ocp-cluster-id test-001 nise --start-date 2018-06-03 --ocp --ocp-cluster-id test-001 --insights-upload https://cloud.redhat.com/api/ingress/v1/upload nise --start-date 2018-06-03 --ocp --ocp-cluster-id test-001 --insights-upload /local/path/upload_dir nise --ocp --ocp-cluster-id my-cluster-id --static-report-file ocp_static_data.yml
Generated reports will be generated in monthly .csv files with the file format <Month>-<Year>-<Cluster-ID>.csv.
Below is an example usage of nise for OCP running on AWS data:
# First ensure that the resource_id and dates in both AWS and OCP static report files match nise --aws --static-report-file aws_static_data.yml nise --ocp --ocp-cluster-id my-cluster-id --static-report-file ocp_static_data.yml
Generated AWS reports will be generated in monthly .csv files with the file format <Month>-<Year>-<Report Name>.csv.
Generated OCP reports will be generated in monthly .csv files with the file format <Month>-<Year>-<Cluster-ID>.csv.
Note: To upload to AZURE, you must have AZURE_STORAGE_ACCOUNT and AZURE_ACCOUNT_KEY set in your environment.
Below is an example usage of nise for AZURE data:
nise --start-date 2019-08-01 --azure nise --start-date 2019-08-01 --azure --azure-storage-name container --azure-report-name cur nise --start-date 2019-08-01 --azure --azure-storage-name /local/path/container --azure-report-name cur nise --start-date 2019-08-01 --azure --azure-storage-name /local/path/container --azure-report-name cur --azure-storage-report-prefix my-prefix nise --start-date 2019-08-01 --azure --azure-storage-name /local/path/container --azure-report-name cur --azure-storage-report-prefix my-prefix --static-report-file example_azure_static_data.yml nise --azure --static-report-file azure_static_data.yml
To add an AZURE-local provider:
{
"name": "Test Azure Source",
"type": "AZURE-local",
"authentication": {
"credentials": {
"subscription_id": "12345678-1234-5678-1234-567812345678",
"tenant_id": "12345678-1234-5678-1234-567812345678",
"client_id": "12345678-1234-5678-1234-567812345678",
"client_secret": "12345"
}
}, "billing_source": {
"data_source": {
"resource_group": {
"directory": --azure-storage-report-prefix,
"export_name": --azure-report-name
},
"storage_account": {
"local_dir": "/tmp/local_container",
"container": ""
}
}
}
}
--gcp-bucket-name could be an local file name or a bucket. When --gcp-bucket-name matches a file on disk,
the generated reports will be written to that file. If --gcp-bucket-name does not match a file on disk,
nise will attempt to upload the gnerated report to a bucket with that name. When this is the case
the GOOGLE_APPLICATION_CREDENTIALS environment variable must be set, and the given bucket-name must match
and existing bucket that is accessable by the service account indicated in GOOGLE_APPLICATION_CREDENTIALS.
For more information about GOOGLE_APPLICATION_CREDENTIALS see the Google Authentication Docs..
Below is an example usage of nise for GCP data:
nise --gcp --start-date 2018-06-03 --end-date 2018-06-08 nise --gcp --start-date 2018-06-03 --end-date 2018-06-08 --gcp-report-prefix my-gcp-data nise --gcp --start-date 2018-06-03 --end-date 2018-06-08 --gcp-report-prefix my-gcp-data --gcp-bucket-name my-gcp-bucket
Generated reports will be generated in daily .csv files with the file format <Report-Prefix>-<Year>-<Month>-<Day>.csv.
Please refer to Contributing.