/aws_costs_collector

Primary LanguagePythonMIT LicenseMIT

⚙ Project With Cost Management Engine Job

It is a repository that contains the entire project for cost management engine logic in aws lambda.

Job is responsible for automatically obtaining management costs and making them available on S3.

(files in config is only for glue jobs configurations DO NOT FORGET CHANGE VALUES IN THE FILES BEFORE EXECUTION!)

📝 PREREQUISITES

What is needed before running the script.

Libraries used in this project (included into dockerfile!):

🔗 JOB ENVIRONMENT VARIABLES (AWS Lambda)

--SECRET_NAME: Name of the secret with connection informations.

  👉 Example: secret_test

--REGION_NAME: Name of region.

  👉 Example: us-east-1

--ENV: Cost environment to obtain.

  👉 Example: DEV|PRD

--FILE: Parameters file path.

  👉 Example: s3://utils/scripts/parameters.json

--OUTPUT_PATH: Output of csv path.

  👉 Example: s3://utils/costs

--PYTHONIOENCODING: Set default encoding.

  👉 Example: utf8

--SNOW_WAREHOUSE: Warehouse to connect in snowflake.

  👉 Example: DEV_COMMON_MONITORING

Parameters like ACCESS_KEY, SECRET_KEY, SNOW_USER and SNOW_PASS is in AWS sercret manager

📍 JOB CREATION

To create the job:

  1. Copy the project files to a location on your local machine;
  2. Change the information you want in the code;
  3. Run the below code to build your image:

docker build -t <USER_ID>.dkr.ecr.<REGION>.amazonaws.com/<REPO_IN_ECR> .

  👉 Example: docker build -t 1234567890.dkr.ecr.us-east-1.amazonaws.com/costs .

  1. Run the below code to publish your image on ECR:

  docker push <USER_ID>.dkr.ecr.<REGION>.amazonaws.com/<REPO_IN_ECR>

  👉 Example: docker push 1234567890.dkr.ecr.us-east-1.amazonaws.com/costs

  1. If you get error "no basic auth credentials" to publish image, make sure if your docker config.json has the below informations:
{
  "credsStore": "ecr-login"
}
  1. Create a lambda function with this image as template.
  2. Make sure that the lambda function has a sufficient timeout (5 minutes recommended).
  3. Make sure that the lambda function has a sufficient memory (512MB recommended).

📋 GETTING STARTED

Some examples of configuration files used in this script:

  • JSON Parameters Estructure:
{
  "fetch": {
    "start_date": "2020-06-01",
    "end_date": ""
  },
  "aws": {
    "enable": true,
    "tag_key": "project",
    "multiply_factor_enable": false,
    "multiply_factor_amount": 1,
    "multiply_factor_unit": "USD"
  },
  "snowflake": {
    "enable": true,
    "account":"<*your-account*>"
    "multiply_factor_enable": true,
    "multiply_factor_amount": 3,
    "multiply_factor_unit": "USD"
  },
  "databricks":{
    "account_id": "",
    "credentials_id": "",
    "storage_configuration_id": ""
  }
}

🔨 BUILD WITH

✒️ AUTHORS

  • Gabriel Praça - 💻 Developer