/ClashOfClans_API

This utilizes a static IP address in GCP to pull data from the Clash of Clans API, store it and serve it via data studio. There is also a API client with a sleep of functions you can use to pull, store, and view CoC facts and dimensions.

Primary LanguagePython

LinkedIn


Logo

Clash of Clans API - Automatically ETL, Store, Report Data with GCP

This python code uses python to get Clash of Clans statistics on a schedule, using Cloud Scheduler, BigQuery using Pub/Sub and Cloud Functions, Virtual Private Cloud, Cloud Nat, Cloud Router, and Data Studio.

Table of Contents
  1. About The Project
  2. Prerequisites & Instructions
  3. Contact

About The Project

  • This project gives functions the user can use to pull data about fact data like clans, players, war logs, global rankings, and dimension data like locations, player labels, clan labels, leauges, etc. (Backend API Code in "apiHelper.py", get response in df form using "main.py" functions)

  • This project, also was deployed entirely in GCP. It uses Cloud Scheduler and Pub/Sub to trigger a Cloud Functions 2x daily. This Cloud Function runs Python code that pulls data from Clash of Clans API through a static API provided by the Virtual Private Cloud, Cloud NAT & Cloud Router. The API data is stored in BigQuery and served as a Data Studio report emailed daily.

  • Note: There is a small cost to running the VPC/NAT/Router combo. It's the only way to get a static IP egress needed for the Clash of Clans API call. My estimates are ~$13/month. Depending on how many clans you lead, how serious of a player you are, and if you use the VPC for another project, it might be worth the investment.

Process Map

Built With

Prerequisites & Instructions

  1. Installing all Required Packages
pip install -r requirements.txt

  1. Open a Google Cloud Platform Account and create a new project.

  2. Open the IAM & Admin page, set up permissions and obtain an API key. Replace this key with the \CloudFunction ".json" key.

set up permissions

  1. Ensure a default Virtual Private Cloud exists in GCP.

set up cloud fucntions

  1. Create a Serverless VPC Access Connector in GCP.

set up serverless vpc access connector

  1. Create a Cloud Router in GCP.

set up cloud router

  1. Create a Cloud NAT in GCP.

set up cloud nat

  1. Create a SuperCell account, and Create an API Key to access Clash of Clans data. Use the Static IP Address from the NAT Gateway Step. Plug this into the cloud functions main.py as the API_KEY. Make another with your home IP Adress if you want to test locally.

set up permissions

  1. Set up a Pub/Sub Topic.

create generic pub/sub topic

  1. Set up a Cloud Scheduler to send the Pub/Sub topic trigger to Cloud Functions 2x daily.

set up cloud scheduler to trigger cloud function through pub/sub

  1. Set up the Cloud Function with the following trigger, code, and connection settings.
  • Trigger Settings set up cloud fucntions
  • Code Settings set up cloud fucntions
  • Connection Settings set up cloud fucntions
  1. Use BigQuery to check query your computers' stats over time.

use BigQuery to query the stats

  1. Create a Data Studio Report connecting to BigQuery. Set report to send daily via email.

use BigQuery to query the stats

  1. Test locally and adjust the cloud function python script by using the apiHelpers.py file and main.py file. Integrate customized functions into your automated ETL process!

Contact

Jared Fiacco - jaredfiacco2@gmail.com

Another GCP Project of Mine:

This project was inspired by: