/bigdata-on-gcp

How put your BigData on GCP.

Primary LanguageGo

Bigdata on Google Cloud Plateform

WeSpeakCloud

Please follow this link to get the SpeakerDeck presentation

Objectif

This project is used to:

  • get data from Twitter and Mastodon
  • convert to a common object
  • improve data
  • store data
  • create statistics
  • get data by GSheet and REST API

Prerequisite

To be installer

Configuration

Clone the current project. Create config file in the root path of the project.

#/bin/bash

export CONSUMER_KEY="***"
export CONSUMER_SECRET="***"
export ACCESS_TOKEN="***"
export ACCESS_SECRET="***"


export MASTODON_SERVER="https://linuxrocks.online"
export MASTODON_CLIENT_ID="***"
export MASTODON_CLIENT_SECRET="***"
export MASTODON_LOGIN="***"
export MASTODON_PASSWORD="***"

export CLIENT_ID="***"
export CLIENT_SECRET="***"

CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN and ACCESS_SECRET are used to connect to Twitter.

MASTODON_SERVER, MASTODON_CLIENT_ID, MASTODON_CLIENT_SECRET, MASTODON_LOGIN and MASTODON_PASSWORD are used to connect to Mastodon.

Création de la plateform

Création de l'infrastructure

Go to "iac". And launch "create-all.sh".

Tests

Flux temps réel

Go to:

API

Génération d'aggregas

curl -X POST https://public.gcp-wescale.slavayssiere.fr/aggregator/stats | jq .

curl -X GET https://public.gcp-wescale.slavayssiere.fr/aggregator/stats/1 | jq .
curl -X GET https://us-central1-slavayssiere-sandbox.cloudfunctions.net/laststat | jq .
curl -X GET https://us-central1-slavayssiere-sandbox.cloudfunctions.net/getstat?id=1 | jq .

curl -X GET http://public.gcp-wescale.slavayssiere.fr/aggregator/top10 | jq .

Observability

Connect to bastion and create local SSH tunnel:

sudo ssh -i /Users/slavayssiere/.ssh/id_rsa admin@bastion.gcp-wescale.slavayssiere.fr -L 80:admin.gcp.wescale:80

To view Istio observability stack, please follow: