A solution for collecting Internet volume for some key words with CI/CD workflow automation.
- Cronjob to generate URL
- Send URLs to RabbitMQ
- Queue URLs
- Consume URLs from RabbitMQ
- Parse URL data
- Generate result and store into InfluxDB
- Keep data from Crawler
- Read data from InfluxDB
- Show the result figure
Dev branch for development, with a CI workflow for test/build (push) images to dockerhub; after some feature or bug released/fixed, contributor can raise a pull request merge dev branch to master.
The pull request for master branch will need at least one reviewer approval comment to accept the request.
After merge the request to master, ArgoCD will sync the application status to cluster.
Github Action use for CI workflow, run pytest and build (push) image to dockerhub.
ArgoCD use for install helm charts on GCP cluster.
(Grafana dashboard)
