A robust end-to-end CI/CD pipeline demonstrating modern DevOps practices through the integration of industry-standard tools. This project automates the complete software development lifecycle from code commit to production deployment and observability of a Python Flask application.
🔍 For detailed setup and usage instructions, please see How to Use Guide.
- Overview
- Core Principles
- Architecture
- Key Features
- Components
- Workflow
- Directory Structure
- Documentation
- Contributing
- License
This project showcases a complete CI/CD pipeline with integrated monitoring that automates the software development lifecycle. It leverages modern DevOps tools to create a scalable, maintainable, and secure solution for continuous delivery and monitoring of a Python Flask application.
The implementation focuses on DevOps best practices including:
- Infrastructure as Code (IaC)
- Containerization
- Continuous Integration
- Continuous Deployment with GitOps
- Automated Testing
- Code Quality Enforcement
- Comprehensive Monitoring
- Multi-Environment Support
This project embodies several core DevOps principles:
- Automation 🔄: Eliminate manual processes through automation of building, testing, and deployment
- Continuous Integration 🛠️: Frequent code integration with automated verification
- Continuous Delivery 🚀: Reliable, low-risk deployments through automation
- GitOps 📝: Git as the single source of truth for infrastructure and application deployment
- Shift Left 🔍: Early testing, security scanning, and quality checks
- Infrastructure as Code 🏗️: Define and version infrastructure alongside application code
- Observability 📊: Comprehensive monitoring and metrics collection
- Environment Parity 🔄: Consistent configurations across environments
The architecture integrates several key components:
- Version Control (GitLab): Central repository for application code, Helm charts, and deployment configurations
- CI Pipeline (Jenkins): Orchestrates building, testing, and artifact creation processes
- Artifact Storage (Nexus): Securely stores versioned application binaries
- Quality Gates (Sonarqube): Enforces code quality standards and test coverage
- Infrastructure Provisioning (Terraform): Manages Kubernetes infrastructure declaratively
- GitOps Engine (ArgoCD): Ensures deployment state matches Git definitions
- Container Orchestration (Kubernetes): Manages application containers
- Monitoring Stack: Tracks application and infrastructure health and performance
The workflow connects these components into a seamless pipeline where code changes automatically flow through verification, building, and deployment stages while maintaining observability.
- Complete CI/CD Automation: End-to-end pipeline from code commit to deployment
- GitOps with ArgoCD: App of Apps pattern for multi-environment deployments
- Infrastructure as Code: Terraform-managed Kubernetes with reusable modules
- Multi-Environment Support: Separate dev and prod configurations
- Advanced Flask Application: Rate limiting, metrics collection, and health monitoring
- Comprehensive Testing: Automated functional, security, and metrics tests
- Code Quality Enforcement: Static analysis, security scanning, and Sonarqube integration
- Artifact Management: Versioned binary storage with Nexus
- Containerized Development: Docker Compose for consistent local environment
- Detailed Monitoring: Prometheus metrics collection with Grafana dashboards
- Alerting: Configurable thresholds with Alertmanager integration
GitLab serves as the central source code repository, providing:
- Version control for application code
- CI/CD integration with Jenkins
- Repository for deployment configurations
- Separate branch for ArgoCD configurations
Jenkins orchestrates the CI process with a pipeline that:
- Runs automated tests with pytest
- Performs static code analysis with Ruff
- Conducts security scanning with Bandit
- Analyzes code quality with Sonarqube
- Builds application binaries with PyInstaller
- Uploads artifacts to Nexus repository
- Updates deployment configurations
- Creates dedicated ArgoCD branch
Nexus provides a central repository for:
- Storing versioned application binaries
- Managing latest and timestamped releases
- Providing a reliable artifact source for deployments
Terraform enables infrastructure as code by:
- Creating and configuring Kind Kubernetes clusters
- Setting up ArgoCD with the App of Apps pattern
- Managing Kubernetes resources with reusable modules
- Connecting cluster to external services like Nexus
ArgoCD implements GitOps practices by:
- Using the App of Apps pattern for hierarchical management
- Automatically synchronizing Git changes to the cluster
- Supporting multiple environments (dev, prod)
- Self-healing deployments that maintain desired state
- Providing visibility into deployment status and history
Helm charts provide declarative application management:
- Environment-specific configurations via values files
- Consistent deployment templates
- Support for versioned releases
- Integration with Nexus for artifact retrieval
The monitoring stack delivers comprehensive observability:
- Application metrics from the Flask
/metricsendpoint - Pipeline performance metrics from Jenkins
- Container and system metrics from cAdvisor
- Custom dashboards for different aspects of the system
- Alerting based on defined thresholds
The CI/CD workflow follows these steps:
- Code Commit 📝: Developer pushes changes to GitLab main branch
- CI Pipeline 🧪: Jenkins tests, analyzes, builds and packages the application
- Artifact Storage 📦: Binary is versioned and stored in Nexus
- GitOps Update 🔄: Jenkins updates Helm chart and ArgoCD branch
- Automatic Deployment 🚀: ArgoCD detects changes and syncs applications to Kubernetes
- Multi-Environment Deployment 🌐: Applications deploy to dev and prod environments
- Continuous Monitoring 📊: Prometheus collects metrics from all components
- Performance Visualization 📈: Grafana displays real-time metrics and trends
This workflow embodies the principle of continuous delivery by providing a reliable, repeatable path to production with built-in quality gates and observability.
basic-ci-cd/
├── appflask/ # Flask application source
│ ├── agent/ # Jenkins agent configuration
│ ├── appflask/ # Application source code
│ ├── argocd-apps/ # ArgoCD application definitions
│ │ ├── apps/ # App of Apps child applications
│ │ └── helm/ # Helm charts for applications
│ ├── tests/ # Test suites
│ └── test_scripts/ # Monitoring test scripts
│
├── docs/ # Documentation files
│ ├── how-to-use.md # Comprehensive usage guide
│ ├── monitoring/ # Monitoring documentation
│ ├── progresses/ # Project progress tracking
│ └── subjects/ # Project requirements
│
├── srcs/ # Docker environment files
│ ├── docker-compose.yaml # Service composition
│ └── requirements/ # Service-specific files
│
├── terraform/ # Terraform configurations
│ ├── environments/ # Environment-specific configs
│ ├── modules/ # Reusable modules
│ │ ├── cluster/ # Kind cluster module
│ │ └── k8s_resources/ # Kubernetes resources modules
│ └── scripts/ # Helper scripts
│
├── Makefile # Build automation
└── README.md # This documentation
Comprehensive documentation is available in the docs directory:
- How to Use Guide: Detailed setup and usage instructions
- Infrastructure Architecture: Monitoring system architecture
- Monitoring Documentation: Metrics collection setup and pipeline metrics
- Code Standards: Coding standards and guidelines
- Technologies: Documentation for GitLab, Jenkins, and Nexus
- Project Requirements: Original project requirements
Contributions are welcome! Please feel free to submit a Pull Request.
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add some amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
Developed with ❤️ by [Nihilantropy]