Airbyte provides the largest catalog of 300+ connectors for APIs, databases, data warehouses, and data lakes.

AWS Deployment:

Install Docker Engine and Kubernetes plugin on your workstation.
Clone the project repository from GitHub:

git clone https://github.com/amigo-nishant/Capstone-Project-Clone.git

Navigate into the Airbyte directory:

cd Capstone-Project-Clone

Start Service using helm
Access in your browser by visiting - http://public-IP:8000
You will be asked for a username and password. that's username --> airbyte and password --> password.
Enter basic details and login

Deploy Airbyte on Kubernetes using Helm

Install Helm, kubernetes, Docker and setup cluster
Add Helm Repository
To deploy on Kubernetes, first add the Helm repository:

helm repo add airbyte https://airbytehq.github.io/helm-charts

Then update the repository index:

helm repo update

You can search for available Airbyte charts using:

helm search repo airbyte

It'll produce output similar to below:

NAME                            	CHART VERSION	APP VERSION	DESCRIPTION
airbyte/airbyte                 	0.49.9       	0.50.33    	Helm chart to deploy airbyte
airbyte/airbyte-api-server      	0.49.9       	0.50.33    	Helm chart to deploy airbyte-api-server
airbyte/airbyte-bootloader      	0.49.9       	0.50.33    	Helm chart to deploy airbyte-bootloader
airbyte/connector-builder-server	0.49.9       	0.50.33    	Helm chart to deploy airbyte-connector-builder-...
airbyte/cron                    	0.49.9       	0.50.33    	Helm chart to deploy airbyte-cron
airbyte/metrics                 	0.49.9       	0.50.33    	Helm chart to deploy airbyte-metrics
airbyte/pod-sweeper             	0.49.9       	0.50.33    	Helm chart to deploy airbyte-pod-sweeper
airbyte/server                  	0.49.9       	0.50.33    	Helm chart to deploy airbyte-server
airbyte/temporal                	0.49.9       	0.50.33    	Helm chart to deploy airbyte-temporal
airbyte/webapp                  	0.49.9       	0.50.33    	Helm chart to deploy airbyte-webapp
airbyte/worker                  	0.49.9       	0.50.33    	Helm chart to deploy airbyte-worker

Deploy Airbyte, default Deployment and if you don't need to customize your deployment, you can deploy Airbyte with default values. Run the following command:

helm install release_name airbyte/airbyte --dry-run --debug

Replace release_name with your desired release name. The release name should only contain lowercase letters and dashes, and it must start with a letter.

DevSecOps Project

Introduction

Welcome to our open source DevSecOps project! This repository serves as an educational resource for developers who want to learn how to deploy a micro-service application to PRODUCTION using DevSecOps principles.

Tools Covered:

Linux
Git and GitHub
Docker
Kubernetes
Docker-compose
Jenkins CI/CD
SonarQube Scan
SonarQube Quality Gates
Trivy
Prometheus & Grafana

Pre-requisites to implement this project:

AWS EC2 instance (Ubuntu) with instance type t2.large and root volume 29GB.

Installation of JAVA

sudo apt update
sudo apt install fontconfig openjdk-17-jre
java -version

Installation of Jenkins

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins

Setup Jenkins

Public-IP:8080 (Jenkins running)
sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Docker and docker-compose installled

 sudo apt-get update
 sudo apt-get install docker.io -y
 sudo apt-get install docker-compose -y
 sudo usermod -aG docker $USER
 sudo reboot

Install trivy --> https://aquasecurity.github.io/trivy/v0.18.3/installation/

SonarQube Server installed

 docker run -itd --name sonarqube-server -p 9000:9000 sonarqube:lts-community

Steps for Jenkins CI/CD:

Access Jenkins UI and setup Jenkins

Plugins Installation:
- Go to Manage Jenkins, click on Plugins and install all the plugins listed below, we will require for other tools integration:
  - SonarQube Scanner (Version2.16.1)
  - Sonar Quality Gates (Version1.3.1)
  - Docker (Version1.5)
  - Kubernetes

Go to SonarQube Server and create token
- Click on Administration tab, then Security , then Users and create Token.
- Create a webhook to notify Jenkins that Quality gates scanning is done. (We will need this step later)
  - Go to SonarQube Server, then Administration , then Configuration and click on Webhook , add webhook in below Format:
  http://<jenkins_url>:8080/sonarqube-webhook/
  
  Example:
```
    http://34.207.58.19:8080/sonarqube-webhook/
```

Go to Jenkins UI Manage Jenkins , then Credentials and add SonarQube Credentials.

Now, It's time to integrate SonarQube Server with Jenkins, go to Manage Jenkins , then System and look for SonarQube Servers and add SonarQube.

Go to Manage Jenkins , then tools , look for SonarQube Scanner installations and add SonarQube Scanner.

Note: Add name as Sonar

Configure the Jenkins Job PROD

create a jenkins job "PROD deployment"
select pipeline
Pipeline -> SCM --> credentials --> Username and PAT --> Secret name "jenkins-secret"
ALso before triggering the build check make sure to give jenkins and docker permission

sudo usermod -aG docker jenkins
sudo systemctl restart docker
sudo systemctl restart jenkins
docker ps -a
docker start <sonarqube-container-ID>
Trigger the pipeline

To start jenkins as a user for PROD environment

ssh to jenkins instance
sudo su jenkins
cd /var/lib/jenkins/workspace/
mkdir -p ~/.docker/cli-plugins/
curl -SL https://github.com/docker/compose-cli/releases/download/v2.0.0-rc.1/docker-compose-linux-amd64 -o ~/.docker/cli-plugins/docker-compose
chmod +x ~/.docker/cli-plugins/docker-compose
Trigger the pipeline
Thr WebApp service is exposed on port 8000
Username - "airbyte" & password "password"
Create plugins and test connectors.

Save1980/Capstone-Project