/mlops-gitops-template

The MLOps setup utilising GitOps principals for local development with kind, dvc, argo and seldon core

Primary LanguagePython

MLOps GitOps Template

Work in progress...

Infrastructure setup

Cluster base infrastructure

Get kind (ref):

brew install kind

Get kubectl (ref):

brew install kubectl

Get helm (ref):

brew install helm

Create kind cluster:

kind create cluster

Get istio (ref):

curl -L https://istio.io/downloadIstio | sh -

cd istio-1.11.3

export PATH=$PWD/bin:$PATH
istioctl install --set profile=demo -y

kubectl label namespace default istio-injection=enabled

Folow the actual instructions on the webside

Seldon Core

Create namespaces (ref):

kubectl create namespace seldon-system
kubectl create namespace seldon

Enable istio in the namespaces:

kubectl label namespace seldon-system istio-injection=enabled
kubectl label namespace seldon istio-injection=enabled

Install the platform using helm (ref):

helm install seldon-core seldon-core-operator \
    --repo https://storage.googleapis.com/seldon-charts \
    --set usageMetrics.enabled=true \
    --namespace seldon-system \
    --set istio.enabled=true

Create istio gareway for seldon projects (ref):

kubectl apply -f ./infra/seldon/istio-gateway.yaml

Port-forward istio gateway ingress:

kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80

Additional resource for referencing: ref

Development

Install requirements:

pip install -r requirements/production.txt -r requirements/development.txt

Get example data:

bash get_data.sh

Formatter:

black .

Type checking:

mypy models

Testing:

python -m pytest --cov=models

Validate model:

python -m models validate

Train model:

python -m models train --id test