Mistral 7b instruct not working in pet store for local LLM

[lukemurraynz]

Describe the bug

Hi, all. Working on a blog article, following a mix of local documentation + Intelligent app workshop, but instead of going Falcon, I've gone with the Mistral 7b model. and at the end - the switch of the pet store app to use it.

I can prompt the model locally from the cluster using:

kubectl run -it --rm --restart=Never curl --image=curlimages/curl -- curl -X POST "http://workspace-mistral-7b-instruct/generate" -H "accept: application/json" -H "Content-Type: application/json" -d "{"prompt":"What is your fav
orite ice cream flavor?"}"

However, using a pet store is not working, and it responds with 'Production Description' text instead.

Screenshots

This is my config map:

kubectl apply -n pets -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: ai-service-configmap
data:
USE_LOCAL_LLM: "True"
AI_ENDPOINT: "http://workspace-mistral-7b-instruct/chat"

apiVersion: apps/v1
kind: Deployment
metadata:
name: ai-service
spec:
replicas: 1
selector:
matchLabels:
app: ai-service
template:
metadata:
labels:
app: ai-service
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: order-service
image: ghcr.io/azure-samples/aks-store-demo/ai-service:latest
ports:
- containerPort: 5001
envFrom:
- configMapRef:
name: ai-service-configmap
resources:
requests:
cpu: 20m
memory: 50Mi
limits:
cpu: 30m
memory: 85Mi
startupProbe:
httpGet:
path: /health
port: 5001
initialDelaySeconds: 60
failureThreshold: 3
timeoutSeconds: 3
periodSeconds: 5
readinessProbe:
httpGet:
path: /health
port: 5001
initialDelaySeconds: 3
failureThreshold: 3
timeoutSeconds: 3
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
port: 5001
failureThreshold: 3
initialDelaySeconds: 3
timeoutSeconds: 3
periodSeconds: 3

apiVersion: v1
kind: Service
metadata:
name: ai-service
spec:
type: ClusterIP
ports:

• name: http
  port: 5001
  targetPort: 5001
  selector:
  app: ai-service
  EOF

Hoping someone can point me in the direction on whats happening here, whether its a bug, or what needs changing.

[pauldotyu]

Thanks for raising this issue @lukemurraynz! Will take a closer look and let you know.