/Azure-Infra-Project

a parallelised, HA system I built + architected for an LLM automation startup!

Primary LanguageTypeScript

Azure Infra Project

A highly-available, parallelised system I built, architected, and hosted on Azure for an LLM automation startup!

System Architecture Diagram

Architecture-Diagram

Things I considered

Key Design Choices

More faster, low-cost LLM calls than fewer costly, slow ones

Rationale: way cheaper + faster; can self-host models

  • (e.g, 3x Llama 3 70b instead of 1x Claude 3.5 Sonnet)

Decent time spent on system design / architecture planning

Rationale: wanted to ensure HA + reliability in production

  • (All non-dependent code/jobs execute asynchronously!)

Use of Windmill automation engine over a cloud function

Rationale: Very flexible + robust abstractions generally

  • (E.g, async job parallelisation, native S3 support, auth, etc)

Scoping

Scoped in:

  • Azure VM + Blob S3 configuration + deployment
    • Self-hosting of a locked-down Windmill instance
  • Robust scripts to handle all aspects of Matter lifecycle
    • Including parallelisation of every potential blocker
  • API endpoint deployment
    • Token/auth management

Scoped-out (due to time/recovering from stitches)

  • Features:
    • Calculate avg word confidence per-doc/page
    • Overlapping page contexts (did previously for a RAG)
  • Reliability/Scaling
    • Kubernetes Cluster deployment
    • Highly Available Failovers (did previously for homelab)

Pros/Cons

Pros

  • Scalability (replicas, Kube Helm Chart)
  • Security (handles team IAM/scoped token auths etc)
  • Flexibility (any langs; Py/TS/JS/Bash/Go etc etc etc)
  • Pog state management (async; share across scripts)
  • Interoperability (Easy to chain scripts as workflow)
  • Extensibility (simple to integrate external API's etc)

Cons

  • Single point of failure (one partially-scaled Azure VM)
    • But: in prod, would use their Kube Helm Chart
  • Slight reliance on 3rd party platform (Windmill)
    • But: open source; and can move scripts anytime

Thought Process

Choice: Python+TypeScript on Windmill thru Azure VM

  • Tried Azure Durable Functions first (painful)
    • Heard from (someone) it was mid && slow long-term
  • Knew that either way, I'd be writing code (Py/JS/TS) to parse the JSON data
    • Spun up VM on Azure to allow rapid prototyping
  • Went w/Windmill for 2 main reasons
    • Good async/replica support (+, Kube Helm Chart) In prod, allows efficient scalability variable workloads
    • Flexible language support + state management Easy to maintain workflows + team collaboration etc