/local-llm

Running large language models (LLMs) locally using Langchain, Ollama and Docker.

Primary LanguageSvelte

Local LLM with Langchain, Ollama and Docker

A proof-of-concept for running large language models (LLMs) locally using Langchain, Ollama and Docker.

Requirements

  • Docker
  • Docker Compose

Quickstart

  1. Build and run the services with Docker Compose: docker compose up --build
  2. Create a .env file in the root of the project based on .env.example: cp .env.example .env.
  3. (Optional) You can change the chosen model in the .env file. Refer to Ollama's model library for available models.
  4. The service will be available at:
    1. As a SvelteKit frontend at http://localhost:8080
    2. As LangChain's UI at http://localhost:8000/chain/playground
    3. In the terminal, e.g.: curl 'http://localhost:8000/chain/invoke' --data-raw '{"input":{"text":"hi"}}'

API Endpoints

  • /chain/playground: Provides an interactive UI to test the model.
  • /chain/invoke: A REST API endpoint for programmatic interaction with the model.