/llama-cpp.el

A client for llama-cpp server

Primary LanguageEmacs LispGNU General Public License v3.0GPL-3.0

Llama-cpp in Emacs

The readme is mostly written by CodeLLama. 😆

Llama package for Emacs provides a client for the llama-cpp server. It allows you to ask llama for code completion and perform tasks within specified regions of the buffer.

Table of Contents

  1. Installation
  2. Usage
  3. Customization
  4. Models
  5. License

Installation

To install this package, add it to your Emacs configuration:

(use-package llama-cpp
  :ensure t)

Make sure to have llama-cpp server running and accessible from the host specified in llama-cpp-host variable (default is "localhost"). The server should be accessible via HTTP on the port specified in llama-cpp-port (default is 8080).

Usage

This package provides llama-cpp-complete as a public API for other packages to use.

To cancel a running llama process, use M-x llama-cpp-cancel.

Chat

You can start a Llama chat session using the command M-x llama-cpp-chat-start.

After providing the necessary input, you can execute M-x llama-cpp-chat-answer to request a response from Llama.

To complete text from the llama buffer, use M-x llama-cpp-chat-complete.

Code

M-x llama-cpp-code-region-task asks for a description of your task and then calls the llama to provide an answer. The task context is taken from the region you select when calling this function.

Customization

The following customization options are available:

  • llama-cpp-host: the host of the llama-cpp server (default is "localhost").
  • llama-cpp-port: the port of the llama-cpp server (default is 8080).
  • llama-cpp-params: parameters for the llama-cpp /completion request.
  • llama-cpp-chat-prompt: sets the chat prompt to start the chat with.
  • llama-cpp-chat-input-prefix: string to prefix user inputs with.
  • llama-cpp-chat-input-suffix: string to suffix after user inputs with.
  • llama-cpp-code-lang-modes: an alist mapping major modes to their language names.
  • llama-cpp-code-region-prompt: a prompt template for code region tasks.

Models

You will need to set llama-cpp-chat-prompt-prefix, llama-cpp-chat-input-prefix and llama-cpp-chat-input-suffix according to the model you use. Some examples are listed below.

dolphin-2.2.1-mistral-7B

(setq llama-cpp-chat-prompt-prefix "<|im_start|>system
"
      llama-cpp-chat-input-prefix "<|im_end|>
<|im_start|>user
"
      llama-cpp-chat-input-suffix "<|im_end|>
<|im_start|>assistant
")

Phind-CodeLlama-34B-v2

(setq llama-cpp-chat-prompt-prefix "### System Prompt
"
      llama-cpp-chat-input-prefix "

### User Message
"
      llama-cpp-chat-input-suffix "

### Assistant
")

WizardLM-1.0-Uncensored-CodeLlama-34B

(setq llama-cpp-chat-prompt-prefix ""
      llama-cpp-chat-input-prefix "
USER: "
      llama-cpp-chat-input-suffix "
ASSISTANT: ")

Samantha-1.11-70B

(setq llama-cpp-chat-prompt ""
      llama-cpp-chat-input-prefix "
USER: "
      llama-cpp-chat-input-suffix "
ASSISTANT: ")

License

This package is distributed under the GNU General Public License version 3 or later. Please refer to the official repository for more information.