/llama-node

Believe in AI democratization. llama for nodejs backed by llama-rs, llama.cpp and rwkv.cpp, work locally on your laptop CPU. support llama/alpaca/gpt4all/vicuna/rwkv model.

Primary LanguageRustMIT LicenseMIT

LLaMA Node

llama-node: Node.js Library for Large Language Model

GitHub Workflow Status NPM npm npm type definitions twitter

Official Documentations

LLaMA generated by Stable diffusion

Picture generated by stable diffusion.

Introduction

This project is in an early stage, the API for nodejs may change in the future, use it with caution.

This is a nodejs library for inferencing llama, rwkv or llama derived models. It was built on top of llama-rs, llama.cpp and rwkv.cpp. It uses napi-rs for channel messages between node.js and llama thread.

Currently supported models (All of these should be converted to GGML format):

Supported platforms:

  • darwin-x64
  • darwin-arm64
  • linux-x64-gnu (glibc >= 2.31)
  • linux-x64-musl
  • win32-x64-msvc

Node.js version: >= 16

Installation

  • Install llama-node npm package
npm install llama-node

  • Install anyone of the inference backends (at least one)

    • llama.cpp
    npm install @llama-node/llama-cpp
    • or llama-rs
    npm install @llama-node/core
    • or rwkv.cpp
    npm install @llama-node/rwkv-cpp

Manual compilation

Please see how to start with manual compilation on our contribution guide

Acknowledgments

This library was published under MIT/Apache-2.0 license. However, we stronly recommend you to cite our work/our dependencies work if you wish to reuse the code from this library.

Models/Inferencing tools dependencies

Some source code comes from