It was made following the Ollama API documentation.
[dependencies]
ollama-rs = "0.1.9"
// By default it will connect to localhost:11434
let ollama = Ollama::default();
// For custom values:
let ollama = Ollama::new("http://localhost".to_string(), 11434);
Feel free to check the Chatbot example that shows how to use the library to create a simple chatbot in less than 50 lines of code. You can also check some other examples.
These examples use poor error handling for simplicity, but you should handle errors properly in your code.
let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();
let res = ollama.generate(GenerationRequest::new(model, prompt)).await;
if let Ok(res) = res {
println!("{}", res.response);
}
OUTPUTS: The sky appears blue because of a phenomenon called Rayleigh scattering...
Requires the stream
feature.
let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();
let mut stream = ollama.generate_stream(GenerationRequest::new(model, prompt)).await.unwrap();
let mut stdout = tokio::io::stdout();
while let Some(res) = stream.next().await {
let responses = res.unwrap();
for resp in responses {
stdout.write(resp.response.as_bytes()).await.unwrap();
stdout.flush().await.unwrap();
}
}
Same output as above but streamed.
let model = "llama2:latest".to_string();
let prompt = "Why is the sky blue?".to_string();
let options = GenerationOptions::default()
.temperature(0.2)
.repeat_penalty(1.5)
.top_k(25)
.top_p(0.25);
let res = ollama.generate(GenerationRequest::new(model, prompt).options(options)).await;
if let Ok(res) = res {
println!("{}", res.response);
}
OUTPUTS: 1. Sun emits white sunlight: The sun consists primarily ...
let res = ollama.list_local_models().await.unwrap();
Returns a vector of Model
structs.
let res = ollama.show_model_info("llama2:latest".to_string()).await.unwrap();
Returns a ModelInfo
struct.
let res = ollama.create_model(CreateModelRequest::path("model".into(), "/tmp/Modelfile.example".into())).await.unwrap();
Returns a CreateModelStatus
struct representing the final status of the model creation.
Requires the stream
feature.
let mut res = ollama.create_model_stream(CreateModelRequest::path("model".into(), "/tmp/Modelfile.example".into())).await.unwrap();
while let Some(res) = res.next().await {
let res = res.unwrap();
// Handle the status
}
Returns a CreateModelStatusStream
that will stream every status update of the model creation.
let _ = ollama.copy_model("mario".into(), "mario_copy".into()).await.unwrap();
let _ = ollama.delete_model("mario_copy".into()).await.unwrap();
let prompt = "Why is the sky blue?".to_string();
let res = ollama.generate_embeddings("llama2:latest".to_string(), prompt, None).await.unwrap();
Returns a GenerateEmbeddingsResponse
struct containing the embeddings (a vector of floats).