A Grafana plugin designed to centralize access to LLMs, providing authentication, proxying, streaming, and custom extensions. Installing this plugin will enable various pieces of LLM-based functionality throughout Grafana.
Note: The Grafana LLM App plugin is currently in Public preview. Grafana Labs offers support on a best-effort basis, and there might be breaking changes before the feature is generally available.
Prerequisites:
- Any Grafana Cloud environment (including Free)
- API connection details from an account with OpenAI or Azure OpenAI
Steps:
- In your Grafana instance, open Administration → Plugins
- Select "All" instead of "Installed" and search for "LLM"
- Click "Install via grafana.com"
- On the LLM's plugin page, you should see your instance listed; click "Install plugin"
- Return to Grafana, and search installed plugins, reloading until the LLM plugin is listed (this may take a minute or two)
- Configuration: choose your provider (OpenAI or Azure) and fill in the fields needed
- Save settings, then click "Enable" (upper right) to enable the plugin
To install this plugin, use the GF_INSTALL_PLUGINS
environment variable when running Grafana:
GF_INSTALL_PLUGINS=grafana-llm-app
or alternatively install using the Grafana CLI:
grafana cli plugins install grafana-llm-app
The plugin can then be configured either in the UI or using provisioning, as shown below.
To provision this plugin you should set the following environment variable when running Grafana:
OPENAI_API_KEY=sk-...
and use the following provisioning file (e.g. in /etc/grafana/provisioning/plugins/grafana-llm-app
, when running in Docker):
apiVersion: 1
apps:
- type: 'grafana-llm-app'
disabled: false
jsonData:
openAI:
url: https://api.openai.com
secureJsonData:
openAIKey: $OPENAI_API_KEY
To provision the plugin to use Azure OpenAI, use settings similar to this:
apiVersion: 1
apps:
- type: 'grafana-llm-app'
disabled: false
jsonData:
openAI:
provider: azure
url: https://<resource>.openai.azure.com
azureModelMapping:
- ["gpt-3.5-turbo", "gpt-35-turbo"]
secureJsonData:
openAIKey: $OPENAI_API_KEY
where:
<resource>
is your Azure OpenAI resource name- the
azureModelMapping
field contains[model, deployment]
pairs so that features know which Azure deployment to use in place of each model you wish to be used.
The vector services of the plugin allow some AI-based features (initially, the PromQL query advisor) to use semantic search to send better context to LLMs (and improve responses). Configuration is in roughly three parts:
- 'global' vector settings:
enabled
- whether to enable or disable vector services overallmodel
- the name of the model to use to calculate embeddings for searches. This must match the model used when storing the data, or the embeddings will be meaningless.
- 'embedding' vector settings (
embed
):type
- the type of embedding service, eitheropenai
orgrafana/vectorapi
to use Grafana's own vector API (recommended if you're just starting out).grafanaVectorAPI
, iftype
isgrafana/vectorapi
, with keys:url
- the URL of the Grafana VectorAPI instance.authType
- the type of authentication to use, eitherno-auth
orbasic-auth
.basicAuthUser
- the username to use ifauthType
isbasic-auth
.
- 'store' vector settings (
store
):type
- the type of vector store to connect to. We recommend starting out withgrafana/vectorapi
to use Grafana's own vector API for a quick start. We also supportqdrant
for Qdrant.grafanaVectorAPI
, iftype
isgrafana/vectorapi
, with keys:url
- the URL of the Grafana VectorAPI instance.authType
- the type of authentication to use, eitherno-auth
orbasic-auth
.basicAuthUser
- the username to use ifauthType
isbasic-auth
.
qdrant
, iftype
isqdrant
, with keys:address
- the address of the Qdrant server. Note that this uses a gRPC connection.secure
- boolean, whether to use a secure connection. If you're using a secure connection you can set theqdrantApiKey
field insecureJsonData
to provide an API key with each request.
- Currently Azure OpenAI is not supported as an embedder.
- Grafana Vector API used in
embedding
andstore
can be optionally different. - If you want to enable the PromQL Query Advisor, set up the Grafana vector API - we'll walk you through loading the data you need for that feature. If you're interested in building your own vector-based features on the Grafana platform, we do also support OpenAI embeddings and Qdrant.
Grafana VectorAPI Store + Grafana VectorAPI Embedder example
apps:
- type: grafana-llm-app
jsonData:
openAI:
provider: openai
url: https://api.openai.com
organizationId: $OPENAI_ORGANIZATION_ID
# openAI:
# provider: azure
# url: https://<resource>.openai.azure.com
# azureModelMapping:
# - ["gpt-3.5-turbo", "gpt-35-turbo"]
vector:
enabled: true
model: BAAI/bge-small-en-v1.5
embed:
type: grafana/vectorapi
grafanaVectorAPI:
url: <vectorapi-url> # e.g. http://localhost:8889
authType: no-auth
# authType: basic-auth
# basicAuthUser: <user>
store:
type: grafana/vectorapi
grafanaVectorAPI:
url: <vectorapi-url> # e.g. http://localhost:8889
authType: no-auth
# authType: basic-auth
# basicAuthUser: <user>
secureJsonData:
openAIKey: $OPENAI_API_KEY
# openAIKey: $AZURE_OPENAI_API_KEY
# vectorEmbedderBasicAuthPassword: $VECTOR_EMBEDDER_BASIC_AUTH_PASSWORD
# vectorStoreBasicAuthPassword: $VECTOR_STORE_BASIC_AUTH_PASSWORD
OpenAI Embedder + Qdrant Store example
apiVersion: 1
apps:
- type: grafana-llm-app
jsonData:
openAI:
provider: openai
url: https://api.openai.com
organizationId: $OPENAI_ORGANIZATION_ID
vector:
enabled: true
model: text-embedding-ada-002
embed:
type: openai
store:
type: qdrant
qdrant:
address: <qdrant-grpc-address> # e.g. localhost:6334
secureJsonData:
openAIKey: $OPENAI_API_KEY
- Note: openai embed type uses the setting from
openAI
automatically
To make use of this plugin when adding LLM-based features, you can use the helper functions in the @grafana/experimental
package.
First, add the correct version of @grafana/experimental
to your dependencies in package.json:
{
"dependencies": {
"@grafana/experimental": "1.7.0"
}
}
Then in your components you can use the llm
object from @grafana/experimental
like so:
import React, { useState } from 'react';
import { useAsync } from 'react-use';
import { llms } from '@grafana/experimental';
import { PluginPage } from '@grafana/runtime';
import { Button, Input, Spinner } from '@grafana/ui';
const MyComponent = (): JSX.Element => {
const [input, setInput] = React.useState('');
const [message, setMessage] = React.useState('');
const [reply, setReply] = useState('');
const { loading, error } = useAsync(async () => {
const enabled = await llms.openai.enabled();
if (!enabled) {
return false;
}
if (message === '') {
return;
}
// Stream the completions. Each element is the next stream chunk.
const stream = llms.openai
.streamChatCompletions({
model: 'gpt-3.5-turbo',
messages: [
{ role: 'system', content: 'You are a cynical assistant.' },
{ role: 'user', content: message },
],
})
.pipe(llms.openai.accumulateContent());
// Subscribe to the stream and update the state for each returned value.
return stream.subscribe(setReply);
}, [message]);
if (error) {
// TODO: handle errors.
return null;
}
return (
<div>
<Input value={input} onChange={(e) => setInput(e.currentTarget.value)} placeholder="Enter a message" />
<br />
<Button type="submit" onClick={() => setMessage(input)}>
Submit
</Button>
<br />
<div>{loading ? <Spinner /> : reply}</div>
</div>
);
};
The messages
parameter is the same as OpenAI's concept of messages
.
The .subscribe
method can take a few different forms. The "callback form" shown here is the more concise form. Another form allows more specific callbacks based on conditions, e.g. error
or complete
which can be useful if you want to do specific UI actions like showing a loading indicator.
-
Update Grafana plugin SDK for Go dependency to the latest minor version:
go get -u github.com/grafana/grafana-plugin-sdk-go go mod tidy
-
Build backend plugin binaries for Linux, Windows and Darwin:
mage -v
-
List all available Mage targets for additional commands:
mage -l
-
Install dependencies
npm install
-
Build plugin in development mode and run in watch mode
npm run dev
-
Build plugin in production mode
npm run build
-
Run the tests (using Jest)
# Runs the tests and watches for changes, requires git init first npm run test # Exits after running all the tests npm run test:ci
-
Spin up a Grafana instance and run the plugin inside it (using Docker)
npm run server
-
Run the E2E tests (using Cypress)
# Spins up a Grafana instance first that we tests against npm run server # Starts the tests npm run e2e
-
Run the linter
npm run lint # or npm run lint:fix
The LLM example app can be a quick way to test out changes to the LLM plugin.
To use the example app in conjunction with the LLM plugin:
- Clone the llm example app
- Update the following fields in
docker-compose.yaml
in the llm example app
- comment out # GF_INSTALL_PLUGINS: grafana-llm-app
- Add the following volume:
<some-parent-path>/grafana-llm-app/dist:/var/lib/grafana/plugins/grafana-llm-app
- Follow the instructions in the llm example app to run the app
- Bump version in package.json (e.g., 0.2.0 to 0.2.1)
- Add notes to changelog describing changes since last release
- Merge PR for a branch containing those changes into main
- Go to drone here and identify the build corresponding to the merge into main
- Promote to target 'publish'
- Push a new tag to the repo (e.g.,
git tag -a llmclient/v0.X.X -m "llmclient v0.X.X release"
)