sakti/parseable

Cloud native log observability platform - ingest high volume logs and query the data in simple, efficient, low latency.

Quick Start | Documentation | Live Demo

Parseable is a cloud native, log storage and observability platform. Parseable is written in Rust and index free by design. Parseable is available as a single binary / Docker image and can be deployed on a wide range of platforms.

It ingests log data via HTTP POST calls and exposes a query API to search and analyze logs. It is compatible with all the standard logging agents via HTTP output plugins.

Features

• Highly compressed log data storage with Parquet.
• Use standard SQL for querying log data.
• Auto inferred schema for log streams.
• Dashboard to query the log data.
• Compatible with existing logging agents.
• Scale with scaling up the node.
• Configurable alerting.
• Distributed, multi-node cluster.
• Log data visualization via Parseable UI.
• Access control for users and groups.
• Log data retention and export.
• Kafka plugin to ingest log data.
• Grafana plugin to visualize log data.
• Anomaly detection within log data.

Getting Started

Docker

Parseable docker image is available on Docker hub. Please change the environment variables as relevant.

cat << EOF > parseable-env
P_S3_URL=https://minio.parseable.io:9000
P_S3_ACCESS_KEY=minioadmin
P_S3_SECRET_KEY=minioadmin
P_S3_REGION=us-east-1
P_S3_BUCKET=parseable
P_LOCAL_STORAGE=/data
P_USERNAME=parseable
P_PASSWORD=parseable
EOF

mkdir -p /tmp/data
docker run \
  -p 8000:8000 \
  --env-file parseable-env \
  -v /tmp/data:/data \
  parseable/parseable:latest \
  parseable server

Kubernetes

Parseable helm chart is available at Parseable charts repository.

helm repo add parseable https://charts.parseable.io/
helm repo update
kubectl create namespace parseable
helm install parseable parseable/parseable --namespace parseable --set parseable.demo=true

Binary

Parseable binary is available on Github releases. Please download the latest release for your platform, also make sure to change the environment variables as relevant.

export P_S3_URL="https://minio.parseable.io:9000"
export P_S3_ACCESS_KEY="minioadmin"
export P_S3_SECRET_KEY="minioadmin"
export P_S3_REGION="us-east-1"
export P_S3_BUCKET="parseable"
export P_LOCAL_STORAGE="./data"
export P_USERNAME="parseable"
export P_PASSWORD="parseable"
chmod +x parseable
./parseable

Parseable dashboard is available at http://localhost:8000. Credentials to login to the dashboard are the values you set in the environment variables.

📝 Parseable is in alpha stage and will evolve over time. There may be breaking changes between releases. Please give us your feedback in Slack, or Issues.

Using Parseable

<stream-name> is the name of the stream you want to create. <basic-auth-header> is the basic auth header value generated from username & password of the user you created in the environment variables. You can generate the basic auth header value using the following command.

echo -n '<user-name>:<password>' | base64

Create a stream

curl --location --request PUT 'http://localhost:8000/api/v1/logstream/<stream-name>' \
--header 'Authorization: Basic <basic-auth-header>'

Send events to the stream

curl --location --request POST 'http://localhost:8000/api/v1/logstream/<stream-name>' \
--header 'X-P-META-meta1: value1' \
--header 'X-P-TAG-tag1: value1' \
--header 'Authorization: Basic <basic-auth-header>' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "id": "434a5f5e-2f5f-11ed-a261-0242ac120002",
        "datetime": "24/Jun/2022:14:12:15 +0000",
        "host": "153.10.110.81", 
        "user-identifier": "Mozilla/5.0 Gecko/20100101 Firefox/64.0", 
        "method": "PUT", 
        "status": 500, 
        "referrer": "http://www.google.com/"
    }
]'

For complete Parseable API documentation, refer to Parseable API Ref Docs.

Live Demo

You can also try out Parseable on our https://demo.parseable.io. Credentials to login to the dashboard are parseable / parseable.

Contributing

Refer to the contributing guide here.

Contributors