datakit: A Go repository from IluckySi

Website • Doc

Read this in other languages.

DataKit is an open source, integrated data collection agent, which provides full platform (Linux/Windows/macOS) support and has comprehensive data collection capability, covering various scenarios such as host, container, middleware, tracing, logging and security inspection.

Key Features

Support collection of metrics, logging and tracing
Fully support Kubernetes ecology
Pipeline: Simple structured data extraction
Supports third-party data import:
- Telegraf
- Prometheus
- Statsd
- Fluentd
- Filebeats
- Function
- Tracing
  - OpenTelemetry
  - DDTrace
  - Zipkin
  - Jaeger
  - Skywalking

Changelog

All DataKit changelog refers to here.

Minimal Requirements

OS	Arch	Install Path
Linux Kernel 2.6.23+	amd64/386/arm/arm64	`/usr/local/datakit`
macOS 10.12+(Why)	amd64	`/usr/local/datakit`
Windows 7+/Server 2008R2+	amd64/386	64-bit：`C:\Program Files\datakit` 32-bit：`C:\Program Files(32)\datakit`

Install DataKit

We can directly obtain the DataKit installation command from guance cloud. Most of the installation commands seems like that:

Linux & Mac

DK_DATAWAY="https://openway.guance.com?token=<YOUR-TOKEN>" bash -c "$(curl -L https://static.guance.com/datakit/install.sh)"

Windows

$env:DK_DATAWAY="https://openway.guance.com?token=<YOUR-TOKEN>";Set-ExecutionPolicy Bypass -scope Process -Force; Import-Module bitstransfer; start-bitstransfer -source https://static.guance.com/datakit/install.ps1 -destination .install.ps1; powershell .install.ps1;

Kubernetes DaemonSet

For more documentations about DataKit installation, see here.

Build From Source

DataKit building relies on some external tools/libs, we must install them all before compile the source code.

We recommend to build source on Ubuntu 20.04+, other linux distribition may failed to install these dependencies. We do not support build DataKit on Windows.

Please build the project with make, we haven't testing with Golang/VSCode IDEs

Setup Golang

Install and setup Golang(1.18.3+):

export GOPRIVATE=gitlab.jiagouyun.com/*
export GOPROXY=https://goproxy.cn,direct
export GOPATH=~/go            # depends on your local settings
export GOROOT=~/golang-1.18.3 # depends on your local settings
export PATH=$GOROOT/bin:~/go/bin:$PATH

Install other tools

!!! Do not install these dependencies under datakit source code dir.

make: apt-get install make
gcc: apt-get install gcc
gcc-multilib: apt-get install -y gcc-multilib
tree: apt-get install tree
packr2: go install github.com/gobuffalo/packr/v2/packr2@v2.8.3
goyacc: go install golang.org/x/tools/cmd/goyacc@latest
lint related:
- lint: go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.46.2
eBPF related:
- clang 10.0+: apt-get install clang
- llvm 10.0+: apt-get install llvm
- kernel headers
  - apt: apt-get install -y linux-headers-$(uname -r)
Documentation exporting:
- waque 1.13.1+

Build

Clone code

$ mkdir -p $GOPATH/src/gitlab.jiagouyun.com/cloudcare-tools
$ cd $GOPATH/src/gitlab.jiagouyun.com/cloudcare-tools

$ git clone https://github.com/GuanceCloud/datakit.git   # may be blocked by GFW
$ git clone https://jihulab.com/guance-cloud/datakit.git # jihulab mirror

$ cd datakit

Building

make

If building ok, all binaries are generated under dist:

dist
├── [4.0K]  datakit-linux-amd64
│   ├── [ 72M]  datakit
│   └── [4.0K]  externals
│       ├── [ 14M]  logfwd
│       └── [10.0M]  oracle
├── [4.0K]  local
│   ├── [ 26M]  installer-linux-amd64
│   └── [ 228]  version
└── [4.0K]  standalone
    └── [4.0K]  datakit-ebpf-linux-amd64
		        └── [ 38M]  datakit-ebpf

We can build all platforms(Linux/Mac/Windows) with following command:

make testing

Basic Usage

We can use help command to see more usage of DataKit:

datakit help

# Or

./dist/datakit-linux-amd64/datakit help

Contributing

Before contributing, check out some guideline of DataKit:

Read architecure introduciton
Read development guideline

Full Documentation

For full documents of DataKit, see

DataKit Doc

IluckySi/datakit