gosense
gosense
is a lightweight hardware monitor. It aims to deliver a small
statically linked binary (cf. 7.9 MB for go version go1.13 linux/amd64, 2.1 MB
stripped and compressed), which has lower runtime overhead and increased safety
when compared with alternatives.
Each monitor implements a cache with an asynchronous goroutine to periodically update itself. A shared REST server services requests from all monitors. The cache allows for low overhead constant time observations. This is important for some monitors (e.g. hard disks) because updates may take many minutes.
tl;dr
buck build //experimental/dwat/gosense
buck test //experimental/dwat/gosense/...
buck run //experimental/dwat/gosense &
curl --insecure localhost:8080/api/sys/sensors
fg
^C
Implementation
Two example monitors can be found here and here.
-
Creating a monitor is as simple as writing an
Update() ([]byte, error)
function, starting a cache with it, and registering an HTTP handler. See main.go for an example. -
Update functions may block, though functions which timeout will panic.
-
Update functions which fail to run successfully the first time will be ignored.
-
After an update returns an error the cache will not return stale data. Instead it will return the error packaged in a JSON object until the next successful update.
Design
-
Monitoring is a p0 capability. This means it must work no matter what else has failed, because we will depend on it to get the system back into a healthy state. Therefore there can be no external dependencies.
-
Monitors should have minimal overhead so as not to disturb the actual workload.
-
Non-responsive monitors should panic instead of failing silently.
-
Complex or unsafe monitoring should be done within a process to protect other monitors sharing the same http server process.
-
Consider using libraries or other API based data sources when available since they are often lower overhead and higher fidelity.
Security
TODO(dwat): Configure http server to load a certificate.
Performance
Benchmarks
go test -cpuprofile cpu.prof -memprofile mem.prof -benchmem -bench .
lmsensors library
goos: linux
goarch: amd64
pkg: experimental/dwat/gosense/pkg/lmsensors
Benchmark/c.Get()-56 790041016 1.50 ns/op 0 B/op 0 allocs/op
Benchmark/c.UpdateWithTimeout()-56 289 4102465 ns/o 921400 B/op 1883 allocs/op
PASS
ok experimental/dwat/gosense/pkg/lmsensors 3.118s
sensors
cli
classic fork and parse Note that the cost of the sensors
command is not measured (because it runs in
another process). TODO(dwat): measure.
goos: linux
goarch: amd64
pkg: experimental/dwat/gosense/pkg/lmsensors/classic
Benchmark/c.Get()-56 774242150 1.46 ns/op 0 B/op 0 allocs/op
Benchmark/c.UpdateWithTimeout()-56 192 6261189 ns/o 75265 B/op 247 allocs/op
PASS
ok experimental/dwat/gosense/pkg/lmsensors/classic 3.167s
Profiling
go tool pprof -http="$(hostname):9090" ./gosense memprofile.out
open localhost:9090 &
Even smaller binaries
Cf. https://blog.filippo.io/shrink-your-go-binaries-with-this-one-weird-trick/
go build -ldflags="-s -w"
upx ./gosense
ls -lh ./gosense
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2018
UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018
File size Ratio Format Name
-------------------- ------ ----------- -----------
6008832 -> 2158064 35.91% linux/amd64 gosense
Packed 1 file.
-rwxr-xr-x. 1 dwat users 2.1M Sep 24 19:18 ./gosense
References
Linux Monitor Sensors (lmsensors)
GitHub: https://github.com/mdlayher/lmsensors Wiki: https://wiki.archlinux.org/index.php/Lm_sensors FAQ: https://hwmon.wiki.kernel.org/faq