Best practices documentation page

[wsargent]

• Wrap blindsight loggers with a feature flag solution that lets you turn logging on and off at runtime.
• Use conditional logging with tracer bullet logging.
• Write diagnostic logs out to /tmpfs, using a rolling file appender. On exception, you can take a snapshot of the debug/trace logs.
• If you write debug and trace logging, keep it and make it useful for production.
• Do not include sensitive data in your logs.
• Consider putting a logging schema together.
• Use terse-logback for the appenders.

Rather than build an in-memory ringbuffer API and have codde for it, may be better to simply use tmpfs and write to a file with rotation.

Especially since debugging/tracing data can be written out continuously via logstash-logback-encoder in a better binary format than raw JSON.

https://github.com/FasterXML/jackson-dataformats-binary#binary-formats-included

tmpfs may use swap space if necessary, so it will respond to memory pressure if the system is under load.

https://unix.stackexchange.com/questions/424341/mount-var-logs-as-tmpfs-with-help-of-overlayfs-to-save-changes-sometimes/424510#424510

https://blog.dustinkirkland.com/2016/01/data-driven-analysis-tmp-on-tmpfs.html

Using a tmpfs filesystem sidesteps the problem of having disk IO be the bottleneck, and means that the debug/trace data is available across process, while also being ephemeral. If there's an exception, look at those files.