ChainSafe/forest-iac

Daily snapshot process logs are convoluted and bloated

Closed this issue · 2 comments

Issue summary

The _script_run logs are displayed weirdly. Metrics are part of the _script_run log (line by line, separated by commands), which is unhelpful given that there is a separate metrics file.

...
+ echo '2023-10-21 15:21:17 peer_tipset_epoch{PEER="12D3KooWQUH9UbPc2BUBKw5EMrrvYxTBkbYDKi3Be5UdnMXgYf8u"} -1'
+ IFS=
+ read -r line
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2023-10-21 15:21:17 peer_tipset_epoch{PEER="12D3KooWQUn93ou1Wy2USK2BHRRJmyR6aQxwwya89NU7p75bGTEH"} -1'
+ IFS=
+ read -r line
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2023-10-21 15:21:17 peer_tipset_epoch{PEER="12D3KooWQVy9g4snnHNfRtR1NepM4KtcSrN87mJSVFHJopZV2QR1"} -1'
+ IFS=
+ read -r line
++ date '+%Y-%m-%d %H:%M:%S'
...

The logs also end up being huge:

172K    mainnet_2023-10-21T14_25_30_daemon.txt
45M     mainnet_2023-10-21T14_25_30_metrics.txt
73M     mainnet_2023-10-21T14_25_30_script_run.txt

Cleanup the daemon logs and reduce the amount of metrics collected.

Other information and links

@LesnyRumcajs, Considering gathering only the logs labeled as "warn" and "error" for the demon logs, and also look into increasing the collection interval with making anything complex.

INFO logs are OK as well. Otherwise, it'd be too hard to reason what is running there.