Daily snapshot process logs are convoluted and bloated
Closed this issue · 2 comments
LesnyRumcajs commented
Issue summary
The _script_run
logs are displayed weirdly. Metrics are part of the _script_run
log (line by line, separated by commands), which is unhelpful given that there is a separate metrics file.
...
+ echo '2023-10-21 15:21:17 peer_tipset_epoch{PEER="12D3KooWQUH9UbPc2BUBKw5EMrrvYxTBkbYDKi3Be5UdnMXgYf8u"} -1'
+ IFS=
+ read -r line
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2023-10-21 15:21:17 peer_tipset_epoch{PEER="12D3KooWQUn93ou1Wy2USK2BHRRJmyR6aQxwwya89NU7p75bGTEH"} -1'
+ IFS=
+ read -r line
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2023-10-21 15:21:17 peer_tipset_epoch{PEER="12D3KooWQVy9g4snnHNfRtR1NepM4KtcSrN87mJSVFHJopZV2QR1"} -1'
+ IFS=
+ read -r line
++ date '+%Y-%m-%d %H:%M:%S'
...
The logs also end up being huge:
172K mainnet_2023-10-21T14_25_30_daemon.txt
45M mainnet_2023-10-21T14_25_30_metrics.txt
73M mainnet_2023-10-21T14_25_30_script_run.txt
Cleanup the daemon logs and reduce the amount of metrics collected.
Other information and links
samuelarogbonlo commented
@LesnyRumcajs, Considering gathering only the logs labeled as "warn" and "error" for the demon logs, and also look into increasing the collection interval with making anything complex.
LesnyRumcajs commented
INFO logs are OK as well. Otherwise, it'd be too hard to reason what is running there.