retrieve_minified_logs-linux-v1 tool doesn't collect anything after upgrade to 0.4.11
matsuro-hadouken opened this issue · 8 comments
Since beginning when my node popup on network
{"PERSISTED_NODE_ID":"11afdb30f8d0994e","VERSION":"0.4.8"}
2021-01-21T22:44:52.726258281Z: INFO: Starting out of band catch-up
2021-01-21T23:10:56.140752441Z: INFO: Completed out of band catch-up
Everything was smooth until upgrade to 0.4.11
By absolute luck, without thinking of log loss I manage to grab data using
retrieve_minified_logs-linux-v1 utility.
At the end of concordium-testnet-node-532b...6f60.log got expected uptime which is
runtime:356
baked:6
After successful upgrade to 0.4.11 retrieve_minified_logs-linux-v1 stop retrieve anything, data in log always the same
{"VERSION":"0.4.11","PERSISTED_NODE_ID":"11afdb30f8d0994e"}
runtime:0
baked:0
I found another utility in concordium-software folder, which is concordium-node-retrieve-logs, by running this get healthy output, which is general log and doesn't have runtime: and baked:0 ( well expected )
My thoughts retrieve_minified_logs-linux-v1 pointed in to wrong path and can't get to the log from the active 0.4.11 container.
Baker never been offline for the long time, never experience any issues and never been jailed. The only time I stop node is for log collection porpoise, which takes approximately 1 minute or less since is no logs visible for to the utility anymore.
Above situation make hard to succeed in ongoing challenges, since logs submission with uptime is mandatory.
Update:
Based on previously successfully collected log from 0.4.8 noticed two events been collected
Block arrived and Baker: Finished bake block
Example:
INFO: Skov: Block 56abc8eb961818c66b696bc3942e9a3c7b47a21dd43931d8167b8547106022d4 (3) arrived
INFO: Baker: Finished bake block 3440c48b27e7045597b526fdd61dc971c3d391b91ff80e482ae67bdd19987c43
Using tool from concordium-software ( concordium-node-retrieve-logs ) folder I collect information from running container, then run simple filter:
cat "1$" | grep -v "is finalized" | grep -e "INFO: Skov: Block" -e "Finished bake block "
From this output I can see my baker and node work as expected, take been compared with explorer and match each other, timestamp correct and follow explorer.
Hi,
Both retrieve_minified_logs-linux-v1 and concordium-node-retrieve-logs use docker logs concordium-client for retrieving the logs, so a wrongly configured path is not the issue.
When upgrading the node, the logs are cleared.
The minified log tool will go through the container logs and will, for every hour of logs, increment the runtime counter.
So if you ran the minified log tool multiple times within the first hour of upgrading your
node, the results would look like what you described.
I can see that your node has been up and running for at least a day now. Please let us know if the issue persists.
Regarding the baker challenges, you should simply submit your logs from before
the upgrade along with the final logs.
Uptime counter on explorer reset every time node restart, in my case I have 99.9% uptime since initial deploy under timestamp
according to log collected from 0.4.8 version 2021-01-21T23:10:56.140752441Z: INFO: Completed out of band catch-up
Currently baker run for about 740 hours and retrieve_minified_logs-linux-v1 doesn't collect any information at all, show incorrect uptime null.
Here is full log after upgrade to 0.4.11
https://www.dropbox.com/s/a8hgpiu3pl5lkeq/concordium-testnet-node-full-after-upgrade-to-0.4.11.zip
Log above start:
2021-02-05T19:14:43.286145809Z: INFO: Starting up concordium_node version 0.4.11!
This is the time where my baker switched from 0.4.8 to .4.11
End of log:
2021-02-21T03:12:25.872558543Z: INFO: Skov: Block c927b ... ea47 (0) arrived
This is 367 hours 58 minutes.
Log collected previously on 0.4.8 shows runtime:356
367+356 = 723 hours ( log from 0.4.8 collected with retrieve_minified_logs-linux-v1 attached )
Currently, tool still doesn't collect anything.
Here in my submission #104, additional log file which collected by
cat "$1" | grep -v "is finalized" | grep -e "INFO: Skov: Block" -e "Finished bake block " just as example.
Expected behavior from retrieve_minified_logs-linux-v1:
Show approximately 367 hours of runtime.
concordium-testnet-node-532bf8c39bf576bc8ec6696bb44559f40be602c133f083b9fcacbc411f5b6f60.log
Hi again,
Thanks for taking the time to share this issue and your logs with us.
With your logs were able to find the bug causing this issue in the minified log tool.
We will soon release an updated version of the minified log tool.
Best regards,
- Kasper, Concordium
@Bargsteen Happy to help, took ages to collect and analyze all this data.
It will be nice to know, why all my work on recent intensive been rejected and didn't present in final score table.
This important for me to understand. Thank you !
@matsuro-hadouken: Your last comment is that about testnet3 rewards? If so, did you try to follow the reward feedback process to get clarity while it was still open?
@matsuro-hadouken: The updated version of the tool is available now (but not published to logs.md yet).
@matsuro-hadouken: Your last comment is that about testnet3 rewards? If so, did you try to follow the reward feedback process to get clarity while it was still open?
Was focused on technical challenge, maintain my baker perfectly, PR submissions according instructions and challenge rules, watching for bugs, made some reports. This is it.
@matsuro-hadouken: You are not answering my questions, so I am not even sure what your complain really is about.
Anyways, the feedback round for testnet3 rewards was open for at least a week where you could have asked testnet3 reward questions. Now it is closed, so there is nothing I can do there. If what you describe above is your testnet4 contribution, it sounds like you have a better chance to get rewards this time. I will internally forward your contribution in helping us solve the logging bug, but I don't promise anything, except our sincere thank you ;-)