Derive progress % from logs
Closed this issue · 0 comments
musicformellons commented
I've successfully ran planetiler on a planet file using large RAM machines. I'm now trying to save some costs by running on smaller RAM machines. The process still runs but using like 1.5% of CPU, free RAM is at 1.5%. I'm wondering if this process is still making progress. Can you help me interpret the logs as in 'what defines best overall progress', and can I somehow derive an expected time to completion?
Here is a recent fragment of the logs:
{"level":"info","time":1709616588,"message":"stdOut log: cpus: \u001b[34m0\u001b[0m gc: 0% heap: 817M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616588,"message":"stdOut log: read( 0%) -> (11/13) -> process( 1% 0%) -> (408/53k) -> write( 0%)"}
{"level":"info","time":1709616598,"message":"stdOut log: \u001b[m\u001b[0m29:25:27 INF [osm_pass2] - nodes: [ 16B 100% 0/s ] 226G ways: \u001b[32m[ 13M 1% 85/s ]\u001b[0m rels: [ 0 0% 0/s ] features: [ 700M 491/s ] 55G blocks: [ 2M 86% 0/s ]"}
{"level":"info","time":1709616598,"message":"stdOut log: cpus: \u001b[34m0\u001b[0m gc: 0% heap: 822M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616598,"message":"stdOut log: read( 0%) -> (11/13) -> process( 1% 0%) -> (909/53k) -> write( 0%)"}
{"level":"info","time":1709616609,"message":"stdOut log: \u001b[m\u001b[0m29:25:37 INF [osm_pass2] - nodes: [ 16B 100% 0/s ] 226G ways: \u001b[32m[ 13M 1% 76/s ]\u001b[0m rels: [ 0 0% 0/s ] features: [ 700M 589/s ] 55G blocks: [ 2M 86% 0/s ]"}
{"level":"info","time":1709616609,"message":"stdOut log: cpus: \u001b[34m0\u001b[0m gc: 0% heap: 824M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616609,"message":"stdOut log: read( 0%) -> (11/13) -> process( 1% 0%) -> (439/53k) -> write( 0%)"}
{"level":"info","time":1709616619,"message":"stdOut log: \u001b[m\u001b[0m29:25:47 INF [osm_pass2] - nodes: [ 16B 100% 0/s ] 226G ways: \u001b[32m[ 13M 1% 86/s ]\u001b[0m rels: [ 0 0% 0/s ] features: [ 700M 568/s ] 55G blocks: [ 2M 86% 0/s ]"}
{"level":"info","time":1709616619,"message":"stdOut log: cpus: \u001b[34m0\u001b[0m gc: 0% heap: 823M/9.5G direct: 4M postGC: \u001b[34m899M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616619,"message":"stdOut log: read( 0%) -> (11/13) -> process( 1% 1%) -> (1.5k/53k) -> write( 0%)"}
{"level":"info","time":1709616629,"message":"stdOut log: \u001b[m\u001b[0m29:25:58 INF [osm_pass2] - nodes: [ 16B 100% 0/s ] 226G ways: \u001b[32m[ 13M 1% 108/s ]\u001b[0m rels: [ 0 0% 0/s ] features: [ 700M 795/s ] 55G blocks: [ 2M 86% 0/s ]"}
{"level":"info","time":1709616629,"message":"stdOut log: cpus: \u001b[34m0\u001b[0m gc: 0% heap: 820M/9.5G direct: 4M postGC: \u001b[34m899M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616629,"message":"stdOut log: read( 0%) -> (11/13) -> process( 0% 1%) -> (1k/53k) -> write( 0%)"}
{"level":"info","time":1709616639,"message":"stdOut log: \u001b[m\u001b[0m29:26:08 INF [osm_pass2] - nodes: [ 16B 100% 0/s ] 226G ways: \u001b[32m[ 13M 1% 79/s ]\u001b[0m rels: [ 0 0% 0/s ] features: [ 700M 484/s ] 55G blocks: [ 2M 86% <1/s ]"}
{"level":"info","time":1709616639,"message":"stdOut log: cpus: \u001b[34m0\u001b[0m gc: 0% heap: 826M/9.5G direct: 4M postGC: \u001b[34m905M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616639,"message":"stdOut log: read( 0%) -> (11/13) -> process( 1% 1%) -> (1.1k/53k) -> write( 0%)"}
Another thing is that I'm running from docker container. I see instructions just mentioning running on straight java. Anything against running in a container (I do have the impression performance is worse than what I see in your performance table...)?