onthegomap/planetiler

Derive progress % from logs

Closed this issue · 0 comments

I've successfully ran planetiler on a planet file using large RAM machines. I'm now trying to save some costs by running on smaller RAM machines. The process still runs but using like 1.5% of CPU, free RAM is at 1.5%. I'm wondering if this process is still making progress. Can you help me interpret the logs as in 'what defines best overall progress', and can I somehow derive an expected time to completion?

Here is a recent fragment of the logs:

{"level":"info","time":1709616588,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 817M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616588,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  0%) -> (408/53k) -> write( 0%)"}
{"level":"info","time":1709616598,"message":"stdOut log: \u001b[m\u001b[0m29:25:27 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%   85/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  491/s ] 55G   blocks: [   2M  86%    0/s ]"}
{"level":"info","time":1709616598,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 822M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616598,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  0%) -> (909/53k) -> write( 0%)"}
{"level":"info","time":1709616609,"message":"stdOut log: \u001b[m\u001b[0m29:25:37 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%   76/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  589/s ] 55G   blocks: [   2M  86%    0/s ]"}
{"level":"info","time":1709616609,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 824M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616609,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  0%) -> (439/53k) -> write( 0%)"}
{"level":"info","time":1709616619,"message":"stdOut log: \u001b[m\u001b[0m29:25:47 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%   86/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  568/s ] 55G   blocks: [   2M  86%    0/s ]"}
{"level":"info","time":1709616619,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 823M/9.5G direct: 4M postGC: \u001b[34m899M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616619,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  1%) -> (1.5k/53k) -> write( 0%)"}
{"level":"info","time":1709616629,"message":"stdOut log: \u001b[m\u001b[0m29:25:58 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%  108/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  795/s ] 55G   blocks: [   2M  86%    0/s ]"}
{"level":"info","time":1709616629,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 820M/9.5G direct: 4M postGC: \u001b[34m899M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616629,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 0%  1%) ->  (1k/53k) -> write( 0%)"}
{"level":"info","time":1709616639,"message":"stdOut log: \u001b[m\u001b[0m29:26:08 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%   79/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  484/s ] 55G   blocks: [   2M  86%   <1/s ]"}
{"level":"info","time":1709616639,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 826M/9.5G direct: 4M postGC: \u001b[34m905M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616639,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  1%) -> (1.1k/53k) -> write( 0%)"}

Another thing is that I'm running from docker container. I see instructions just mentioning running on straight java. Anything against running in a container (I do have the impression performance is worse than what I see in your performance table...)?