[FEATURE] Merge with already existing tiles
Opened this issue · 2 comments
Is your feature request related to a problem? Please describe.
I'm currently looking into ways to build in a very constrained system, as a good measure, I want to be able to work with 4GB Ram and 30 GB disk max.
To achieve this goal I'm trying to implement an incremental build, for example for Germany. Its available either as a full dump, 3.9G or as 16 dumps, one for each state (See GeoFabrik page)
Describe the solution you'd like
I would like to see a functionality like tilemaker --merge. This way it should be possible to download each state separately, process and merge on the go, using a script to loop for example.
My expectation is that this would need less resources for lookup data structures etc for a single iteration, I don't mind if this would take considerably longer (5 times longer runtime would be acceptable, since in this scenario processing - CPU - is way cheaper than storage)
Describe alternatives you've considered
I've considered using tilemaker, but the memory management isn't very transparent, it's not even capable to give an estimation or hints how much memory operations might cost, it just fails. And using some sort of swap file don't really work either, it doesn't seem to use it efficiently.
Additional context
There has been significant ground work in #639, this added the --append flag, but also left out the mbtiles format for now.
The main issue here is certainly not, as a code comment suggests, the insertion, but merging of overlapping tiles.
Additionally it might be desirable to implement #536 to be able to merge with preexisting tiles in a directory structure.
And while we are at it, this can also be a possible way to implement #658.
Are you trying to build Germany? Or Europe? The least I've seen work for the planet in one go is 16gb although you might be able to get away with even less. All of Europe might work with as little as 4gb.
I don't think it would be too hard to support an mbtiles merge mode, just want to see if we can support your use case with a single pass of planetiler and no merging.
My goal is to generate parts of Europe (Germany alone is working) within 6 hours on a very constrained system. this system is just a runtime Frankenstein of a GitHub Runner (Free Tier)...