Workflow from DEM to QM
colek42 opened this issue · 32 comments
Thank you for making this open source and your informative talk. We are looking to implement a workflow for world topo coverage using QM. What does your workflow look like going from DEM to a fully tiled set? Right now we are unsure how to best create the triangles from DEMS. Thank you again for you work.
I'm also in the process of using this module to create a global tileset from SRTM+ data. Here's my code: https://github.com/rolker/cesium-tools
The base SRTM data can be found here: ftp://topex.ucsd.edu/pub/srtm15_plus/
My goal is to create a few levels of topography and bathymetry worldwide, with an area of high resolution data around an area of interest.
I hope this helps!
We used commercial software to create the triangulation/mesh. But there are a few open source options out there (Grass Gis for instance).
For creating a new mesh from a regular grid you usually first create contour lines and then triangulate.
Also you should make sure you adapt your mesh density with the level of detail (around 5000 nodes max per tile) . You can either simplify existing meshes or create fewer contour lines.
My workflow involves loading several meshes in postgis and cutting right through it. You will need to re triangulate some parts at the tile edges. But that's pretty much it.
The initial project where this code was created can be found here:
https://github.com/geoadmin/3d-forge
Note that the lighting is not correct yet, as it still creates unit vector pointing in a wrong direction.
As the triangles winding order is not ensured, we need to compute both and keep the correct one.
Also to generate the geodetic profile make sure you use GlobalGeodetic, I saw this wasn t the case.
https://github.com/loicgasser/quantized-mesh-tile/blob/master/quantized_mesh_tile/global_geodetic.py
I have some hack-a-thon code I wrote some time ago that uses the 2nd derivative of the height to select areas of interest. https://gist.github.com/colek42/f071b3c259bf118680ce90d253812b6d
It is very messy and probably buggy as well --- it was written during a hack-a-thon.
A good tool for handling meshes is http://meshlab.sourceforge.net/
@colek42 thank you, I will have a look.
@loicgasser I'm working on porting your code to golang and creating an entire workflow. I should have enough to make my repo public in a couple weeks. It is not a funded project, so it will take a bit of time to get done.
Great! I would be intreseted even if my GO coding skills are suboptimal at the moment.
I am working on combining this lib with https://github.com/WoLpH/numpy-stl to create 3D stl files for 3d prints from quantized mesh tiles. If you have a suggestion, you're welcome.
@loicgasser I was going to use https://github.com/fogleman/simplify for simplification, stl generation from qm would probably fit well into the workflow. I am going to put a few hours in this weekend to get my tile loader working. I was going to start with HGT files from the SRTM dataset unless you have any other suggestions. I would like to stay away from GDAL due to memory leaks in the current c bindings and the complication it adds to the build process. If you want I'll give you read/write access to the repo after that portion is finished.
Hi, yes I d be interested to see that. Thanks for the pointer. SRTM is a good start. My idea concerning stl was to use existing quantized mesh tiles and combine / merge / transform them into stl files. An easy and cheap way to do it. I ll put it out there when it s ready as well.
I´m playing around with https://github.com/jdugge/GridToTIN (cloned to https://github.com/umeier/GridToTIN). Still a lot of work to do, but looks interesting.
Hey Loic. Great work on this library.
I have files in .tif and am aiming for quantized mesh output. Any suggestions or detailed steps would be fantastic.
Should I use Grass GIS and then your library?
Yes anything that can perform the triangulation.
@umeier did you get anything out of GridToTIN?
It seems that you already did some work compared to the original version but does it work ? What is there left to do ?
When you have time @loicgasser, it would be immensely useful if you could explain how to perform the triangulation in steps. It would be amazing for people like me who are new to manipulating terrain but want to work with the output. There is not much information out there in truth.
I think from there I will be able to use this library.
Sorry, had no time to play with GridToTin in depth.
@slandersson look for the Garland/Heckbert-Algorithm.
Okay I have had some success in generating TINs using meshlab
Steps are:
-
gdal: convert .tif to .asc
-
meshlab: apply 2 filters (Invert faces orientation and Simplification: Quadric Edge Collapse) [see http://imgur.com/a/FEYgW ignoring the first box]. Seems to be fairly nice.
-
Convert .ply to WKT with a small script I wrote so it can be read by this library
-
???
I think I am just missing the tiling. At what stage should I do it and how @loicgasser , @colek42 ?
Hi, there are some explanation about the tiling scheme in this issue
geoadmin/3d-forge#99
Now that you have generated the mesh, you just need to cut it using the tiling scheme described above.
So I guess you want be using anything that can perform this operation.
I used postgis, for that, but you may have another option. I just had to fix a bug where the height computation is incorrect on tile edges. Here is a plpsql function that takes care of it.
https://github.com/geoadmin/3d-forge/blob/master/forge/sql/_interpolate_height_on_plane.sql
The script I used is:
https://github.com/geoadmin/3d-forge/blob/master/forge/lib/tiler.py
It is a bit of a mess but I know some people got it to work for them.
I need to update this project and start using this module for instance. The best would be a docker ready to use. ;)
At the moment I´m playing around with a python wrapper around Garland´s terra program. Not documented and tested and definitely still a lot to do,
https://github.com/umeier/pyterra
Thank you for the info. Always eager to find new tools.
Hi! We recently published C++ command line tool https://github.com/heremaps/tin-terrain
which implements the whole flow:
raster (tif) -> mesh (obj) -> quantized-mesh tile-set + Cesium.js example
it implements Garland´s terra algorithm for mesh simplification + one hirechical algorithm which provides even better quality.
@loicgasser @colek42 Any feedback appreciated!
@delfrrr I actually don't work with GIS much anymore. I wish this was around a couple years ago!
@delfrrr I have a similar workflow, but a terra-implementation in java, without looking any deeper in your code...
Do you solve the problem with possible gaps on the tile-edges, when neighbour-tiles don't share the same vertices?
That looks promising. I was working with LiDAR data. I'll let you know when I have the chance to test it.
@thiloSchlemmer we do buffer when mesh a tile and then cut a mesh; in most cases it works well;
@cleeus and @ylian-here can tell more;
While reading this thread I am not able to figure out a way to start from a raster DEM (such as a GeoTiff for example) and to end up with terrain tiles ready for Cesium.
I may be able to produce a *.stf file storing TIN out of my DEM by using some SAGA GIS tools, such as:
- http://www.saga-gis.org/saga_tool_doc/7.5.0/tin_tools_0.html
- http://www.saga-gis.org/saga_tool_doc/7.5.0/tin_tools_1.html
But they are not very well documented for the moment, even if they seem to produce something I can work with.
But then, I miss the step on how to import the TIN as geometries in the quantized-mesh-tile python framework:
Do you really need to store all the triangles in a PostGIS database?
If yes, what datatype are you using? One listed here: https://postgis.net/workshops/postgis-intro/3d.html , or others?
Or having them as files-on-disk is OK?
If yes, in which format(s) do I need to store them (.*ply, *.obj, *.stl, ...)?
And then, how to load them as convenient geometries as described in the tutorial: https://quantized-mesh-tile.readthedocs.io/en/latest/tutorial.html ?
Even better would be a third party python lib (or bindings to lower language libs) to directly feed the quantized-mesh-tile a raster DEM. Let us know if you know any. But that may stay a dream for some time I guess.
Sorry for the bunch of questions. I wish I could help, such as describing a workflow, but I have not (yet) been enough in contact with all these specific aspects and I miss some technical part, especially when dealing with binary data.
I ported Martini to Cython: pymartini. With it I can convert a 512x512 pixel heightmap to a mesh in about 25ms. (The algorithm only works on square heightmaps)
I'm also working on a faster quantized mesh encoder. I've found so far that a lot of the encoding steps can be parallelized well with numpy ufuncs, or with a little bit of Cython.
@kylebarron great! Thanks for the info!
I think I meant to add in the previous comment that depending on my progress trying to speed up encoding, I'd be happy to point out some suggestions for performance improvements, but I think a PR to integrate it here would probably be large/take a lot of time.
As an update, I finished porting the encoding portions of quantized-mesh-tile to use fast Numpy broadcasting. It encodes a mesh with 100k coordinates and 180k triangles in 20ms. That includes computing header data, including the bounding sphere and occlusion point, but assumes the input data is already topologically correct. Creating TerrainTopology and TerrainTile with the same dataset was taking 5-10 seconds with this package.
If you're interested I'm happy to talk more and give suggestions about where you can improve performance. My goals for my package are significantly simpler, essentially only mesh encoding, and assuming input data is correct.
Hey, that sounds great! Suggestions are always welcome and I plan on maintaining this small library.
just saw https://github.com/kylebarron/quantized-mesh-encoder . happy to see that the ecosystem is evolving.
Hey I've been working with QM recently. Everything goes all right, despite filling missing data. I mean My data doesn't fit perfectly for tiles and I have to manually add corner-data so that Cesium won't stop working, but it sometimes breaks integrity of data on end of tiles(Looks like different height of single corner from adjacent tiles) . Any suggestions how to fill missing data efficently?