GIS integration?
Closed this issue · 8 comments
I'd like to pull sun exposure into QGIS as a layer or, if necessary, ArcGIS Pro or ArcMap. Mapbox's documentation suggests this should be possible using WMTS but, from the ShadeMap API key email, it's unclear how the WMTS endpoint would be constructed or how to specify a date (and, if working with shadow layers, time). It's similarly unclear if there's a good way to retrieve shadow information an area of interest and save it out as a GeoJSON file for import as a layer (or GeoTIFF or any other supported format—I've no specific requirements on the details).
Before I start experimenting, I wanted to open this issue to see if anyone else has already identified an approach or if there might be development plans in this direction. If it matters, ShadeMap use on my part is for an educational (research) project.
There are some JavaScript plugins for QGIS and Python modules for wrapping JavaScript functions. So it might be possible to use the current ShadeMap API with a localhost key under PyQGIS or ArcPy. A better PyQGIS option might be QgsMapBoxGlStyleRasterSource.toRasterLayer(), though that's a QGIS internal API and hence there's no example of setting up the QgsMapBoxGlStyleConversionContext which seems needed to configure the raster source.
The video for the Chrome extension doesn't suggest it's relevant here and, while a .gpx or .kml can be imported for shade profiling, there doesn't appear to be support for going the opposite direction.
Excellent question.
WMTS relies on fetching pre-rendered or run-time computed map tiles from a server. ShadeMap's shade and sun exposure layers ARE NOT pre-rendered map tiles downloaded from a server.
Each time your drag or zoom, ShadeMap re-traces each pixels path towards the sun checking for collisions with terrain or buildings. The layers are not stored and downloaded, instead the visualization is computed and displayed by your browser in real-time.
Once the data is displayed, I could make it downloadable in a GeoTIFF format from your browser so it could be imported into another program, but this would be a manual process consisting of you dragging the map to the appropriate location and then pressing a download button, but possibly could also be automated using web browser automation tools such as puppeteer.
Let me know if a GeoTIFF download option would be of use to you and I apologize for the confusion.
Hi Ted, thank you for the quick reply! Nothing I'm doing requires high temporal resolution of sun exposure. So I, at least, can definitely get by with picking a middle year in the analemma cycle and manually downloading one GeoTIFF on an average day for each month.
My study area is about 1200 km² and 100 m spatial resolution is required, which sets a GeoTIFF lower bound of a few hundred kB compressed. I wouldn't expect these to be challenging, either to rendering in browser or to underlying DEM resolution. However, for a direct export like that it'd be nice to be able to specify the CRS and the raster cells' alignment to the CRS (nothing fancy as they match 100 m intervals in the CRS, e.g. one cell's coordinates are something like (119200, 209100) to (119300, 209200)). If it'd be an export locked to, say, WGS84 then closer to a 10 m resolution would be better for resampling into the project CRS and grids.
I'm guessing these considerations may mean raytracing for GeoTIFF would ideally operate with a separate, and likely higher, resolution than tracing for browser display? For example, if it was one screen pixel per GeoTIFF pixel at MapBox zoom level 13 a browser window of about 4700x4500 pixels would be required. Well above the 1080p of the highest resolution LCDs I have. That said, 1:1 at 1080p from ShadeMap would be an improvement over reinventing ShadeMap by running calculations on a raster stack from GRASS's r.horizon.
Could you test if I generated this GeoTIFF correctly? It's of downtown Seattle. I'm not sure if the quality is as good as what's on the website but I'm making good progress. Any feedback would be helpful.
I think I've got it. Here is another downtown Seattle along with zoom level 13 of the Cascades. Let me know if this quality is good enough for your use case.
Any feedback would be helpful.
Awesomely fast turnaround! And the 2.4 m-ish resolution at mid-latitudes looks good.
Both UGgemPzP.tiff and Wmh1cx_P.tiff are three band rasters with sparse histograms in the range of 0-255 on each channel, though, which is the same as what I get from using the download button to pull a shademap.tiff. I'd expect a sun exposure raster to have a single channel whose values correspond to hours, minutes, seconds or some other unit of time (given the browser mouseover, 32 bit floating point hours seems a natural default but 16 bit minutes would likely yield smaller files) and a fairly continuous histogram. I'm also not spotting a formula users can run through a raster calculator to convert display colors back to hours of sunlight.
It looks like if the browser zoom's set right the mouseover can be used to read three cells' hours of sunlight. One could pull those cells' RGB values in GIS, solve three equations in three unknowns, and use the results to build a raster calculator formula. Without testing more points it's unclear if it's a linear colormap, though, and it looks like colormap stretch might mean this time consuming process has to be repeated, or at least checked, for every download.
Am I missing an easier way? If something's forcing translation through a colormap I'm not spotting UX options to change to single band greyscale and lock units to something like tenths of hours (e.g. cell byte value 112 = 11.2 hours sunlit).
Since the saved shade layers are single channel booleans they mostly avoid colormap issues. Using false = 0 for shaded areas and true for lit ones strikes me as natural. But, from a GIS perspective, I'll suggest it's unusual to code a boolean as 0 and 255 instead of 0 and 1. Not a big deal and other folks might mention other use cases but, for all the years I've done GIS things, I can't think of a situation where the boolean convention wasn't 0 and 1.
Hey Todd,
Thanks for the feedback. I made some updates this morning and they are live on shademap.app. The changes I made are as follows:
For shadow and sun, I updated the values to 0 for shadow and 1 for sun. One pet peeve is that the file viewer on MacOS just displays them as black where before it would display the 255 value as white. Also, when I load a new file into app.geotiff.io, it displays 0 as white and 1 as black. Not sure why. Is there any program that gives you a decent preview of a 0/1 encoded GeoTIFF?
For sun exposure, the values are 0-255 and you multiple the value by 6 to get the number of minutes in the sun. For example, 17 would be 17*6=102 minutes or 1 hour and 42 minutes in the sun. I went the simplest route and encoded it into a single byte, but I can switch to a 16-bit encoding if the granularity is not enough. It just gets more complicated for me because I need to get the byte ordering right.
Also, one thing I wanted to mention: for sun exposure, ShadeMap slices the sun exposure time range into 32 even time periods, calculates the position of the sun and shadow location for each and then merges them together. This seems good enough since even in the arctic summer, each time period is 45 minutes. In a 12 hour day, the interval is half that, or 22.5 minutes. The number of iterations is configurable and I could add it to the Settings dialog if you need smoother data.
Let me know if I missed anything.
Cool cool. Looks good to me for encoding. Seems maybe a bit confusing the help text on save says the max is 255 since that's 25.5 hours. But people should be able to figure out the actual max is 240.
When I save sun exposure for my study area I get a GeoTIFF with 65 x 65 m cells, though. So could there be a setting to ask for saving at a zoom level that's higher than the display zoom level? Looks like I could zoom in, save something like 16² GeoTIFFs per month, and tile them. Since that'd be over 3000 manual steps to do all 12 months of a year an automated option would be a major time saver.
A base temporal resolution of six minutes seems fine to me. If someone wants greater accuracy then it's probably time to take a look at DEMs and see how often they're good enough to provide it. That starts to get into the angular width of the sun and how exactly sunlit is defined geometrically, which I suspect implies some O(p²) type casting of rays from p points in the raster cell to p points on the solar disc (I'm assuming right now it's one ray from cell center to sun center).
I also suspect it would be valuable to be able to have enough sun exposure time periods to hit the six minute resolution. The accuracy may not really be there to support the precision but oversampling in time and space does should reduce estimation bias at the cell level. Probably it averages out fairly well across a whole raster but that's not the same as averaging out over a specific handful of cells at a particular location of interest.
Is there any program that gives you a decent preview of a 0/1 encoded GeoTIFF?
Any GIS or tool for working with remotely sensed data should do fine as those are codes with at least a basic understanding of symbology and histogram stretching. Most photographically oriented tools I know expect .tiffs to have image data and are typically useless for viewing GeoTIFFs. Often they seem to fall apart in decompression or handling pixels which aren't 24 bit RGB. For example, Windows 10's File Explorer and Photo Viewer think 0/1 shade maps and single channel sun exposure aren't valid TIFF files.
IrfanView does ok by default with sun exposure and it's just shift+u to histogram stretch shademaps. There should be something with equivalent functionality for macOS but I don't know offhand.