name attribute for objects and GLTF / GLB import/export
JBatUN opened this issue · 17 comments
First and foremost, excellent work - and thank you.
We are working on a 3d visualization (maps, graphs, etc.) library that leverages rgl to generate data-driven scene files for editing in Blender. There are a few features that would be very helpful for the capability:
1.] When creating an object (i.e. points, lines, segments, shapes, etc.), it would be ideal if objects can be given a name/id and other data attributes from the data source. For example: rgl.points(x, y, z, id = countries, data = meta, color ="lightgray"). The goal would be that when the scene is exported (Obj) and opened in a 3d authoring tool (Blender), all data generated objects are identifiable in the outliner / scenegraph.
2.] Considering the growing use of GLTF / GLB as a format for 3d scenes, would you consider adding the capability as a feature request? This would make it possible to leverage all aspects of the scene such as materials and textures.
Thank you for your consideration.
Every object receives a unique integer id. You could associate strings with these in various ways depending on your needs, but I think that's out of scope for rgl.
I wasn't familiar with the GLTF / GLB format, but I've just taken a look at the spec at https://www.khronos.org/registry/glTF/specs/2.0/glTF-2.0.html .
I'd certainly welcome a contribution to read & write scenes in that format, though it might make sense to put it in a separate package at least to start. Let me know if you are interested in working on it.
I'd like to leave this open, for the GLTF/GLB suggestion.
I'd like to chime in on point 1: There's a lot of value in being able to assign a user-specific ID (integer or otherwise) to each object, in particular for package authors using rgl to construct scenes. Allowing a package author to specify their own ID to a particular object allows for a much easier automatic programmatic manipulation of a scene, without manually keeping track of the IDs associated with each object. I (as a package author) can try to keep track of the individual integer IDs in some session/rgl device specific environment, but that can get messy real quick (particularly because you can open any number of simultaneous rgl devices, and now you're relying on the user not clearing/changing these hidden environment variables).
For example, in rayshader I have a workaround that creates a "user ID" by setting the ambient color to a specific value, which only works because I forgo using rgl's lighting system entirely. This enables me to do things like "clear all the lines associated with data in the scene, but keep lines representing axes" or "remove the triangle mesh associated with a scale bar but keep all the other triangle meshes", or "remove all meshes/triangles/lines associated with water but keep everything else" without manually tracking every ID associated with every scene. I do have to reconstruct this information from the material info every single time the user performs one of these actions, however. As a package developer, having rgl be able to optionally track the "context" of the displayed objects internally would simplify this process immensely.
I can see why you might consider it out of scope, but a single "user_id" integer/factor that prints with rgl.ids() would be quite useful for anyone writing a package that heavily uses rgl for complex, interactive visualizations. The current ID values feel more like an exposed internal rgl variable than something that the user should be interacting with directly: you have to manually query and save the IDs each time as they change every time you plot the same visualization in an R session. If a package developer instead is able to set their own invariants, it's much easier to write functions that pop/remove objects based on the actual content of the scene, rather than these arbitrary IDs.
I'm not convinced, but if you show me some code that you'd like to be able to write, I'll look at it.
Sure! Here's a real life example, using rayshader's water rendering function render_water(). This function calls a make_water() function that processes the elevation data to draw a layer of water on top of a 3D map. Note the get_ids_with_labels() function: this processes objects based on their actual context in the scene (everything related to water including lines), which makes manipulating the scene much easier than tracking transient IDs.
Note: I've editing out non-essential code from the actual implementation for this discussion.
render_water = function(heightmap, waterdepth=0, watercolor="lightblue",
zscale=1, wateralpha=0.5, waterlinecolor=NULL, waterlinealpha = 1,
linewidth = 2, remove_water = TRUE) {
if(remove_water) {
idlist = get_ids_with_labels()
remove_ids = idlist$id[idlist$raytype %in% c("waterlines", "water")]
rgl::pop3d(id=remove_ids)
}
make_water(heightmap/zscale,waterheight=waterdepth/zscale,wateralpha=wateralpha,watercolor=watercolor)
}Here's the make_water() function. This either draws two triangles that represent a flat surface of water (if the input elevation data has no NA values, which are interpreted as holes) or processes the elevation data to draw a triangle mesh. Note here I set the same ambient value "#000003" regardless of whether I use the triangle object or the surface object to draw the water surface. This is the hack I'm currently using to get the proposed functionality.
make_water = function(heightmap,waterheight=mean(heightmap),watercolor="lightblue",zscale=1,wateralpha=0.5) {
heightmap = heightmap/zscale
na_matrix = is.na(heightmap)
waterheight = waterheight/zscale
if(all(heightmap >= waterheight, na.rm=TRUE)) {
warning("No water rendered--all elevations above or equal to water level. Range of heights: ",
min(heightmap,na.rm = TRUE)*zscale,"-", max(heightmap,na.rm = TRUE)*zscale, ". Depth specified: ",
waterheight * zscale)
} else {
heightlist = make_water_cpp(heightmap, na_matrix, waterheight)
if(length(heightlist) > 0) {
fullsides = do.call(rbind,heightlist)
fullsides[,1] = (fullsides[,1] - nrow(heightmap)/2)
fullsides[,3] = -(fullsides[,3] + ncol(heightmap)/2 - 1)
fullsides = fullsides[nrow(fullsides):1,]
}
if(all(!na_matrix)) {
triangles3d(matrix(c(-nrow(heightmap)/2+1, nrow(heightmap)/2, -nrow(heightmap)/2+1,
waterheight,waterheight,waterheight,
ncol(heightmap)/2,-ncol(heightmap)/2+1,-ncol(heightmap)/2+1),3,3), lit=FALSE,
color=watercolor,alpha=wateralpha,front="filled",back="culled",texture=NULL,ambient = "#000003")
triangles3d(matrix(c(-nrow(heightmap)/2+1, nrow(heightmap)/2, nrow(heightmap)/2,
waterheight,waterheight,waterheight,
ncol(heightmap)/2,ncol(heightmap)/2,-ncol(heightmap)/2+1),3,3), lit=FALSE,
color=watercolor,alpha=wateralpha,front="filled",back="culled",texture=NULL,ambient = "#000003")
if(length(heightlist) > 0) {
rgl::triangles3d(fullsides,lit=FALSE,color=watercolor,alpha=wateralpha,front="fill",depth_test="less",texture=NULL,ambient = "#000003")
}
} else {
if(length(heightlist) > 0) {
rgl::triangles3d(fullsides,lit=FALSE,color=watercolor,alpha=wateralpha,front="fill",
texture=NULL,ambient = "#000003")
}
basemat = matrix(waterheight,nrow(heightmap),ncol(heightmap))
basemat[is.na(heightmap)] = NA
rgl.surface(1:nrow(basemat)-nrow(basemat)/2,1:ncol(basemat)-ncol(basemat)/2,basemat,color=watercolor,alpha=wateralpha,
lit=FALSE,texture=NULL,ambient = "#000003")
}
}
}The get_ids_with_labels() processes the rgl data to do the reverse mapping, taking ambient values and returning the scene description of those objects (I've edited out some of the non-essential parts of this rather verbose function). This function wouldn't be necessary with an internal rgl user ID (and I could regain use of the rgl lighting system).
get_ids_with_labels = function(typeval = NULL) {
ambient_encoder = c("#000001" = "surface","#000002" = "base","#000003" = "water",
"#000004" = "lines","#000005" = "waterlines","#000006" = "shadow",
"#000007" = "basebottom", "#000008" = "textline", "#000009" = "raytext",
"#000010" = "north_symbol", "#000011" = "arrow_symbol",
"#000012" = "bevel_symbol", "#000013" = "background_symbol",
"#000014" = "scalebar_col1", "#000015" = "scalebar_col2",
"#000016" = "text_scalebar", "#000017" = "surface_tris",
"#000018" = "path3d", "#000019" = "points3d", "#000020" = "polygon3d")
get_rgl_material = getFromNamespace("rgl.getmaterial", "rgl")
idvals = rgl::rgl.ids()
material_type = list()
material_properties = vector("list", nrow(idvals))
for(i in 1:nrow(idvals)) {
material_type_single = get_rgl_material(id=idvals[i,1])
material_type[[i]] = ambient_encoder[material_type_single$ambient]
#....other unrelated code
}
idvals$raytype = unlist(material_type)
retval = idvals
if(!is.null(typeval)) {
if(any(typeval %in% ambient_encoder)) {
retval = retval[retval$raytype %in% typeval,]
}
}
return(retval)
}Here's my ideal desired interface (to replace the above code in render_water()/make_water()) using character descriptions, which makes reading exactly what the function's doing much more clear. The only changes are replacing the ambient argument with a user_id argument in triangles3d and rgl.surface:
make_water = function(heightmap,waterheight=mean(heightmap),watercolor="lightblue",zscale=1,wateralpha=0.5) {
...
if(all(!na_matrix)) {
triangles3d(matrix(c(-nrow(heightmap)/2+1, nrow(heightmap)/2, -nrow(heightmap)/2+1,
waterheight,waterheight,waterheight,
ncol(heightmap)/2,-ncol(heightmap)/2+1,-ncol(heightmap)/2+1),3,3), lit=FALSE,
color=watercolor,alpha=wateralpha,front="filled",back="culled",texture=NULL, user_id = "water")
triangles3d(matrix(c(-nrow(heightmap)/2+1, nrow(heightmap)/2, nrow(heightmap)/2,
waterheight,waterheight,waterheight,
ncol(heightmap)/2,ncol(heightmap)/2,-ncol(heightmap)/2+1),3,3), lit=FALSE,
color=watercolor,alpha=wateralpha,front="filled",back="culled",texture=NULL, user_id = "water")
if(length(heightlist) > 0) {
rgl::triangles3d(fullsides,lit=FALSE,color=watercolor,alpha=wateralpha,front="fill",depth_test="less",texture=NULL, user_id = "water")
}
} else {
if(length(heightlist) > 0) {
rgl::triangles3d(fullsides,lit=FALSE,color=watercolor,alpha=wateralpha,front="fill",
texture=NULL, user_id = "water")
}
basemat = matrix(waterheight,nrow(heightmap),ncol(heightmap))
basemat[is.na(heightmap)] = NA
rgl.surface(1:nrow(basemat)-nrow(basemat)/2,1:ncol(basemat)-ncol(basemat)/2,basemat,color=watercolor,alpha=wateralpha,
lit=FALSE,texture=NULL, user_id = "water")
}
}
}
render_water = function(heightmap, waterdepth=0, watercolor="lightblue",
zscale=1, wateralpha=0.5, waterlinecolor=NULL, waterlinealpha = 1,
linewidth = 2, remove_water = TRUE) {
if(remove_water) {
rgl::pop3d(user_id="water")
}
make_water(heightmap/zscale,waterheight=waterdepth/zscale,wateralpha=wateralpha,watercolor=watercolor)
}But equally as useful for a package author (and potentially less annoying from an implementation point of view) would just be an integer ID, where the mapping is known to the author.
make_water = function(heightmap,waterheight=mean(heightmap),watercolor="lightblue",zscale=1,wateralpha=0.5) {
...
if(all(!na_matrix)) {
triangles3d(matrix(c(-nrow(heightmap)/2+1, nrow(heightmap)/2, -nrow(heightmap)/2+1,
waterheight,waterheight,waterheight,
ncol(heightmap)/2,-ncol(heightmap)/2+1,-ncol(heightmap)/2+1),3,3), lit=FALSE,
color=watercolor,alpha=wateralpha,front="filled",back="culled",texture=NULL, user_id = 1234)
triangles3d(matrix(c(-nrow(heightmap)/2+1, nrow(heightmap)/2, nrow(heightmap)/2,
waterheight,waterheight,waterheight,
ncol(heightmap)/2,ncol(heightmap)/2,-ncol(heightmap)/2+1),3,3), lit=FALSE,
color=watercolor,alpha=wateralpha,front="filled",back="culled",texture=NULL, user_id = 1234)
if(length(heightlist) > 0) {
rgl::triangles3d(fullsides,lit=FALSE,color=watercolor,alpha=wateralpha,front="fill",depth_test="less",texture=NULL, user_id = 1234)
}
} else {
if(length(heightlist) > 0) {
rgl::triangles3d(fullsides,lit=FALSE,color=watercolor,alpha=wateralpha,front="fill",
texture=NULL, user_id = 1234)
}
basemat = matrix(waterheight,nrow(heightmap),ncol(heightmap))
basemat[is.na(heightmap)] = NA
rgl.surface(1:nrow(basemat)-nrow(basemat)/2,1:ncol(basemat)-ncol(basemat)/2,basemat,color=watercolor,alpha=wateralpha,
lit=FALSE,texture=NULL, user_id = 1234)
}
}
}
render_water = function(heightmap, waterdepth=0, watercolor="lightblue",
zscale=1, wateralpha=0.5, waterlinecolor=NULL, waterlinealpha = 1,
linewidth = 2, remove_water = TRUE) {
if(remove_water) {
rgl::pop3d(user_id=1234)
}
make_water(heightmap/zscale,waterheight=waterdepth/zscale,wateralpha=wateralpha,watercolor=watercolor)
}As a non-package related data analysis example, let's say we're plotting three datasets via points3d(). Here, the user could specify a label describing the data as the year, and then pop the data off using that description (rather than the rgl ID). What's nice about this interface is it leads to more reproducible code: unlike a hardcoded ID, you could run this code multiple times within the same R session and still get the same result, and it's more obvious what you're trying to remove from the plot.
data_2019 = matrix(rnorm(300,mean=0,sd=0.1),ncol=3)
data_2020 = matrix(rnorm(300,mean=1,sd=0.1),ncol=3)
data_2018 = matrix(rnorm(300,mean=-1,sd=0.1),ncol=3)
points3d(data_2020, user_id = "2020")
points3d(data_2019, user_id = "2019")
points3d(data_2018, user_id = "2018")
pop3d(user_id = "2018")Basically, right now rgl's ID interface provides the basics for describing the graphical elements and accessing ID values for removing objects, but provides no contextual link to the underlying data. I emphasized popping objects off the scene because it's the most simple use case, but I also use this function to get each object's context to reconstruct and write OBJ files with custom materials and build pathtraced scenes with custom materials that depend on the object, so I've found many use cases for this type of workflow.
Making the id a material property would mean it would be pretty easy. Not all objects have material properties, but I think you typically wouldn't need an id on the ones that don't, because you don't usually have many of them: lights, viewpoints, and subscenes don't have materials.
I think a string id wouldn't be too hard to manage.
Okay, I'll think about doing this.
Okay, I've made a first attempt at the user_id suggestion on the user_id branch. Please try it out, and make suggestions. It is currently implemented as a material property on objects that have those.
-
I don't like the name
user_id: it sounds as though it is the id of the user, not an id applied by the user to the object.
In addition, there are other potential uses for this besides just an id. I haven't thought of a better one yet, but possibilities are "tag", "meta", "info". Other suggestions are welcome. -
It hasn't been tested very carefully yet. The most likely problems will come from functions that accept material properties in their ... arguments but don't pass them on to all the objects they create.
decorate3d()was an example of this. -
It accepts a single string value. Arguments for allowing other kinds of things could be made.
Agreed on user_id: I'm a fan of either tag or meta. I've run some initial tests with some data analysis workflows and pop3d() and it seems to be working great, but I'll write a rayshader branch to use this new feature and see if I run into any issues.
I've switched the name to tag. Still to do:
- add a function to retrieve ids from tags
- update the Javascript related functions to support selection of objects by tag
Reopened to keep suggestion of GLTF support.
I've just started the https://github.com/dmurdoch/rgl2gltf repository for a package to implement glTF/GLB support. At the moment, it can read geometry information and convert it to an rgl mesh. I plan to add materials and write methods.
I'm not sure that this belongs in rgl itself; there's already a fair amount of code there.
https://github.com/dmurdoch/rgl2gltf is now good enough to close this issue.
https://github.com/dmurdoch/rgl2gltf is now good enough to close this issue.
FYI, I just finished an initial version of rayshader with the new tag feature and it works great! No problems and really cleaned up and simplified a good deal of code. Thanks! Any planned timeline on the next CRAN update?
I'm planning to integrate some parts of rgl2gltf into rgl, but I'm not sure how much yet, so there won't likely be a new release before January unless CRAN asks for one.
Actually, things went more quickly than I thought they would, so I am hopeful to have a new release in the next week or two. I have run reverse dependency tests, and have identified one problem that I need to address: The Rpdb package assumes that ids3d will return two columns, and gives an error with the revised version.
There are several possible fixes for this:
- I could add an argument to
ids3d()to request thetagcolumn, defaulting toFALSE. - I could drop the
tagcolumn fromids3d()and add a new function to return it. - I could let the return value depend on whether or not there were any non-empty tags.
- I could inform the
Rpdbmaintainer that their package needs changes.
If I do 1 or 2, then your package requesting tags will not work with older versions of rgl. But I think you will need a version dependency anyway, so maybe this isn't a big issue.
If I do 1, then either you would need to add tags = TRUE to all calls to ids3d, or write your own wrapper function.
If I do 2, you would need to change your calls to that function, and I would need to write a new help page. If you need both the type and the tag, your code would be more complicated.
I don't like 3, because it makes return values less predictable. I don't like 4, because Rpdb has been stable for many years, so I shouldn't force an update.
Number 1 (what you implemented) works for me: I'm indeed going to be introducing a version dependency so explicitly requesting tags when needed is no issue.
It's on CRAN, on the third try: 0.108.3.