Question about deformers

[fire]

How are you doing the deformers?

I'm looking into making deformers for skeletons via ml and found this project.

https://github.com/dgovil/MLDeform

My proposal is to deform skeletal meshes with MLDeform via https://github.com/openxla/iree and godot engine 4.0

[cloudofoz]

Hi, thank you for your interest in my project. Your idea seems quite interesting even if I can't be of a great help with ML actually. In my Godot addon there is a double way communication between a deformable mesh and all the deformers that can affect it. So the mesh always knows when it needs to be updated. Each mesh surface has an istance of SurfaceData ("dm_surface_data.gd"), a class that keeps the original vertex positions and calculate all the deformations from the deformers.

[cloudofoz]

A further note.

As far as I know, Godot 4 by itself doesn't allow to use a GPU shader just to deform vertices while keeping the original materials on the mesh. as they are.

This "forced" me to calculate the deformations on CPU.
For your project maybe it could be useful to understand if a simplified animation through ML still needs to be calculated on GPU to be of a great benefit over the current animation system.

[ChildLearningClub]

Hi @cloudofoz,

First of all thank you for your project.

It does seem that it is possible to still use a GPU shader while keeping the original materials mesh. The main negative being that it appears that it still requires the user to convert their StandardMaterial3D to ShaderMaterial from the Material drop down menu in the inspector tab (from what I understand it is not exposed as an option in gdscript).

From this point it is just a matter of writing shader code strings to the .shader.code file directly.

			# Apply additional shader code to each of the materials within the mesh
			for surface_material in multimesh_to_influence.multimesh.mesh.get_surface_count():
				# Create duplicate material to edit
				var mesh_surface_duplicate = multimesh_to_influence.multimesh.mesh.surface_get_material(surface_material).duplicate()

				var mesh_surface_shader = mesh_surface_duplicate.shader
				merge_shaders(mesh_surface_shader, shader_string_text.uniform_string(), "shader_type spatial;")
				merge_shaders(mesh_surface_shader, shader_string_text.vertex_string(), "void vertex() {")
				merge_shaders(mesh_surface_shader, shader_string_text.fragment_wind_string(), "void fragment() {")

				# Add edited duplicate shader to mesh
				mesh_surface_duplicate.shader.code = mesh_surface_shader.code
				# Add edited shader back to the surface it came
				multimesh_to_influence.multimesh.mesh.surface_get_material(surface_material).set_shader(mesh_surface_duplicate.shader)

func merge_shaders(mesh_surface_shader, insert_string : String, find_string : String) -> void:
	# Find the find_string in the shader.code
	index = mesh_surface_shader.code.find(find_string) + find_string.length()
	# Insert the insert_string at the position we found above
	modified_shader = mesh_surface_shader.code.left(index) + "\n" + insert_string + mesh_surface_shader.code.substr(index)
	# copy the modified shader code back to the original shader code
	mesh_surface_shader.code = modified_shader

The shader_string_text. : uniform_string() vertex_string() and fragment_wind_string() all just come from a file I have saved as a resource.

extends Resource
class_name BlastShaderStrings
## Description here

## Another desciption
var uniform : String = """
// Added shader code
uniform vec3 blast_pos = vec3(0.0);
uniform float blast_power = 0.5;
uniform float blast_radius = 20.0;
// End added code
"""

## And another desciption
var vertex : String = """
	// Added shader code
	vec3 world_vert = (MODEL_MATRIX * vec4(VERTEX, 1.0)).xyz; // model space to world space
	vec3 direction = world_vert - blast_pos;
	direction.y = 0.0;
	direction = normalize(direction);
	float dist = distance(blast_pos, world_vert);
	float power = smoothstep(blast_radius, 0.0, dist);
	direction = (vec4(direction, 1.0) * MODEL_MATRIX).xyz; // world space direction to model space
	VERTEX += direction * power * blast_power * (1.0 - UV.y);
	// End added code
	"""

var fragment_wind_motion : String = """
	// Added shader code
	// End added code
"""


## Put description here
func blast_uniform_string() -> String:
	return uniform

## And here
func blast_vertex_string() -> String:
	return vertex

## And here
func blast_fragment_wind_string() -> String:
	return fragment_wind_motion

Anyway, I know it's not the best solution but it works, and just thought I would share it in case you find any use of it. :)

2024-01-02.09-23-03.mp4

Your Plugin hard at work.

[cloudofoz]

Hi @ChildLearningClub, thank you for your interest in my project.

My first approach with this add-on involved a GPU shader. Unfortunately when I was investigating on Discord about the possibility of merging shaders, it seemed that no one was aware of the possibility of automatically converting a StandardMaterial3D to a ShaderMaterial, including me.

I've read your code and it seems like an interesting and workable way to solve the problem, thanks for sharing it with me!
Unfortunately I'm a bit busy at the moment, but I'll look into this further as soon as I can.

Best regards and good luck with your project,

Claudio

[cloudofoz]

Hi @ChildLearningClub,

I finally had some time today to study your helpful tip a little better:

• I confirm that, at the moment, it is not yet possible to convert a StandardMaterial3D into a ShaderMaterial via gdscript.

The problem is that the material conversion plugin in the source code uses StandardMaterial3D::get_shader_rid() to get the shader but, unfortunately, this method is not exposed in gdscript.

However, I discovered that there is an open proposal to solve this issue: godotengine/godot-proposals#8121

• There is also a small "design" issue with a "shader approach" in my add-on. It is currently designed so that you can add multiple deformers of different types to any mesh (e.g. spherical and bend deformers all together). This means being able to handle all these different situations through a shader as well.

Maybe I could just create a simplified GPU version as a separate add-on (DeformableMeshGPU).
This could be quite interesting to do, and it will be even more interesting when we are able to convert a StandardMaterial3D to a ShaderMaterial inside a gdscript.

Thanks again for your interest in my project.

Claudio

[ChildLearningClub]

Creating a separate “DeformableMeshGPU” plugin might be a good option 👍. I would be one of the first to use it ;).

Edit: also I do want to point out that my solution above does have issues. If you play back the video before and after the explosion you will notice that there is a slight change in the trees colors. I’m pretty sure that had more to do with the order of my shader code then anything, but just wanted to mention that.