How can I add detail layer at runtime using GDScript?

still using Godot 3 :)

I don't think you can do that in a straightforward way, there is no direct API to do this. It was only developped for in-editor usage, so you have to do several steps and investigate.
You may check how the plugin does it in the editor (ignoring the editor-specific logic):

godot_heightmap_plugin/addons/zylann.hterrain/tools/detail_editor/detail_editor.gd

Line 106 in 9c6e5c6

func _add_layer():

The idea is that a detail layer is 2 things:

A density map in HTerrainData (currently with _edit_add_map, which was only thought to be done in editor at the time)
A node using that density map

So adding a detail layer means to create a new map (texture) to HTerrainData in the DETAIL channel, and then add a detail layer node which has a layer_index equal to the index of that map. Multiple detail layer nodes can use the same density map.

The code might be this:

	var node := HTerrainDetailLayer.new()
	var map_index := terrain.data._edit_add_map(HTerrainData.CHANNEL_DETAIL)
	node.layer_index = map_index
	terrain.add_child(node)

But if you want to re-use an existing density map:

	var node := HTerrainDetailLayer.new()
	node.layer_index = map_index
	terrain.add_child(node)

(untested)

Note that this doesnt actually create a texture yet, but I think that will happen on the first call to terrain_data.get_texture(HTerrainData.CHANNEL_DETAIL, map_index). So if you procedurally generate the map, you might want to first modify the image before creating the node, with terrain_data.get_image(HTerrainData.CHANNEL_DETAIL, map_index). This is just a small optimization so it doesn't try to upload multiple times.

So I created a scene containing only a HTerrainDetailLayer node, where I already set the grass texture.

Then I have this code in my world generation script:

	# Create terrain node
	var terrain = HTerrain.new()
	
	# Shader
	terrain.set_shader_type(HTerrain.SHADER_CLASSIC4_LITE)
	terrain.set_shader_param("u_ground_uv_scale", 10.0)
	
	# Set Data
	terrain.set_data(terrain_data)	
	terrain.set_texture_set(texture_set)

	terrain.name = "rwg_terrain"
	add_child(terrain)
	
	# Detail Layer	
	var node = Detail_layer.instance()
	var map_index = terrain_data._edit_add_map(HTerrainData.CHANNEL_DETAIL)
	node.layer_index = map_index
	terrain.add_child(node)
	var detailImg = terrain_data.get_image(HTerrainData.CHANNEL_DETAIL, map_index)
	detailImg.lock()
	
	for x in range(img_width):
		for y in range(img_width):
			detailImg.set_pixel(x, y, Color(1,1,1,1)) # make grass available everywhere for now
	
	terrain_data.set_image(HTerrainData.CHANNEL_DETAIL, detailImg, map_index)
	terrain_data.get_texture(HTerrainData.CHANNEL_DETAIL, map_index)
	detailImg.unlock()

and I added this set_image function inside the hterrain_data.gd script:

func set_image(map_type: int, img : Image, index := 0):
	_maps[map_type][index].image = img

But nothing happens so far. Do you have another hint for me, I think I'm still doing something wrong that I can't get?

Not really sure what could be wrong here, other than the method you added to hterrain_data.gd. You should not need to do that, because _edit_add_map does it already, so all you need is to get the image.

I tried the following with the current version and it worked:

extends Node

const HTerrain = preload("res://addons/zylann.hterrain/hterrain.gd")
const HTerrainData = HTerrain.HTerrainData
const HTerrainDetailLayer = preload("res://addons/zylann.hterrain/hterrain_detail_layer.gd")
const HTerrainTextureSet = HTerrain.HTerrainTextureSet

func _ready():
	var terrain_data := HTerrainData.new()
	terrain_data.resize(513)
	
	var detail_map_index := terrain_data._edit_add_map(HTerrainData.CHANNEL_DETAIL)
	var detail_map_image := terrain_data.get_image(HTerrainData.CHANNEL_DETAIL, detail_map_index)
	
	for y in detail_map_image.get_height():
		for x in detail_map_image.get_width():
			# Make some recognizable pattern to confirm that the result comes from here
			var d := maxf(cos(x * 0.1) + sin(y * 0.1), 0.0)
			detail_map_image.set_pixel(x, y, Color(d, d, d, 1.0))
	
	var terrain := HTerrain.new()
	terrain.set_data(terrain_data)
	
	var texture_set := HTerrainTextureSet.new()
	texture_set.insert_slot(0)
	texture_set.set_texture(0, HTerrainTextureSet.TYPE_ALBEDO_BUMP, load("res://icon.svg"))
	terrain.set_texture_set(texture_set)
	
	var detail_layer := HTerrainDetailLayer.new()
	detail_layer.layer_index = detail_map_index
	terrain.add_child(detail_layer)
	
	add_child(terrain)