Example for sending external texture/ draw data to clients

[tom-robo]

Hi @ggerganov

Thanks for your work on imgui-ws, it's a great implementation and should be very useful.

Just wondering if you could provide a quick example for properly rendering external opengl textures in imgui windows.

I can successfully render using normal imgui in a glfw window, however get a blank screen when viewing via imgui-ws.

I have looked at the setTexture function in the header file, however am unsure how this ties in with a texture already rendered using imgui and whether I would need to setTexture for imgui-ws when constructing the original opengl3 texture, etc.

I am also unsure if a rendered opengl texture inside a imgui window is counted inside ImGuiDrawData() when imguiWS.setDrawData(ImGui::GetDrawData()); is called. A quick example or a few pointers would be great.

Kind regards,

Tom.

[ggerganov]

Hi, you have correctly noted that the rendered texture in the ImGui server is not part of the ImGuiDrawData() that is streamed to the clients. It is however completely possible to add support for rendering images on the clients. I think the easiest way to explain this is to add an example and probably I will do that soon.

For now, here are a few tips:

• you need to stream your texture data (i.e. images) to your clients. This is tricky, because if the images that you render are big and change often, then you will generate a lot of traffic
• the setTexture function is your starting place. It can be used to stream texture data which is associated with some id. Underneath it uses the incppect library that helps mitigate and simplify the issues in the previous point
• currently the example streams only one texture - this is ImGui's font texture
• on the client side, when rendering vertex data with a texture, we need to bind the corresponding texture. However, currently we hardcoded only binding texture_id == 1 which is the font texture:

imgui-ws/src/imgui-ws.js

Lines 260 to 268 in 1eb94e0

	if (clip_x < this.canvas.width && clip_y < this.canvas.height && clip_z >= 0.0 && clip_w >= 0.0) {
	this.gl.scissor(clip_x, this.canvas.height - clip_w, clip_z - clip_x, clip_w - clip_y);
	// todo : temp hack
	if (texture_id == 1) {
	this.gl.activeTexture(this.gl.TEXTURE0);
	this.gl.bindTexture(this.gl.TEXTURE_2D, this.tex_font_id);
	}
	this.gl.drawElements(this.gl.TRIANGLES, n_elements, this.gl.UNSIGNED_SHORT, 2*offset_idx);
	}

• you should update this code to bind other textures that you have streamed via setTexture

Not sure if this makes a lot of sense, but as I said - providing an example should make it easier.

[tom-robo]

Thanks for your quick reply and clarifying a few points. I think I was tentatively coming to that conclusion, glad you cleared it up.

To answer you visually, correct me where I'm wrong, I think this is what you said:

I guess ImDrawData has the required texIDs to match with those stored in the streamed data array.

I'll have to look at your snippet a little bit more (I haven't done any Java or WebGL before) but from what I gather, we are iterating through vertex arrays and binding textures to them which we are creating from the streamed data array from the server?

Thanks again, I'll probably ask you a bit more tomorrow.

[ggerganov]

That's exactly right!

On the javascript side:

• the following piece of code should be changed like this:

     if (texture_id == 1) { 
         this.gl.activeTexture(this.gl.TEXTURE0); 
         this.gl.bindTexture(this.gl.TEXTURE_2D, this.tex_font_id); 
     } 

into

     this.gl.activeTexture(this.gl.TEXTURE0); 
     this.gl.bindTexture(this.gl.TEXTURE_2D, this.tex_map[texture_id]);  

• notice instead of the tex_font_id we now have a map tex_map that we need to populate. Here it might need some more work - I have to figure out myself what would be the best way. The goal is to try to generalise the following code to handle any number of textures instead of just one:

imgui-ws/src/imgui-ws.js

Lines 152 to 181 in 1eb94e0

	incppect_tex_font: function(incppect) {
	if (this.tex_font_abuf === null || this.tex_font_abuf.byteLength < 1) {
	this.tex_font_abuf = incppect.get_abuf('imgui.textures[%d]', 1);
	} else if (this.tex_font_id === null) {
	imgui_ws.init_font(this.tex_font_abuf);
	}
	},
	init_font: function(tex_font_abuf) {
	var tex_font_uint8 = new Uint8Array(tex_font_abuf);
	var tex_font_int32 = new Int32Array(tex_font_abuf);
	const width = tex_font_int32[2];
	const height = tex_font_int32[3];
	var pixels = new Uint8Array(4*width*height);
	for (var i = 0; i < width*height; ++i) {
	pixels[4*i + 3] = tex_font_uint8[16 + i];
	pixels[4*i + 2] = 0xFF;
	pixels[4*i + 1] = 0xFF;
	pixels[4*i + 0] = 0xFF;
	}
	this.tex_font_id = this.gl.createTexture();
	this.gl.bindTexture(this.gl.TEXTURE_2D, this.tex_font_id);
	this.gl.texParameteri(this.gl.TEXTURE_2D, this.gl.TEXTURE_MIN_FILTER, this.gl.LINEAR);
	this.gl.texParameteri(this.gl.TEXTURE_2D, this.gl.TEXTURE_MAG_FILTER, this.gl.LINEAR);
	//this.gl.pixelStorei(gl.UNPACK_ROW_LENGTH); // WebGL2
	this.gl.texImage2D(this.gl.TEXTURE_2D, 0, this.gl.RGBA, width, height, 0, this.gl.RGBA, this.gl.UNSIGNED_BYTE, pixels);
	},

[ggerganov]

@tom-garford I just added support for multiple textures - Grayscale, RGB and RGBA. Checkout the new example for more info:

https://github.com/ggerganov/imgui-ws/tree/master/examples/textures-null

By the way, it turned out to be a bigger change than I initially anticipated.