Efficiently animating labels

[edherbert]

Hello.

I have a question regarding labels and glyph animations. I have a dialog box in my project. I'd like the text to gradually appear from the left as the character speaks it. This means that certain glyphs would be invisible, and the ones which are being spoken animate in their opaqueness. Really I want to know what the most efficient way to do this is.

I see Colibri can do this with rich text, where I can specify sections of the text to be a different colour or font or whatever. I could use this, and gradually animate in the opacity of each glyph, however that poses some problems as sections of the text have to be specified upfront as rich text. This means each character would have to be specified as a rich text section if my understanding is correct, which to me seems inefficient.

If I went a more complicated route, I could determine which parts of the text are animated and which aren't and only specify the active ones as rich text, just to limit the number of entries. However I would then have to set new rich text values each frame during the animation.

I'm wondering, is there a lower level solution to this problem which might be simpler, or something else you can suggest to solve this problem?

As well as this, in future I'd like to do things like animate glyphs, like in this example from Mario Odyssey below.

I get the impression that I'd want to do that sort of thing as close to the vertex shader as possible, so again I'm curious what the most user friendly/efficient solution is.

Thanks very much for any input you can give.

[darksylinc]

This problem can be tackled from multiple approaches.

But I think the most simple one (and also fast) is a vertex shader customization that looks at the XY position on screen and animates/fades based on that.

Even easier may be the C++ variation presented at the end

Shader version

Our vertex shader boils down to this:

// Vertex shader
@property( colibri_text )
	uint vertId = uint(gl_VertexID) % 6u;
	outVs.uvText.x = (vertId <= 1u || vertId == 5u) ? 0.0f : float( input.blendIndices.x );
	outVs.uvText.y = (vertId == 0u || vertId >= 4u) ? 0.0f : float( input.blendIndices.y );
	outVs.pixelsPerRow		= input.blendIndices.x;
	outVs.glyphOffsetStart	= input.tangent;
@end

// Later in pixel shader:
glyphCol = bufferFetch1( glyphAtlas, int( inPs.glyphOffsetStart +
										  uint(floor(inPs.uvText.y) * float(inPs.pixelsPerRow) +
											   floor(inPs.uvText.x)) ) );

// In C++ the vertex we send to the vertex shader looks like this:
struct GlyphVertex
{
	float x;
	float y;
	uint16_t width;  // aka blendIndices.x
	uint16_t height; // aka blendIndices.y
	uint32_t offset; // aka glyphOffsetStart
	uint32_t rgbaColour;
	float clipDistance[Borders::NumBorders];
};

Which tells us several things:

1. Each letter is 6 vertices (2 triangles).
2. Same glyphOffsetStart means the same letter. e.g. all letters "e" from the same font and font size will share the same glyphOffsetStart
3. gl_VertexID / 6 tells you the character location (watch out GLSL needs to subtract worldMaterialIdx[inVs_drawId].w first). Each letter individually is represented by "gl_VertexID / 6".

Animating like in Mario should be easy:

Add custom_vs_posExecution and randomly displace letters based on their .xy location and width height:

@piece( custom_vs_posExecution )
  float randomOffset = fbm( worldPos.x * userMultiplier / blendIndices.x ); // Experiment whether div by blendIndices.x is necessary
  outVs_Position.y += randomOffset;
@end

Note: for a good noise function like fbm see ShaderToy from Iñigo Quilez

Fade in

Fade in is a bit more tricky unless we assume each label is a single line only because then fade in is only left to right (or right to left for RTL languages), because we could certainly use more information being sent from C++ (like line number the glyph belongs to) for full support.

But if we assume single line only then fading is simple:

@piece( custom_vs_posExecution )
    // outVs_Position.x is in range [-1; 1] so we must bring it to range [0; 1]
    outVs.colour.a = lerp( 0.0f, outVs.colour.a, ((outVs_Position.x + 1.0f * 0.5f) - animatedValue) / renderTargetWidth );
@end

Last but not least, all text uses the same shader, so you probably want to overload HlmsColibri in order to distinguish Labels that require your special modifications. You'll need to overload HlmsColibri::calculateHashForPreCreate with something like:

void MyHlms::calculateHashForPreCreate( Renderable *renderable, PiecesMap *inOutPieces )
{
    HlmsColibri::calculateHashForPreCreate( renderable, inOutPieces );

    const Ogre::Renderable::CustomParameterMap &customParams = renderable->getCustomParameters();
    if( customParams.find( myValueToIdentifyCustomization ) != customParams.end() )
    {
        setProperty( "my_customization", 1 );
    }
}

C++ alternative

It will be more powerful if Label::_fillBuffersAndCommands is modified so that when addQuad is called, you do whatever you want to glyph data and RGBA colour (for fade in) or you can offset the the position (for Mario-like animation).

You could create your own MyCustomLabel and override _fillBuffersAndCommands with your own implementation.

The key parts are:

// Rendering letter's shadow (if active)
addQuad( textVertBuffer,                                           //
		 topLeft + shadowDisplacement,                             //
		 bottomRight + shadowDisplacement,                         //
		 shapedGlyph.glyph->width, shapedGlyph.glyph->height,      //
		 shadowColour, parentDerivedTL, parentDerivedBR, invSize,  //
		 shapedGlyph.glyph->offsetStart,                           //
		 canvasAr, invCanvasAr, derivedRot );

// Rendering actual letter:
addQuad( textVertBuffer, topLeft, bottomRight,                        //
						 shapedGlyph.glyph->width, shapedGlyph.glyph->height,         //
						 richText.rgba32, parentDerivedTL, parentDerivedBR, invSize,  //
						 shapedGlyph.glyph->offsetStart,                              //
						 canvasAr, invCanvasAr, derivedRot );

You'd offset topLeft and bottomRight for Mario animation, and use your own rgba32 colour where Alpha is fading in instead of richText.rgba32 and shadowColour

_fillBuffersAndCommands gets called every frame, so you can make animate it even if the text doesn't change.

If the changes done to _fillBuffersAndCommands to have animated text aren't too expensive I could even consider adding it to core, since those animations come in handy.

[edherbert]

Hi dark_sylinc.

Thanks as always for your detailed comment. I've been able to spend some time on this this afternoon and got this:

I wasn't aware I could intercept the addQuad function like that. I've gone with the c++ solution. While the shader tricks are definitely interesting, doing it on the cpu makes much more sense to me from a portability standpoint, and will also integrate better with the scripting language I use to do gui.

You mention integrating this with core as well. Honestly, keeping my stuff as close to mainline is good for me, so I'd be happy to merge it in. The changes are on this branch
edherbert@851c1fd
It's not done yet, I need to clean it up and test but really I'd be interested to see whether you think foundationally it's merge worthy.
I've created a new widget type, AnimatedLabel. I've made the addQuad function virtual, and really the only thing the animatedLabel changes is that function. It does however keep track of some data per shape which contains the modifiable data. So there is a memory overhead when using AnimatedLabel which shouldn't be present in regular labels. As for making some frequent functions virtual, the impression I get is a decent compiler should be able to inline them anyway. I'll do some profiling some point soon to see.

Thanks very much

[darksylinc]

Regarding the virtuals, I think the sensible approach here is to do what I did for Ogre:

virtual_l1 void myFunc();

Where virtual_l1 would be a macro defined at CMake time:

#if COLIBRI_FLEXIBILITY_LEVEL > 1
    #define virtual_l1 virtual
#else
    #define virtual_l1
#endif

That way users who want the raw system don't have to pay runtime overhead of using virtual in a hot path.

[edherbert]

It's an interesting point.

I think there's a question of whether AnimatedLabel the class makes its way into colibri core, or we just settle for the optional virtual functions and I move the label code to my own code base. If we went with flexibility levels it doesn't seem there's much sense in including it as a unique class.

You're right that these settings can come in handy, and I do think someone else will want something like this at some point as well. Originally I was planning to include an example of it in the demo app. However in this case we wouldn't be able to showcase it if the user has not enabled the virtual option, which then means we'd have to macro in and out some code in the demo app as well as in other places. Personally I think that's a bit gruesome, so we could just settle for a comment explaining what the benefit of enabling the virtual is.