Text rendering
Opened this issue · 65 comments
Text rendering is not supported and not planned. This is an absurdly complex task and the Rust ecosystem doesn't provide basically any libraries to implement this.
We need:
- Font parser: ttf-parser.
- Text shaper: rustybuzz or all-sorts.
- Font database with a font fallback mechanism.
- High-quality glyph rasterization library. Like ab_glyph_rasterizer, but with FreeType level of quality.
- Code in
tiny-skiathat ties it all together.
I think woff2 is better than ttf.
@yisar WOFF2 is a file format organization and compression scheme to optimize transmission of fonts over the internet for embedding in websites. The font payload you get after transfer and decompression in the end is the same. These days whether packaged as TTF, OTF, or WOFF2 what you are getting in terms of shapes and font functionality is identical. And hence you need a TTF parser to handle WOFF2 ;-) And WOFF2 doesn't make much sense for local font rendering, it's more CPU time to optimize away something that isn't even a bottleneck when working with local font files.
Maybe https://github.com/servo/font-kit is also suitable for tiny-skia, I've used it 3 years ago: https://github.com/rust-canvas/rust-canvas/blob/master/src/canvas/context_2d.rs#L226
@Brooooooklyn font-kit is a wrapper over system libraries, which is no go. I want a pure Rust solution.
Take a look at ‘fontdue’ by @mooman219
It may meet your requirements: https://github.com/mooman219/fontdue
@kettle11 fontdue is just a rasterizer right now. We need a lot more.
@kettle11 this is kinda a small rundown of my opinion of the situation
It may meet your requirements: https://github.com/mooman219/fontdue
Fontdue is a no-std raster and layout tool for non-shaped text over a small subset of font features found in glyphbrush/ab_glyph/rusttype. This feature subset we all support isn't freetype grade for reasons mentioned below. In terms of runtime performance, it's much faster than glyphbrush/ab_glyph/rusttype, all of which use the raster from font-rs. It does exact coverage anti aliasing just like font-rs, so the quality is the same.
This is an absurdly complex task and the Rust ecosystem doesn't provide basically any libraries to implement this.
This is fairly accurate. The problem is that overall, the crates mentioned above (no-std crates with nice APIs) in rust are very limited in their scope over the domains they cover. Freetype font engine grade text includes a whole slew of hard problems. A non exhaustive list of features we're generally missing include:
- Gzip decompression for woff if you really care about that
- RGB Subpixel anti aliasing/blending/positioning
- Font config (including fallback)
- Stem darkening
- Shaping to any degree
- Color emoji/color in general
- Variation axis transformations
- BiDi
- CJK locale behavior
The big one here is shaping, which in pure rust is immature and greatly influences the design of implementing some of these features like BiDi. Any serious work from any no-std font-engine can really start once that's settled in my opinion, so we're basically hoping that either rustybuzz or all-sorts mature. The rest of the features aren't nearly as insurmountable by comparison.
A quick note: Pathfinder actually does a ton of these things like stem darkening, RGB AA/Positioning, color emoji, and such, but I'm assuming that's out of the question given its kinda hard to integrate with and also on the gpu.
I wanted to add linebreaking to that list as well. It's wildly more complex than the spec implies. The unicode linebreak annex defines the bare minimum you should handle, but is not freetype grade in the slightest. There's a ton of nuance and opinions in the space and across platforms that add/remove line break opportunities, handle trailing whitespace, etc.
This feels like making a working group territory, but also there's not a ton of people that actually care about pure rust solutions :'( . I think the xi zulip is the most concentrated group so far, but there's not a big push for no-std solutions there. I'd be happy to break anything out of fontdue / work on a small tool for one of these features if you see an opportunity. I really want to see great no-std rust text in the future.
@mooman219 Afaik, freetype does only rendering. It doesn't do layout in any way. At least on linux, it's up to pango/Qt.
Also, Skia has it's own paragraph implementation on top of harfbuzz. This what flutter use (or plan to use).
PS: Why do you care about no-std solutions?
@mooman219 Afaik, freetype does only rendering. It doesn't do layout in any way. At least on linux, it's up to pango/Qt.
Whoops, I'm conflating pango in with freetype again for no good reason. s/pango/freetype where it matters in my original comment.
Also, Skia has it's own paragraph implementation on top of harfbuzz. This what flutter use (or plan to use).
Yep! I've read a lot about it the past few weeks which is why I wanted to mention linebreaking as a problem that still needs to be solved in pure rust, even if it's just cloned from Skia.
PS: Why do you care about no-std solutions?
This is mostly a personal opinion. I think that a huge benefit of having the ecosystem in pure rust is going the next step to being no-std. Fontdue for example is used in some no-std environments including a small kernel and on consoles, and there's really no reason to not be no-std for the majority of the stack as far as I can tell, unless you're aware of something significant? Font config / fallback behavior may be system dependent, but that can be optional. The only thing preventing allsorts from being nostd is that it depends on a std gzip library for woff.
no_std is kinda an abstract term. fontdue uses alloc, so it's not pure no_std in my opinion. ttf_parser is, because it doesn't use heap at all.
In this sense, tiny-skia also can be no_std, because the only thing from std I'm suing is Vec.
No-std is a well defined term since it's a crate level attribute. I like to think alloc is in the spirit of no-std.
Alloc is reasonably free to add on most no-std environments. Requiring all the threading/networking/file system/etc components is really heavy. Making tiny-skia no-std and just needing alloc would be great for some of the common no-std environments like consoles
I've been using fontdue for rasterization and basic layout, both in a work project and a personal project. I don't have any complicated text layout or shaping issues to handle, but I've been very pleased with the results so far. Someone with sharper eyes than mine may be able to spot issues, I'm somewhat of a layman when it comes to seeing slight issues with text rendering.
@bschwind You have a very large text. It will always look good. The problem is in the tiny one.
To test shaping, you can simply try rendering Arabic text.
To test the layout, you can use a mix of Arabic and English.
You have a very large text. It will always look good. The problem is in the tiny one.
Ahhh okay, today I learned :) Makes sense though!
I'm sure Arabic text or Arabic + English wouldn't turn out well currently. As mooman219 mentioned, it just does basic rasterization and layout at the moment.
Anyway,
High-quality glyph rasterization library. Like ab_glyph_rasterizer, but with FreeType level of quality.
I think it would be productive to make an exhaustive list of what this means and see if we can those on the radar of the existing pure-rust rasters.
@mooman219 All I want to see, as I've mentioned a while ago, is a detailed comparison between different libraries. Here is PNG generated by fontdue, here is one by freetype, etc. As a library author, you should sell your library first. Performance is great, but this in not what I care about.
Font's rendering is one of the areas that I really don't want to go into, because it's too subjective.
High-quality glyph rasterization library. Like ab_glyph_rasterizer, but with FreeType level of quality.
I’d expect this (and parsing as necessary for this) to be in scope for tiny-skia eventually. Everything else (shaping, layout, bidi, fallback, …) could be built separately on top, though that doesn’t mean smooth integration can’t exist.
Compare with Cairo v.s. Pango (and pangocairo integration).
@SimonSapin I actually haven't looked into how Skia renders glyphs yet, so I'm not sure. It clearly depends on freetype, so I assume it uses it for rasterization (or maybe just for outlining). It also has it's own minimal TrueType parser for some additional info I guess.
Barely worth a mention here, but KAS-text implements:
- BiDi layout
- line breaking
- very primitive rich text
The library vaguely follows the relevant Unicode specifications but isn't fully compliant.
Caveats: the API won't suit all users, likely significant redesign is needed for freetype-quality and performance.
@dhardy I don't know when I would have time to implement text support, but it would probably be implemented from scratch using ttf-parser + rustybuzz + fontdb + ab_glyph_rasterizer/fontdue/own renderer. I.e. a pure Rust solution. Any non-Rust dependencies are forbidden.
@RazrFalcon the only non-Rust dep in KAS-text is HarfBuzz, which is optional (and probably will be replaced soon).
In any case, I was more pointing out another "piece" for the Rust font ecosystem, for anyone here interested.
@dhardy What about font-kit?
Sure. I'm still not sure what API I want, but it probably will be low-level enough to be used in resvg. Basically, I need a very good font fallback support (which I don't have) and an ability to modify position of each glyph (each glyph should also have some metadata left, so I could match it with an original string).
You're right, font-kit is used as a crutch to find fonts. Still need a native-Rust fontconfig. It may be worth trying the recent rust_fontconfig crate.
KAS-text does include the string index with each glyph, but no, I wouldn't recommend resvg depending on kas_text.
rust_fontconfig
Nice. I didn't know about this crate. I've actually wanted to write one myself, but fontconfig is an untested mess, so I've ended up writing fontdb. Also, looks like rust_fontconfig doesn't support fontconfig's disk cache, which will make it ultra-slow.
This is the main problem with pure Rust font db: system libs doesn't scan fonts each time.
@dhardy I don't know when I would have time to implement text support, ...
You have been pretty active and progressed with ttf-parser, rustybuzz and fontdb.
How about the missing peace of a glyph raserizer?
I'm asking, since OrbTK as a pure rust UI toolkit did fully integrate tiny-skia for 2D rendering. It is still lagging a freetype grade solution for text rendering and handling. This is needed to support i.e multiline text blocks. So I'm highly interested in any progress in this front.
@rzerres it's done (couple of weeks ago). It appears I didn't enable the feature for docs.rs so you'll have to build the docs locally with the raster feature. See Cargo.toml. You still need some driving code; see text_pipe.rs.
Do let me know how you get on; I'm open to design changes. It's not FreeType grade; a couple of major omissions are support for embedded objects (e.g. emojis) and scalability to larger texts (possibly by stitching together "paragraphs"; I think some redesign is needed but haven't thought too much about it). Font fallback support is also not complete.
@rzerres I added doc to the raster module: https://docs.rs/kas-text/0.3.3/kas_text/raster/index.html
swash may (eventually) be a another option in this space, though currently omits text layout.
Thanks @dhardy.
Right now I'm quite busy. But will have a look. Did you got feedback from @RazrFalcon conserning his needs to get a crate that links to resvg and `tinyskia'?
@rzerres resvg will probably forever use just rustybuzz. SVG text layout it too complicated and SVG-specific. Not much you can delegate to a separate library.
The scope of text support in tiny-skia is just a singe-line text layout for now. No paragraphs and other complex features.
@RazrFalcon sadly i do know :)
For OrbTk we are hunting for a feasable solution to support multiline TextBoxes. Sure a little bit of markdown parsing would be much appreciated. I started to just allow newlines.... but then ... selection and cursor handling is starting to get a nightmare. The deeper you think about it, the more the generic solution is needed.
I am sorry if this is out of the scope of this issue, but as far as I could understand, femtovg implements a few things that have been mentioned here (using ttf-parser, rustybuzz, unicode-bidi and unicode-segmentation).
Sure, there are multiple libs like this. I simply haven't had time to look into this. Mainly because I don't need it myself. I'm still not sure what scope text support in tiny-skia should have.
Have you looked into swash? I tried out the demo and was very impressed with the quality, couldn't distinguish it from FreeType rendering in Chromium. Afaik swash supports both hinting and sub-pixel anti-aliasing.
Damn, I tried it too and it's really impressive. I think I'm gonna try to use tiny-skia in combination with swash for my project.
I've seen some mentions of how the difficult part of rendering text is when it is small, so I tried with font size 8 in some parts of the demo and it looks like this:
Adding references to solutions that come up, cosmic-text says that they implemented their own Font fallback.
Font fallback is also a custom implementation, reusing some of the static fallback lists in browsers such as Chromium and Firefox. Linux, macOS, and Windows are supported with the full feature set. Other platforms may need to implement font fallback capabilities.
@saona-raimundo Yes, Cosmic-text is probably the way to go, since it is the most complete and the highest quality font rendering library in pure Rust out there. I think there isn't much choice if not to use it ¯_(ツ)_/¯
+1 for cosmic-text, in a short time it has become extremely popular and seems to be almost becoming the de-facto solution for Rust.
I've created a piet-tiny-skia crate. Among other things, it adds support for text rendering via the cosmic-text crate.
Even though piet-tiny-skia is released under the MPL, if someone wanted to port the text rendering code from there to this crate I would be happy to allow it.
@notgull Great! I still plan to implement my own text layout library for tiny-skia one day.
Is there any plan for implementing text rendering in tiny-skia, anytime soon??
Nope. Maybe in a couple of years.
Nope. Maybe in a couple of years.
I've read the thread and as you're very serious about implementing your own text rendering system, I would like you to look at stb_truetype.h. I hope you find it useful.
stb_truetype.h simply renders glyphs. It doesn't do layout. So until I write a layout library, or at least exact the one one form resvg, tiny-skia would not have any text.
OK, got it!
@RazrFalcon fontdue supports complex text layout. Actually, even Cosmic text itself has a layout engine, though not as complete (no ellipsize or wrapping)
@RazrFalcon whoops I mixed the libaries 😵💫
Anyways, I still think that cosmic text is advanced enough for 75% of the use cases. It would still be great to have some good enough text support in the meantime rather than no text support at all.
@RazrFalcon What do you imagine under a NSAttributedString Rust implementation? How should/could the API & internal architecture look like?
I'm currently experimenting with a concept somewhat akin to NSAttributedString, aiming for rich text handling. The first design goal is to build something that takes in a (english) text & attributes and outputs a tiny-skia-path. Does something similar already exist?
It's a early Proof of Concept and based on usvg’s text_to_paths implementation,
thus I’d appreciate insights on several design choices:
- IntervalTree (like
rust-lapper) for attribute management—efficient for overlapping attributes? - Structuring attributes—partial struct for direct access or a HashMap for flexibility?
- Tokenizing text into
TextFragment,WordSeparator,Linebreak, etc., beforehand for simpler transformation application, like text layout, emoji, etc. handling
and ofc any thoughts in general.
Thanks 🙌
@bennoinbeta Yes, the idea is to extract usvg text handling code into a separate crate and make a nice, high-level API around it. The problem is that recently I had no free time, so I never actually looked into it. Therefore I cannot comment much.
The main issue with the current usvg implementation, and yours as well, is that the library should not care about outlines or rendering in general.
It should accept text + fonts + styles and produce a list of glyphs with positions (aka layout). It might provide utilities to outline or even render glyphs, but it should be a separate step.
This is needed to properly implement bitmap fonts, aka emojis.
Another usvg, or rather SVG, limitation is the lack of paragraphs. Aka multi-line text. That's a very complicated topic on its own and I have no expertise here. Especially when hyphenation is involved.
A yet another usvg limitation is the lack of glyphs caching. Which is a must.
Also, the API should provide a way to distinguish between logical and visual indexing. To properly support BIDI text.
usvg does handle it internally, sans a couple of bugs, but it would be hard to provide a decent and fast high-level API for it.
Think of it as character-to-glyph index and vise-versa mapping. In RTL text, the first glyph is the last character.
And so on and so on... It would be nice to have a library like this, but it's an extremely hard task.
@RazrFalcon No worries, I'll give it a try as I've a bit free time right now (but not the expertise in Rust and especially text layout, shaping, .. so don't expect too much). But learning by doing :) The current iteration offers the following API:
let text = String::from("Hello, world!\nשלום עולם!\nThis is a mix of English and Hebrew.");
let attrs_intervals = vec![
AttrsInterval {
start: 0,
stop: 10,
val: Attrs::new()
.font_family(FontFamily::Monospace)
.font_weight(400)
.font_size(24.0),
},
AttrsInterval {
start: 10,
stop: text.len(),
val: Attrs::new()
.font_family(FontFamily::Serif)
.font_weight(400)
.font_size(12.0),
},
];
let mut attributed_string = AttributedString::new(
text,
attrs_intervals,
AttributedStringConfig {
bbox: Vec2::new(100.0, 100.0),
..Default::default()
},
);
// Tokenize the text (String) in logical intervals for further processing
attributed_string.tokenize_text(&mut fonts_cache);
// Apply the layout like linebreaks, letter spacing, ..
attributed_string.layout();
// Create tiny-skia-path
let path = attributed_string.to_path(&mut fonts_cache);Right now, it's early work in progress and mostly a mixture of concepts from usvg and cosmic-text. It's also still bound to tiny-skia-path but its possible to make it optional later and instead iterate over the glyphs as needed (like in to_path() implementation).
Would appreciate your thoughts on the current API and (maybe) implementation as I've no plan what I'm doing and am figuring it out on the go..
Implementation details: Initially the text is divided into spans (tokenize_text (1)), each corresponding to an attribute interval for potential caching and determining direction (ltr or rtl). Each span contains shape tokens like TextFragment, WordSeparator, Linebreak, .. which encapsulate glyphs. This assigns glyphs a logical context for subsequent processing in the layout (2) step. Idk whether I'm on the right track and whether this approach makes any sense and is efficient & performant in any means.
Thanks 🙌
@bennoinbeta the attribution is vaguely similar to my FormattableText excepting that it doesn't allow custom trait impls and doesn't require font face lookup be done in advance (an optimisation for when the result will be drawn many times).
How do you plan on handling overlapping start..stop regions and uncovered regions? I solved that by having defaults for everything and only encoding the start. It looks like your system doesn't allow merging attributes from overlapping regions anyway?
... this is not the right place to discuss this type of detail however.
@dhardy Yeah probably not the right place @RazrFalcon ?
How do you plan on handling overlapping start..stop regions and uncovered regions?
I'm using rust-lapper and added a function(divide_overlaps_with()) to merge overlapping intervals (See: sstadick/rust-lapper#23 , not merged yet though)
self.attrs_intervals.divide_overlaps_with(|overlaps| {
let mut merged_attrs = Attrs::new();
for &attrs in overlaps.iter() {
merged_attrs.merge(attrs.clone());
}
return merged_attrs;
});For uncovered regions I structured the Attrs struct in a way that all attributes are optional and made them only accessible via getter methods in which I apply the defaults (if None). However, I actually don't like this approach but couldn't figured out a better way in the Rust ecosystem yet.
Thanks for sharing FormattableText, I'll have look. Does it maybe already accomplish what I'm trying to build. Like a more abstract library positioning glyphs (supporting layout, ..) with an API like the AttributedString one from Apple than e.g. cosmic-text? Thanks 🙌
@bennoinbeta the kas-text library is designed to handle text layout from an str (optionally attributed) to positioned glyphs, with some (optional) helpers towards rendering. It covers roughly the same ground as cosmic-text, but older, and more designed for explicit cache control (stateful), so not the simplest API.
@bennoinbeta Not being much of an expert here as well, I would try to clarify what resvg needs:
- Static layout and not dynamic. All we care about is a heavily styled text layout. We don't care about caching, performance, content bounds resize and any kind of user input. It will not be used to render text at 60Hz. Which leads to a very different design.
- Stateful API (I hope I use this term correctly). We create an AttributedString, call the
layout()and get all the info we need (positions, glyphs, attached styles (like a gradient in SVG), underlines, etc). No need for a callback mess likeNSLayoutManager. - Ability to post-process the layout. For example to implement text-on-path. To do so, we need a quite low-level access to the final layout.
Overall, I appreciate your effort, but I'm not sure I would be able to use it.
Text support is such a fundamental feature that I would rather write it myself (one day I hope) to understand all the intricacies. Especially if we account all the missing features in usvg, like shape layout, bitmap fonts, caching, proper font fallback, embedded fonts, glyphs rasterization (aka SVG and COLR fonts), etc.
It would be nice to offload it to some true text expert, maybe as a part of a bigger project, but that's highly unlikely. There are probably like 5 people who know how the text works.
I'm not even really sure what I want from a AttributedString-like library myself, therefore I cannot judge your implementation either.
@RazrFalcon Understood, and thanks for the insights :)
I'm hopeful you'll either find a skilled collaborator for the text layout or manage to allocate some time for it eventually. Looking forward to an expert-crafted abstract AttributedString-like library. In the meantime, I'll use my hacky and not so thought through implementation, given the lack of alternatives. cheers :)
I've just implemented an example demonstrating integration of parley with tiny-skia using skrifa to do scaling and hinting. Emoji rendering is currently missing which is:
- Partly a limitation of
skrifa(bitmap emoji formats). This can be worked around be usingswashinstead. - Partly a limitation of tiny-skia. There are missing gradient type(s) (we need both Radial and Conic/Sweep) for full COLRv1 support.
- Partly just a matter of writing integration code to implement skrifa's ColorPainter trait on top of tiny-skia (need code to manage clips and transforms). This would immediately enable COLRv0 fonts and would also enable COLRv1 if the gradients types are implemented.
I've learned to appreciate tiny-skia's smaller API. I'd be fine if text handling was entirely absent here.
Would this provide a reason to implement?
No. fontations is still an alpha. And we already have rustybuzz + ttf-parser. That's not the problem. What is missing is a layout library.
As an update to the last comment here, for those reading today. Fontations is now shipping in Chrome, Rustybuzz has been moved to the Harfbuzz organization and is being migrated to the Fontations stack. Conic and Sweep gradients are still the main thing preventing Fontations users from rendering COLRv1 with tiny-skia.
No idea. There will be no more commits from me. At least in the near feature.
So unless someone is brave enough to tackle this issue it would stay the same.
Yeah, we're likely to do radial and sweep gradients, since it means we will have a complete software renderer for COLRv1 on Parley/Fontations. Right now the only one we have that does radial and sweep gradients is Vello, but for a variety of reasons Vello is not currently ideal on the CPU.