Complex Text on Simple Devices - Unicode Conference...We were able to fit in 128 mb devices, where...

Complex Text on Simple Devices

Pedro Navarro Sr. Software Engineer / Streaming Client Technologies

Background

1

Gibbon● Codename for our JavaScriptCore-based application

framework. It provides objects to create UI elements, access the device and perform video playback.

● Written in portable C++ targeted to the lowest common denominator

● Runs in Consumer Electronic devices (TVs, Blu-Ray players, Roku devices), Android TV and Game Consoles (from the Wii to the PS4 Pro)

2

Netflix UIPowered by Gibbon since 2013

Constraints● We are targeting devices with very low capabilities

(128mb total RAM) and, in many cases, with a Read Only file system.

● We don’t control the host we are running on, so we have to be ready to work with old compilers and different versions, and combinations, of all third party libraries we use

● Very long release cycles: except for game consoles it takes 12 months from the time we provide our SDK until there are devices in the market with it.

● No upgrades! We are part of the device’s firmware.

● Small footprint. Our binaries and fonts have to be as small as possible, as flash storage is scarce

● Different graphics platforms. We need to run on DirectFB, OpenGL and Game Console graphics APIs.

3

OLD DEVICELuckily, we didn’t have to support this one.

2013’s Text Engine● Our first iteration of Gibbon’s Text Layout Engine was

very simple and provided just 1:1 mapping between characters in a text string and glyph indices in a font.

● The character set was WGL-4, which we extended later to add additional Latin glyphs.

● Support for CJK was added by introducing fall fallbacks.

4

2013 Supported Writing Systems■ Latin (425 languages)■ Cyrillic (106 languages)■ Greek (1 language)■ Chinese (5 languages)■ Japanese (1 language)■ Hangul (1 language)

Global Launch● For the global launch we had to be ready well in advance

because of the long release times.

● We defined our own Character Set (NGL-2), to standardize our fonts and our content.

● Supporting Complex Scripts meant that we had to integrate text shaping and BiDi processing.

● Research showed that the vast majority of devices we needed to support, besides game consoles, would be low performance set-top boxes.

● It’s important for us to get pixel fidelity between platforms, so the UI doesn’t have to account for differences.

5

Global launch candidates:■ Arabic (38 languages)■ Hebrew (9 languages)■ Ge’ez■ Georgian■ Armenian■ Tibetan (4 languages)■ Khmer■ Lao■ Thai■ Burmese

Indic writing systems:■ Bengali (5 languages)■ Devanagari (19 languages)■ Gurmukhi■ Gujarati (2 languages)■ Kannada (4 languages)■ Malayalam (2 languages)■ Oriya (2 languages)■ Tamil■ Telugu

Global Text Layout Engine Features● Font Handling

○ Modular fonts○ Aliasing, fallbacks and substitution○ Synthetic bold and italic support

● Text Shaping○ Context based glyphs○ Ligatures (substitution)○ Positioning○ Reordering

● Text Layout○ Rich text○ Bidirectional support○ Line breaking (word wrapping)

6

Font fallbacks

7

Font fallbacks / Font linking

8

● Font linking automatically picks glyphs from other fonts, if not present in the active one, that offer the Unicode range where the missing glyph is.

● Font linking lets us ships fonts for each script as needed, making the deployments modular.

● Design points:○ A writable file system is not guaranteed, so Fontconfig’s cache solution would not work for us. Fontconfig might not be

available on the system so we would have to supply ours.○ We are not generic: we control the fonts that are in our system and the content we are going to display.○ We know the font we want to use for every writing system.

Font fallbacks

9

● Build time:

At font licensing time, subset the font to leave only the glyphs for that particular writing system (no latin in CJK fonts, for example) plus the space (U+0020)

Scan the fonts at build time and write to a configuration file which Unicode Blocks the font has glyphs in.

● Run time:

When a glyph is not found in the current font, search for fonts that can supply the needed Unicode Block, sorted by language and priority. Keep going down the list until a match is found.

Once a match is found, keep using the same font until a new Unicode Block is needed.

Spaces are always considered to be part of the current Unicode Block, so we keep spacing consistent by using the space glyph for each script’s font.

Helvetica, Sans, serif

fonts/Arial_for_Netflix-R.ttf

-136,-621+143x1864

-1006,-665+2222x1864

000000-00007F

Text Layout

10

Text Layout

11

● Infrastructure:

ICU for BiDi, Script categorization and line breaking.

Freetype for rasterization.

Harfbuzz for text shaping.

● Itemization:

White space is collapsed according to the HTML5 rules.

Fonts are resolved before shaping, so we shape the longest possible run of the same font. We don’t fall back to the base font for spaces.

We add synthesized bold and oblique styles to the list of available fonts.

We try to find locales, specified in a or by inferring it from the script, to use ICU’s dictionary based line breaking when available.

Attributes

[00] 0 - 0: [00:000-000] Japanese 20 [LTR]

[01] 1 - 2: [00:000-000] Traditional_Chinese 20 [LTR]

[02] 3 - 3: [00:000-000] Japanese 20 [LTR]

[03] 4 - 4: [00:000-000] Traditional_Chinese 20 [LTR]

[04] 5 - 7: [00:000-000] Japanese 20 [LTR]

Text direction runs

[00] 0 - 7: LTR (0:0-0)

Embedding levels: 0 0 0 0 0 0 0 0

Visual map: 0 1 2 3 4 5 6 7

Visual embeddings: 0 0 0 0 0 0 0 0

Text script runs

[00] 0 - 2: Hani

[01] 3 - 3: Kana

[02] 4 - 4: Hani

[03] 5 - 6: Hira

[04] 7 - 7: Hani

Text locale runs

[00] 0 - 7: ja

Word breaks

[00] 0 - 0: |驩|

[01] 1 - 2: |檤 |

[02] 3 - 3: |サ|

[03] 4 - 4: |捯|

[04] 5 - 5: |ひ|

[05] 6 - 6: |ろ|

[06] 7 - 7: |驚|

Line breaks

Attributes

[00] 0 - 0: [00:000-000] Japanese 20 [LTR] Hani

[01] 1 - 2: [00:000-000] Traditional_Chinese 20 [LTR] Hani

[02] 3 - 3: [00:000-000] Japanese 20 [LTR] Kana

[03] 4 - 4: [00:000-000] Traditional_Chinese 20 [LTR] Hani

[04] 5 - 6: [00:000-000] Japanese 20 [LTR] Hira

[05] 7 - 7: [00:000-000] Japanese 20 [LTR] HaniSample text layoutDebug information our itemizer provides about a text object

驩檤サ捯ひろ驚

Text Layout

12

● Text layout:

Emphasis on being one-pass. We forget the text string as soon as we have itemized it.

Harfbuzz buffers are referenced by multiple “items”. Each item has a harfbuzz buffer starting and ending offset.

We never shape text again. If we need to left/right trim, we operate directly on the items by modifying the offsets. For each font, we keep an in-memory codepoint to glyph index for all spacing characters.

We don’t support hyphenation or justification.

For BiDi reordering we operate directly on the runs, as each run has an embedding level property.

A run can have any number of sub-runs associated with it, for emphasis marks or rubies.

Layouts are cached, as they are expensive, and a change in container attributes can trigger a relayout or reitemize.

Itemizer layout - Bounds: [0,0+146x22] - Desired: [0,0+300x200]

Padding: [0x0] - Indent: 0

Mirror: false

[00] Line: [0,0+146x22] | Dir: RTL | Padding: 0+0

[00] Text run: Bounds: [0,0+20x21] | Ascent: -18

Direction: LTR: [00:000-000]

Buffer offsets: 0 - 0

Buffer contents: gid6521=0




Buffer contents: gid6606=1|gid3=2












Buffer contents: gid77=5|gid104=6


Last word mark present: 0




Cache Reuse: 1[1]/0 (0/0)

DisplayList(0xdcdb2ee0) pixels=2,780 size=300x200: Text: txt:'驩檤サ捯ひ

ろ驚'

Sample text layoutDebug information our itemizer provides about a text object

驩檤サ捯ひろ驚

Text Layout Facts

13

● We were able to fit in 128 mb devices, where we have only 20-30 mb available for our app.

● Text layout is, by far, the most expensive operation. Smart caching of text layouts helped us reach 30-45 fps when scrolling movie titles:

○ Try to never itemize text a second time.○ When changing the container dimensions or

alignment adjust the layout lines.○ Relayout text if the source text string hasn’t changed.○ To-do: diff attributes and reshape only the ones that

have changed.○ For ellipsized text, we keep the original layout and

ellipsize it when the container changes.

● After optimization, and refactoring our glyph cache, performance is slightly worse than the previous version that didn’t do BiDi or text shaping.

Arabic UIDebug information our itemizer provides about a text object

Challenges

14

Code Challenges● Using different libraries for the different itemization functions required re-encoding the text several times:

○ Fetch strings from the server [UTF-8]○ Assign them to JavaScript objects [UCS-2]○ Get strings from JavaScriptCode [UTF-8]○ Parse XML with libExpat○ Send the parsed text to ICU [UTF-16]○ Shape the text with Harfbuzz [UTF-32]

● For the same reason, itemizing a text string required iterating it up to 5 times:○ Find fonts○ Find direction (BiDi)○ Find the scripts○ Find locales and line breaking points

15

Text Layout Challenges● Where to place the ellipsis on BiDi Text? When ellipsizing the text “He said السالم علیكم” there are two options:

He said ... السالم

He said السالم …

We picked the second one, where the ellipsis is placed following the base direction of the text paragraph.

● Which format should the ellipsis have? For the text: Good Morning○ Use the font style of the text being ellipsized? Good Morn...○ Use the base style? Good Morn…○ Safari used the font style of the page! Good Morn…

We picked the second one. We tested Windows, OS X, Pango, Chrome, Safari, Internet Explorer, Edge and Firefox and none of them rendered and positioned the ellipsis the same way (as of March 2015).

16

Text Layout Challenges● Cursor movement: some systems move the cursor visually (our approach and the most popular one) and some others do it

logically (Windows).

● Cursor deletion: no clear standard on how to, for example, delete compound clusters. Some systems delete each individual member, some systems delete the whole clusters, some systems do it only with Indic scripts…

Our choice was to delete the whole cluster only when stacking diacritics. To future-proof the solution we implemented all cursor handling in JavaScript. Our C++ code exports all layout attributes, plus the output of ICU’s Character and Grapheme’s iterators.

17

Conclusion

18

Our Conclusion● The functionality of the Linux open source text stack is fantastic. Integrating the different components was hard for us.

Documentation about library usage and best practices was hard to find and we had to rely on the harfbuzz mailing list for tips and tricks about the different components. (Thanks guys!)

● There is not an integrated solution for text itemization and, as such, many CPU cycles are spent calling different libraries to perform different itemization steps, each one with its own supported text encoding. We believe it would be good to complement harfbuzz and Freetype with an open source itemization/layout engine.

● Layout specific issues, like ellipsizing or cursor movement have undefined behavior and it’s up to the implementor to follow what they think it’s best. Perhaps that’s something that Unicode should standardize?

19

Questions?Pedro Navarro Sr. Software Engineer / Streaming Client Technologies

[email protected]@pedronavf

20

Complex Text on Simple Devices - Unicode Conference...We were able to fit in 128 mb devices, where...

Documents

Transcript of Complex Text on Simple Devices - Unicode Conference...We were able to fit in 128 mb devices, where...