dymokomi/luce-browser-foundation
Source of release 0.6.1
0.6.1 / docs / regions / r07.md
# Region r07: LibUnicode without ICU (`web_unicode`) — what other regions need to know
Region r07 ports the LibUnicode LibWeb uses and replaces ICU, which the donor links for most of
it. Each fragment's header describes it, and `docs/namemap.tsv` lists every name.
## What replaces ICU
- **The generator.** `tools/gen_ucd/` is a Luce program. It reads:
- `data/ucd/`: `ppucd.txt`, `nfc.txt`, `uts46.txt` and the emoji sequence files (Unicode
17.0.0);
- `data/icu/`: locale lists dumped from ICU 78.2 (the available locales, the likely subtags and
the right-to-left scripts).
Both directories keep their license files beside the data.
- **The generated tables.** `generated_ucd_properties`, `generated_ucd_casing`,
`generated_ucd_emoji`, `generated_ucd_idna` and `generated_icu_locales` hold hex strings, about
850 KB in all, with ICU's numbering for the categories, properties and scripts. `test.sh`
regenerates them into a temporary directory and compares the result with the committed files.
- **The hand-written fragments:**
- `ucd`: property lookup;
- `casing`: full case mapping, titlecasing, case closure and the Halfwidth-Fullwidth mapping,
with the tr, lt, nl and hy rules;
- `break_iterator`, `break_grapheme`, `break_word`, `break_line` and `break_sentence`: ICU's
rule files, including the word rule statuses and ICU's navigation semantics;
- `text_buffers`;
- `idna` and `idna_2`, from ICU's `uts46.cpp` and `punycode.cpp`;
- `normalize`, over luce-std's `unicode.normalize`.
## Checked locally against ICU 78.2
The harness is not committed. It checked:
- every per-code-point property, for every code point;
- grapheme, line and sentence boundaries on 60,000 random strings;
- word boundaries and rule statuses, except for Han, Kana and Thai, which need ICU's dictionaries;
- casing in the root, tr, lt and nl locales, titlecasing and fullwidth mapping on 30,000 strings;
- IDNA on about 26,000 inputs, in both of LibURL's option sets.
## Tests
- UCD's GraphemeBreakTest, WordBreakTest, SentenceBreakTest and LineBreakTest pass. The
dictionary word cases are skipped, as ICU skips them.
- IdnaTestV2's `toAsciiN` column passes. A lone A4_2 on a trailing root dot counts as a success,
as it does in ICU.
- Ported from Ladybird: TestUnicodeCharacterTypes, TestSegmenter, TestIDNA,
TestUnicodeNormalization and TestLocale.
- The casing tests are pinned to ICU.
- At integration:
- the tests moved off the cores r07 had used while ak was stubs. They now go through the real
AK String entry points: `normalize`, `idna_to_ascii`, `for_each_boundary(String)` and
`expand_range_case_insensitive`.
- TestLocale's parse and canonicalize cases were added (`tests_locale_2`).
Those canonicalize cases that need CLDR alias data stay in place as comments (see Deviations).
## Changes other regions must know
- **AK String methods live in web_unicode.** LibUnicode defines these AK::String and Utf16String
methods: `to_lowercase`, `to_uppercase`, `to_titlecase`, `to_casefold`, `to_fullwidth`,
`equals_ignoring_case`, `find_byte_offset_ignoring_case` and `trim_whitespace`. ak cannot
import web_unicode, so they are `web_unicode.string_*` and `web_unicode.utf16_string_*`.
`ak/stub_r07_web_unicode.lucb` is removed. ak's `Icu78UnicodeString` and
`EmptyOrStringOrIcu78UnicodeString` are dropped too: nothing uses them.
- **Locale functions:**
- `locale_id_remove_extension_type` and `locale_id_for_each_extension_of_type` take an
`ExtensionKind` instead of a type parameter;
- the callback is a `Function1[Extension*, IterationDecision]`.
- **The Locale parsers** use ak's `GenericLexer[u8]`, as Locale.cpp does. Before integration they
used a local `LocaleLexer`.
- **Signature changes:**
- `char_direction_to_bidi_class` takes a `u32`;
- `apply_extensions_to_locale` takes `LocaleId*`s;
- `ranges_equal_ignoring_case` takes code-point spans.
- **Types edited by hand:**
- `SegmenterImpl` holds a `BreakIterator*` and the new `SegmentedText` enum;
- `ResolvedLocale` holds byte spans;
- `GENERAL_CATEGORY_*` and `PROPERTY_*` are `let` constants.
The ICU-only declarations are removed: `PropertyName`, `ADDITIONAL_NAME`, the case-closure
caches, `types_external` and `types_generated_{brkiter,uniset,locid}`.
## Deviations from ICU
- No dictionary segmentation, so CJK and South-East Asian words are not split.
- No Greek-specific uppercasing.
- Locale canonicalization is syntax-only, with no CLDR alias replacement. TestLocale's alias
cases are therefore not ported:
- the ks, ms, tz and ca value aliases, and t's m0 alias;
- the language, territory, script, variant, subdivision and complex subtag aliases.
They stay in `tests_locale_2` as comments. The "yes" → "true" keyword rule of LocaleData
(ICU-21367) is ported.
- Likely subtags come from the dumped table plus UTS #35.
- The segmenter uses the root rules for every locale.
- IDNA's ContextO checks are not ported; Ladybird never enables them.
- ICU's data quirks are kept: ccc comes from `nfc.txt`, and U+2985 and U+2986 are not mapped to
fullwidth.
## Fixed at integration
- **`locale_data_canonicalize` kept dangling views.** It kept the keys of the "yes" keywords as
views into a short String on the stack. When a later key overwrote that String, the wrong
keyword got "yes" back (`en-u-ka-yes-kb-yes`). The keys are now a
`HashTable<String>`, as the donor's `HashTable<ByteString>`.
- **`is_locale_available`** looks its String table up by a StringView through the new
`ak.StringStringViewTraits`, which conforms to `HashCompatibleTraits[String, StringView]`.
- **ak's Vector and GenericLexer** replace the local `ViewList`, `vector_items` and `LocaleLexer`.