Luce packages
LuceLuce BaseLuciaOS

dymokomi/luce-browser-foundation

Source of release 0.6.1

0.6.1 / docs / regions / r07.md

# Region r07: LibUnicode without ICU (`web_unicode`) — what other regions need to know

Region r07 ports the LibUnicode LibWeb uses and replaces ICU, which the donor links for most of
it. Each fragment's header describes it, and `docs/namemap.tsv` lists every name.

## What replaces ICU

- **The generator.** `tools/gen_ucd/` is a Luce program. It reads:
  - `data/ucd/`: `ppucd.txt`, `nfc.txt`, `uts46.txt` and the emoji sequence files (Unicode
    17.0.0);
  - `data/icu/`: locale lists dumped from ICU 78.2 (the available locales, the likely subtags and
    the right-to-left scripts).

  Both directories keep their license files beside the data.
- **The generated tables.** `generated_ucd_properties`, `generated_ucd_casing`,
  `generated_ucd_emoji`, `generated_ucd_idna` and `generated_icu_locales` hold hex strings, about
  850 KB in all, with ICU's numbering for the categories, properties and scripts. `test.sh`
  regenerates them into a temporary directory and compares the result with the committed files.
- **The hand-written fragments:**
  - `ucd`: property lookup;
  - `casing`: full case mapping, titlecasing, case closure and the Halfwidth-Fullwidth mapping,
    with the tr, lt, nl and hy rules;
  - `break_iterator`, `break_grapheme`, `break_word`, `break_line` and `break_sentence`: ICU's
    rule files, including the word rule statuses and ICU's navigation semantics;
  - `text_buffers`;
  - `idna` and `idna_2`, from ICU's `uts46.cpp` and `punycode.cpp`;
  - `normalize`, over luce-std's `unicode.normalize`.

## Checked locally against ICU 78.2

The harness is not committed. It checked:

- every per-code-point property, for every code point;
- grapheme, line and sentence boundaries on 60,000 random strings;
- word boundaries and rule statuses, except for Han, Kana and Thai, which need ICU's dictionaries;
- casing in the root, tr, lt and nl locales, titlecasing and fullwidth mapping on 30,000 strings;
- IDNA on about 26,000 inputs, in both of LibURL's option sets.

## Tests

- UCD's GraphemeBreakTest, WordBreakTest, SentenceBreakTest and LineBreakTest pass. The
  dictionary word cases are skipped, as ICU skips them.
- IdnaTestV2's `toAsciiN` column passes. A lone A4_2 on a trailing root dot counts as a success,
  as it does in ICU.
- Ported from Ladybird: TestUnicodeCharacterTypes, TestSegmenter, TestIDNA,
  TestUnicodeNormalization and TestLocale.
- The casing tests are pinned to ICU.
- At integration:
  - the tests moved off the cores r07 had used while ak was stubs. They now go through the real
    AK String entry points: `normalize`, `idna_to_ascii`, `for_each_boundary(String)` and
    `expand_range_case_insensitive`.
  - TestLocale's parse and canonicalize cases were added (`tests_locale_2`).

  Those canonicalize cases that need CLDR alias data stay in place as comments (see Deviations).

## Changes other regions must know

- **AK String methods live in web_unicode.** LibUnicode defines these AK::String and Utf16String
  methods: `to_lowercase`, `to_uppercase`, `to_titlecase`, `to_casefold`, `to_fullwidth`,
  `equals_ignoring_case`, `find_byte_offset_ignoring_case` and `trim_whitespace`. ak cannot
  import web_unicode, so they are `web_unicode.string_*` and `web_unicode.utf16_string_*`.
  `ak/stub_r07_web_unicode.lucb` is removed. ak's `Icu78UnicodeString` and
  `EmptyOrStringOrIcu78UnicodeString` are dropped too: nothing uses them.
- **Locale functions:**
  - `locale_id_remove_extension_type` and `locale_id_for_each_extension_of_type` take an
    `ExtensionKind` instead of a type parameter;
  - the callback is a `Function1[Extension*, IterationDecision]`.
- **The Locale parsers** use ak's `GenericLexer[u8]`, as Locale.cpp does. Before integration they
  used a local `LocaleLexer`.
- **Signature changes:**
  - `char_direction_to_bidi_class` takes a `u32`;
  - `apply_extensions_to_locale` takes `LocaleId*`s;
  - `ranges_equal_ignoring_case` takes code-point spans.
- **Types edited by hand:**
  - `SegmenterImpl` holds a `BreakIterator*` and the new `SegmentedText` enum;
  - `ResolvedLocale` holds byte spans;
  - `GENERAL_CATEGORY_*` and `PROPERTY_*` are `let` constants.

  The ICU-only declarations are removed: `PropertyName`, `ADDITIONAL_NAME`, the case-closure
  caches, `types_external` and `types_generated_{brkiter,uniset,locid}`.

## Deviations from ICU

- No dictionary segmentation, so CJK and South-East Asian words are not split.
- No Greek-specific uppercasing.
- Locale canonicalization is syntax-only, with no CLDR alias replacement. TestLocale's alias
  cases are therefore not ported:
  - the ks, ms, tz and ca value aliases, and t's m0 alias;
  - the language, territory, script, variant, subdivision and complex subtag aliases.

  They stay in `tests_locale_2` as comments. The "yes" → "true" keyword rule of LocaleData
  (ICU-21367) is ported.
- Likely subtags come from the dumped table plus UTS #35.
- The segmenter uses the root rules for every locale.
- IDNA's ContextO checks are not ported; Ladybird never enables them.
- ICU's data quirks are kept: ccc comes from `nfc.txt`, and U+2985 and U+2986 are not mapped to
  fullwidth.

## Fixed at integration

- **`locale_data_canonicalize` kept dangling views.** It kept the keys of the "yes" keywords as
  views into a short String on the stack. When a later key overwrote that String, the wrong
  keyword got "yes" back (`en-u-ka-yes-kb-yes`). The keys are now a
  `HashTable<String>`, as the donor's `HashTable<ByteString>`.
- **`is_locale_available`** looks its String table up by a StringView through the new
  `ak.StringStringViewTraits`, which conforms to `HashCompatibleTraits[String, StringView]`.
- **ak's Vector and GenericLexer** replace the local `ViewList`, `vector_items` and `LocaleLexer`.