dymokomi/luce-browser-foundation
Source of release 0.6.0
0.6.0 / docs / regions / r02.md
# Region r02: AK text (`ak`) — what other regions need to know
Region r02 ports AK's strings, views, builders and text helpers: String, StringBase, StringData,
FlyString, StringView, StringUtils, StringBuilder, StringNumber, StringConversions, ByteString,
ByteStringImpl, CharacterTypes, UnicodeUtils, Utf8View, Utf16View, Utf32View, Utf16String,
Utf16StringBase, Utf16StringData, Utf16FlyString, GenericLexer, Base64, Hex, IPv4Address,
IPv6Address, LexicalPath and SourceLocation. It also ports AK/MemMem.h, which no region owned.
Each fragment's header describes it, and `docs/namemap.tsv` lists every name: renamed and added
ones are marked `ported` or `addition`, and templates expanded into typed functions are marked
`dropped`.
simdutf, fast_float and dragonbox are replaced, not ported:
- `simdutf.lucb` and `simdutf_base64.lucb` hold scalar versions of the simdutf entry points AK
calls, named `simdutf_*`. Their results match simdutf 7.4.0's scalar kernels, which every SIMD
kernel also matches.
- Integer parsing follows fast_float's `parse_int_string`.
- Floats are parsed with fast_float's grammar; the accepted prefix goes to `strings.parse_f64` /
`parse_f32` (correctly rounded), and out-of-range values become ±inf.
- Shortest digits (`convert_to_decimal_exponential_form_f32/_f64`) use Burger–Dybvig over a
bignum.
## Signature and type changes against the skeleton
### General rules
- `Optional<size_t>` is `usize?`, not `u64?`.
- A function that returns a view into a String, FlyString, Utf16String or Utf16FlyString takes its
owner by pointer, because short strings (up to 7 bytes) live inside the owner. Many by-value
parameters of the skeleton became pointers for the same reason, or because C++ takes `const&`:
- String, FlyString, ByteString, StringBase;
- StringData, Utf16StringData, ByteStringImpl;
- Utf16View and the code point iterators, where the callee keeps a pointer or compares.
- C++ constructors taking `nullptr_t` or a Badge lost their `arg0: void*?` parameter.
- `copy_characters_to_buffer(buffer, size)` takes a `u8[]` span.
### Templates became concrete functions
- `to_number<T>`, `parse_number<T>`, `parse_first_number<T>` and `parse_hexadecimal_number<T>`
exist once per arithmetic type, with the suffixes `_i8 _i16 _i32 _i64 _u8 _u16 _u32 _u64 _f32
_f64` (hexadecimal for integers only):
- `parse_number_string_view_i32(view, trim_whitespace, base_) -> i32?`;
- `parse_first_number_utf16_view_f64(...) -> ParseFirstNumberResult[f64]?`;
- the members `{string_view,string,byte_string,utf16_view,utf16_string_base,utf16_fly_string}_to_number_T(this, ...)`.
- `number<T>`:
- `string_number_i64`, `_u64` and `_f64`;
- `utf16_string_number_i64` and `_u64`;
- `byte_string_number_i64`, `_u64` and `_f64`.
Every narrower integer widens to i64 or u64 and prints the same digits.
`create_string_from_number` is `create_string_from_number_{string,utf16_string}_{i64,u64}`.
- `join` takes spans with the default `"{}"` format only:
- `string_builder_join_{string_view,string,fly_string,utf16_view}`;
- `string_join_{string_view,string}`;
- `byte_string_join_string_view`;
- `utf16_string_join_{string_view,utf16_view}`.
- `consume_decimal_integer<T>` is `generic_lexer_consume_decimal_integer_T` (integers). It returns
`T!` and fails with `ak_errno_error` "EINVAL" or "ERANGE".
### Callbacks and predicates
Lambdas passed to templates become structs that conform to interfaces declared in
`text_support.lucb` (DESIGN.md §2.7, case 2):
| Interface | Receives | Stands for the lambda of |
| --- | --- | --- |
| `Utf8ByteCallback`, `FallibleUtf8ByteCallback` | one UTF-8 byte | `code_point_to_utf8`, `try_code_point_to_utf8` |
| `Utf16CodeUnitCallback`, `FallibleUtf16CodeUnitCallback` | one UTF-16 code unit | `code_point_to_utf16`, `try_code_point_to_utf16` |
| `BytesCallback`, `ShortStringBytesCallback` | the string's bytes | `replace_with_new_string`, `replace_with_new_short_string` |
| `StringViewCallback` | a part; answers `IterationDecision!` | `StringView::for_each_split_view` |
| `Utf8ViewCallback` | a part | `Utf8View::for_each_split_view` |
| `Utf16ViewCallback` | a part; answers `IterationDecision` | `Utf16View::for_each_split_view` |
| `CharPredicate` | a byte; answers a bool | a `Function<bool(char)>` |
| `CodePointPredicate` | a code point; answers a bool | a `Function<bool(u32)>` |
- The functions that took `Function<bool(char)>` or `Function<bool(u32)>` are generic over
`P: CharPredicate` or `P: CodePointPredicate` and take `const P*`: `split_view_if`,
`ByteString::split_view`, `Utf8View::for_each_split_view`, and the GenericLexer predicate
functions.
- `IsAnyOf`, `IsNotAnyOf`, `CharPredicateFunction`, `CodePointIsAnyOf` and
`CodePointPredicateFunction` are ready-made predicates.
- `is_path_separator()` and `is_quote()` are functions, because a StringView constant is not a
constant expression.
### GenericLexer
- The struct keeps its name, `GenericLexer[TCharType]`:
- the char lexer is `generic_lexer_*` over `GenericLexer[u8]`, which the engine's stubs already
use;
- the char16_t lexer is `utf16_generic_lexer_*` over `GenericLexer[u16]`.
- Its fields are `m_input: StringView` (char lexer), `m_utf16_input: Utf16View` (char16_t lexer)
and `m_index`. FormatParser still embeds `GenericLexer[u8]`.
- The skeleton's `_opaque` overloads are `_string_view`.
- `Result<u32, UnicodeEscapeError>` is the payload enum `UnicodeEscapeResult` (`success(code_point)`
and `failure(reason)`).
- `LineTrackingLexer.m_line_start_positions` is `RedBlackTree[usize, usize]*`.
### Base64
- The simdutf option structs are gone from the signatures.
- `decode_base64_into` and `decode_base64url_into` return `DecodeBase64IntoResult`, with the cases
`decoded(count)` and `invalid(invalid_base64)`.
### Types fixed by hand
- `StringData` is `align(8)` without the flexible-array field. The bytes, or a
`StringDataSubstringData` for a substring, follow the struct at `sizeof(StringData)`.
`StringDataSubstringData` has its fields.
- `ByteStringImpl` loses `m_inline_buffer`: the characters and a NUL follow the struct.
`EmptyByteStringImpl` is an impl followed by an 8-byte zero buffer, with no reference count.
- `max_short_string_byte_count` and `string_max_short_string_byte_count` are `let 7`;
`utf16_flag` is `let usize = 63`.
- `Utf8View`'s `utf8_encoded_byte_data` table and `Utf8ViewLeadingByte` are filled in.
- The trait structs are filled in: `StringTraits`, `FlyStringTraits` and `Utf16StringTraits` in
`types_traits.lucb`, the fly-string table traits, and `AsciiCaseInsensitiveFlyStringTraits`.
## Departures from DESIGN.md §2.6
- `ak.string("…")` is `string_from_str("…")`. A module function named `string` would collide with
the many parameters named `string`, since luce-base forbids shadowing. The same pattern gives
`fly_string_from_str`, `utf16_string_from_str`, `utf16_fly_string_from_str` and
`byte_string_from_str`. `sv("…")` exists as designed, and `utf16_sv("ascii")` is its UTF-16
counterpart.
- `==` on FlyString does not compile: FlyString holds a StringBase, whose union has no equality.
Compare with `fly_string_eq_fly_string` (the eight bytes), `string_equals` or `string_base_eq`;
the Utf16 types have their `_eq_` functions too.
- Value constructors: `string_new`, `string_base_new`, `fly_string_new`, `utf16_string_new`,
`utf16_fly_string_new`, `utf16_view_new`, `string_builder_new[_usize|_mode|_mode_usize]`,
`string_view_from_bytes`, `utf16_view_from_utf16`, `utf16_view_from_ascii`,
`byte_string_from_string_view` and `fly_string_from_string`. `String {}` is the empty string;
a zero-initialized String is the invalid (null) sentinel.
## Deviations from the donor
- Creating a String or Utf16String from a StringBuilder copies the bytes. It does not adopt the
builder's ByteBuffer heap buffer, so StringData's layout does not depend on ByteBuffer's; the
builder is cleared as when its buffer is leaked. The builder still reserves the header prefix.
- Utf8View and Utf16View do not cache their `mutable` code point counts: they are values, and
writing through a `const` receiver would be undefined. A view keeps a count it was given; the
data blocks still cache theirs.
- `StringBuilder::appendvf` is declared but never defined in the donor, so it traps.
- `try_append_escaped_for_json` writes the `"\\u{:04x}"` escape directly instead of calling r03's
formatting.
- No reference counts and no destructors (DESIGN.md §3.3). A substring keeps its superstring
alive by pointing at it.
## GC follow-up (r04 / the engine)
The FlyString and Utf16FlyString tables (`s_all_fly_strings`, `s_all_utf16_fly_strings`) are
module globals. They and every interned string live in `memory.heap`, whatever allocator is
current (see `INTEGRATION.md`, "Interned strings"):
- Their bucket storage is outside the collected heap, so it needs no root.
- Their entries should be weak: the C++ removes an entry in StringData's destructor. This needs
interned data in the collected heap and a blob finalizer that calls
`did_destroy_fly_string_data` / `did_destroy_utf16_fly_string_data`. gc has no blob finalizers,
so interned strings are never released (a FIXME in `fly_string.lucb`).
String data blocks hold no managed pointers except a substring's superstring, so they can be
atomic blobs, apart from substrings.
## For ak (r01)
- **`string_hash`:** it must sign-extend bytes ≥ 0x80, because `char` is signed on the donor's
targets. Otherwise AK's hash values, and the iteration orders that depend on them, differ.
r02 calls it as `string_hash[u8](...)` and, in `utf16_view_hash`, as `string_hash[u16](...)`.
r01's current signature is non-generic (`const u8*?`), which breaks the u16 call, as r05's
scratch merge found. r01 should keep the C++ template's two instantiations, char and char16_t.
- **HashTable lookups** (FlyString and Utf16FlyString tables, `fly_string_table_find_view`,
`fly_string_table_find`, `utf16_fly_string_create_fly_string_from_cache_*`,
`utf16_fly_string_construct_utf16_string`):
- r02 expects `hash_table_find_t` and `hash_table_find_u32_tt_unary_predicate` to answer a
pointer to the element, or `none`.
- r02 passes predicate structs with `func matches(entry: const T*) -> bool`.
- r05's scratch merge found that r01 takes an `ak.Function1[const T*, bool]` and returns a
non-optional `HashTableIterator[T]`.
- This must be reconciled at merge. Either r01 adds an element-pointer lookup, or these few r02
functions (all in `fly_string.lucb` and `utf16_fly_string.lucb`) switch to a `Function1` and
compare the iterator with `end()`.
- `hash_table_set` is called with `T`.
- **ByteBuffer:**
- StringBuilder assumes a zero-initialized ByteBuffer is an empty buffer, or r01 provides an
initializer that `string_builder_construct*` can call.
- It uses `byte_buffer_size`, `capacity`, `data`, `data_const`, `span_const`, `end_pointer`,
`set_size`, `resize`, `try_resize`, `ensure_capacity`, `try_ensure_capacity`, `append_u8`,
`try_append_u8`, `try_append_span_u8`, `is_inline`, `leak_outline_buffer` and
`copy_span_u8`.
- **Vector:** r02 uses `vector_append_t`, `vector_try_append_t` and `vector_size`. The tests read
`Vector.m_outline_buffer` and `m_size`.
- **Other stubs:** LineTrackingLexer calls the RedBlackTree stubs (`red_black_tree_insert`,
`find_largest_not_above_iterator`, the iterator `key`/`deref`, `base_red_black_tree_size`). It
has no test yet.
- **Imports:** `text_support.lucb` imports `memory` and `string_conversions_1.lucb` imports
`strings`. A module may import each once, so drop the duplicate at merge if `module.lucb` also
imports them.
## Tests and the two gating flags
The ported donor tests from `Tests/AK` cover String, StringView, StringUtils, CharacterTypes,
Utf8View, ByteString, FlyString, Utf16View, Utf16String, Utf16FlyString, GenericLexer,
StringConversions, Base64, Hex, IPv4Address, IPv6Address and LexicalPath. `tests_to_number` and
`tests_text_support` add tests pinned to the donor for code without donor tests:
- simdutf (checked against the donor's libsimdutf);
- MemMem, UnicodeUtils, StringBuilder, StringNumber and Utf32View.
Regions r01 and r03 were ported in parallel, so `tests_support.lucb` has two flags:
- `var ak_core_is_ported: bool = false` gates the tests that reach r01's Vector, ByteBuffer,
HashTable, string_hash or streams. Set it to `true` after merging r01.
- `var ak_format_is_ported: bool = false` gates the tests that reach r03's vformat. Set it to
`true` after merging r03.
A gated test returns at its start, so on this branch it passes without checking anything. All
r01-gated tests were run locally against scratch stand-ins of the r01 functions (never committed)
and pass; the r03-gated ones have never run. Interning is safe under a test's fixed-buffer
allocator (see "GC follow-up"); the Utf16FlyString tests reset the table so that they count
from zero, as the donor's do.
## Compiler issue
A `#` comment at the end of a `catch failure:` line inside parentheses did not parse. See
`.donors/luce-browser-tools/compiler-issues/comment_on_catch_line_in_parens.lucb`. Fixed in Luce
0.8.13.