Luce packages
LuceLuce BaseLuciaOS

dymokomi/luce-browser-html

Source of release 0.5.1

0.5.1 / docs / regions / r20.md

# Region r20: the HTML tokenizer (`html_syntax`) — what other regions need to know

Region r20 ports Ladybird's HTML tokenizer: `HTMLTokenizer`, `HTMLToken`, the named character
reference matcher (`Entities`) and the generated `NamedCharacterReferences` tables, with their
generator. It is the whole of `luce-browser-html`. Each fragment's header describes it, and
`docs/namemap.tsv` lists every name. The stub fragment `stub_r20_html_tokenizer.lucb` is gone.

| Fragment | Donor |
| --- | --- |
| `entities` | `Entities.cpp`, `Entities.h`, the inline functions of the generated `NamedCharacterReferences.h` and the lookup of `NamedCharacterReferences.cpp` |
| `named_character_reference_tables` | the tables of the generated `NamedCharacterReferences.cpp`, written by `tools/gen_entities` |
| `html_token` | `HTMLToken.h`, `HTMLToken.cpp` |
| `html_tokenizer_1` | `HTMLTokenizer.cpp` lines 1-319: the macros, the input stream, `next_token`'s frame |
| `html_tokenizer_2` … `_8` | the 80 states of `next_token`, in the donor's order |
| `html_tokenizer_9` | `HTMLTokenizer.cpp` lines 2881-3076 and the inline members of `HTMLTokenizer.h` |

## Signature and type changes against the skeleton

### For the engine's HTMLParser (region of `HTML/Parser/HTMLParser.cpp`)

- **The parser hook (DESIGN.md §4.1.3).** `HTMLTokenizer::set_parser(Badge, HTMLParser&)` is
  `html_tokenizer_set_parser(tokenizer, adjusted_current_node_is_foreign: func(void*) -> bool,
  parser: void*)`. The parser passes itself and a function answering
  `adjusted_current_node() && adjusted_current_node()->namespace_uri() != Namespace::HTML`; the
  markup declaration open state calls it for `<![CDATA[`. Without a parser (syntax highlighting),
  CDATA is a bogus comment, as in C++. `html_tokenizer_visit_edges` visits nothing: the parser owns
  the tokenizer, and the context pointer is the parser itself.
- `Optional<HTMLToken>` (a SentinelOptional in AK) is `HtmlToken?`. `next_token` answers it.
- The Badge overloads lose their Badge: `html_tokenizer_switch_to_html_parser(t, state)` is
  `switch_to(Badge<HTMLParser>, State)`; `html_tokenizer_switch_to_html_tokenizer_state(t, state)`
  is the public `switch_to(State)`; `html_tokenizer_parser_did_run(t)`;
  `html_token_set_start_position` / `_end_position` take no Badge.
- `html_token_doctype_data(token) -> const HtmlTokenDoctypeData*` and
  `html_token_last_attribute_const(token) -> const HtmlTokenAttribute*` answer pointers (the C++
  `const&`); `html_token_last_attribute(token) -> HtmlTokenAttribute*` and
  `html_token_ensure_doctype_data(token) -> HtmlTokenDoctypeData*` are the mutable ones.
- `HTMLToken() = default` is `html_token_construct(&token, .invalid)`; `HTMLToken::Attribute {}`
  is `html_token_attribute_new()` (empty name and value, no prefix or namespace, zero positions).
- The `for_each_attribute` overloads take an `ak.Function1[HtmlTokenAttribute*,
  ak.IterationDecision]` (const: `const HtmlTokenAttribute*`); inside this module the lambdas that
  only walk the list are loops over `html_token_tag_attributes`.
- **Tokens are copies.** HTMLToken is move-only in C++; in the port a `move` is a copy
  (DESIGN.md §3.3 rule 1), so two copies of a tag token share its attribute list and two copies of
  a DOCTYPE token share its DoctypeData. The tokenizer's own `move(m_current_token)` goes through
  `html_tokenizer_move_current_token`, which leaves the current token without its attribute list
  or DOCTYPE data as the moved-from C++ OwnPtr is null, so the tokenizer never changes a token it
  has handed out.
- `HTMLTokenizer::state_name` answers `str`.

### Types fixed by hand

- `HtmlTokenAttribute` and `HtmlTokenDoctypeData` were opaque in the skeleton; their fields are
  filled in `types_html_token.lucb` (the C++ default member initializers are applied where the port
  creates them).
- `HtmlTokenizer.m_insertion_point: isize?` and `m_old_insertion_points: ak.Vector[isize?]` (the
  generator wrote `i64`): the donor's `ssize_t`, compared and assigned with `m_current_offset`.
- The four `extern` tables of `NamedCharacterReferences.h` are not `var` pointers but the `let`
  arrays of `named_character_reference_tables.lucb` (`g_named_character_reference_nodes` …);
  `g_named_character_reference_codepoints_lookup` (a `static` in C++) is private.

## How next_token is structured

`next_token` is one C++ function with a label per state, entered by `goto` from the state
machine's macros. luce-base has no `goto`, so (see `html_tokenizer_1`'s header):

- each `BEGIN_STATE(X)` is a function `html_tokenizer_x_state(this, current_input_character,
  stop_at_insertion_point) -> TokenizerStep`, and `html_tokenizer_run_state` is the `switch`;
- `next_token` is three loops: `start_of_function` (hands out a queued token, checks `m_aborted`),
  `consume_next` (the `for (;;)`: the pause check and `next_code_point`) and `reconsume` (runs the
  current state on the same character);
- each macro is a function of the same name that does what the macro does to the tokenizer and
  answers where to go: `SWITCH_TO` and `continue;` → `.consume_next` (the macro's own pause
  check and `CONSUME_NEXT_INPUT_CHARACTER` are exactly the loop head), `RECONSUME_IN` →
  `.reconsume`, `SWITCH_TO_RETURN_STATE` → `.start_of_function`, `return x` → `.returned(x)`;
- the two `goto AnythingElse…` labels inside states fall through to the anything-else code.

Every state keeps its spec comment and its `ON` order.

## Deviations from the donor

- `log_parse_error(SourceLocation = current())` takes `io.Location = luce.location`. The
  `dbgln_if(TOKENIZER_TRACE_DEBUG, …)` messages are ported under `if tokenizer_trace_debug`
  (`false`, as in the donor build).
- `StringBuilder::append(char)` called with a `u32` (three end tag name states) is
  `string_builder_append_u8(builder, (u8)code_point)`, the implicit conversion's low byte.
- `m_character_reference_code` arithmetic wraps (`*%`, `+%`), as C++'s u32 does.
- The generator of the tables, `Meta/Generators/generate_libweb_html_named_character_references.py`,
  is ported as the Luce program `tools/gen_entities`, reading the copied `data/Entities.json` (DESIGN.md
  §5). `test.sh` regenerates the tables and compares them with the committed fragment. The C++
  header the script also writes is ported by hand (the types, `entities.lucb`).

## Donor quirks kept

- Asking `next_token` again after it answered the end-of-file token re-runs the current state's
  EOF branch. In the comment, DOCTYPE-identifier and some other states that branch touches the
  moved-from current token, and the donor fails a VERIFY (`is_comment()`, `is_doctype()`); the
  port fails the same `assert`. TestHTMLTokenizer's `run_tokenizer` loops until none, so the
  donor's own harness crashes on inputs like `<!-- unterminated`; the pinned cases stop after EOF.
  HTMLParser stops at EOF, so this does not arise there.
- Character tokens are emitted without `will_emit`, so their end position is 0:0, and a streaming
  tokenizer that backtracks over an overconsumed named character reference can lose its source
  positions (they become 0:0). The pinned cases record both.

## Tests

- `tests_html_tokenizer`: all 16 cases of `Tests/LibWeb/TestHTMLTokenizer.cpp`, including the
  regression hash of `tokenizer-test.html` (its 594 bytes are embedded).
- `tests_html_tokenizer_cases_1`, `_2`: 144 inputs covering every state (DOCTYPE, comments,
  markup declarations, tags and attributes, character references in text and attributes,
  newline normalization and positions, and RCDATA, RAWTEXT, script data, PLAINTEXT and CDATA
  entered as the parser enters them), each token dumped with all its fields and compared with the
  donor's dump. The expected dumps come from a local oracle linking the reference build
  (`.donors/luce-browser-tools/html_tokenizer_oracle`: `oracle < cases.txt`); the fragments are
  generated from its output.
- `tests_html_tokenizer_scenarios`: the input stream API against the same oracle
  (`oracle --scenarios`): streaming input with a CR and a named reference split across chunks,
  insertion points with `document.write`-style insertion, nested insertion points, `abort`, and a
  windows-1252 source.
- `tests_entities` and `tests_entities_table`: every one of the 2231 named character references of
  `Entities.json` through the matcher; the matcher's state after each code point for 15 words
  (`oracle --matcher WORD`); the second code points and the unique index lookup.
- `tests_html_token`: the HTMLToken members the tokenizer does not reach (the tree builder's
  `adjust_*`, lookups, `for_each_attribute` with `Break`, the self-closing acknowledgement,
  `normalize_attributes`, `to_string`), the parser hook, `switch_to` and `state_name`.
- `tests_gc_stress`: the regression case and three pinned cases with a `gc.Heap` as the current
  allocator collecting before every allocation (664 and 49 collections).

`./test.sh` checks the module with `-W` (any output fails), runs the 55 test blocks, and
regenerates the tables. All pass.

## Compiler issues

None found.