Luce packages
LuceLuce BaseLuciaOS

dymokomi/luce-browser-engine

Source of release 0.6.0

0.6.0 / docs / regions / r21.md

# Region r21: HTML tree construction — what other regions need to know

Region r21 ports `HTML/Parser/HTMLParser.cpp/.h` (the tree builder: every insertion mode, foreign
content, the adoption agency algorithm, foster parenting, "the end" and `HTMLParserEndState`,
fragment parsing and serialization, dimension and legacy color values), `StackOfOpenElements`,
`ListOfActiveFormattingElements`, `HTMLEncodingDetection`, `IncrementalDocumentParser`,
`SpeculativeHTMLParser` and `SpeculativeMockElement` (`ParserScriptingMode.h` has only its enum).
The stub fragment `stubs/stub_r21_html_tree_construction.lucb` is gone; all 184 of its functions
are ported. The engine's `stubs/stub_r20_html_tokenizer.lucb` (the old `set_parser` signature) is
gone too: the parser uses `html_syntax.html_tokenizer_set_parser` (HANDOFF §5 item 4).

## Fragments (in `html/parser/` unless noted)

| Fragment | Donor |
| --- | --- |
| `html_encoding_detection` | `HTMLEncodingDetection.cpp` |
| `stack_of_open_elements` | `StackOfOpenElements.cpp/.h` |
| `list_of_active_formatting_elements` | `ListOfActiveFormattingElements.cpp/.h` |
| `html_parser_1` | `HTMLParser.cpp` 1-557: port helpers, integration points, constructors, `run`, `the_end`, `HTMLParserEndState` |
| `html_parser_2` | 558-1132: dispatch, quirks, initial / before html / before head, insertion location, creating and inserting elements |
| `html_parser_3` | 1134-1727: in head (+ declarative shadow roots), in head noscript, character insertion, after head |
| `html_parser_4` | 1729-2191: implied end tags, after body, reconstructing formatting elements, adoption agency, special category |
| `html_parser_5`, `_6`, `_7` | 2193-3441: in body (one C++ function, continued across three functions in order; `AnyOtherEndTag` and `BRStartTag` labels are functions), token adjustments |
| `html_parser_8` | 3443-3931: parser-blocking scripts, text, table contexts, in row / cell / table text |
| `html_parser_9` | 3933-4537: in table body / table / caption / column group / template |
| `html_parser_10` | 4539-5057: frameset modes, foreign content (`ScriptEndTag` label is a function), reset the insertion mode |
| `html_parser_11` | 5059-5510: fragment parsing, factories, escaping, fragment serialization |
| `html_parser_12` | 5512-5892 and `HTMLParser.h`'s inline members: dimension / legacy color / align values, speculative parser, abort, inserting an element |
| `incremental_document_parser`, `speculative_html_parser`, `speculative_mock_element` | the matching `.cpp` |
| `html/tag_names`, `html/attribute_names`, `svg/tag_names`, `math_ml/tag_names` | the FlyString definitions of `HTML/TagNames.cpp`, `HTML/AttributeNames.cpp`, `SVG/TagNames.cpp`, `MathML/TagNames.cpp` |
| `tests_html_parser_support`, `tests_html_encoding_detection`, `tests_html_parser_1`, `tests_html_parser_2` | tests (Ladybird has no unit tests for these files) |

## What other regions must know

### Name tables (r22, r55, r13b and everyone comparing tag or attribute names)

`web_initialize_strings` now interns `HTML::TagNames`, `HTML::AttributeNames`, `SVG::TagNames`
and `MathML::TagNames` (`html_tag_names_initialize_strings`, … generated locally from the
donor headers' X-macro lists) and the stack of open elements' `s_base_list`. They were ported
ahead of r22 (AttributeNames' `is_boolean_attribute` stays in r22's stub), r55 and r13b because
the parser compares every token against them. `HTML::EventNames` are still not interned (r22).

### Using the parser (r17 DocumentLoading, r28, r57 web_test)

- `html_parser_create_with_uncertain_encoding(document, bytes, mime_type)` sniffs the encoding
  (BOM, transport charset, prescan of the first 1024 bytes, then windows-1252 for invalid UTF-8,
  else UTF-8), creates the parser (which sets the document's parser and encoding) and
  `html_parser_run_url(parser, url)` parses to completion and runs "the end".
- Scripting disabled: `<script>` elements are created (their setters and `prepare_script` are
  r27's) and `<noscript>` in head is parsed as with scripting disabled. The speculative parser
  only starts for a pending parser-blocking script, so phase 1 never reaches its fetches.
- Every created element goes through `dom_create_element`; every insertion outside fragment
  parsing goes through `html_relevant_similar_origin_window_agent` (r29) for the custom element
  reactions stack.

### Signature and type changes

- Tokens: the parser's functions take `html_syntax.HtmlToken*` (C++ `HTMLToken&` and
  `HTMLToken const&`); `create_element_for`, `insert_foreign_element` and `insert_html_element`
  took the token by value in the stub. `HTMLToken::make_start_tag(...)` temporaries go through the
  private `html_parser_insert_html_element_for_tag_name`.
- `html_list_of_active_formatting_elements_create_own_token -> HtmlToken*` (never null),
  `_find_index -> usize?` (was `u64?`), `_entries_const` and
  `html_stack_of_open_elements_elements_const` answer const pointers (C++ `const&`); the stack's
  `has_in_scope_impl` take the list as `const ak.Vector[ak.FlyString]*` (private).
- `html_parser_insertion_mode_name -> str`; `html_log_parse_error(location: io.Location =
  luce.location)` as r20's tokenizer (module.lucb imports `io`); `escape_string<ViewType>` is
  `html_escape_string_utf8_view` / `_utf16_view` (private).
- Types filled: `HtmlParserAdjustedInsertionLocation` (`parent`, `insert_before_sibling`),
  `HtmlListOfActiveFormattingElementsEntry` (`element`, `token: html_syntax.HtmlToken*?`),
  `HtmlStackOfOpenElementsLastElementResult` (`element`, `index: isize`);
  `html_s_quirks_public_ids` is a `let str[55]` (was a `var` Vector).
- Other regions' stubs (signatures only): r27's `html_script_element_set_parser_document`,
  `_set_force_async`, `_set_already_started` and `_prepare_script_t` lost their `[T]` (C++
  `Badge<T>` parameters).
- LibJS: `vm_has_running_execution_context`, `vm_save_execution_context_stack`,
  `vm_clear_execution_context_stack`, `vm_restore_execution_context_stack` (`Vm` gained
  `m_saved_execution_context_stacks`, whose contexts gather_roots reports).
- Closure (`stubs/stub_closure.lucb`, namemap rows): `fetch_legacy_extract_an_encoding` (ported
  whole), `fetch_body_wait_for_sniff_bytes` and `fetch_body_incrementally_read` (P2 traps; the
  task destination is the object form of C++'s variant).

## Deviations and notes

- C++ `goto AnythingElse` / `goto value` / `goto spaces` are flags that skip the entries between
  the jump and its label; `reconstruct_the_active_formatting_elements`' Rewind/Advance/Create are
  two loops. handle_in_body is split into `html_parser_handle_in_body`,
  `_continued` and `_continued_2`, called in order (each part keeps its entries' C++ order).
- `issue_speculative_fetch` and the `rel=preload` branch of `speculative_fetch` trap
  "unported (P2)" (create_potential_CORS_request, Fetch::Fetching::fetch,
  translate_a_preload_destination are outside the closure). `IncrementalDocumentParser` cannot
  visit `m_body` (Fetch's Body is opaque until P2).
- The static adjustment tables of `adjust_svg_tag_names` / `adjust_svg_attributes` /
  `adjust_foreign_attributes` are `str` tables interned where they are used.
- Donor quirks kept: the foreign-content "any other end tag" step 2 logs a parse error when the
  names *match* (the C++ test is inverted); `parse_legacy_color_value` of a whitespace-only
  string fails a VERIFY (`input[0]` on an empty ByteString) and the port traps likewise.

## Tests

`tests_html_encoding_detection` (6 tests): BOM sniffing, the prescan and the sniffing algorithm
for 28 inputs, the 1024-byte limit, the transport charset, extracting an encoding from a meta
element (12 inputs), prescan_get_attribute, and 32 legacy color values — all against a local
oracle compiling the donor's HTMLEncodingDetection.cpp and parse_legacy_color_value with the
reference build's AK, LibTextCodec and LibGfx. `tests_html_parser_1` (9 tests): the stack of
open elements and the list of active formatting elements (scopes, popping, Noah's Ark, markers).
`tests_html_parser_2` (21 tests): which_quirks_mode (20 DOCTYPEs, oracle values), the special
category and integration points, the dispatcher, implied end tags, resetting the insertion mode
(document and fragment cases), the insertion-mode entries that only pop, the adoption agency's
early returns, foster-parenting insertion locations, token adjustments, escaping, fragment
serialization and an html5lib-format dump of a hand-built tree, dimension-value failures, and
the constructors on r17's `test_document_in_realm` (parser and encoding set on the document,
encoding sniffing through create_with_uncertain_encoding).

### Scenario tests to add once the blockers land

Building a tree end to end traps today: appending the DOCTYPE or any node runs
`invalidate_node_style` (r42), creating any element runs
`invalidate_style_after_custom_element_state_change` (r42) and the HTML element constructors
(r22 `HTMLElement`, r26/r27 the element classes), inserting an element outside fragment parsing
needs `relevant_similar_origin_window_agent` (r29), and `the_end`'s readiness change fires an
event (r29). Then add, comparing `parser_test_dump` (html5lib format) with the reference build's
DOM: the html5lib tree-construction cases (tests1-26, adoption01/02, tables01, template,
foreign-fragment, math, svg, noscript01 with scripting disabled, domjs-unsafe), whole documents
through `create_with_uncertain_encoding` + `run(url)` in each encoding, `parse_html_fragment` for
each context element kind, declarative shadow roots, the `option`/`selectedcontent` popped hook,
form association, and `parse_dimension_value` / `parse_table_child_element_align_value` values
(their style values are r34's).

## Compiler issues

None found. (`let x = f() else:` with a block and an empty `pass` statement do not exist in the
language; written as `if x == none:` blocks and explicit `if ...: break` loops.)