dymokomi/luce-browser-engine
Source of release 0.6.0
0.6.0 / docs / regions / r57.md
# Region r57: the web_test runner — what other regions need to know
Region r57 adds `tests/web_test`, the headless runner of Ladybird's Layout, Ref and Crash tests over the engine
(DESIGN.md §6), the phase-1 copy of those tests in `tests/libweb`, the expected-failures lists in
`tests/expected_failures`, and a `./test.sh` step that runs them (so CI runs them on macOS and Linux). r57 had no stub
fragment: it is new code, plus ports of the parts of `Tests/LibWeb/test-web` and `Services/WebContent` a native runner
needs.
## Using it
```sh
luce-base build tests/web_test -o build/web_test
build/web_test # all four suites, one worker per CPU
build/web_test layout -f block-and-inline/ -v
build/web_test --one tests/libweb/Layout/input/acid1.html --mode layout # one test, result on stdout
build/web_test screenshot -f css- # the Screenshot suite (phase 2 tests, below)
build/web_test --update-failures # after reviewing build/web_test_results
```
`./test.sh` builds it and runs `build/web_test layout ref crash screenshot` last. **The run fails on an unexpected failure and on
an unexpected pass.** A region whose port makes tests pass (or newly fail) reviews `build/web_test_results` and runs
`build/web_test --update-failures`, then commits the rewritten `tests/expected_failures/*.txt` with its port. The lists
only shrink by design; their notes (the first `unported:` function a test reaches) change as regions land, and
`--update-failures` keeps them current. A reason set by hand (`font`, `libm`, …) is kept by `--update-failures`.
**Paths in the notes are relative to the working directory** (added with region r48): the driver records a trap's
message with the directory it runs in (the repository root under `./test.sh`) left out of every path it names
(`relative_to_working_directory` in `driver_run`), so a note reads `loading a document that is not HTML
(tests/libweb/Ref/expected/...xht)` whatever the checkout is called, and regions no longer rewrite those notes when
they update the lists from another worktree. Run web_test from the repository root to keep them that way.
Results: `build/web_test_results/summary.txt`, `results.tsv` (path, suite, outcome, last stage, detail), and for each
test that did not pass `PATH.log` (outcome, last stage, trap message, the worker's stderr) and, for Layout,
`PATH.actual.txt` (what it dumped, also partially), and for a Ref or Screenshot test whose comparison failed
`PATH.actual.png`, `PATH.expected.png` and `PATH.diff.png` (test-web's three: the diff has the differing pixels red),
written by the worker with luce-png (`--images`). results.tsv gives a failed comparison's statistics (pixel error
count and maximum error, or the fuzzy range it missed).
## Fragments (tests/web_test, ORDER)
| Fragment | What |
| --- | --- |
| `module` | command line, `main`: driver or worker (`--one`) |
| `worker_platform` | the worker's Core event loop (timers, deferred invocations; traps "stalled" when nothing can happen), EventLoopPlugin, the test fonts and FontPlugin in layout-test mode, the main thread VM (WebContent's `main.cpp`) |
| `worker_page_client` | the headless `PageClient` (WebContent's `PageClient.cpp`: palette, 1920x1080 screen, DPR 1, zoom 1, the 60 fps frame timer that queues "update the rendering", `page_did_finish_loading`, `page_did_take_screenshot`) |
| `worker_load` | the page (`PageHost` + `HeadlessWebView::initialize_client`: fresh top-level traversable, 800x600 viewport, visible, scripting disabled), loading a file as `SVGDecodedImageData`/`load_html_document` do (DESIGN §6.1 step 3), the load wait, `afterFontsAndPaint` (fonts.ready + two native animation frames), screenshots |
| `worker_modes` | Layout (`request_internal_page_info`'s three dumps, each written as soon as it is made), Ref (`Internals::load_reference_test_metadata`, the reference loaded into the same page, fuzzy compare), Crash; a test or reference that is not HTML traps `unported (P2)` with its extension |
| `worker_screenshot` | Screenshot (`run_screenshot_test`: the fuzzy configurations without their references, the expected PNG decoded with luce-png into an unpremultiplied BGRA8888 bitmap), `write_screenshot_failure_results` (PNGs encoded with luce-png), image references of Ref tests |
| `fuzzy` | `Tests/LibWeb/test-web/Fuzzy.cpp` |
| `driver_collect` | `collect_dump_tests` / `collect_ref_tests` / `collect_crash_tests` / `collect_screenshot_tests`, `support_file_patterns`, `-f` globs (`*GLOB*`), `TestConfig.ini` |
| `driver_run` | workers through luce-std `process.Command`, `-j` at a time, `--timeout` (30 s), judging their logs and results |
| `driver_report` | expected failures, results directory, summary with the top traps |
| `tests_web_test` | 11 tests (below) |
## What other regions must know
- **Where every test stops now (main at 9647a3f, r50 merged):** all 1,474 tests that run (845 Layout, 587 Ref, 42
Crash; 16 are skipped by `TestConfig.ini`, 1 of them because it needs an HTTP server) trap in
`Gfx::SharedImageBuffer::create` (r50's `create_shareable_bitmap_backing_stores`: shared memory images are not in
luce-browser-render), reached from `Navigable::set_viewport_size` while the page is set up. So no Layout test matches
its dumps or its layout tree yet.
- **Without rendering** (`--no-rendering`, below) the 845 Layout tests stop at: 299 r51a's
`Web::Painting::ScrollState::create_scroll_frame_for` (engine half of r51a, `stubs/stub_r51a_display_list.lucb`,
from `ViewportPaintable::assign_scroll_frames` in `Document::update_layout`), 180 `GridFormattingContext` (r47),
137 `FlexFormattingContext` (r46), 113 `TableWrapper` (r48), 53 `SVGSVGElement` (r55), 21 `HTMLInputElement` (r24),
18 `MarkerPaintable::create` (r53), 17 `HTMLButtonElement` (r25), 2 `FieldSetPaintable::create` (r53),
2 `HTMLTextAreaElement` (r25). The expected-failures notes of Layout tests say both stops.
- **Rendering paints.** "Update the rendering" paints every visible document (step 22), so the faithful path (the
page visible, as a headless view is) needs backing stores, r51a/b, r52 and the CPU player (r54) before any Layout
dump. The driver therefore reruns each Layout test that trapped or crashed with `--no-rendering`: the page is one
that is never rendered, as Ladybird's pages of SVG-as-image documents are (the client answers
`is_svg_page_client`, so the navigables have no backing stores or rendering thread and "update the rendering"
skips the documents), it stays hidden, fonts.ready and animation frames are not awaited, and the dumps are taken
right after the load. The summary reports how many of those match all three dumps and the layout tree, and their
top traps. It is a diagnostic, never a pass.
- **Partial dumps count:** the layout tree is compared even when the worker trapped later (in the paint or
stacking-context dump); "layout tree matches" in the summary counts those.
- **The document path** (`worker_load`): a file: `FetchResponse` (URL list « file URL of the absolute path »,
`Content-Type: text/html`), `NavigationParams` as `tests_html_browsing`, `Document::create_and_initialize`, then
`Navigable::set_ongoing_navigation(empty)`, `set_active_document`, `Document::make_active`, and
`HTMLParser::create_with_uncertain_encoding` + `run(url)`; "the end" completes the load on the event loop and
`page_did_finish_loading(url)` reaches the client. A Ref test's reference is loaded the same way into the same page.
- **A worker reports its progress** on stderr as `web_test: stage platform|page|created|parsed|loaded|fonts-ready|
painted|screenshot|layout-tree|paint-tree|stacking-context-tree|reference-loaded|done`; the summary counts where the
failing tests stopped.
## Deviations
- `TraversableNavigable::create_a_fresh_top_level_traversable` is done by the runner without its AD-HOC emulated
geolocation position (`Geolocation::GeolocationCoordinates` is a phase-4 closure stub).
- The replaced document is not destroyed: `SVGDecodedImageData` destroys it, but a navigation unloads the previous
document and destroys it only when it is not salvageable (`Document::unload`, step 16), and `Document::destroy`
reaches `MessagePort::for_each_message_port` (phase 4).
- Palette: WebContent's `setup_palette` (zeroed theme, magenta/cyan window colors, not dark). test-web then sends
`Base/res/themes/Default.ini` (`Gfx::load_system_theme` is not ported); both are light, and no phase-1 test uses the
`-libweb-palette-*` keywords.
- Fonts: the worker loads `tests/libweb/fonts` (SerenitySans and Noto Emoji as Ladybird's resource fonts, plus
Lato-Bold and Noto Sans Tifinagh, which the CSS font tests need there); no system fonts (DESIGN §6.1).
- `[LoadFromHttpServer]` tests are skipped (no HTTP server in phase 1). The `test-wait`/`reftest-wait` classes are not
watched (they need script).
- Images are decoded and encoded with luce-png (DESIGN §7.0), not Ladybird's ImageDecoder and PNGWriter. A Ref test
whose reference is a `.png` compares the screenshot with the decoded image (test-web loads it as an image document);
no Ref test at the pin has one. Screenshot rebaselining (`--rebaseline`) is not ported: the expected PNGs are
Ladybird's, rendered by Skia (DESIGN §6.4).
- The copy (`tests/libweb/README.md`): r57 copied 845 Layout, 602 Ref, 43 Crash inputs — HTML documents without
script and resources and not WPT support files. DESIGN §1.3's 854/627/47 counted the XML documents (`.svg`, `.xht`,
`.xhtml`) and `.pdf` too, which need phase 2's XML parser. The phase-2 tests came later (below).
## Tests
`tests/web_test/tests_web_test.lucb` (11, all passing; Ladybird has none for test-web): Fuzzy.cpp's range and
configuration parsing (AK's `split` leaving out empty parts, `to_number` trimming whitespace) and matching (identical,
within ranges, the first configuration for the compared reference, sizes); a bitmap through luce-png and back, and the
three PNGs of a failed comparison; Screenshot expectation paths and comparison statistics; `-f` globs; support files; `TestConfig.ini`
groups and directory keys; reading a worker's last stage and trap; splitting the layout tree off the three dumps;
expected-failures lists; the worker's event loop (deferred invocations first, due timers in registration order,
single-shot timers once, stopped timers never, waiting for a later timer).
Local tools (not committed): `luce-browser-tools/web_test/copy_tests.py` refreshes `tests/libweb` from the pin.
## Phase 2 tests
The phase-2 copy (branch p2/tests) adds every Layout, Ref and Crash test that needs no script, and the Screenshot
suite: tests that load resources (images, style sheets, `@font-face`, frames, SVG-as-image, media) and the XML
documents (`.svg`, `.xht`, `.xhtml`) and the PDF, with every file they reach (`tests/libweb/README.md`): 922 Layout,
820 Ref, 52 Crash and 67 Screenshot tests. `web_test` gained the `screenshot` suite, and luce-png 0.4.1 (with
luce-raster 0.2.0) as a test-side dependency in `package.prisma` and `bootstrap/PACKAGES`. `./test.sh` lets the
warnings in those two packages' code pass (they have five at the pinned commits) and fails on the engine's own.
Where the tests stood when they were copied (with main's P2 skeleton, 555b9e3, merged; the expected-failures lists
name each test's first trap):
| | Tests | Pass | Fail | Trap | Skipped |
| --- | ---: | ---: | ---: | ---: | ---: |
| Layout | 922 | 855 | 0 | 67 | 0 |
| Ref | 820 | 471 | 23 | 289 | 37 |
| Crash | 52 | 38 | 0 | 13 | 1 |
| Screenshot | 67 | 30 | 21 | 14 | 2 |
The first traps, which drive the phase-2 regions:
| Tests | First trap (stub) | What reaches it |
| ---: | --- | --- |
| 189 | `loading a document that is not HTML` (156 `.xht`, 30 `.svg`, 3 `.xhtml`; web_test's `worker_require_html`) | XML documents as tests or references: the XML parser (LibXML replaced) and real navigation |
| 93 | `Web::HTML::create_potential_CORS_request` (`stub_p2d_loader_resources`) | image loading: `<img>` (87), SVG `<image>` (4), `<feImage>`, an SVG `<use>` of an external file (46 Layout, 38 Ref, 7 Screenshot, 2 Crash) |
| 45 | `Web::Fetch::Infrastructure::Request::create` (`stub_p2b_fetch_http_1`) | CSS `url()` resources: background and mask images, `@font-face` sources (30 Ref, 7 Layout, 6 Screenshot, 2 Crash) |
| 20 | `Web::Fetch::Infrastructure::request_priority_from_string` (`stub_p2b_fetch_http_2`) | `<link rel=stylesheet>` (17 Ref, 2 Layout, 1 Crash) |
| 17 | `unported (P3): JS::Array::create_from` (`stub_closure`) | `@font-feature-values` (CSSFontFeatureValuesMap) in the font-variant-alternates tests |
| 8 | `unported (P4): Web::CSS::EasingFunction::from_style_value` (`stub_closure`) | unchanged from phase 1 |
| 5 | `unported (P4): Web::HTML::AudioTrackList::AudioTrackList` (`stub_closure`) | `<video>`/`<audio>` |
| 5 | `Core::Promise<Empty>::construct` (`stub_p2d_loader_resources`) | iframes (navigating a child navigable) |
| 1 | `loading a document that is not HTML (.pdf)` | `Layout/input/pdf-viewer.pdf` |
Of the 21 Screenshot tests that render but differ from Ladybird's PNG, 13 are listed `raster` (right picture, pixels
not Skia's: glyph and edge anti-aliasing, gradient dithering, blur) and 8 `bug`: all 8 are taller than the viewport and
paint a scrollbar thumb at x 791–798 that Ladybird's full-document screenshot does not have (fixed on
fix/fidelity-text: test-web runs WebContent with `--disable-scrollbar-painting`, and so does the worker now); `css-gradients` (wrong
gradients), `outer-box-shadow` (blur), `css-color-functions` (glyphs) and `css-transform-box` differ elsewhere too, the
other four elsewhere by at most 2 a channel.
## Compiler issues
None. (`test` is a reserved word and `trap` a core name, so neither can name a variable or an enum case.)