Luce packages
LuceLuce BaseLuciaOS

dymokomi/luce-browser-css

Source of release 0.5.1

0.5.1 / docs / regions / r30a.md

# Region r30a: CSS syntax (`css_syntax`) — what other regions need to know

The `css_syntax` module is CSS Syntax Level 3 below the object model, ported from Ladybird's
`CSS/CharacterTypes.h`, `CSS/Number`, `CSS/SerializationMode.h`, `CSS/Parser/Token`,
`CSS/Parser/Tokenizer`, `CSS/Parser/TokenStream.h`, `CSS/Parser/ComponentValue` and
`CSS/Parser/Types`, plus the half of `CSS/Serialize.cpp` that tokens use (DESIGN.md §4.1.3). Each
fragment's header says which donor lines it holds, and `docs/namemap.tsv` lists every name.

## Fragments

| Fragment | Donor |
| --- | --- |
| `format_support` | the module's `String::formatted` / `appendff` / `append(String)` spellings |
| `character_types` | `CSS/CharacterTypes.h` |
| `number` | `CSS/Number.h`, `CSS/Number.cpp` |
| `serialize` | `CSS/Serialize.cpp` 18-180: escape a character (as code point), serialize an identifier, a string, a URL, a number |
| `component_value` | `CSS/Parser/ComponentValue.h`, `.cpp` |
| `token` | `CSS/Parser/Token.h`, `.cpp` |
| `token_stream` | `CSS/Parser/TokenStream.h` (with `StateTransaction`) |
| `types` | `CSS/Parser/Types.h`, `.cpp` |
| `tokenizer_1`, `_2`, `_3` | `CSS/Parser/Tokenizer.h`, `Tokenizer.cpp` 1-576, 577-962, 963-1242 |

## The two cuts into the engine

### `CSS/Serialize` moved in part

`Token::to_string`, `Function::to_string` and `Number::serialize` call the identifier, string, URL
and number serializers, so those live here (DESIGN.md §4.1.3, "file split by function"):

| C++ | css_syntax |
| --- | --- |
| `escape_a_character`, `escape_a_character_as_code_point` | same names |
| `serialize_an_identifier(StringBuilder&, StringView)` / `(StringView)` | `serialize_an_identifier_string_builder` / `_string_view` |
| `serialize_a_string(...)` | `serialize_a_string_string_builder` / `_string_view` |
| `serialize_a_url(...)` | `serialize_a_url_string_builder` / `_string_view` |
| `serialize_a_number(StringBuilder&, double)` / `(double)` | `serialize_a_number_string_builder` / `serialize_a_number_f64` |

**For r30b (engine, `CSS/Serialize`):** delete these ten stubs
(`css_serialize_an_identifier_string_builder`, …, `css_serialize_a_number_f64`, and the two
`css_escape_a_character*`) and call the `css_syntax.` functions instead; the rest of
`Serialize.cpp` stays in the engine.

### `ErrorReporter` through a hook

`AtRule::for_each_as_*` and `QualifiedRule::for_each_as_declaration_list` report misplaced
children with `ErrorReporter::the().report(InvalidRuleLocationError { outer, inner })`, and the
ErrorReporter lives in the engine. css_syntax calls

```luce
pub var invalid_rule_location_reporter: (func(ak.FlyString, ak.FlyString) -> unit)?
```

with the error's `outer_rule_name` and `inner_rule_name`; until it is set nothing is reported.
**For r30b:** at start-up (with the ErrorReporter), set it to a function that does
`css_error_reporter_report(css_error_reporter_the(), CssParsingError.css_invalid_rule_location_error(...))`.

## Signature and type changes against the skeleton

- **`TokenStream[T]` functions are constrained** `[T: TokenStreamToken[T]]` where the C++ template
  body calls something on a T (`is_empty`, `discard_a_token`, `discard_whitespace`,
  `has_next_token`, the constructors, `dump_*`). `TokenStreamToken[T]` (in
  `types_token_stream`) is the template's implicit requirements: `is`, `to_string`,
  `to_debug_string` and `make_eof` (whose `if constexpr` is a method that ignores its receiver).
  Token and ComponentValue conform. **Engine generics over a TokenStream** (`css_parser_parse_a_*[T]`
  in the r30b stubs, …) must add the same constraint when their bodies call these functions.
- **Value-form constructors** (`TokenStream stream { tokens };`):
  `token_stream_create(vector)`, `token_stream_create_span(span)`. The stream views the tokens,
  which must outlive it, as in C++.
- `token_stream_construct_vector(this, tokens: ak.Vector[T])` keeps the stub's by-value vector (the
  span points into the vector's heap buffer).
- `token_stream_of_single_token(token: const T*)` (was `token: T`): the C++ takes `T const&` and
  the stream views it; a by-value parameter would dangle.
- `token_stream_tokens_since` answers `const T[]` (was an unmapped `void*?`).
- `TokenStream.m_marked_indexes` is `ak.Vector[usize]` (was `u64`).
- **`StateTransaction` is filled in** (was opaque): `m_parent`, `m_token_stream_index: usize*`
  (the stream's `m_index`; the C++ holds `TokenStream<T>&` but only reads and restores the index,
  so one non-generic type serves every T), `m_saved_index`, `m_commit`. Its destructor is
  explicit:

  ```luce
  var transaction = css_syntax.token_stream_begin_transaction(&tokens)
  defer css_syntax.state_transaction_destroy(&transaction)
  ...
  css_syntax.state_transaction_commit(&transaction)
  ```

  `state_transaction_create_child(&transaction)` gives a child that must also be destroyed; the
  parent must not move while the child lives (as in C++).
- `Token.create*`'s `String original_source_text = {}` default has no constant Luce form: pass
  `ak.string_new()`. `token_new()` is `Token {}` (a zeroed Token holds the invalid-String sentinel,
  not empty strings); use it for `SimpleBlock.end_token` / `Function.name_token` defaults.
- Additions: `number_create(type, value)`; `component_value_from_token`, `_from_function`,
  `_from_simple_block`, `_from_guaranteed_invalid_value` (the C++ implicit and explicit
  conversions); `component_value_eq` (the defaulted `operator==`) and `component_values_eq`
  (`Vector<ComponentValue>::operator==`).
- The Tokenizer.cpp code point classes (`is_eof`, `is_quotation_mark`, …, `is_e_2` for `is_E`),
  `log_parse_error` and Token.h's `clamp_to_single_precision` are private, as their C++ internal
  linkage.
- Accessors that return `T const&` in C++ (`ComponentValue::token/function/block`,
  `TokenStream::next_token/consume_a_token/peek_token`, `Token::ident` …) return the value. It
  shares the strings and vectors, so nothing observable changes.

## Tests

`./test.sh` checks both modules with `-W` (any output fails) and runs `luce-base test` on
css_syntax (15 tests, interpreter and `--native`):

- `Tests/LibWeb/CSSTokenizer`: the 15 corpus files, copied to `tests/css_tokenizer`, driven the way
  `test-css-tokenizer.py` drives `css-tokenizer` (encodings from `*.css.encoding`).
- `tests/css_tokenizer_extra`: 11 edge-case files (numbers, escapes, urls, hashes, comments,
  strings, filtered code points, multibyte columns, UTF-16BE, Shift_JIS, empty) whose expected
  dumps were written by the donor's `css-tokenizer` locally.
- `TestCSSTokenStream.cpp`: `basic` and `marks`.
- Focused cases pinned to the donor through a local probe program: `Token::to_string` of every
  token type, `Number::to_string` (including AK's `{:.6}` of 1e20, which prints
  `18446744073709551615.555556` in the donor and here), `round_to_nearest_integer`, identifier,
  string and URL serialization, ComponentValue strings and source text, TokenStream over
  ComponentValues, transactions (rollback, commit through a child), and the rule-list walkers
  with their reports.

Test bodies run `with memory.heap:` because interned FlyStrings outlive the per-test allocator
(`luce-browser-foundation/docs/regions/r02.md`); without it a later test reads a FlyString table
entry whose memory the runner has reused.

## Compiler issues

None found.