Repository navigation
LALR(1) parser from official MySQL grammar - #429
Conversation
🤖 Lexer benchmarkChanges to lexer-related files were detected and triggered a benchmark:
Note: Hosted runners are noisy, and absolute numbers vary. Treat the results with caution and verify them locally. To reproduce locally: |
df8874b to
70d642d
Compare
b3c39da to
1f88932
Compare
Add a new monorepo package for a MySQL parser generated from the official MySQL grammar. This commit sets up the package metadata; the source, tooling, and documentation follow in later commits.
Bring the MySQL lexer and the token and node classes over from the mysql-on-sqlite package unchanged, so the later adaptation to the official grammar is reviewable as a focused diff, and register src/ as the package Composer classmap (the WordPress-style file names rule out PSR-4).
296a9c5 to
0f841c5
Compare
Compile the grammar from the official MySQL sources: fetch sql_yacc.yy and lex.h at a pinned, checksum-verified mysql-server tag; run a pinned Bison build (Docker, version-asserted) to produce the automaton; compact the automaton into plain PHP ACTION/GOTO tables (about 7% of the dense cells); and derive the keyword table and token constants from lex.h, failing the build on any unresolved terminal. bin/build-grammar (composer run build-grammar) runs the pipeline end to end.
a9f8619 to
aca2c38
Compare
Commit the LALR(1) parse table produced by bin/build-grammar: a plain PHP array that compacts the grammar's dense ACTION/GOTO automaton to about 7% of its cells. Regenerate with composer run build-grammar. The token-level data (keyword table, paren-gated function keywords, and token constants) is generated into the lexer itself; see the next commit.
A table-driven LALR(1) shift-reduce runtime (WP_Parser) over a WP_Parser_Grammar that expands a compact, generated ACTION/GOTO parse table, building a WP_Parser_Node AST. The grammar is unambiguous for LALR(1), so the loop is deterministic, with no conflict handling or backtracking. A rule that matches nothing produces no node, so empty optional rules are absent from the tree. This is grammar-agnostic: it knows nothing about MySQL, only how to run an LALR(1) parse table. Adapt the copied parse-tree primitives to the package: the runtime builds each node in a single step, so the old recursive parser's merge_fragment() is dropped, and the node and token docblocks no longer reference that parser.
Wire the generated MySQL parse table into the generic LALR(1) runtime through a factory: WP_MySQL_Parser_Factory::create_parser() builds a WP_Parser over a WP_Parser_Grammar loaded from src/mysql-parse-table.php. The grammar is expanded once and shared between created parsers; create_grammar() exposes a fresh grammar for callers that want their own. This is the only piece that knows the parser is being used for MySQL.
Cut the generated parse table from 190 KB to 177 KB (-7%) with no behavior change: most shifts on a given terminal go to the same successor state, so those cells are stored as bare token lists (action_row_shift_tokens) and restored from a per-terminal target table (action_shift_targets) when the grammar is constructed. The smaller file also parses faster on a cold opcache.
Bring the query corpus extracted from the MySQL server test suite, with the tooling that generates it, into the package: data/mysql-server-query-corpus/ plus a bin/build-corpus orchestrator (composer run build-corpus) that fetches the mysql-test directory at the pinned tag and extracts the queries. The SQLite driver package keeps its own copy for now; it will be retired when the driver is ported to this package.
Measure the corpus parse rate and end-to-end (lex + parse) throughput, with warmup and timed passes. The parser accepts 99.76% of the ~69k corpus queries.
Cover the token stream, the scanner (the exhaustive unit suite ported from the SQLite driver), the parser runtime, token value and name resolution, generated grammar-data invariants, and a corpus regression test pinning the exact acceptance tally. Run the suite on the oldest and newest supported PHP versions in CI.
Tokenizing a whole statement routed every token through the pull iterator (next_token -> produce -> scan_lexeme -> read_next_token -> enqueue_token), adding ~4 method calls plus token-queue bookkeeping per token over a plain scan-and-emit loop. Give remaining_tokens() a tight fast path that emits the common single-token lexemes inline and delegates only the rare multi-token ones (@, WITH ROLLUP, end markers) to the buffered producers. The pull API is unchanged and the output is byte-identical; ~24% faster (no JIT) / ~16% (JIT) end-to-end over the MySQL server corpus.
The pull iterator buffered produced tokens in a dynamic $token_queue drained by index. A scan step yields at most two grammar tokens, so a single $pending_token slot suffices: next_token() returns the first and holds the second. The multi-token producers (@, WITH ROLLUP, end markers) now append to a caller-supplied array, shared directly by both next_token() and remaining_tokens() — removing the queue bookkeeping and the duplicated drain in the fast path. A make_token() helper unifies token construction. Output is byte-identical and throughput is unchanged (the multi-token cases were already off the hot path); this is a structural cleanup.
| list( $type, $start, $length ) = $this->lookahead; | ||
| $this->lookahead = null; | ||
| } else { | ||
| // Inlined scan_lexeme(): skip whitespace and comments, then scan. |
There was a problem hiding this comment.
we could remove all comments like that one, LLMs love to leave them for continuity between commits, but it doesn't really mean anything anymore
| // Token constants share the class with the lexer's own constants; | ||
| // grammar tokens are the non-negative ints that are not SQL modes | ||
| // (the scanner sentinels are negative, the SQL modes are bit flags). | ||
| foreach ( ( new ReflectionClass( self::class ) )->getConstants() as $name => $value ) { |
There was a problem hiding this comment.
Reflections look concerning, especially in such a prominent code path. What's the performance hit?
There was a problem hiding this comment.
Getting back to the LALR(1) work and your review here.
This comes from the old lexer, and it is not a hot path. Only get_name() and __toString() call this, both are debug-only, and no production code calls them.
| $type = self::LESS_OR_EQUAL_OPERATOR; | ||
| } | ||
| } elseif ( '>' === $next_byte ) { | ||
| $this->bytes_already_read += 1; // Consume the '>'. |
There was a problem hiding this comment.
These comments seem pretty redundant.
| // in the range of U+0080 to U+FFFF before looking at further bytes. | ||
| // If it can't, bail out early to avoid unnecessary UTF-8 decoding. | ||
| // Identifiers are usually ASCII-only, so we can optimize for that. | ||
| $byte_1 = ord( |
There was a problem hiding this comment.
An inline utf-8 parser? :D Let's see if it makes any sense to adapt the one @dmsnell built for WordPress core, that's less maintenance, less opportunity to have an error, and probably more performance.
There was a problem hiding this comment.
Getting back to the LALR(1) work and your review here.
This comes from the old lexer and targets unquoted MySQL identifiers specifically. It is not a full UTF-8 parser (unquoted MySQL identifiers allow only ASCII a-z, A-Z, 0-9, _, $, and Unicode U+0080-U+FFFF), and that makes it much smaller and simpler.
This implementation also optimizes for the fact that the majority of real MySQL identifiers are ASCII only (the strspn path with an early "could be multi-byte" gate) and is probably the fastest thing we can do in PHP for this use case.
The tests carried over from the old lexer cover every BMP code point and every 2-byte and 3-byte sequence against PCRE to ensure the implementation is 100% correct.
The core _wp_scan_utf8() solves a different problem. It finds the longest valid UTF-8 run, which includes spaces, operators, and 4-byte characters. We’d still need the identifier rules on top of it, and since this package doesn’t depend on WordPress, we’d have to copy it over.
| } | ||
|
|
||
| private function read_number(): ?int { | ||
| // @TODO: Support numeric-only identifier parts after "." (e.g., 1ea10.1). |
There was a problem hiding this comment.
Should this be addressed before merging?
| * A backslash with any other character represents the character itself. | ||
| * That is, \x evaluates to x, \\ evaluates to \, and \🙂 evaluates to 🙂. | ||
| */ | ||
| $preg_quoted_backslash = preg_quote( $backslash ); |
There was a problem hiding this comment.
Can we just compute this on paper and inline it in the next line? :-)
| * https://github.com/mysql/mysql-workbench/blob/8.0.38/library/parsers/grammars/MySQLLexer.g4 | ||
| * https://github.com/mysql/mysql-workbench/blob/8.0.38/library/parsers/mysql/MySQLBaseLexer.cpp | ||
| */ | ||
| class WP_MySQL_Lexer { |
There was a problem hiding this comment.
I'd love to understand better how was this was adapted from the default lexer
|
Really good work here @JanJakes! I've left some non-blocking comments and will proceed with merging 🎉 |
## Release `3.0.0-rc.6` Version bump and changelog update for release `3.0.0-rc.6`. **Changelog draft:** * Don't take a write lock for SET statements (fixes "database is locked" on connect) ([#443](#443)) * Add REVERSE() to user defined functions ([#434](#434)) * LALR(1) parser from official MySQL grammar ([#429](#429)) **Full changelog:** v3.0.0-rc.5...release/v3.0.0-rc.6 ## Next steps 1. **Review** the changes in this pull request. 2. **Push** any additional edits to this branch (`release/v3.0.0-rc.6`). 3. **Merge** this pull request to complete the release. Merging will automatically build the plugin ZIP and create a [GitHub release](https://github.com/WordPress/sqlite-database-integration/releases). > [!NOTE] > This is a **pre-release**. It will not be deployed to [WordPress.org](https://wordpress.org/plugins/sqlite-database-integration/).
Both of our MySQL lexers, the new `mysql-parser` lexer and the `mysql-on-sqlite` lexer that the driver still uses, read some queries differently from MySQL. Some valid queries didn't parse, and some invalid ones did. **This change fixes these cases, so both lexers now read them like MySQL's own lexer:** - `t.1`, `t.1st`, and `t.1e5` are qualified names. Before, `.1` was read as a decimal number. - A keyword right before a dot and a name is an identifier, so `order.id` works, while `@@session.sql_mode` keeps its meaning. - A name right after a dot is always an identifier, so `t.x'41'` is the column `x` followed by a string, not a hex literal. - A dot that ends a number doesn't make the next word a name, so `SELECT 1.FROM dual` now parses. - The name after `@@` can't be empty, so `SELECT @@ version` and `SELECT @@'version'` are now syntax errors. It's also never a number or a literal, so `@@123` is a variable name. A dot after a quoted name or a space still starts a number, as in MySQL. All cases were checked against **MySQL 8.4**, and one more query from the MySQL server test corpus now parses. **The driver now stores CHECK and DEFAULT expressions without spaces around dots and after `@@`.** It parses these expressions again to create the SQLite table. With spaces, `t . order` contains the `ORDER` keyword, and `@@ max_connections` has an empty variable name. Before, a constraint like `CHECK (t.order > 0)` failed with an internal error. **The lexers now track a state that each token sets for the next one, like MySQL's `next_state`.** The states use MySQL's names, written out in full (for example, `STATE_IDENTIFIER_SEPARATOR` for `MY_LEX_IDENT_SEP`), so the code is easy to compare with `sql_lex.cc`. Lexing speed stays within noise of trunk, with and without opcache and JIT. The PR also applies smaller improvements from the review of the new lexer: - Removes comments that only repeated the code or described its history. - Writes the backslash pattern in token unescaping as a plain literal. - Keeps the unquoted identifier characters in one shared constant. Follows up on #429. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved handling of qualified SQL names, including names that begin with digits or match reserved words. * Corrected parsing and display of system-variable references in stored expressions. * Rejected invalid or missing names after `@@` while preserving support for backtick-quoted names. * Improved parsing of SQL statements that previously failed. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…#607) The `mysql-parser` lexer still used the token names of the MySQL Workbench lexer it was ported from, such as `OPEN_PAR_SYMBOL` or `INT_NUMBER`. The AST nodes already use names from the MySQL grammar (`sql_yacc.yy`), so code that uses the parser had to mix two naming schemes. **This PR names all lexer tokens after the MySQL grammar:** - Tokens use their `sql_yacc.yy` names, such as `IDENT`, `TEXT_STRING`, `NUM`, or `GT_SYM`. - Single-character tokens have no usable grammar name, so they are named after the character, such as `LEFT_PAREN`, `COMMA`, or `ASTERISK`. - The end marker is `YYEOF`, as in the C code that Bison generates. - `SINGLE_QUOTED_TEXT` and `DOUBLE_QUOTED_TEXT` were two names for one token, so they are now one `TEXT_STRING`. Keywords are still looked up by their SQL spelling in `WP_MySQL_Lexer::KEYWORDS`. The README now explains how to refer to grammar rules, keywords, and other tokens. **The lexer now defaults to MySQL 8.4.10**, the version of its grammar and query corpus. With the old 8.0.38 default, version comments like `/*!80400 ... */` were skipped. Three corpus queries with six-digit version comments now parse too. The PR also fixes a few leftovers: - The grammar generators now use the MySQL tag of the fetched sources. Before, the build could write the old `mysql-8.4.3` tag into the parse table header. - The unused `WP_MySQL_Lexer::get_token_id()` is removed. - The README no longer points to a token data file that doesn't exist. - A `read_quoted_text()` call no longer passes an argument that the method ignores. <details> <summary>All renamed token constants</summary> | Old | New | |---|---| | `OPEN_PAR_SYMBOL`, `CLOSE_PAR_SYMBOL` | `LEFT_PAREN`, `RIGHT_PAREN` | | `OPEN_CURLY_SYMBOL`, `CLOSE_CURLY_SYMBOL` | `LEFT_BRACE`, `RIGHT_BRACE` | | `COMMA_SYMBOL`, `DOT_SYMBOL`, `COLON_SYMBOL`, `SEMICOLON_SYMBOL`, `AT_SIGN_SYMBOL` | `COMMA`, `DOT`, `COLON`, `SEMICOLON`, `AT_SIGN` | | `PLUS_OPERATOR`, `MINUS_OPERATOR`, `MULT_OPERATOR`, `DIV_OPERATOR`, `MOD_OPERATOR` | `PLUS`, `MINUS`, `ASTERISK`, `SLASH`, `PERCENT` | | `BITWISE_AND_OPERATOR`, `BITWISE_OR_OPERATOR`, `BITWISE_XOR_OPERATOR`, `BITWISE_NOT_OPERATOR`, `LOGICAL_NOT_OPERATOR` | `AMPERSAND`, `PIPE`, `CARET`, `TILDE`, `EXCLAMATION` | | `EQUAL_OPERATOR`, `NULL_SAFE_EQUAL_OPERATOR`, `NOT_EQUAL_OPERATOR` | `EQ`, `EQUAL_SYM`, `NE` | | `GREATER_OR_EQUAL_OPERATOR`, `GREATER_THAN_OPERATOR`, `LESS_OR_EQUAL_OPERATOR`, `LESS_THAN_OPERATOR` | `GE`, `GT_SYM`, `LE`, `LT` | | `ASSIGN_OPERATOR`, `SHIFT_LEFT_OPERATOR`, `SHIFT_RIGHT_OPERATOR` | `SET_VAR`, `SHIFT_LEFT`, `SHIFT_RIGHT` | | `LOGICAL_AND_OPERATOR`, `LOGICAL_OR_OPERATOR`, `CONCAT_PIPES_SYMBOL` | `AND_AND_SYM`, `OR2_SYM`, `OR_OR_SYM` | | `JSON_SEPARATOR_SYMBOL`, `JSON_UNQUOTED_SEPARATOR_SYMBOL` | `JSON_SEPARATOR_SYM`, `JSON_UNQUOTED_SEPARATOR_SYM` | | `IDENTIFIER`, `BACK_TICK_QUOTED_ID` | `IDENT`, `IDENT_QUOTED` | | `SINGLE_QUOTED_TEXT`, `DOUBLE_QUOTED_TEXT`, `NCHAR_TEXT` | `TEXT_STRING`, `NCHAR_STRING` | | `INT_NUMBER`, `LONG_NUMBER`, `ULONGLONG_NUMBER`, `DECIMAL_NUMBER`, `FLOAT_NUMBER` | `NUM`, `LONG_NUM`, `ULONGLONG_NUM`, `DECIMAL_NUM`, `FLOAT_NUM` | | `BIN_NUMBER`, `HEX_NUMBER` | `BIN_NUM`, `HEX_NUM` | | `NULL2_SYMBOL`, `NOT_SYMBOL`, `NOT2_SYMBOL` | `NULL_SYM`, `NOT_SYM`, `NOT2_SYM` | | `WITH_SYMBOL`, `ROLLUP_SYMBOL`, `WITH_ROLLUP_SYMBOL` | `WITH`, `ROLLUP_SYM`, `WITH_ROLLUP_SYM` | | `END_MARKER` | `YYEOF` | `PARAM_MARKER`, `UNDERSCORE_CHARSET`, and `END_OF_INPUT` keep their names. </details> ## API changes The package isn't published yet, and the driver still uses its own lexer, so no released code is affected. - **Renamed** token constants in `WP_MySQL_Lexer` (see the table above). `get_name()` returns the new names too, except for tokens with a keyword spelling, such as `=` for `EQ` or `NULL` for `NULL_SYM`. - **Removed** `WP_MySQL_Lexer::get_token_id()`. - **Changed** the default `mysql_version` from `80038` to `80410`. ## Merge notes - #432 needs the new names when it's rebased. - #596 merges without a Git conflict, but it adds four uses of `NCHAR_TEXT` and merges the two `get_value()` conditions into one. Whichever PR lands second must rename them to `NCHAR_STRING` and reformat the condition. Follows up on #429. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Updates** * The MySQL parser now defaults to MySQL 8.4.10 and uses MySQL grammar terminal names for its public tokens. Applications that use the previous token names or the removed token-ID lookup method may need updates. * **Documentation** * Expanded guidance on token and AST naming, keyword handling, special terminals, token constants and IDs, and end markers. Added examples for querying descendant nodes and tokens. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Note
The changed line numbers are misleading—about 115,000 added lines is just a testing query corpus.
(Copied to the new
mysql-parserpackage frommysql-on-sqlite.)LALR(1) parser from official MySQL grammar
A new experimental
packages/mysql-parserpackage that implements a universal LALR(1) parser and builds a MySQL parse table from the official MySQL grammar.This is the initial implementation, not used anywhere in the driver yet.
A full driver migration to this new parser is AI-prototyped in #432.
What it does
What it doesn't do yet
Benchmarks
Measured on MacBook Pro M4 Max on PHP 8.4, the package's 8.4.10 corpus ~70k queries, end-to-end (lex + parse), best of 5 timed passes after 2 warmups:
This parser is over 5× faster without JIT and over 4.5× faster with JIT. Cold boot is a bit slower; warm boot is faster. The memory footprint is a bit higher, and the overall size about 14 KB higher.
Recognize-only
The same lex+parse runs but building no AST, measuring only recognition without AST allocation:
Dropping AST construction lifts both by ~1.5–2×, but the gap stays around ~4.2–5.8×.