Skip to content

fix(extraction): grammar loads started together wait for one another - #3303

Open
Ahaeflig wants to merge 1 commit into
colbymchenry:mainfrom
Ahaeflig:fix/grammar-load-queue
Open

Ahaeflig wants to merge 1 commit into
colbymchenry:mainfrom
Ahaeflig:fix/grammar-load-queue

Conversation

@Ahaeflig

Copy link
Copy Markdown

Grammars that load at the same moment break each other on main, and stay broken for the life of the process.

The problem

loadGrammarsForLanguages loads grammars one at a time, because web-tree-sitter corrupts a grammar loaded while another is loading (tree-sitter#2338). That only holds within one call. Two calls started together each run Parser.init and load side by side. A grammar loaded that way stays cached, and every later parse of its language throws "Incompatible language version" until the process exits.

Calls can overlap wherever grammars load lazily. A daemon's sync loads the grammars its files need, while a request may load another for branch guards (graph/branch-guards.ts) or syntax tokens (extraction/syntax-tokens.ts). loadGrammarsForLanguages is also exported.

To reproduce, start six loads together in a fresh process (loadGrammarsForLanguages(['lua']), (['rust']), (['python']), (['lua', 'rust']), (['go']), (['typescript', 'python'])), then parse a line of each language. On main, three runs in a row gave:

  • Parser.init ran 6 times.
  • All 6 loads overlapped.
  • Lua, Go or TypeScript (a different set each run) threw Incompatible language version 3538997. Compatibility range 13 through 15. on its first parse.

What changes

  • initGrammars initializes the runtime once for callers that arrive together.
  • Every load joins one queue that skips a grammar an earlier call has loaded.

What I ran

On Linux x64 (WSL2), Node 24.19.0, against main at f72c9d2:

  • __tests__/grammar-load-concurrency.test.ts (new). It starts the six loads together, and requires one Parser.init, one load per grammar, never two loads at once, and a clean parse of every language. On main it fails at the first assertion: Parser.init is called 6 times.
  • Full suite, with the native kernel staged and CODEGRAPH_KERNEL_EXPECT=1: 11,070 passed, 2 expected failures, 58 skipped (platform-gated), 0 failed.

Not run: macOS and Windows. The change has no platform-specific code.

This came out of the LuaJIT → Rust bridge work, which loads grammars during synthesis. It's split out so it can be reviewed and merged on its own.

🤖 Generated with Claude Code

loadGrammarsForLanguages loads grammars one at a time, because
web-tree-sitter corrupts a grammar loaded while another one is loading
(tree-sitter#2338). That held only within one call. Two calls started
together, like a daemon's sync beside a request that loads a grammar for
branch guards or syntax tokens, each ran Parser.init and loaded side by
side. A grammar loaded that way stayed cached, and every later parse of its
language threw "Incompatible language version" until the process exited.

initGrammars now initializes the runtime once for callers that arrive
together, and every load joins one queue that skips a grammar an earlier
call has loaded. The new test starts six loads together: on main the
runtime initializes six times, and parses fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant