Skip to content

feat(lua,rust): a literal LuaJIT operation links to the Rust handler it selects - #3305

Draft
Ahaeflig wants to merge 3 commits into
colbymchenry:mainfrom
Ahaeflig:feat/lua-rust-operations
Draft

Ahaeflig wants to merge 3 commits into
colbymchenry:mainfrom
Ahaeflig:feat/lua-rust-operations

Conversation

@Ahaeflig

Copy link
Copy Markdown

Builds on #3304. A LuaJIT program that calls one shared Rust export with an operation name now reaches the handler that operation selects, not every handler behind the export.

-- bridge.lua
local lib = require("ffi").load("core")
local M = {}
function M.call(op, body) return lib.dispatch(op, #op, body, #body) end
return M

-- caller.lua
local bridge = require("bridge")
function run_task() return bridge.call("task.run", "{}") end
#[no_mangle]
pub unsafe extern "C" fn dispatch(op: *const u8, op_len: usize, body: *const u8, body_len: usize) {
    let op = std::str::from_utf8(std::slice::from_raw_parts(op, op_len)).unwrap_or("");
    match op { "task.run" => run(), "task.read" => read(), _ => () }
}

With FFI links alone, run_task → dispatch is a plain call, so run_task reaches both run and read, and every Lua caller of dispatch shows up among the callers of run. With this change, run_task links to run with the hop LuaJIT → Rust operation `task.run` via `dispatch` . The call into dispatch is kept as a transport hop that walks stop at, so read is no longer reachable from run_task.

What it does

  • Follows a literal operation to the handler it selects. It goes through Lua wrappers and modules that forward a parameter, and through Rust functions, methods, closures and declarative macros that forward it, into a literal match, and links the Lua call to the handler the arm selects (lua-rust-operation). Strings may cross as a pointer and length or as a CStr.
  • Refuses what may not run.
    • An arm a guard, a #[cfg], a binding or a wildcard may pre-empt.
    • An operation an earlier return always takes, judged on the string tests the source spells.
    • A shadowing loop or let binding.
  • Keeps the shared export as a boundary. The Lua call into an export that dispatches operations is a transport reference (metadata.transportOnly). Every walk stops at it: BFS, DFS, callers, callees, impact, paths, explore's test notes and the viewer's walks. An export that maps strings to values is no dispatcher, and its calls stay plain.
  • Resolves Rust paths through the Cargo identity. Cross-crate paths and crate:: need declared modules and local path dependencies.
  • Extends the analysis store. Each Rust file's dispatch analysis and module declarations join the store feat(lua,rust): a LuaJIT FFI call links to the Rust export with its symbol #3304 adds, so a sync re-analyzes only what changed. A Cargo.toml edit refreshes synthesis on sync.

docs/design/lua-rust-bridge.md has the rules, the limits and the validation.

What I ran

On Linux x64 (WSL2), Node 24.19.0, against main at f72c9d2 with the grammar fix and #3304:

  • Public corpora. The same fifteen repositories as feat(lua,rust): a LuaJIT FFI call links to the Rust export with its symbol #3304, indexed from scratch on both extraction paths. Each one gets exactly the FFI edges feat(lua,rust): a LuaJIT FFI call links to the Rust export with its symbol #3304 gives it, both paths stay byte-identical, and node rows match main's. None of them has an export that dispatches operations.
  • The monorepo. Its native library has one export that dispatches about 40 operations through a match. This change adds 125 operation edges and turns its 4 calls into the export into transport hops. A script checked the flows on that index:
    • a network client's send forwards three literal operations and reaches exactly those three handlers;
    • two functions that each send one operation reach only that operation's handler;
    • codegraph_explore labels each hop with its operation and the Rust line that dispatches it.
  • Cost, on the monorepo, median of three alternating rounds on the kernel path: a fresh index took 21.9 s, against 15.0 s with the FFI links alone and 14.8 s on main. The pass took 6.1 s of it, analyzing the Rust the exports reach (162 of 181 files) and 104 Lua files. A sync took 2.2 to 2.6 s against main's 1.9 to 2.1 s, the pass about 0.5 s of it. The design doc has the table.
  • Tests. The full suite with the kernel staged and CODEGRAPH_KERNEL_EXPECT=1: 11,369 passed, 2 expected failures, 58 skipped (platform-gated), none failed. With CODEGRAPH_KERNEL=0, the bridge's tests and every Lua and Rust resolution test: 49 files, 546 passed. The module-declaration test fails when the stored declarations are keyed without the file's text; I checked it by breaking the code.

Not done

  • A public corpus. I found no public repository that sends a literal operation through one export from LuaJIT. The closest, leejw51/bfvegas, routes formats through a table first. This part is validated on the private monorepo and synthetic fixtures only, and I'd understand if that's not enough to merge it.
  • Agent A/B. It has not been run.
  • A narrower Rust reach. A fresh index of the monorepo takes about 7 s longer, because the reach from the exports follows every call that forwards a value, so it analyzes 162 of its 181 Rust files. Following only the operation's own value would cut that; I've left it for a follow-up.
  • Other platforms. macOS and Windows weren't run.

This branch carries the commits of #3303 and #3304 until they merge; review only the last one. It takes EXTRACTION_VERSION 30, or the next free one after #3304.

🤖 Generated with Claude Code

Ahaeflig and others added 3 commits November 16, 2026 01:53
loadGrammarsForLanguages loads grammars one at a time, because
web-tree-sitter corrupts a grammar loaded while another one is loading
(tree-sitter#2338). That held only within one call. Two calls started
together, like a daemon's sync beside a request that loads a grammar for
branch guards or syntax tokens, each ran Parser.init and loaded side by
side. A grammar loaded that way stayed cached, and every later parse of its
language threw "Incompatible language version" until the process exited.

initGrammars now initializes the runtime once for callers that arrive
together, and every load joins one queue that skips a grammar an earlier
call has loaded. The new test starts six loads together: on main the
runtime initializes six times, and parses fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ymbol

A LuaJIT program calls its Rust library through the FFI by symbol, so the
Lua call named a C function no Lua file defines and every flow stopped at
it. A synthesis pass, run only where a Lua file names "ffi" and a Rust file
declares an export, links each call through a proven FFI receiver to the
one Rust C-ABI export with that symbol (lua-rust-ffi).

A receiver is proven from require("ffi") through ffi.C, ffi.load and
pcall(ffi.load), their aliases, table members and function returns, and a
library another module returns as a member or from a function, as an
OpenResty binding's cdefs.clib. Shadowing, reassignment, a write through a
function's parameters or a receiver the analysis can't resolve, and a value
handed to code in another file withdraw the proof. A symbol two exports
share links neither; ffi.cdef asm labels rename the symbol.

The pass analyzes the Lua files that name "ffi" and, round by round, the
files holding a string that can name a module found to hand out a library,
since require reads only a literal, and scans the Rust files that name
no_mangle or export_name for their exports. Each file's analysis is stored
under the index directory, keyed by a fingerprint of the analyzers and
grammars and by the file's path and text, so a sync re-analyzes only what
changed. A file nested too deep, unreadable as a chunk or over its work
budget is declined, and the pass never throws, so no other pass loses its
edges. Lua and Rust files are synthesis inputs only once the bridge is open.

Validated on twelve public corpora, kernel and wasm giving identical graphs,
and on three controls left byte-identical; see docs/design/lua-rust-bridge.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it selects

A LuaJIT program that calls one shared Rust export with an operation name,
call("task.run", ...) into match op { "task.run" => run(), ... }, made every
Lua caller reach every handler through the export. The bridge now follows a
literal operation through Lua wrappers and modules that forward it, and Rust
functions, methods, closures and declarative macros that forward it, into
the match, and links the Lua call to the one handler its arm selects
(lua-rust-operation). The call into the shared export is kept as a
transport reference that every walk treats as a boundary, explore's test
notes and the viewer's walks included.

Arms are tried in order: a guard, a #[cfg] arm, a binding or a wildcard
that may take a literal leaves it unresolved, and an operation an earlier
return always takes never reaches its arm. A C string read with CStr is its
text. Positions skip comments and attributes; for, while-let and let-chain
bindings shadow; crate:: paths go through the Cargo identity. An export that
maps strings to values is no dispatcher. A Rust function nested too deep or
over its budget is declined.

The analysis store also keeps each Rust file's dispatch analysis and module
declarations, so a sync re-analyzes only what changed; a Cargo.toml edit
refreshes synthesis on sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant