Hintergewerk — The Work Behind the Surface
Inspiration
Every renovated wall tells the truth exactly once: while it's open. Then we close it, and the knowledge is gone — living in a few blurry photos, or in nothing at all. Months later, a drill meets a water pipe nobody remembered. Existing documentation means scrolling through photo chaos, or a BIM model that presumes your house was planned digitally. The existing stock of buildings has no memory. So we gave it one.
What it does
Hintergewerk — German for "the work behind" — captures what an open wall reveals, in VR, with your hands. Pinch to draw a cable run along the wall; two hands set the depth; under thirty seconds per object. Each object becomes permanent: spatially anchored, typed (power, water, heating, studs), timestamped, signed with the installer's name. While you draw, a floor plan grows in your other hand — every corridor appearing as you confirm it. Weeks later, the overlay returns exactly where you left it — not as a thin line, but a glowing safety corridor that escalates in color, intensity and panned audio as your hand approaches. The drill stops before the pipe does. Look and pinch, and the evidence appears: photo, depth, who documented it — measured or estimated. One gesture exports a floor plan per room with legend and scale as PDF, plus machine-readable, BIM-ready JSON.
How we built it
Unity 6 with the Meta XR SDKs. The core is deliberately SDK-agnostic: a versioned data schema stores geometry anchor-relative, so anchor drift and re-localization can never corrupt a captured run. Snap points, a 5 cm grid and magnetic endpoints absorb hand-tracking tremor; a ±5–6 cm safety corridor replaces the illusion of millimeter lines. The in-hand floor plan is the same top-down projection that renders the exported PDF — one source of truth, live instead of on paper — both drawing the room outline from the Quest Scene API. Rooms sort themselves: each run joins the room whose wall it touched, every room gets its own plan, one JSON holds them all. The PDF export is a dependency-free emitter written from scratch; 109 automated tests cover the core.
Challenges we ran into
Hand tracking lives at 1–2 cm of tremor — so we engineered precision into the system instead of pretending it's a motor skill. The simulator ships no Scene anchors and can't script gaze, forcing honest fallbacks: with no room model, drawing happens on a visibly amber, estimated plane — never on invisible fake walls (a device bug we caught and fixed). An automated test scene once shipped as the start scene with scripted fake hands — caught on device, fixed with an editor tool. One simulator session surfaced five real bugs — a dead menu selection, drawing stuck at two points, double-bound anchors — all fixed.
What we learned
The biggest lesson: honesty is a feature, not a disclaimer. Every object carries a visible confidence level — measured, estimated or scanner; "not found" never means "not there"; lost anchors stay "not locatable" in the record, never deleted. We never claim to see through walls — we remember what was documented. Persistence is demonstrated, not claimed: runs survive full app restarts, stored anchor-relative — we won't quote multi-day numbers until we've measured them.
What's next
Scanner readings joining the memory, whole houses with staircases, LiDAR imports, IFC export for planning firms — and agents that propose new routing around what the house already knows.