inference stack compiler
Fifty years of software reuse let the compiler delete and specialize, but never restructure. AI knocks that wall down — and once ideas are the unit of reuse, every project gets to be bespoke. Scratchy is what that looks like for inference.
The entire forward pass of an architecture, written as the math.
gemma4-moeThe stock config.json for one instance of that architecture.
gemma-4-26b-a4b-itOne preset, which replaces the dense emission rather than adding to it.
fp8-dynamic-per-channelLonger-form writing on why the stack is shaped the way it is.
Shared libraries, source modules, bundlers, AoT-compiled crates — every era let the toolchain specialize a little more, and every one of them stopped at the same wall. What happens when the unit of reuse becomes an idea, and what that bought us on IBM Spyre.
Here is the hot take: AI coding agents will eliminate the software system as a concept. In its place every solution will be bespoke, span the full stack, and will be generated from scratch.
100 Mi for Spyre.
The runtime is the binary. There is no graph executor to ship.
Independent of model size. 12 s for 8B on Spyre.
Most inference engines hand-write a Python/CUDA module per architecture, then rely on a runtime graph executor to pick kernels. Scratchy takes the opposite approach: you write the forward declaratively, and the compiler emits the whole pass ahead of time.
Rust's Turing-complete procedural macros run the entire pipeline at expansion — op tape, reroll, layer classes, slot coloring, lifetimes, barriers, arena sizes, kernel selection — and emit static tapes of each target's own types.
Everything is a constant. That turns complex analysis into arithmetic, and hands the optimizer enough constants to cut register pressure in the kernels that matter.
crates/models/arch/dsl/llama.py
Scratchy's first target is the IBM Spyre AIU — and that is the real test, because novel silicon is where the library era has nothing to offer you.
Each Spyre core has a 2 MiB scratchpad, of which 1,677,721 bytes are yours. Every tile of every operation in the forward pass must fit. Overflow it and you do not get an error message: you get DtException 1535 on the card, minutes later, about a tile you can no longer inspect. The dominant cost of new hardware isn't writing kernels — it's the feedback loop from a nameless on-card fault back to the line of math that caused it.
Because scratchy knows every tile size at compile time, that fault becomes a cargo build error on your laptop. A general-purpose runtime structurally cannot do this: it doesn't know the shapes until it is already running, on the card, where the only channel back to you is an integer.
Supporting silicon with no ecosystem behind it cost zero lines of model code. The same 25 architectures, the same 1,580 lines of DSL, the same 22-line LLaMA. An 8B model boots in 12 s from a 330 Mi image.
Reuse-by-code quietly puts a population threshold on what hardware is allowed to exist — “support” means a vendor maintaining a general backend inside someone else's general framework until the market justifies the headcount. Reuse-by-idea drops that threshold to one team with a compiler.
Each target is its own binary over one common CLI.
spyreIBM Spyre AIU · requires the Spyre build toolkit
scrApple Silicon · requires macOS
Install a recent Rust toolchain, then name your target, your model and your quant. Every build names its own scope — naming zero models is a build-time panic, not a silent empty binary.
Apple Silicon · Llama 3.2 3B · MLX 4-bit
model/<stem> — one checked-in config. Widen with model/<arch> or model/all.
quant/<preset> — compiles that quantization instead of dense/bf16.
-Fserve -Fbench — beyond the default scr chat CLI.
Feature scoping, model and quant selection, air-gapped builds.
The whole-forward DSL, and what the proc macro actually emits.
All 25 supported models in the DSL, diffable two at a time.
The recipe for teaching scratchy a model it has never seen.
Building and shipping an image for the AIU.
PR workflow, CI gates, and the architecture invariants.
The repository, issues, and pull requests.