llm-d¶

July 27, 2026
in llm-d, Deep Dives
9 min read

From Simulation to Production Part II: Soft-Reflective Flow Control for llm-d

Part 2 of the sim2real series. A case study in closing the AI-native loop across a second problem class: dispatch prioritization.

Introduction

The first article in this series traced a complete traversal of the AI-native loop for llm-d's admission control layer: an AI agent framework (Nous) used a high-fidelity simulator (BLIS) to discover a probabilistic, proactive admission control policy that cuts critical-traffic TTFT by up to 97% at near-saturation workloads. That article restates the core methodology: simulation lets the loop run at machine speed, reserving expensive GPU time for validating the few candidates that actually merit real-hardware testing.

This article applies the same loop to a different problem: flow control. The distinction matters. Admission control acts at the gate: it decides whether an incoming request enters the system at all, and may reject it. Flow control acts at the dispatch queue: it decides which already-admitted request gets sent to a backend server next. The two mechanisms protect different points in the pipeline and neither replaces the other.

The concrete outcome is the soft-reflective ceiling policy: a new flow control plugin for llm-d-router that reduces critical-class TTFT p99 by up to 98% at near-capacity load without rejecting a single request. It is now merged into llm-d-router, and users can enable it today with a single YAML change.

May 13, 2026
in llm-d, Deep Dives
10 min read

From Simulation to Production: How an AI-Native Pipeline Discovered a Better Admission Controller for llm-d

A case study in closing the AI-native loop: observe, reason, change, validate, deploy.

Introduction

An AI-native system is one that continuously and autonomously closes the loop from observation to action to deployment, with AI as the primary agent driving this process. Rather than humans manually directing each improvement, humans establish objectives and boundaries while the system autonomously executes the cycle, at machine speed.

April 9, 2026
in BLIS, llm-d
1 min read

The Physics of High-Fidelity Distributed Inference Platform Simulation

Production LLM inference platforms are distributed systems where routing policies, admission control, autoscaling, and engine-level scheduling all interact to determine latencies and throughput. How do you explore how different policies and configurations affect these KPIs before deploying to production? Testing a new routing policy or autoscaling threshold on live traffic risks cascading bugs across the fleet, while building separate test environments burns GPU-hours and still cannot predict interactions between cluster-level policies and engine-level batch dynamics.

March 5, 2026
in BLIS, llm-d
1 min read

Why Simulate Before You Scale

Deploying large language models in production is one of the most expensive infrastructure decisions an organization can make. A single high-end GPU costs upwards of $30,000, and a production cluster can run into millions per year. Yet most teams make their first scaling decisions based on rough estimates, vendor benchmarks, or — worst of all — trial and error on live hardware.

What if you could test your deployment plan before spending a dollar on GPUs?