BLIS¶

August 6, 2026
in BLIS, Deep Dives
1 min read

What We've Learned About Modeling LLM Latency

The last post in the BLIS series covered how BLIS models the engine, data plane, and control plane to predict full-pipeline latency without touching a GPU. What it didn't answer is whether the predictions can be trusted.

That comes down to one number. Everything BLIS reports at the cluster level rests on its estimate of how long a single forward pass takes, so if that's off, nothing above it can be right. This post is about how we estimate it, how well it holds up, and where it falls short.

The headline, up front: fit once on H100, BLIS predicts held-out configurations (six models across three GPU types) at 6.7% median end-to-end error, roughly 200× faster than running them for real, and those predictions have already steered serving policies we later confirmed on a physical cluster: a better admission controller and soft reflective flow control for llm-d. The rest of this post is how we get there, and where it still falls short.

June 5, 2026
in BLIS, llm-d
1 min read

BLIS: Evolving llm-d at Simulation Speed

Deploying llm-d is not just a question of choosing a model server and adding GPUs. In a production inference deployment, operators have to choose routing policies, admission behavior, batching settings, KV-cache reuse strategies, prefill/decode placement, and autoscaling rules under concrete TTFT, ITL, throughput, and cost constraints.

These choices are coupled. A routing change that improves cache locality can concentrate load. A prefill/decode threshold that helps one workload can hurt another. An admission policy that protects critical traffic can reduce total served volume. A change in any one policy can shift TTFT, inter-token latency, throughput, SLO compliance, and accelerator cost in ways that are difficult to predict analytically.

The only reliable way to confirm those tradeoffs is to measure them in a GPU-backed llm-d cluster. But using cluster runs as the first step in every policy or capacity-planning experiment is too slow and expensive. BLIS provides a faster inner loop: a calibrated discrete-event simulator for distributed inference systems like llm-d. Developers can evaluate candidate policies and deployment configurations locally, then reserve cluster validation for the candidates most likely to matter.

April 9, 2026
in BLIS, llm-d
1 min read

The Physics of High-Fidelity Distributed Inference Platform Simulation

Production LLM inference platforms are distributed systems where routing policies, admission control, autoscaling, and engine-level scheduling all interact to determine latencies and throughput. How do you explore how different policies and configurations affect these KPIs before deploying to production? Testing a new routing policy or autoscaling threshold on live traffic risks cascading bugs across the fleet, while building separate test environments burns GPU-hours and still cannot predict interactions between cluster-level policies and engine-level batch dynamics.

March 5, 2026
in BLIS, llm-d
1 min read

Why Simulate Before You Scale

Deploying large language models in production is one of the most expensive infrastructure decisions an organization can make. A single high-end GPU costs upwards of $30,000, and a production cluster can run into millions per year. Yet most teams make their first scaling decisions based on rough estimates, vendor benchmarks, or — worst of all — trial and error on live hardware.

What if you could test your deployment plan before spending a dollar on GPUs?