Mini-vLLM
A small inference engine with paged KV cache and continuous batching.
A roughly 500-line, single-GPU inference engine implementing the core ideas behind vLLM: paged KV cache, a block allocator, and continuous batching. It keeps one file per concept so the data flow remains inspectable.
- Paged KV cache: one shared GPU pool with fixed-size blocks and a per-sequence block table.
- Block allocator: ref-counted physical blocks returned to the pool when a sequence finishes.
- Continuous batching: an FCFS scheduler slots new requests into the running batch between decode steps.
- Engine loop:
add_request → schedule → prefill → decode → sample → free blocks.
Benchmarked on H100: 2.6× decode throughput, 3.5× more concurrent sequences, and substantially lower worst-case time to first token under a 16-request burst.
