Skip to content

feat(kv): add CUDA VMM KV storage with fixed addresses - #194

Open
XinhaoTheo wants to merge 2 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc192-vmm-kv-storage
Open

XinhaoTheo wants to merge 2 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc192-vmm-kv-storage

Conversation

@XinhaoTheo

@XinhaoTheo XinhaoTheo commented Nov 17, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?

Size the KV pool from the memory that is free after warmup and CUDA Graph capture (RFC #192). The captured graphs must stay valid.

2. What is the highest-leverage variable for that outcome?

The KV tensor addresses. A captured graph keeps the KV pointers, the strides and the size of the first dimension (cuda_graph.py#L121). If the addresses stay fixed, the engine can add KV capacity after capture.

3. How will this work move that variable rather than create busywork?

StableKVStorage reserves the full virtual address range at startup. It adds physical memory later with CUDA VMM. The addresses, strides and stored values do not change. The tests show that a graph that was captured before a commit writes correctly to the new blocks.

4. Who is the customer or user, and what do they actually need?

PR necessity

A PR is necessary. No current storage path keeps the KV addresses fixed when the capacity increases. This is not a bugfix.

Modification

This PR adds vllm_rlt/core/kv_storage.py. The engine does not use it yet, so no behavior changes.

  • PoolSpec(block_shape, dtype) gives the shape and dtype of one block in one pool. A pool is not specific to K or V, so other page layouts ([KV] Pack recurrent depth planes for partially recurrent models #147, feat(kv): store INT4/NVFP4 KV pages in the cache manager and read them with Triton #174) can add pools.
  • StableKVStorage(specs, max_blocks=..., initial_blocks=..., device=...):
    • It reserves one virtual address range for each pool. Each pool has one tensor with the shape [max_blocks, *block_shape].
    • commit(blocks) adds physical memory from the start of each range, in multiples of the granule (2 MiB on H100). The capacity can only increase.
    • committed_bytes gives the physical bytes. The PyTorch allocator does not count them.
    • As with torch.empty, the memory is released when no reference to the storage or to its tensor views remains. The release synchronizes the device first. There is no explicit release, because it would leave the remaining views on unmapped memory.
  • vmm_unavailable_reason(device) gives the reason for a fallback: not a CUDA device, a ROCm build, not Linux, or no VMM support.
  • The driver binding uses ctypes and libcuda.so.1. It does not need a build step or a new dependency.

Out of scope: the KVCacheManager integration (#195), the startup order change and the measured final capacity (other RFC #192 contributors), HIP VMM for ROCm, and VMM memory for NIXL RDMA.

Related PR

Accuracy Test

This PR does not change an execution path, so the model outputs do not change.

The tests in tests/test_kv_storage.py check three things:

  • The addresses stay fixed after commit().
  • A graph that was captured before a commit writes correctly to the new blocks.
  • The last reference releases the memory, and the PyTorch allocator does not count it.

All tests passed on H100 (Modal) at 594dc63. The CPU suite also passed, except 12 tests in test_async_speculative_sampling_plan.py that need Triton. Triton is not available on macOS. The token equality test against torch.empty comes with the integration in #195.

Benchmark and Profiling

Micro-benchmark on one H100 80GB HBM3, measured at 7aaa7c6. 594dc63 changes only when the release starts, not the release code. The KV geometry comes from the ByteDance/Ouro-1.4B config.json: 24 layers, 16 KV heads, head dim 128, block size 16, BF16. One block of one pool is 1.5 MiB.

Lifecycle cost (K and V pools together, median of 3 runs):

Size of each pool VMM construct VMM commit() VMM release torch.empty allocate torch.empty free
1 GiB 2.7 ms 2.0 ms 1.3 ms 1.8 ms 1.3 ms
8 GiB 2.4 ms 3.2 ms 8.8 ms 5.2 ms 7.6 ms
32 GiB 2.6 ms 143 ms 37 ms 180 ms 41 ms

Rounding: the granule is 2 MiB. The unused physical memory is 1 MiB to 2 MiB for K and V together. The usable block count is equal to the requested block count.

Bandwidth (both storages in memory, 6 interleaved rounds, median):

Workload VMM torch.empty Difference
Triton paged_attention, batch 32, context 4096, 24 layers 2932.9 GB/s 2933.3 GB/s −0.02% (noise)
Device copy, 12 GiB 3033.0 GB/s 3033.9 GB/s −0.03% (all rounds)

The attention outputs of the two storages are bitwise equal.

Profiling: not applicable, because no execution path changes. The end-to-end A/B and the profiles come with the integration in #195.

Conclusion

This PR contains only the KV storage. The KVCacheManager integration, the token equality test and the end-to-end A/B are in #195.

  • Add StableKVStorage: fixed addresses, commit(), release on the last reference, and the fallback reason.
  • Test it on GPU: fixed addresses, graph replay on new blocks, and memory release.
  • Measure the lifecycle cost, the rounding and the bandwidth on H100.
  • Apply the review: remove close(), make the release test independent of other processes, and rebase onto main.
  • Run CI (it waits for maintainer approval).

@kunkunblueberry kunkunblueberry left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this is a useful piece. I read the diff and ran your two files on a clean upstream/main checkout on an A100 (torch 2.14+cu130), so the observations below are reproduced, not inferred. I am the author of #195, the end-to-end variant of the same RFC #192 step, so the overlap is discussed at the end.

Reviewed head: 7aaa7c6 (base c338aaa, 11 commits behind main).

Blocking

1. commit() has no upper bound — vllm_rlt/core/kv_storage.py:320

The docstring says blocks must not exceed max_blocks, but nothing enforces it. With 3 MiB blocks and max_blocks=1 (4 MiB reserved), commit(2) asks for 6 MiB and fails inside the driver:

commit(2) raised: RuntimeError : CUDA driver error 1: invalid argument

An integration mistake would surface as an opaque driver error during startup. if not 1 <= blocks <= self.max_blocks: raise ValueError(...) turns it into a usable message. Note that when the block is smaller than the granule the round-up hides it: max_blocks=4 with 64 KiB blocks, commit(5) silently returns 4.

2. close() leaves the object looking alive — :336

After close() on a 64-block / 2 MiB store:

before close: committed_blocks = 64 committed_bytes = 2097152
after  close: committed_blocks = 64 committed_bytes = 2097152
after  close: tensors still exposed = 1
commit after close -> 64
write through a stale view after close: no error

The VA has already been freed at that point, so this is a silent-corruption window rather than a crash: commit() is a no-op that reports success, and a surviving view still writes. A _closed flag that raises from commit() and from tensors, plus zeroed counters, makes the misuse loud.

3. ruff format --check . fails, which is what cpu-tests.yml runs (pinned ruff==0.13.2)

Would reformat: tests\test_kv_storage.py
Would reformat: vllm_rlt\core\kv_storage.py

Reproduced with 0.13.2 itself, so it is not a version artifact. ruff check . passes.

Non-blocking

4. initial_blocks=0 fails deep inside torch — :308: RuntimeError: The specified pointer resides on host memory and is not registered with any CUDA device. The constraint comes from building the tensors out of the pool before any mapping exists, not from CUDA, so an explicit validation would be clearer.

5. No capacity gate for the consumer. tensors spans max_blocks, and "do not access the blocks after committed_blocks" is documentation only. Once KVCacheManager is wired (your unchecked item), allocating past committed_blocks is a sticky illegal address that poisons the CUDA context for the whole process. A published <= committed <= max split, where publishing requires committed_blocks, makes that unreachable; your TODO already lists the free-pool update, and the invariant is what keeps it safe.

6. No CI evidence attached — statusCheckRollup is empty and mergeStateStatus is UNSTABLE, so the H200/H100 runs are not visible from the PR. Rebasing onto current main would also help, since the base is 11 commits behind.

7. Test oracle depends on absolute driver memory — test_close_returns_memory_the_torch_allocator_never_counted compares mem_get_info() values. That is fine on a dedicated Modal box, but another tenant's allocations move the same counter, so it flakes on a shared device (I hit exactly this in my own suite). A sizeable mapping plus release accounting from the storage itself is more stable.

Verified as correct, for the record

  • The lifetime story holds: torch.as_tensor keeps the __cuda_array_interface__ provider alive, so views keep the pool alive. Reproduced: after del storage the finalizer did not run and the surviving view stayed writable.
  • weakref.finalize receives a module-level function rather than a bound method, which avoids the self-reference keepalive trap.
  • close() is idempotent, a failed cuMemMap releases its handle, and memory_reserved() stays untouched, so "the PyTorch allocator does not count this memory" is accurate.
  • Driving the real write_graph_kv kernel in the graph test is a stronger check than a torch-only write.

Overlap with #195

#195 implements the same RFC #192 step end-to-end (storage + KVCacheManager capacity states + startup growth + CLI flag) and also adds tests/test_kv_storage.py, so the two will conflict. Two storage modules for one mechanism is the part worth avoiding. My suggestion:

  • keep a single storage layer — your PoolSpec API is more general (it can host the #147/#174 layouts) and your vmm_unavailable_reason probe is better than mine;
  • port items 1, 2 and 5 into it;
  • then I rebase #195 onto your module, drop my vmm_pool.py and rename my test file.

I am happy to do that rebase once 1-3 are addressed, if you prefer to keep the storage module where it is.

Add StableKVStorage for RFC InferMatrixLab#192. It reserves one virtual address range
for each KV pool. The tensor of each pool has the maximum block count from
the start. commit() adds physical memory to more blocks. It does not change
a tensor address, a stride or a stored value. Because of this, a CUDA Graph
that was captured before a commit also writes correctly to the new blocks.

The driver binding uses ctypes and libcuda.so.1. It does not need a build
step or a new dependency. vmm_unavailable_reason() gives the reason when a
device cannot use this storage.

The engine does not use this storage yet.
@XinhaoTheo

Copy link
Copy Markdown
Author

Thanks for the review. I rebased #194 onto main (33d950d). For each item:

  1. Not done. A request above max_blocks already stops with a driver error, so it does not fail silently. commit(5) returns 4 on a 4-block storage, because the return value is the number of committed blocks. The caller must use this value. I do not add a second check for the same contract.
  2. Done, with a different fix. I removed close(). The memory is released only when no reference to the storage or to its views remains, as with torch.empty. Thus a stale view cannot occur, and a _closed flag is not necessary. For an earlier release, remove all references to the storage and its views.
  3. Not done. With the CI version (ruff 0.13.2) on Linux and macOS, ruff format --check . gives "139 files already formatted". Your output shows Windows paths. I think that your checkout changed the line endings.
  4. Not done. The error occurs in the constructor, before the storage is used. The comment on that line gives the cause.
  5. Not done in feat(kv): add CUDA VMM KV storage with fixed addresses #194. The published/committed gate is part of the KVCacheManager integration, which is your part of the split. committed_blocks gives the value for the gate.
  6. Done. I rebased onto main. The CI workflow waits for maintainer approval (action_required), so it has not run yet.
  7. Done. The release test does not read the free memory of the device. It checks that the views keep the pools alive, that the last reference releases them, and that torch.cuda.memory_reserved() does not change. A failed release in the finalizer now fails the test.

GPU: tests/test_kv_storage.py passes on H100 (5 passed).

Overlap: I agree. Please rebase #195 onto vllm_rlt/core/kv_storage.py, remove vmm_pool.py and rename your test file.

close() released the memory also when tensor views remained. A remaining
view then wrote to unmapped memory without an error. Now the memory is
released only when no reference to the storage or to its views remains,
as with torch.empty.

The release test does not read the free memory of the device, because
other processes on a shared device change it. The test checks the pool
lifetime, and a failed release in the finalizer fails the test.
@XinhaoTheo
XinhaoTheo force-pushed the feat/rfc192-vmm-kv-storage branch from 7aaa7c6 to 594dc63 Compare November 17, 2026 16:50
@XinhaoTheo
XinhaoTheo marked this pull request as ready for review November 17, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants