Repository navigation
[Bugfix] Detect Gemma 3's sliding windows from mlx-vlm caches - #1377
Open
syang-genai wants to merge 1 commit into
Open
syang-genai wants to merge 1 commit into
syang-genai wants to merge 1 commit into
Conversation
mlx-vlm defines its own RotatingKVCache and KVCache rather than subclassing mlx-lm's, so the cache-factory probe returned None for Gemma 3 and every layer ran as full attention. Accept both libraries' classes; the probe's rules are unchanged. Signed-off-by: syang-genai <191082996+syang-genai@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Gemma 3 (
Gemma3ForConditionalGeneration, loaded through mlx-vlm) gets no sliding windows: every layer runs as full attention. Once a sequence passes the 1,024-token window, greedy output no longer matches mlx-vlm.DefaultModelAdapter._sliding_windows_from_cache_factoryreads each layer's window from the model'smake_cache(). It accepts only mlx-lm'sRotatingKVCacheandKVCache. mlx-vlm defines its own cache classes, which do not subclass mlx-lm's, so the probe returnsNonefor Gemma 3. The config fallback needslayer_types, and Gemma 3 has onlysliding_window_pattern, sosliding_window_per_layerends upNone.I found this while measuring sliding-window decode for #1272: on Gemma 3 4B, every attention call in an in-process trace was a full-attention call.
Changes
_sliding_windows_from_cache_factoryaccepts the rotating and full KV cache classes of both mlx-lm and mlx-vlm. The mlx-vlm import is optional.keep == 0gives itsmax_size, a full cache gives −1, and anything else returnsNone.Tests
test_mlx_vlm_cache_factory_drives_windowsbuilds a small mlx-vlm Gemma 3 language model and checks two things:[512] * 5 + [-1]per six-layer block.It fails on main.
Results
mlx-community/gemma-3-4b-it-4bit: a 1,807-token prompt, then 48 greedy tokens, compared with mlx-vlm's own forward on its own caches.sliding_window_per_layerNone{-1, 1024}Prompts shorter than the window were not affected, because full and sliding attention agree there. Gemma 4 loads through mlx-lm, so it was not affected either.
Testing
pytest tests -m "not slow and not network": 6,217 passedruff check/ruff format --check.VLLM_ENABLE_V1_MULTIPROCESSING=0 python gemma3_window_parity.py(not included in this PR).