Skip to content

[Bugfix] Detect Gemma 3's sliding windows from mlx-vlm caches - #1377

Open
syang-genai wants to merge 1 commit into
vllm-project:mainfrom
syang-genai:fix/gemma3-sliding-window
Open

syang-genai wants to merge 1 commit into
vllm-project:mainfrom
syang-genai:fix/gemma3-sliding-window

Conversation

@syang-genai

Copy link
Copy Markdown
Contributor

Purpose

Gemma 3 (Gemma3ForConditionalGeneration, loaded through mlx-vlm) gets no sliding windows: every layer runs as full attention. Once a sequence passes the 1,024-token window, greedy output no longer matches mlx-vlm.

DefaultModelAdapter._sliding_windows_from_cache_factory reads each layer's window from the model's make_cache(). It accepts only mlx-lm's RotatingKVCache and KVCache. mlx-vlm defines its own cache classes, which do not subclass mlx-lm's, so the probe returns None for Gemma 3. The config fallback needs layer_types, and Gemma 3 has only sliding_window_pattern, so sliding_window_per_layer ends up None.

I found this while measuring sliding-window decode for #1272: on Gemma 3 4B, every attention call in an in-process trace was a full-attention call.

Changes

  • _sliding_windows_from_cache_factory accepts the rotating and full KV cache classes of both mlx-lm and mlx-vlm. The mlx-vlm import is optional.
  • The probe's rules are unchanged: a rotating cache with keep == 0 gives its max_size, a full cache gives −1, and anything else returns None.

Tests

test_mlx_vlm_cache_factory_drives_windows builds a small mlx-vlm Gemma 3 language model and checks two things:

  • its caches are mlx-vlm classes;
  • the adapter resolves windows [512] * 5 + [-1] per six-layer block.

It fails on main.

Results

mlx-community/gemma-3-4b-it-4bit: a 1,807-token prompt, then 48 greedy tokens, compared with mlx-vlm's own forward on its own caches.

sliding_window_per_layer first divergent token
main None 0 (output is garbled from the first token)
this PR {-1, 1024} none (all 48 tokens match)

Prompts shorter than the window were not affected, because full and sliding attention agree there. Gemma 4 loads through mlx-lm, so it was not affected either.

Testing

  • pytest tests -m "not slow and not network": 6,217 passed
  • ruff check / ruff format --check.
  • Parity script: VLLM_ENABLE_V1_MULTIPROCESSING=0 python gemma3_window_parity.py (not included in this PR).

mlx-vlm defines its own RotatingKVCache and KVCache rather than
subclassing mlx-lm's, so the cache-factory probe returned None for
Gemma 3 and every layer ran as full attention. Accept both libraries'
classes; the probe's rules are unchanged.

Signed-off-by: syang-genai <191082996+syang-genai@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant