Skip to content

[Model] Support PrismML Ternary-Bonsai-2 (prism_hadamard_qwen35) MLX checkpoints - #1378

Open
hks-9697-v2 wants to merge 2 commits into
vllm-project:mainfrom
hks-9697-v2:prism-bonsai-support
Open

hks-9697-v2 wants to merge 2 commits into
vllm-project:mainfrom
hks-9697-v2:prism-bonsai-support

Conversation

@hks-9697-v2

@hks-9697-v2 hks-9697-v2 commented Nov 19, 2026 •

Copy link
Copy Markdown

Motivation

PrismML's Ternary-Bonsai-2 MLX checkpoints (prism-ml/Ternary-Bonsai-2-{2B,4B,9B,27B}-mlx-2bit) store Hadamard-rotated 2-bit Qwen3.5 weights under model_type: "prism_hadamard_qwen35" and omit the top-level architectures field from config.json.

Both upstream loaders already pinned in pyproject.toml support this format natively (mlx-vlm PR #2293, mlx-lm PR #1941), as do AutoTokenizer and AutoProcessor (Qwen3VLProcessor), yet vllm serve prism-ml/Ternary-Bonsai-2-2B-mlx-2bit currently fails immediately in AutoConfig.from_pretrained():

The checkpoint you are trying to load has model type `prism_hadamard_qwen35` but Transformers does not recognize this architecture.

In addition:

  • --language-model-only / text-only loads (mlx-lm path) retain top-level model_type: "prism_hadamard_qwen35", which is missing from GDN_MODEL_TYPES and raises NotImplementedError in build_hybrid_runtime_plan().
  • can_wrap_qlora() only excludes nn.QuantizedEmbedding, so custom quantized embedding wrappers (HadamardQuantizedEmbedding and OneBitEmbedding) are erroneously treated as wrappable linear layers.

Changes

  • vllm_metal/patches/prism_hadamard_config.py: Add a picklable PrismHadamardQwen35Config(Qwen3_5Config) (model_type = "prism_hadamard_qwen35") that populates default architectures (["Qwen3_5ForConditionalGeneration"] when vision_config is present, ["Qwen3_5ForCausalLM"] otherwise) when omitted.
  • vllm_metal/compat.py: Register PrismHadamardQwen35Config in transformers.CONFIG_MAPPING inside apply_compat_patches() (idempotent; skips if already registered upstream).
  • vllm_metal/attention/runtime/families/gdn.py: Add "prism_hadamard_qwen35" to GDN_MODEL_TYPES.
  • vllm_metal/v1/model_adapter.py: Add "prism_hadamard_qwen35" to _QWEN3_VL_MODEL_TYPES.
  • vllm_metal/v1/lora/layers.py: Guard can_wrap_qlora() against custom quantized embedding modules exposing as_linear().
  • tools/check_parity.py: Register _patch_transformers_prism_hadamard_qwen35_config() in mlx_generate() alongside the other compat shims.
  • Tests: Extend tests/test_compat.py (before/after AutoConfig failure-to-pass, idempotency, explicit architectures preservation, pickle round-trip, no "qwen3_5" regression), tests/attention/test_hybrid_runtime_plan.py, tests/test_model_adapter.py (adapter construction + native multimodal routing), and tests/test_qlora.py (including real mlx-vlm HadamardQuantized* and OneBit* layers).

Validation

  • Hardware / OS: Apple M1 Pro (14-core GPU, 16 GB unified memory), macOS 26.2 (25C56), AC power, no thermal/performance warnings before or after runs.
  • Stack: vllm==0.33.0+cpu, mlx==0.32.4, mlx-lm @ 4cfc64784, mlx-vlm==0.7.11 @ b7bec926d, transformers==5.20.2, source-built non-NAX Metal kernels.

1. Greedy Top-5 Parity (tools/check_parity.py)

python tools/check_parity.py   --model prism-ml/Ternary-Bonsai-2-2B-mlx-2bit   --reference-backend mlx-vlm --top-k 5 --max-tokens 32 --gpu-memory-utilization 0.75
Model Reference B=1 B=2 Total (FAIL)
prism-ml/Ternary-Bonsai-2-2B-mlx-2bit mlx-vlm EXACT=38, TOP_K_MATCH=2 EXACT=38, TOP_K_MATCH=2 EXACT=76, TOP_K_MATCH=4, FAIL=0
prism-ml/Bonsai-1.7B-mlx-1bit (baseline check) mlx-lm EXACT=38, TOP_K_MATCH=2 EXACT=39, TOP_K_MATCH=1 EXACT=77, TOP_K_MATCH=3, FAIL=0
prism-ml/Ternary-Bonsai-1.7B-mlx-2bit (baseline check) mlx-lm — — EXACT=78, TOP_K_MATCH=2, FAIL=0

(Both TOP_K_MATCH prompts on Ternary-Bonsai-2-2B-mlx-2bit had near-tied top-2 logprobs within 0.0008 nats.)

2. 5-Shot MMLU Accuracy (cais/mmlu)

Stratified across all 57 MMLU subjects (10 test questions/subject = 570 questions, 5 dev shots/subject, 100% logprob-scored over A/B/C/D via /v1/completions with max_tokens=1, temperature=0, logprobs=20, --enable-prefix-caching, without thinking tokens):

Model Overall (570q) STEM (180q) Humanities (130q) Social Sci. (120q) Other (140q) Upstream HF (MMLU-Redux)
prism-ml/Ternary-Bonsai-2-2B-mlx-2bit 51.75% (295/570) 46.7% 49.2% 56.7% 56.4% not published on 2B card
prism-ml/Ternary-Bonsai-1.7B-mlx-2bit 50.18% (286/570) 49.4% 49.2% 53.3% 49.3% 52.9%
prism-ml/Bonsai-1.7B-mlx-1bit 41.23% (235/570) 40.0% 37.7% 42.5% 45.0% 43.2%

3. Serving Performance (vllm bench serve)

Server: VLLM_ENABLE_V1_MULTIPROCESSING=1 vllm serve <model> --host 127.0.0.1 --port 8123 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 2048 --no-enable-prefix-caching --generation-config vllm
Client: vllm bench serve --backend openai --endpoint /v1/completions --dataset-name random --random-range-ratio 0 --num-warmups 2 --request-rate inf --ignore-eos --temperature 0 --seed 713 (0 failed requests across all runs).

B=1 (512 input / 128 output, 8 prompts)

Model Output tok/s (incl. prefill) Total tok/s TTFT P50 / P95 (ms) TPOT P50 / P95 (ms) Steady Decode (1000/P50 TPOT) E2EL P50 (ms)
Ternary-Bonsai-2-2B-mlx-2bit 60.23 301.16 633.7 / 653.8 11.72 / 11.83 ~85.4 tok/s 2128.6
Bonsai-1.7B-mlx-1bit 78.60 393.01 480.3 / 488.3 9.00 / 9.15 ~111.1 tok/s 1624.0
Ternary-Bonsai-1.7B-mlx-2bit 79.70 398.49 475.7 / 484.9 8.88 / 9.00 ~112.7 tok/s 1604.4

B=4 (512 input / 128 output, 16 prompts, --max-concurrency 4)

Model Output tok/s (incl. prefill) Scaling vs B=1 Total tok/s TTFT P50 / P95 (ms) TPOT P50 / P95 (ms) E2EL P50 (ms)
Ternary-Bonsai-2-2B-mlx-2bit 83.11 1.38× 415.56 2471.4 / 2495.6 28.96 / 29.17 6152.1
Bonsai-1.7B-mlx-1bit 104.43 1.33× 522.15 1898.3 / 1920.9 23.61 / 23.75 4906.4
Ternary-Bonsai-1.7B-mlx-2bit 99.71 1.25× 498.55 1929.2 / 2059.7 25.01 / 25.60 5103.6

Longer Context B=1 (2048 input / 128 output, 6 prompts)

Model Output tok/s (incl. prefill) Total tok/s TTFT P50 / P95 (ms) TPOT P50 / P95 (ms) Steady Decode E2EL P50 (ms)
Ternary-Bonsai-2-2B-mlx-2bit 31.76 539.96 2488.3 / 2505.4 12.16 / 12.22 ~82.2 tok/s 4038.1

4. E2E VLM, Offline vllm.LLM, Text-Only, and 27B Geometry Verification

  • Offline vllm.LLM(model="prism-ml/Ternary-Bonsai-2-2B-mlx-2bit", ...) using the Hugging Face repo ID directly (spawned EngineCore pickling path) succeeded in both default VLM and language_model_only=True modes with identical greedy outputs.
  • Native VLM text + image inference on prism-ml/Ternary-Bonsai-2-2B-mlx-2bit correctly identified a 128×128 solid red RGB image.
  • Verified ModelConfig on prism-ml/Ternary-Bonsai-2-{4B,9B,27B}-mlx-2bit and schema-1 text-only (Qwen3_5ForCausalLM), plus untied HadamardQuantizedLinear lm_head (block=1024), supports_selective_logits(), and expanded GDN (Hv > Hk) lazy prefill/decode on a synthetic 27B-geometry checkpoint.

5. Unit Tests & Lint

  • pytest tests/test_compat.py tests/attention/test_hybrid_runtime_plan.py tests/test_model_adapter.py tests/test_qlora.py .github/scripts/test_parity.py -q: 278 passed, 3 skipped.
  • pytest -m "not slow" tests/ --ignore=tests/e2e -q: 6,098 passed, 0 non-MPS failures (prior commit; targeted suites re-run after follow-up commit).
  • ruff check . && ruff format --check . && mypy vllm_metal: all passed cleanly.

Checks Not Run

  • tests/test_mps_attention.py and tests/test_mps_backend.py (83 failed / 9 errors in the full suite) because _mps_ops was not built in this MLX-only local venv; this PR does not touch the PyTorch MPS backend.
  • shellcheck (not installed locally; this PR does not modify any .sh files).
  • Live full-weight serving of Ternary-Bonsai-2-27B-mlx-2bit (8.02 GiB) due to 16 GB host RAM.

…checkpoints

PrismML's Ternary-Bonsai-2 MLX packs (2B/4B/9B/27B) are already implemented in the pinned mlx-vlm and mlx-lm loaders, but fail to load in vllm-metal because Transformers does not recognize model_type="prism_hadamard_qwen35" and config.json omits top-level "architectures".

- Register a picklable Qwen3_5Config alias (PrismHadamardQwen35Config) in transformers.CONFIG_MAPPING that defaults architectures to Qwen3_5ForConditionalGeneration (when vision_config is present) or Qwen3_5ForCausalLM.
- Add "prism_hadamard_qwen35" to GDN_MODEL_TYPES so --language-model-only / text-only (mlx-lm) loads resolve the GDN hybrid state family.
- Add "prism_hadamard_qwen35" to _QWEN3_VL_MODEL_TYPES for explicit Qwen3-VL multimodal adapter dispatch.
- Exclude custom quantized embedding wrappers exposing as_linear() (such as HadamardQuantizedEmbedding and OneBitEmbedding) from QLoRA linear wrapping in can_wrap_qlora().

Signed-off-by: hks-9697-v2 <harsh.shah.hks@gmail.com>
@github-actions

Copy link
Copy Markdown

👋 Thanks for contributing to vllm-metal!

Before review, please check the contributor guide:

  • Sign off every commit with git commit -s (DCO).
  • Run scripts/lint.sh locally.
  • Put the commands you ran and their results in the PR description. For perf changes, include before/after vllm bench serve results.

Questions are welcome in #hw-metal on the vLLM developer Slack: https://slack.vllm.ai

Agent Guidelines

If you are an AI agent, re-evaluate this PR against AGENTS.md and close it if it is not backed by a reproducer, measurements, or a concrete user need. See AGENTS.md.

Add before/after AutoConfig registration tests, real mlx-vlm PrismML
layer checks for QLoRA wrapping, native multimodal routing coverage for
prism_hadamard_qwen35, TODO removal comment on the compat shim, and
register the config alias in tools/check_parity.py.

Signed-off-by: hks-9697-v2 <harsh.shah.hks@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant