Repository navigation
[Model] Support PrismML Ternary-Bonsai-2 (prism_hadamard_qwen35) MLX checkpoints - #1378
Open
hks-9697-v2 wants to merge 2 commits into
Open
hks-9697-v2 wants to merge 2 commits into
hks-9697-v2 wants to merge 2 commits into
Conversation
…checkpoints PrismML's Ternary-Bonsai-2 MLX packs (2B/4B/9B/27B) are already implemented in the pinned mlx-vlm and mlx-lm loaders, but fail to load in vllm-metal because Transformers does not recognize model_type="prism_hadamard_qwen35" and config.json omits top-level "architectures". - Register a picklable Qwen3_5Config alias (PrismHadamardQwen35Config) in transformers.CONFIG_MAPPING that defaults architectures to Qwen3_5ForConditionalGeneration (when vision_config is present) or Qwen3_5ForCausalLM. - Add "prism_hadamard_qwen35" to GDN_MODEL_TYPES so --language-model-only / text-only (mlx-lm) loads resolve the GDN hybrid state family. - Add "prism_hadamard_qwen35" to _QWEN3_VL_MODEL_TYPES for explicit Qwen3-VL multimodal adapter dispatch. - Exclude custom quantized embedding wrappers exposing as_linear() (such as HadamardQuantizedEmbedding and OneBitEmbedding) from QLoRA linear wrapping in can_wrap_qlora(). Signed-off-by: hks-9697-v2 <harsh.shah.hks@gmail.com>
|
👋 Thanks for contributing to vllm-metal! Before review, please check the contributor guide:
Questions are welcome in Agent GuidelinesIf you are an AI agent, re-evaluate this PR against AGENTS.md and close it if it is not backed by a reproducer, measurements, or a concrete user need. See AGENTS.md. |
Add before/after AutoConfig registration tests, real mlx-vlm PrismML layer checks for QLoRA wrapping, native multimodal routing coverage for prism_hadamard_qwen35, TODO removal comment on the compat shim, and register the config alias in tools/check_parity.py. Signed-off-by: hks-9697-v2 <harsh.shah.hks@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
PrismML's Ternary-Bonsai-2 MLX checkpoints (
prism-ml/Ternary-Bonsai-2-{2B,4B,9B,27B}-mlx-2bit) store Hadamard-rotated 2-bit Qwen3.5 weights undermodel_type: "prism_hadamard_qwen35"and omit the top-levelarchitecturesfield fromconfig.json.Both upstream loaders already pinned in
pyproject.tomlsupport this format natively (mlx-vlmPR #2293,mlx-lmPR #1941), as doAutoTokenizerandAutoProcessor(Qwen3VLProcessor), yetvllm serve prism-ml/Ternary-Bonsai-2-2B-mlx-2bitcurrently fails immediately inAutoConfig.from_pretrained():In addition:
--language-model-only/ text-only loads (mlx-lmpath) retain top-levelmodel_type: "prism_hadamard_qwen35", which is missing fromGDN_MODEL_TYPESand raisesNotImplementedErrorinbuild_hybrid_runtime_plan().can_wrap_qlora()only excludesnn.QuantizedEmbedding, so custom quantized embedding wrappers (HadamardQuantizedEmbeddingandOneBitEmbedding) are erroneously treated as wrappable linear layers.Changes
vllm_metal/patches/prism_hadamard_config.py: Add a picklablePrismHadamardQwen35Config(Qwen3_5Config)(model_type = "prism_hadamard_qwen35") that populates defaultarchitectures(["Qwen3_5ForConditionalGeneration"]whenvision_configis present,["Qwen3_5ForCausalLM"]otherwise) when omitted.vllm_metal/compat.py: RegisterPrismHadamardQwen35Configintransformers.CONFIG_MAPPINGinsideapply_compat_patches()(idempotent; skips if already registered upstream).vllm_metal/attention/runtime/families/gdn.py: Add"prism_hadamard_qwen35"toGDN_MODEL_TYPES.vllm_metal/v1/model_adapter.py: Add"prism_hadamard_qwen35"to_QWEN3_VL_MODEL_TYPES.vllm_metal/v1/lora/layers.py: Guardcan_wrap_qlora()against custom quantized embedding modules exposingas_linear().tools/check_parity.py: Register_patch_transformers_prism_hadamard_qwen35_config()inmlx_generate()alongside the other compat shims.tests/test_compat.py(before/afterAutoConfigfailure-to-pass, idempotency, explicitarchitecturespreservation,pickleround-trip, no"qwen3_5"regression),tests/attention/test_hybrid_runtime_plan.py,tests/test_model_adapter.py(adapter construction + native multimodal routing), andtests/test_qlora.py(including realmlx-vlmHadamardQuantized*andOneBit*layers).Validation
25C56), AC power, no thermal/performance warnings before or after runs.vllm==0.33.0+cpu,mlx==0.32.4,mlx-lm @ 4cfc64784,mlx-vlm==0.7.11 @ b7bec926d,transformers==5.20.2, source-built non-NAX Metal kernels.1. Greedy Top-5 Parity (
tools/check_parity.py)B=1B=2FAIL)prism-ml/Ternary-Bonsai-2-2B-mlx-2bitmlx-vlmEXACT=38,TOP_K_MATCH=2EXACT=38,TOP_K_MATCH=2EXACT=76,TOP_K_MATCH=4,FAIL=0prism-ml/Bonsai-1.7B-mlx-1bit(baseline check)mlx-lmEXACT=38,TOP_K_MATCH=2EXACT=39,TOP_K_MATCH=1EXACT=77,TOP_K_MATCH=3,FAIL=0prism-ml/Ternary-Bonsai-1.7B-mlx-2bit(baseline check)mlx-lmEXACT=78,TOP_K_MATCH=2,FAIL=0(Both
TOP_K_MATCHprompts onTernary-Bonsai-2-2B-mlx-2bithad near-tied top-2 logprobs within0.0008nats.)2. 5-Shot MMLU Accuracy (
cais/mmlu)Stratified across all 57 MMLU subjects (10 test questions/subject = 570 questions, 5
devshots/subject, 100% logprob-scored overA/B/C/Dvia/v1/completionswithmax_tokens=1, temperature=0, logprobs=20,--enable-prefix-caching, without thinking tokens):prism-ml/Ternary-Bonsai-2-2B-mlx-2bitprism-ml/Ternary-Bonsai-1.7B-mlx-2bitprism-ml/Bonsai-1.7B-mlx-1bit3. Serving Performance (
vllm bench serve)Server:
VLLM_ENABLE_V1_MULTIPROCESSING=1 vllm serve <model> --host 127.0.0.1 --port 8123 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 2048 --no-enable-prefix-caching --generation-config vllmClient:
vllm bench serve --backend openai --endpoint /v1/completions --dataset-name random --random-range-ratio 0 --num-warmups 2 --request-rate inf --ignore-eos --temperature 0 --seed 713(0 failed requests across all runs).B=1(512 input / 128 output, 8 prompts)Ternary-Bonsai-2-2B-mlx-2bitBonsai-1.7B-mlx-1bitTernary-Bonsai-1.7B-mlx-2bitB=4(512 input / 128 output, 16 prompts,--max-concurrency 4)B=1Ternary-Bonsai-2-2B-mlx-2bitBonsai-1.7B-mlx-1bitTernary-Bonsai-1.7B-mlx-2bitLonger Context
B=1(2048 input / 128 output, 6 prompts)Ternary-Bonsai-2-2B-mlx-2bit4. E2E VLM, Offline
vllm.LLM, Text-Only, and 27B Geometry Verificationvllm.LLM(model="prism-ml/Ternary-Bonsai-2-2B-mlx-2bit", ...)using the Hugging Face repo ID directly (spawnedEngineCorepickling path) succeeded in both default VLM andlanguage_model_only=Truemodes with identical greedy outputs.prism-ml/Ternary-Bonsai-2-2B-mlx-2bitcorrectly identified a 128×128 solid red RGB image.ModelConfigonprism-ml/Ternary-Bonsai-2-{4B,9B,27B}-mlx-2bitand schema-1 text-only (Qwen3_5ForCausalLM), plus untiedHadamardQuantizedLinearlm_head(block=1024),supports_selective_logits(), and expanded GDN (Hv > Hk) lazy prefill/decode on a synthetic 27B-geometry checkpoint.5. Unit Tests & Lint
pytest tests/test_compat.py tests/attention/test_hybrid_runtime_plan.py tests/test_model_adapter.py tests/test_qlora.py .github/scripts/test_parity.py -q: 278 passed, 3 skipped.pytest -m "not slow" tests/ --ignore=tests/e2e -q: 6,098 passed, 0 non-MPS failures (prior commit; targeted suites re-run after follow-up commit).ruff check . && ruff format --check . && mypy vllm_metal: all passed cleanly.Checks Not Run
tests/test_mps_attention.pyandtests/test_mps_backend.py(83 failed / 9 errors in the full suite) because_mps_opswas not built in this MLX-only local venv; this PR does not touch the PyTorch MPS backend.shellcheck(not installed locally; this PR does not modify any.shfiles).Ternary-Bonsai-2-27B-mlx-2bit(8.02 GiB) due to 16 GB host RAM.