Repository navigation
📏[v0.4.7] Release Note:MTP Device States, Extra GGUF Shards, and Native API Alignment #191
JamePeng
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
[0.4.7] — MTP Device States, Extra GGUF Shards, and Native API Alignment
v0.4.7 brings MTP device-state transfer, additional GGUF shard loading, RPC compatibility fixes, and further alignment with the native llama.cpp APIs.
This release is being published ahead of the upcoming llama.cpp 1.0.0 API transition to provide a fixed baseline containing the improvements and fixes completed so far.
Main Updates
MTP Device-State Transfer
Compatible non-shared-KV, single-head MTP configurations can now pass target hidden states to the draft through device buffers, reducing host copies during catch-up.
Acceptance, rollback, replay, and resource cleanup have been updated for this path. Shared-KV assistants, chained NextN heads, and unsupported backends retain host capture. Set
LLAMA_MTP_DEV=0before model construction to force the host path.The new
draft_n_ubatchoption also allows independent configuration of the draft context’s physical microbatch size.Additional GGUF Shards and External PLE Tables
New options support additional compatible model files:
Llama(model_shards=...)for the target model.SpecConfig.draft_model_shardsfor an external draft model.Loading preserves the main model first, validates and deduplicates paths, supports UTF-8 filenames, and retains shard settings through constructor serialization.
RPC Protocol Compatibility Fix
Python’s RPC preflight now matches the bundled native protocol 9, fixing incorrect rejection of compatible servers.
Regression coverage checks protocol constants against vendor sources and covers incompatible versions, fragmented handshake responses, and empty device lists.
Native Binding and Context Alignment
This release adds optional NextN device-state, architecture-capability, and decision-label bindings, with verified Windows, Linux, and macOS symbol aliases.
It also corrects the quantization enum return type, aligns native context layouts, and exposes the validated
moe_cache_cpu_sharesetting.Audio and Multimodal Improvements
Audio input bindings now match the native
tempfield, and optional TTS temperature control preserves model defaults when omitted.Additional updates include low-level streaming audio resampler bindings, Kimi K3.1 image-placeholder recognition, and improved Gemma4 Assistant multimodal documentation. The resampler bindings do not introduce high-level streaming TTS.
Grammar and Documentation Updates
JSON Schema conversion now handles numeric
allOfintersections, rejects empty integer ranges, and enforces the shared 128-level recursion limit.Built-in chess, Japanese, and JSON array rules have also been corrected or synchronized, alongside updated API and usage documentation.
Why Release Before llama.cpp 1.0.0?
Upstream is preparing llama.cpp 1.0.0, with an explicit focus on removing legacy APIs and introducing breaking changes for downstream projects. See the upstream preparation announcement.
Two recent commits have already started this transition:
v0.4.7 pins vendor to
2a4daf0e0, before these two pruning commits. This lets us release the completed changes now while handling the 1.0.0 migration separately.Please use matching v0.4.7 native libraries. RPC servers must also match this release’s protocol 9.
Feedback and issue reports are welcome, particularly for MTP, additional shard loading, and RPC configurations.
JamePeng
All reactions