| title | Llama Class |
|---|---|
| module_name | llama_cpp.llama |
| source_file | llama_cpp/llama.py |
| class_name | Llama |
| last_updated | 2026-11-15 |
| version_target | latest |
The Llama class is the core, high-level Python wrapper for a llama.cpp model. It handles model loading, memory management (KV cache), tokenization, and generation (both base text completion and chat formatting). It includes advanced features like dynamic LoRA routing, dual-mode hybrid/recurrent checkpointing, speculative decoding, and context shifting.
Llama is the main user-facing entry point for loading a GGUF model and
creating a native llama.cpp context. It exposes completion, chat, tokenization,
embedding, state, sampling, and runtime configuration APIs through one managed
object.
Use Llama when one application needs a general-purpose model interface.
For embedding-only applications, LlamaEmbedding provides embedding-oriented
defaults and additional reranking helpers while inheriting the same model and
context lifecycle.
For manual context submission, see Llama Batch for the internal builders, range limits, output indices, and cleanup rules.
Initialize the model and context. Normal inference construction allocates model
and context resources according to the loading and offloading parameters.
vocab_only=True avoids loading weight tensors, while no_alloc=True simulates
model allocation for inspection and planning rather than inference.
| Parameter | Type | Default | Description |
|---|---|---|---|
model_path |
str |
Required | Model file path (GGUF format) |
model_shards |
Optional[Sequence[str]] |
None |
Extra GGUF files, including external PLE tables. The main model stays first; repeated paths are deduplicated. Native numbered-shard discovery and validation still apply. |
mmproj_path |
Optional[str] |
None |
Optional multimodal projection GGUF. When provided, Llama creates a generic MTMD chat handler; it replaces an explicitly supplied chat_handler. Prefer passing it by keyword. |
n_gpu_layers |
Union[int, Literal["auto", "all"]] |
"auto" |
Maximum number of model layers stored in VRAM: • auto/-1: auto-selected by llama.cpp• all/-2: all possible layers• positive integer N: offload up to N layers • 0: disable layer offload |
cpu_moe |
bool |
False |
Whether to keep all MoE weights on CPU |
n_cpu_moe |
int |
0 |
Number of first N MoE layers to keep on CPU (compatible with cpu_moe) |
split_mode |
int |
LLAMA_SPLIT_MODE_LAYER |
Model GPU split mode: • LLAMA_SPLIT_MODE_NONE: single GPU• LLAMA_SPLIT_MODE_ROW: row-level split• LLAMA_SPLIT_MODE_LAYER: layer-level split |
load_mode |
int (llama_load_mode) |
LLAMA_LOAD_MODE_AUTO |
How model data is loaded. AUTO lets llama.cpp choose from device capabilities; the explicit LLAMA_LOAD_MODE_* values are described below. |
lazy_mode |
int (llama_lazy_mode) |
LLAMA_LAZY_MODE_AUTO |
Controls on-demand reads for architecture-marked tensors when mmap is active. |
main_gpu |
int |
0 |
With LLAMA_SPLIT_MODE_NONE, selects the GPU for the whole model. With ROW, selects the GPU for small tensors and intermediate results. It is ignored with LAYER. RPC devices precede selected local GPUs in the device order. |
tensor_split |
Optional[List[float]] |
None |
Proportional split of tensors across GPUs (max LLAMA_MAX_DEVICES). With RPC, supply one value per selected remote and local GPU device, in device order. |
rpc_servers |
Optional[Union[str, Sequence[str]]] |
None |
RPC endpoints as host:port strings or one comma-separated string. Each endpoint may expose multiple devices. Requires an available RPC backend and compatible server. |
rpc_local_devices |
Optional[Sequence[str]] |
None |
Local GPU device names to include after RPC devices. None selects default local GPUs; [] selects only remote devices. Requires rpc_servers. |
kv_overrides |
Optional[Dict[str, Union[bool, int, float, str]]] |
None |
Key-value overrides for the model metadata. |
use_mmap, use_direct_io, use_mlock |
bool |
False |
Deprecated compatibility arguments. They no longer configure native loading; use load_mode. |
vocab_only |
bool |
False |
Load model metadata and vocabulary without weight tensors. |
check_tensors |
bool |
False |
Validate tensor data while loading the model. This increases load time. |
use_extra_bufts |
bool |
True |
Allow extra backend buffer types, including supported weight-repacking paths. |
no_host |
bool |
False |
Bypass the ordinary host buffer so compatible extra buffer types can be used. Advanced backend option. |
no_alloc |
bool |
False |
Load metadata and simulate model allocation without allocating tensor data. Intended for inspection and planning rather than inference. |
load_mtp |
bool |
False |
Load the target model's NextN/MTP tensors. This is enabled automatically for built-in MTP through speculative; normally it should not be set manually. |
numa |
Union[bool, int] |
False |
NUMA strategy (e.g., GGML_NUMA_STRATEGY_DISTRIBUTE). |
RPC registration is process-wide, but each Llama instance passes its own
explicit device list. RPC model construction rejects row split mode and fails
if a configured server is unavailable; it does not fall back to local GPUs.
See RPC model offloading for setup, lifecycle, and
multimodal device selection.
model_shards accepts additional local paths, including paths on other drives.
The loader uses llama_model_load_from_splits() when distinct extra files are
present. Pass a sequence, not a comma-separated string. Files must exist before
loading; native validation checks tensor and shard compatibility. Constructor
serialization preserves the extra paths.
For a model exported with the vendor converter's --no-ple and --ple-only
options from the same checkpoint:
from contextlib import closing
from llama_cpp import Llama
with closing(Llama(model_path="./model-no-ple.gguf",
model_shards=["./model-ple.gguf"])) as llm:
print(llm.create_completion("Hello", max_tokens=32)["choices"][0]["text"])The native loader also permits an extra shard containing only
per_layer_token_embd.weight to replace an existing PLE table. The table must
match the model. For a separate speculative draft, use
SpecConfig.draft_model_shards.
load_mode replaces the legacy use_mmap, use_direct_io, and use_mlock
arguments. It accepts a member of llama_cpp.llama_load_mode:
| Value | Integer | Description |
|---|---|---|
LLAMA_LOAD_MODE_AUTO |
-1 |
Default. Auto-detect the loading behavior from device capabilities; currently resolves to mmap where supported and ordinary loading otherwise. |
LLAMA_LOAD_MODE_NONE |
0 |
Use no special model-loading mode. |
LLAMA_LOAD_MODE_MMAP |
1 |
Memory-map the model. |
LLAMA_LOAD_MODE_MLOCK |
2 |
Keep the loaded model in RAM rather than allowing it to be swapped or compressed. |
LLAMA_LOAD_MODE_MMAP_MLOCK |
3 |
Memory-map the model and keep its mapped pages in RAM. |
LLAMA_LOAD_MODE_DIRECT_IO |
4 |
Use direct I/O when it is available. |
lazy_mode accepts a member of llama_cpp.llama_lazy_mode:
| Value | Integer | Description |
|---|---|---|
LLAMA_LAZY_MODE_OFF |
0 |
Read complete tensors up front. |
LLAMA_LAZY_MODE_AUTO |
1 |
Default. Lazily read architecture-marked tensors only when they are larger than 4 GiB. Requires mmap. |
LLAMA_LAZY_MODE_ON |
2 |
Read rows of every architecture-marked tensor on demand. Requires mmap. |
import llama_cpp
llm = llama_cpp.Llama(
model_path="models/model.gguf",
load_mode=llama_cpp.llama_load_mode.LLAMA_LOAD_MODE_MMAP_MLOCK,
lazy_mode=llama_cpp.llama_lazy_mode.LLAMA_LAZY_MODE_AUTO,
)The legacy loading arguments are retained only for call compatibility. They no
longer configure the underlying model parameters and may emit a deprecation
warning; set load_mode explicitly instead. Use the following migration
mapping:
| Legacy configuration | Replacement |
|---|---|
use_mmap=False, use_mlock=False |
load_mode=LLAMA_LOAD_MODE_NONE |
use_mmap=True, use_mlock=False |
load_mode=LLAMA_LOAD_MODE_MMAP |
use_mmap=False, use_mlock=True |
load_mode=LLAMA_LOAD_MODE_MLOCK |
use_mmap=True, use_mlock=True |
load_mode=LLAMA_LOAD_MODE_MMAP_MLOCK |
use_direct_io=True |
load_mode=LLAMA_LOAD_MODE_DIRECT_IO |
| Parameter | Type | Default | Description |
|---|---|---|---|
seed |
int |
LLAMA_DEFAULT_SEED |
RNG seed used by the model. LLAMA_DEFAULT_SEED requests a random seed. |
n_ctx |
int |
512 |
Requested total native context capacity. Set to 0 to load from model metadata. Use n_ctx_seq() for the resolved single-sequence window. |
n_keep |
int |
256 |
Preferred number of leading tokens to preserve during automatic context shifting. |
n_batch |
int |
2048 |
Maximum number of tokens in a logical prompt-processing batch. The effective value cannot exceed n_ctx. |
n_ubatch |
int |
512 |
Maximum number of tokens in a physical micro-batch processed by llama.cpp. |
n_seq_max |
int |
1 |
Capacity for independent sequence states. Non-unified KV partitions total context among these slots. Ordinary generation uses sequence 0; embedding calls split automatically at this limit. |
n_rs_seq |
int |
0 |
Experimental recurrent-state snapshots retained per sequence for rollback. 0 disables rollback snapshots. |
n_outputs_max |
int |
0 |
Maximum outputs in a physical batch. 0 uses the effective n_batch; encoder models also use n_batch regardless of this override. |
n_outputs_max_per_seq |
int |
1 |
Maximum outputs per sequence with an attached backend sampler. 0 uses the effective n_outputs_max; larger values are capped at that limit. |
moe_cache_size |
int |
0 |
Experimental device cache budget in bytes for MoE experts kept in host memory, distributed among devices like layers. 0 disables it. Requires a GPU and MoE model; tensor parallelism is unsupported. |
moe_cache_cpu_share |
float |
0.0 |
Experimental share of MoE cache misses computed on CPU instead of uploaded to GPU when caching is enabled. Must be finite and in [0, 1]; 0 uploads all misses. Preserved when serializing and restoring constructor settings. |
n_threads |
Optional[int] |
None |
Number of threads for generation (defaults to CPU count // 2). |
n_threads_batch |
Optional[int] |
None |
Number of threads for batch processing (defaults to CPU count). |
ctx_type |
int |
LLAMA_CONTEXT_TYPE_DEFAULT |
Context implementation selected by llama.cpp. Keep the default unless a model or backend requires another context type. |
| Parameter | Type | Default | Description |
|---|---|---|---|
embeddings |
bool |
False |
Enable embedding extraction alongside logits. Must be True before calling embed() or create_embedding(). |
pooling_type |
int |
LLAMA_POOLING_TYPE_UNSPECIFIED |
Pooling strategy for embedding output. UNSPECIFIED follows model metadata, NONE returns token-level vectors, and RANK returns classifier or reranking output. |
attention_type |
Optional[int] |
LLAMA_ATTENTION_TYPE_UNSPECIFIED |
Attention mode used by the context. None is normalized to UNSPECIFIED, which lets llama.cpp select the model-compatible behavior. |
logits_all |
bool |
False |
Retain logits for every evaluated token instead of only requested outputs. Completion log probabilities require this mode. |
flash_attn_type |
Optional[int] |
LLAMA_FLASH_ATTN_TYPE_AUTO |
Controls when Flash Attention is enabled. None is normalized to AUTO. |
rope_scaling_type |
Optional[int] |
LLAMA_ROPE_SCALING_TYPE_UNSPECIFIED |
RoPE scaling strategy. The unspecified value follows model metadata. |
rope_freq_base |
float |
0.0 |
RoPE base frequency override. 0.0 follows model metadata. |
rope_freq_scale |
float |
0.0 |
RoPE frequency scaling override. 0.0 follows model metadata. |
yarn_ext_factor |
float |
-1.0 |
YaRN extrapolation mix factor. A negative value follows model metadata. |
yarn_attn_factor |
float |
1.0 |
YaRN attention magnitude scaling factor. |
yarn_beta_fast |
float |
32.0 |
YaRN low-correction dimension. |
yarn_beta_slow |
float |
1.0 |
YaRN high-correction dimension. |
yarn_orig_ctx |
int |
0 |
Original YaRN context size. 0 follows model metadata. |
offload_kqv |
bool |
True |
Offload K, Q, and V tensor operations to the selected device when supported. |
no_perf |
bool |
False |
Disable native performance timing collection when True. |
op_offload |
Optional[bool] |
None |
Whether supported host tensor operations may be offloaded to a device. None keeps llama.cpp's default. |
swa_full |
Optional[bool] |
None |
Use a full-size sliding-window-attention cache. None keeps llama.cpp's default. |
kv_unified |
Optional[bool] |
None |
Use a shared KV buffer for all sequences; otherwise total capacity is partitioned across n_seq_max slots. None preserves the native default. LlamaEmbedding forces True. |
type_k / type_v |
Optional[int] |
None |
KV cache data types for keys and values. None uses llama.cpp defaults. |
| Parameter | Type | Default | Description |
|---|---|---|---|
chat_format |
Optional[str] |
None |
String specifying the chat template (e.g., "llama-2", "chatml"). Guessed from GGUF if None. |
chat_handler |
Optional[LlamaChatCompletionHandler] |
None |
Optional custom handler. See Qwen image chat for a custom multimodal handler. |
chat_template_name |
Optional[str] |
None |
Named chat template passed to the generic MTMD handler created by mmproj_path. |
chat_handler_kwargs |
Dict[str, Any] |
{} |
Additional keyword arguments passed to the generic MTMD chat handler created by mmproj_path. |
tokenizer |
Optional[BaseLlamaTokenizer] |
None |
Override the tokenizer used by the high-level API. By default, LlamaTokenizer wraps the loaded model vocabulary. |
draft_model |
Optional[LlamaDraftModel] |
None |
Deprecated stateless draft callback kept for compatibility. New code should use speculative. |
speculative |
Optional[Union[SpecConfig, LlamaSpecEngine]] |
None |
Stateful speculative configuration or engine. Supports the complete begin/draft/process/accept lifecycle; it cannot be combined with draft_model. |
ctx_checkpoints |
int |
16 |
Max hybrid/recurrent prompt checkpoints to keep. 0 disables these snapshots. Ordinary text requests can still extend the live prefix; branching or shorter prompts require a fresh prefill. See zero-checkpoint limits. |
checkpoint_interval |
int |
4096 |
Token interval for saving periodic Hybrid/Recurrent checkpoints during long prompt evaluation. |
checkpoint_on_device |
bool |
False |
Store Hybrid/Recurrent checkpoint tensor payloads in llama_context-owned device buffers via LLAMA_STATE_SEQ_FLAGS_ON_DEVICE. Reduces device-to-host copy overhead, but only one active checkpoint per seq_id is safe. |
last_n_tokens_size |
int |
64 |
Legacy constructor setting retained for serialization compatibility. Configure repetition history with per-request penalty_last_n; zero disables it. |
spm_infill |
bool |
False |
Use the Suffix/Prefix/Middle order for infill instead of Prefix/Suffix/Middle. |
| Parameter | Type | Default | Description |
|---|---|---|---|
verbose |
bool |
True |
Backward-compatible boolean native logging switch. False keeps only error-level llama.cpp / ggml logs; True enables debug-level native logs. If verbosity is provided, verbosity takes precedence over verbose. |
verbosity |
Optional[Union[int, str, bool]] |
None |
Fine-grained llama.cpp-style native runtime log verbosity. Numeric levels: 0=output, 1=error, 2=warning, 3=info, 4=trace, 5=debug. Use verbosity=3 for llama.cpp-style default info logs. String aliases such as "silent", "quiet", "info", "trace", and "debug" are also accepted. |
log_filters |
Optional[Sequence[str]] |
None |
Optional substring filters for native runtime logs. If any provided substring appears in a decoded backend log message, that message is suppressed. The default logger may include built-in filters for noisy low-level logs such as CUDA Graph id %d reuse messages. Pass an empty list [] to disable default substring filtering. |
log_filters_case_sensitive |
bool |
True |
Whether log_filters should match case-sensitively. Defaults to True for predictable low-level backend log filtering. |
Unknown extra constructor keywords are accepted through **kwargs for
compatibility but are not applied to model or context parameters. Use explicit
named parameters from the current signature.
Generates a chat response using the configured chat_format or chat_handler.
It also accepts presence_penalty as a compatibility alias for the primary
present_penalty parameter; an explicitly non-default present_penalty takes
precedence.
import llama_cpp
model = llama_cpp.Llama(model_path="models/qwen2.5-7b-instruct.gguf", n_gpu_layers=-1)
response = model.create_chat_completion(
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain KV caching."}
],
temperature=0.7,
max_tokens=2048
)
print(response["choices"][0]["message"]["content"])Pass response_format={"type": "json_object"} to constrain the response to
valid JSON. Add a schema property when the output must follow a specific JSON
Schema:
response = model.create_chat_completion(
messages=[{"role": "user", "content": "Extract: Ada is 36 years old."}],
response_format={
"type": "json_object",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"age": {"type": "integer"},
},
"required": ["name", "age"],
},
},
temperature=0.0,
)The schema is converted to a llama.cpp grammar. Keep schemas small and explicit for better reliability with compact models.
Chat handlers and model templates that support tool calling accept OpenAI-style
tools and tool_choice arguments:
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the weather for a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}
]
response = model.create_chat_completion(
messages=[{"role": "user", "content": "What is the weather in Shanghai?"}],
tools=tools,
tool_choice="auto",
)
tool_calls = response["choices"][0]["message"].get("tool_calls")Tool syntax and automatic selection depend on the model's chat template. Pass a
specific tool_choice object to force one named function when the active handler
supports it.
This method accepts the same arguments as create_chat_completion() but returns
typed objects from the OpenAI Python package instead of dictionaries. Install
the optional dependency first with pip install openai.
response = model.create_chat_completion_openai_v1(
messages=[{"role": "user", "content": "Say hello in one sentence."}],
max_tokens=64,
)
print(response.choices[0].message.content)With stream=True, it returns an iterator of typed ChatCompletionChunk
objects.
The OpenAI SDK's typed finish_reason currently does not include the local
"abort" extension. If an interrupted response is converted through this
method, the SDK may raise a validation error. Use create_chat_completion()
directly when the caller needs to handle finish_reason="abort".
Generates standard text completion from a raw string prompt.
For OpenAI naming compatibility, both methods accept presence_penalty as an
optional alias. Internally it is normalized to the primary present_penalty
parameter only when present_penalty remains at its default value.
import llama_cpp
model = llama_cpp.Llama(model_path="models/llama-3-8b.gguf")
output = model("The capital of Japan is", max_tokens=10, stop=["\n"])
print(output["choices"][0]["text"])A low-level generator yielding token IDs one by one. Highly customizable with sampling parameters, dynamic LoRA mounting, and control vectors.
import llama_cpp
model = llama_cpp.Llama(model_path="models/llama-3-8b.gguf")
tokens = model.tokenize(b"def fibonacci(n):")
for token in model.generate(tokens, top_k=40, top_p=0.95, temp=0.2):
print(model.detokenize([token]).decode('utf-8'), end="", flush=True)Pass sampling options per request. create_completion(), __call__(), and
create_chat_completion() use temperature; generate() and sample() use
temp. Chat handlers must forward the options they support; a custom handler
can impose additional limits. The OpenAI-compatible server exposes its own
request schema, not every Python sampling option.
| Request parameter | Default | Meaning |
|---|---|---|
temperature / temp |
0.8; chat 0.2 |
Plain logit temperature; with ordinary sampling and dynamic temperature disabled, non-positive values keep only the maximum remaining logit. |
top_k, top_p, min_p, typical_p |
40, 0.95, 0.05, 1.0 |
Candidate filters; disable with <= 0, 1.0, 0.0, and 1.0, respectively. |
min_keep |
0 |
Minimum candidate count for top-p, min-p, typical-p, and XTC. Zero adds no minimum; it does not disable those filters. |
dynatemp_range, dynatemp_exponent |
0.0, 1.0 |
A positive range enables entropy-based temperature between max(0, temp - range) and temp + range. |
top_n_sigma |
-1.0 |
Filter logits by their distance from the maximum in standard deviations; negative disables it. |
xtc_probability, xtc_threshold |
0.0, 0.1 |
Probability and threshold for excluding high-probability choices; zero probability disables XTC. |
repeat_penalty, frequency_penalty, present_penalty |
1.0, 0.0, 0.0 |
Repetition multiplier, frequency-scaled subtraction, and presence subtraction. History includes prompt and accepted generated tokens. |
penalty_last_n, dry_penalty_last_n |
64, 64 |
Separate history lengths. 0 disables the respective history; -1 uses the initialized n_ctx_seq() capacity. |
dry_multiplier, dry_base, dry_allowed_length |
0.0, 1.75, 2 |
DRY repetition penalty; enabled only with nonzero multiplier, base at least 1.0, and positive history. |
dry_seq_breakers |
["\n", ":", "\"", "*"] |
Text sequences that interrupt DRY repetition matching; native limits are described below. |
adaptive_target, adaptive_decay |
-1.0, 0.9 |
A target from 0.0 to 1.0 enables Adaptive-P as the final selector. Negative disables it; decay is in 0.0–0.99. |
mirostat_mode, mirostat_tau, mirostat_eta |
0, 5.0, 0.1 |
Disabled, Mirostat 1, or Mirostat 2 (0, 1, 2); target surprisal and learning rate. Other mode values raise ValueError during sampler construction. |
Ordinary sampling applies user biases/model suppression, standard penalties,
DRY, optional reasoning-budget control and Python callbacks, then top-n-sigma,
top-k, typical-p, top-p, min-p, XTC, and temperature. Adaptive-P replaces the
final greedy/distribution selector when enabled; mild filtering such as min-p
alone is recommended before it. For that setup, disable top-k/top-p with
top_k=0, top_p=1.0 and use temperature=1.0.
Mirostat follows the native common sampler's separate path: plain temperature followed by Mirostat. It bypasses standard/DRY penalties and all the ordinary filters, dynamic temperature, and Adaptive-P. User logit biases, model suppression, Python logits processors, grammar, and reasoning-budget constraints remain active. Sampling controls alter selection after model evaluation; they do not themselves reduce prefill computation.
Native DRY ignores empty breakers, truncates each to 40 UTF-8 bytes, deduplicates the resulting strings, and processes at most 64 distinct breakers. Initialization scans the vocabulary for each distinct breaker, so avoid unnecessarily large lists. Direct C constructors take non-negative history counts; their negative values disable history instead of selecting context size.
The internal LlamaSamplingParams uses different names for several fields:
temp, mirostat, penalty_repeat, penalty_freq, penalty_present, and
dry_sequence_breakers. Applications normally use the request parameters above.
sample() reuses an active generation sampler when one exists; its arguments
only construct a temporary sampler when none is active. A standalone sample()
call does not populate prompt history automatically.
For text completions, ignore_eos=True skips the EOG stop check; it does not
mask EOG logits. Use logit_bias to suppress specific token IDs when needed.
n_ctx() returns the resolved total capacity, matching native llama_n_ctx().
n_ctx_seq() returns the single-sequence limit, matching llama_n_ctx_seq().
Generation budgets, context shifting, and multimodal capacity checks use
n_ctx_seq().
With kv_unified=False, capacity is partitioned among n_seq_max slots. For
example, n_ctx=512, n_seq_max=2 resolves to 512 total and 256 per sequence;
a 300-token completion prompt is rejected before decoding. With
kv_unified=True, each sequence can use the total window, shared with other
active sequences. Native padding can adjust both sizes, so query the initialized
instance rather than assuming n_ctx / n_seq_max is the final limit.
n_seq_max reserves sequence-state capacity, including distinct states for
recurrent models. Ordinary create_completion() and create_chat_completion()
use sequence 0: user/assistant turns in one conversation do not each require
another sequence slot. Increasing this setting does not enable concurrent
requests on the same instance. Keep n_seq_max=1 for ordinary single-conversation
generation unless another feature needs additional sequences.
For an aligned n_ctx=4096, the KV modes resolve as follows:
n_seq_max |
kv_unified |
n_ctx() total |
n_ctx_seq() per sequence |
|---|---|---|---|
| 1 | False |
4096 | 4096 |
| 2 | False |
4096 | 2048 |
| 4 | False |
4096 | 1024 |
| 4 | True |
4096 | 4096, shared among active sequences |
Unused slots do not enlarge sequence 0's window with non-unified KV. With unified
KV, four sequences share 4096 token slots; they do not receive 4096 exclusive
slots each. In this checkout, kv_unified=None keeps the native default of
False; LlamaEmbedding forces True. Raising n_seq_max can increase sequence
state and output-buffer costs, and does not by itself guarantee faster inference.
Low-level method to ingest and evaluate a sequence of tokens. Used internally
to update context memory and logits. When the token count exceeds n_ctx_seq(), it
attempts Context Shifting by discarding older tokens while preserving up to
n_keep leading tokens. It raises RuntimeError if the backend memory cannot
be shifted or the incoming chunk cannot fit.
# Evaluates a chunk of tokens and updates internal state
model.eval(tokens=[1, 453, 234, 987], active_loras=[{"name": "coding_adapter", "scale": 1.0}])If the native abort callback interrupts an in-flight llama_decode() call,
eval() fully resets the context and raises the internal
LlamaDecodeAbort exception. Direct low-level callers should treat that
exception as the end of the current request. The full reset is necessary
because llama.cpp may already have committed an unknown number of physical
micro-batches while the Python token ledger still represents the pre-call
state.
Fatal decode exceptions also reset the high-level model before propagating. The reported error position refers to the cursor before cleanup.
Clears the evaluated sequence state owned by the active context. This includes native KV/recurrent memory, the Python token cursor, cached output boundaries, hybrid checkpoints, and speculative-engine state. It does not unload model weights, reset sampling configuration, or clear a separately configured prompt-cache object.
Reset synchronizes target work, clears the speculative engine (which waits for draft work before clearing its memory and resetting its sampler), then clears target memory. This prevents clearing shared KV while a draft still reads it.
save_state() returns a LlamaState containing owned native state bytes, the
committed token prefix, copied score arrays, seed, optional last valid logits,
and model/context compatibility metadata. LlamaState.nbytes counts its owned
payload, excluding Python object and allocator overhead. The snapshot remains
usable after the source context changes or closes, subject to compatibility at
restore time; it can be pickled and holds no live native handle.
The snapshot does not contain sampler history, sampler RNG progression, or the
draft engine's context and hidden/features state. Restoring the seed does not
make this an exact stochastic continuation. With logits_all=False, the score
payload normally contains only the last valid output row. A snapshot without
valid last logits cannot be sampled immediately; evaluate new tokens first.
load_state() validates token counts, buffer sizes, array shapes, and any stored
compatibility metadata before native mutation. Metadata includes the model path,
file size and modification time, context capacity, vocabulary size, and selected
context parameters; it is not a portable model fingerprint. Do not rely on
snapshots as an interchange format across model files or native library builds.
Legacy snapshots without compatibility metadata still receive structural checks.
Restoration clears draft state and the previous sampler, invalidates registered
hybrid checkpoints, and restores the token cursor and owned output. Validation
errors leave the live state untouched; ordinary exceptions during restoration reset
the context because the native read may already have changed it. Both save and
load reject calls during speculative verification.
The direct load_state() cleanup catches Exception, not KeyboardInterrupt.
After loading into a model with a speculative engine, start a new request with
reset=True and the full text prompt. generate(reset=False) is rejected because
target-only snapshots cannot restore the matching draft state. For ordinary text
models, see the snapshot example.
Requests cancellation of active generation. abort() sets both the Python
generation event and the native callback flag:
- The Python event is checked at generation boundaries on every backend.
- The native callback can interrupt an in-flight
llama_decode()graph, but llama.cpp currently documents this path as CPU-only. - A new high-level completion clears the previous abort flags before starting.
For create_completion() and chat handlers that delegate to the standard text
completion path, both streaming and non-streaming responses finish with
"finish_reason": "abort". A native mid-decode abort also performs a full
context reset, because completed micro-batches cannot be inferred reliably.
After a native mid-decode abort, the interrupted request loses its reusable
KV/checkpoint state. Aborted requests are not written to the configured prompt
cache. Text already emitted by a stream remains valid output; after a full
reset, a later request must evaluate its prompt again.
For a native mid-decode abort, generate() ends iteration normally after the
reset. If cancellation is observed only at a Python generation boundary, it can
stop without forcing that full reset. Direct eval() and
LlamaContext.decode() callers instead receive LlamaDecodeAbort; the latter
is an advanced internal API and does not perform the high-level reset itself.
A KeyboardInterrupt raised inside the generate() loop also clears partial
state and ends generation normally. Completion responses report "abort";
verbose=True prints a cancellation diagnostic to stderr. This catch does not
cover arbitrary caller code, model construction, or direct eval() calls.
After a reset leaves no evaluated tokens, generation cleanup does not allocate
an empty hybrid checkpoint.
Call abort() from another thread, such as a timeout timer or UI cancellation
handler. Do not use one Llama instance for concurrent generation requests;
the abort flags and context state belong to the instance.
attach_threadpool(threadpool, threadpool_batch=None) attaches externally
created ggml_threadpool handles to the native context. The first pool is used
for single-token generation. When threadpool_batch is omitted, llama.cpp uses
the first pool for batch/prompt processing too.
detach_threadpool() waits for pending context work and removes both handles.
Attaching also synchronizes first, so a pool cannot be replaced while work is
still pending. The Python wrapper keeps strong references to the supplied
objects while attached, but ownership stays with the caller: it neither creates
nor frees the native pools.
The package currently exposes the opaque ggml_threadpool_p handle and the
attach/detach calls, but not ggml_threadpool_new() or
ggml_threadpool_free(). Obtain and release pools through the native extension
or integration that created them. Detach before freeing an external pool, and
keep the Llama context alive while the pool is attached. Applications that do
not need shared or specially configured pools should rely on llama.cpp's
automatic threadpool creation and the n_threads / n_threads_batch
constructor parameters.
The Llama class exposes lightweight runtime helpers for adjusting native llama.cpp / ggml logging after initialization.
Note: Native backend logging is process-global because llama.cpp / ggml use a global log callback. Changing verbosity or log filters affects all
Llamainstances in the current Python process.
set_verbosity(verbosity: Union[int, str, bool, None]): Set native runtime log verbosity.get_verbosity() -> int: Return the current native runtime log verbosity.set_log_filters(filters: Sequence[str], case_sensitive: bool = True): Replace substring filters for native runtime logs.add_log_filters(filters: Sequence[str]): Append substring filters.get_log_filters() -> List[str]: Return the current substring filters.clear_log_filters(): Clear all substring filters, including default filters.reset_log_filters(): Restore default substring filters.
from llama_cpp import Llama
llm = Llama(
model_path="models/qwen3.gguf",
verbosity=3, # llama.cpp-style info logs
)
# Temporarily enable debug-level native logs.
llm.set_verbosity(5)
# Suppress noisy backend messages by substring.
llm.add_log_filters([
"CUDA Graph",
"CUDA graph",
"clip_model_loader: tensor",
])
# Return to quiet error-only logging.
llm.set_verbosity(1)The Llama class allows you to load multiple LoRAs into VRAM and apply them dynamically per-generation or per-eval.
load_lora(name: str, path: str): Loads an adapter into VRAM (does not apply it yet).unload_lora(name: str): Releases the specific LoRA from VRAM.list_loras() -> List[str]: Returns names of all registered LoRAs.unload_all_loras(): Forces VRAM release for all loaded adapters.
-
Context Shifting & Prompt Caching:
By default,
generate(reset=True)andcreate_completion()check for the longest matching prefix in existing context memory when speculation is disabled. If the live context ends exactly at the matching prefix, only the new suffix is evaluated. If old context extends beyond the match, generation first truncates or restores the appropriate prefix. Fresh speculative requests reset target and engine together instead. To maximize reuse, keep system prompts static and only append new dialogue to avoid re-evaluating the entire history. If the context limit is reached duringeval, the model attempts a Context Shift, discarding older tokens while preserving up ton_keepleading tokens. Backends whose memory cannot shift raise an error instead. -
Structured Output: Use
response_formatfor JSON or JSON Schema constrained output. See Structured JSON Output for the current API. -
Speculative Decoding:
New code should pass
SpecConfigthrough thespeculativeargument. This enables the stateful begin/draft/process/accept lifecycle, including verification batches, acceptance feedback, recurrent-state rollback, and per-run statistics.Current engines use sequence ID
0. Qwen3.5 image chat supports MTP with compatible MTP heads and a matching vision projector.NGRAM_MAP_KandNGRAM_MAP_K4Vcan consume a completed MTMD prompt handoff; negative media IDs remain in history, but proposals stop before the first negative ID. DFlash-family embedding-batch support is a lower-level capability and does not establish general MTMD chat support. See the support boundaries.Built-in MTP heads
from llama_cpp import Llama from llama_cpp.llama_speculative import SpecConfig, SpeculativeType llm = Llama( model_path="path/to/model-with-mtp.gguf", n_ctx=4096, n_batch=512, n_gpu_layers=-1, speculative=SpecConfig( spec_type=SpeculativeType.DRAFT_MTP, draft_n_max=2, draft_p_min=0.0, ), )
Omitting
draft_model_pathmakesLlamaenable target MTP tensor loading automatically. MTP has been tested with built-in and external MTP models from the Qwen3.5, Qwen3.6, and Qwen3.8 families. See Qwen3.5 MTP image chat for the supported multimodal workflow. Agemma4target paired with a compatible externalgemma4-assistantGGUF uses shared target KV memory; matching MTMD projectors also permit media prefill. The engine handles its target-context and shared-KV workflow automatically.For Qwen3.8 27B,
draft_n_max=2is a starting point. Tune it withexamples.benchmark.benchmark_speculativefor your backend, GPU, quantization, prompt, and sampling settings.External MTP model
llm = Llama( model_path="path/to/target.gguf", n_batch=512, n_gpu_layers=-1, speculative=SpecConfig( spec_type=SpeculativeType.DRAFT_MTP, draft_model_path="path/to/mtp.gguf", draft_n_max=2, draft_n_gpu_layers="all", ), )
External DFlash, DFlash2, or DSpark model
llm = Llama( model_path="path/to/target.gguf", n_ctx=8192, n_batch=512, n_gpu_layers="all", speculative=SpecConfig( # Generic mode selects the head reported by the native model loader. spec_type=SpeculativeType.DRAFT_DFLASH, # or DRAFT_DFLASH2 / DRAFT_DSPARK draft_model_path="path/to/dflash-dflash2-or-dspark.gguf", draft_n_max=7, draft_p_min=0.0, draft_n_gpu_layers="all", # Used by DFlash v1/DSpark; remains inactive for DFlash2. draft_backend_sampling=True, ), )
DFlash, DFlash2, and DSpark require a compatible external draft GGUF. Their effective draft length is limited by the GGUF's trained block size. DFlash2 supports explicit
DRAFT_DFLASH2selection. GenericDRAFT_DFLASHusesllama_model_dflash_variant()to select any DFlash-family head, including DSpark and LiliCorr; explicit types reject mismatched sidecars.draft_p_minfilters token probability for DFlash, selector-transition probability for DFlash2, and predicted acceptance confidence for DSpark. DFlash2 reads unmasked selector rows and does not activate backend vocabulary sampling. Benchmark draft length and threshold together; longer blocks only help when their additional accepted tokens outweigh verification cost.N-gram lookup
llm = Llama( model_path="path/to/model.gguf", n_batch=512, speculative=SpecConfig( spec_type=SpeculativeType.NGRAM_MAP_K, ngram_size_n=8, ngram_size_m=16, ), )
MTP and n-gram draft lengths are independent: draft-family engines use
draft_n_max, while K/K4V usengram_size_m. The implementation keeps the complete[last_verified_token, draft...]verification batch together and limits the effective draft length ton_batch - 1. Withctx_checkpoints=0on a Hybrid/Recurrent target, the target's native rollback window also caps this length.After a generation,
llm.last_speculative_statsexposes acceptance, phase timing, checkpoint, rollback, TTFT, and sustained-generation measurements. See Llama Speculative Decoding for configuration details, supported engines, benchmark commands, and the exact meaning of each statistic. -
Dynamic LoRA Routing:
You can load multiple LoRAs using
load_lora()at startup. Then, pass theactive_lorasparameter to.generate(),.create_completion(), or.create_chat_completion()to dynamically apply them to specific queries without reloading the base model.Multi-LoRA Dynamic Switching Example:
Load multiple adapters and apply them selectively without reloading the base model.
llm = Llama(model_path="base_model.gguf") llm.load_lora("coding", "codellama_adapter.gguf") llm.load_lora("story", "storywriter_adapter.gguf") llm.load_lora("sql_expert", "adapters/sql_lora.gguf") # Use coding adapter llm.create_completion("def sort:", active_loras=[{"name": "coding", "scale": 1.0}]) # Use story adapter llm.create_completion("Once upon a time", active_loras=[{"name": "story", "scale": 0.9}]) # Use sql adapter llm.create_completion("SELECT *", active_loras=[{"name": "sql_expert", "scale": 0.8}])
-
Hybrid & Recurrent Architectures:
The class natively detects Hybrid/Recurrent models (for example LFM2VL/LFM2.5VL, Qwen3.5/3.6, Mamba, RWKV, or specialized SWA models such as Gemma3/4) and automatically enables the
HybridCheckpointCache.Unlike regular Transformer KV caches, Hybrid/Recurrent model memory cannot always be safely truncated token-by-token. The wrapper therefore saves periodic sequence-state checkpoints during long context prefill, allowing rollback to a verified prefix without corrupting recurrent state.
HybridCheckpointCachesupports two checkpoint storage modes:- Host checkpoint mode (
checkpoint_on_device=False, default): checkpoint payloads are serialized into Python-owned bytes. Multiple historical checkpoints perseq_idmay coexist, but their attention prefixes must remain valid in the live context; these are partial snapshots, not independent full-context copies. - Device checkpoint mode (
checkpoint_on_device=True): checkpoint tensor payloads are stored inllama_context-owned device buffers viaLLAMA_STATE_SEQ_FLAGS_ON_DEVICE. Python only keeps the host-visible serialized portion. This reduces device-to-host tensor copy overhead, but only one active checkpoint perseq_idis safe because device payloads are keyed byseq_id.
These prompt-cache checkpoints are distinct from the native recurrent snapshots used by MTP speculative decoding. MTP reserves recurrent snapshot slots from
draft_n_max. N-gram engines do not own a native draft context, so rejection on a Hybrid/Recurrent target depends onHybridCheckpointCache; keepctx_checkpointsgreater than zero for that combination, preferably with on-device storage.Set
ctx_checkpoints=0to avoid Python prompt checkpoint exports. Ordinary text requests can still reuse an exact prompt with valid final output or append to the live prefix; branching and shorter prompts require a full prefill. MTP can use this setting when the target has native recurrent rollback slots and the engine can follow target rollback; draft-side fallback may still capture device checkpoints. See caching and state reuse.This text example requires a compatible Qwen3.5 GGUF:
from contextlib import closing from llama_cpp import Llama with closing(Llama( model_path="./Qwen3.5-9B.gguf", n_ctx=4096, ctx_checkpoints=0, )) as llm: response = llm("The capital of France is", max_tokens=16, temperature=0) print(response["choices"][0]["text"])
For long prompts on GPU-backed Hybrid/Recurrent models, you can enable device-backed checkpoints to reduce device-to-host copy overhead:
llm = Llama( model_path="./Qwen3.6-27B.gguf", n_ctx=32768, n_gpu_layers=-1, ctx_checkpoints=16, checkpoint_interval=4096, checkpoint_on_device=True )
Use
checkpoint_on_device=Falseif you need multiple historical checkpoints for the sameseq_id. Usecheckpoint_on_device=Truewhen fast rollback/checkpointing is more important than keeping many historical checkpoint payloads. Do not disable checkpoints when combining n-gram speculation with a Hybrid/Recurrent target. - Host checkpoint mode (
-
Assistant Prefill:
llama-cpp-pythonsupports Assistant Prefill for message continuation through theassistant_prefill=Trueparameter ofcreate_chat_completion().The default chat-format path copies the input message list, removes the last assistant message before rendering the preceding history, and appends that message's text to the formatted prompt. If the list is empty, the final role is not
assistant, or its content is empty, prefill is ignored (with a warning when verbose logging is enabled). Custom chat handlers may implement different behavior.from llama_cpp import Llama llm = Llama(model_path="path/to/model.gguf") # An interrupted/partial conversation messages = [ {"role": "user", "content": "What are the first 5 planets in the solar system?"}, {"role": "assistant", "content": "The first 5 planets in our solar system are:\n1. Mercury\n2."} ] # Seamlessly continue the generation response = llm.create_chat_completion( messages=messages, max_tokens=50, assistant_prefill=True # <--- Enables seamless continuation ) prefilled_text = messages[-1]["content"] # The generated text continues after the supplied "2." prefix. generated_text = response["choices"][0]["message"]["content"] print(prefilled_text + generated_text)
-
Interrupting Reasoning & Assistant Prefill (Time-boxing):
Use
abort()alongsideassistant_prefill=Trueto request cancellation when a reasoning model (such as Qwen or DeepSeek) exceeds a time limit, then inject bridge text and request continuation as a final answer.A native mid-decode abort resets the context, so the continuation request below re-evaluates the reconstructed message history rather than reusing the interrupted request's KV state. The partial text collected by the caller is preserved in the assistant-prefill message.
import threading from llama_cpp import Llama llm = Llama(model_path="Qwen3.6-27B.gguf", n_ctx=4096, n_gpu_layers=-1) def run_controlled_generation(prompt: str, timeout_seconds: int = 10): messages = [{"role": "user", "content": prompt}] # 1. Set a time bomb to interrupt long <think> phases def timeout_handler(): llm.abort() timer = threading.Timer(timeout_seconds, timeout_handler) timer.start() stream = llm.create_chat_completion( messages=messages, max_tokens=2048, stream=True ) partial_response = "" finish_reason = None for chunk in stream: finish_reason = chunk["choices"][0].get("finish_reason") if finish_reason is not None and finish_reason != "abort": timer.cancel() break if finish_reason == "abort": break delta = chunk["choices"][0]["delta"].get("content", "") if delta: partial_response += delta print(delta, end="", flush=True) # 2. Forced Intervention and Prefill Continuation if finish_reason == "abort": # Inject bridge text to forcefully close the reasoning tag bridge_text = "\n...Wait, I have thought long enough, let's start answering the user.\n</think>\n\n" print(bridge_text, end="", flush=True) prefilled_content = partial_response + bridge_text messages.append({"role": "assistant", "content": prefilled_content}) # Use assistant_prefill=True to seamlessly continue the text block stream_part2 = llm.create_chat_completion( messages=messages, max_tokens=2048, stream=True, assistant_prefill=True ) for chunk in stream_part2: delta = chunk["choices"][0]["delta"].get("content", "") if delta: print(delta, end="", flush=True) run_controlled_generation("Explain quantum mechanics in a way that relates to bugs in code.", timeout_seconds=8)
-
Runtime Logging & Backend Noise Filtering:
Llamasupports fine-grained native llama.cpp / ggml logging throughverbosity. This is more precise than the legacyverboseboolean flag.from llama_cpp import Llama # Legacy behavior: # verbose=False -> error-only logs llm_quiet = Llama( model_path="models/qwen3.gguf", verbose=False, ) # Recommended precise logging: # 0 = output, 1 = error, 2 = warning, 3 = info, 4 = trace, 5 = debug llm = Llama( model_path="models/qwen3.gguf", verbosity=3, # llama.cpp-style default info logs )
For low-level debugging, use
verbosity=5. By default, the logger may suppress known noisy backend messages such as CUDA Graph reuse logs. Passlog_filters=[]to disable all substring filtering.llm = Llama( model_path="models/qwen3.gguf", verbosity=5, log_filters=[], # show all debug logs, including normally filtered ones )
To suppress additional noisy messages, pass substring filters:
llm = Llama( model_path="models/qwen3.gguf", verbosity=5, log_filters=[ "CUDA Graph id", "clip_model_loader: tensor", "ggml_cuda_graph_update_required", ], )
You can also adjust logging at runtime:
llm.set_verbosity(5) llm.add_log_filters(["llama_perf_context_print"]) # Later, return to warning-level logs. llm.set_verbosity(2)
Important: native backend logging is process-global. Runtime changes affect all
Llamainstances in the same Python process.verbose=False vs. verbosity=0: These have distinct behaviors.
verbose=Falsesuppresses wrapper prints and sets native logging to ERROR, unlessverbosityoverrides it.verbosity=0hides severity-based native messages, including errors. Output-level messages remain visible unlessshow_output=Falseis set throughconfigure_logging().
The Llama embedding methods are maintained and use streaming batches. Create
the model with embeddings=True before calling them.
from llama_cpp import Llama, LLAMA_POOLING_TYPE_UNSPECIFIED
llm = Llama(
model_path="path/to/embedding-model.gguf",
embeddings=True,
pooling_type=LLAMA_POOLING_TYPE_UNSPECIFIED,
n_batch=512,
n_ubatch=512,
n_seq_max=8,
kv_unified=True,
)
try:
# Raw sequence embeddings with explicit L2 normalization.
vectors = llm.embed(["query", "document"], normalize=2)
# OpenAI-compatible response.
response = llm.create_embedding(
["query", "document"],
normalize=True,
)
finally:
llm.close()Generate raw embedding values for strings or pre-tokenized inputs.
| Parameter | Type | Default | Description |
|---|---|---|---|
input |
Union[str, List[str], List[List[int]]] |
Required | A single string, a list of strings, or a list containing pre-tokenized token-ID lists. |
normalize |
Union[bool, int] |
False |
False returns raw values, while True applies L2 normalization. Integer modes are listed below. Rank outputs are not normalized. |
truncate |
bool |
True |
Truncate each input to min(n_ctx_seq(), n_ctx(), n_batch), also bounded by n_ubatch for non-causal attention. If disabled, exceeding any applicable limit raises ValueError before decoding that input. |
separator |
Optional[str] |
None |
Split a single string into multiple independent inputs. When set, the result uses the batch return shape. |
return_count |
bool |
False |
Return (result, total_token_count) instead of only the embedding result. |
Normalization modes follow the llama.cpp embedding example:
| Value | Behavior |
|---|---|
False or -1 |
No normalization |
True or 2 |
Euclidean/L2 normalization |
0 |
Scale by the maximum absolute value to a maximum magnitude of 32760 |
1 |
Taxicab/L1 normalization |
Integer greater than 2 |
Corresponding p-norm normalization |
Unlike LlamaEmbedding.embed(), the standard Llama.embed() method defaults to
raw, unnormalized output for backward compatibility.
The return shape depends on the input and pooling type:
| Input / pooling mode | Return shape |
|---|---|
| Single string with sequence pooling | List[float] |
| String list or separator-split string with sequence pooling | List[List[float]] |
LLAMA_POOLING_TYPE_NONE |
One token embedding matrix per input: List[List[float]] for a single string or List[List[List[float]]] for a batch |
LLAMA_POOLING_TYPE_RANK with one classifier output |
A scalar for a single string or a list of scalars for a batch |
LLAMA_POOLING_TYPE_RANK with multiple classifier outputs |
A classifier vector for each input |
Any mode with return_count=True |
(result, total_token_count) |
Use LLAMA_POOLING_TYPE_UNSPECIFIED for ordinary sentence embeddings unless
the model documentation requires a specific sequence pooling strategy.
LLAMA_POOLING_TYPE_NONE is token-level output and should not be used when one
vector per input document is expected.
Embedding requests reset existing generation, output, checkpoint, and draft state before decoding. Cleanup runs on success and on exceptions after execution begins. Outputs are copied before native memory is cleared; callers may retain the returned vectors. Do not interleave embedding requests with an active stream on the same instance.
Wrap sequence or token-level embedding output in an OpenAI-compatible response:
{
"object": "list",
"data": [
{
"object": "embedding",
"embedding": [...],
"index": 0,
}
],
"model": "path/to/embedding-model.gguf",
"usage": {
"prompt_tokens": 12,
"total_tokens": 12,
},
}| Parameter | Type | Default | Description |
|---|---|---|---|
input |
Union[str, List[str]] |
Required | One string or a list of strings. |
model |
Optional[str] |
None |
Model name placed in the response. Defaults to model_path. |
normalize |
Union[bool, int] |
False |
Passed directly to embed(). |
truncate |
bool |
True |
Passed directly to embed(). |
For parallel batches, n_seq_max must cover every sequence ID active in a
single decode batch. The default n_seq_max=1 is valid and processes multiple
inputs sequentially. Increasing it allows more inputs to be decoded in
parallel; for example, n_seq_max=8 permits IDs 0 through 7 in one batch.
n_batch limits logical input tokens, n_ubatch controls the physical token
batch, and n_seq_max limits independent sequences.
The batcher also limits combined input tokens to n_ctx() and each individual
input to n_ctx_seq(); unified KV shares total capacity across inputs.
LlamaEmbedding remains available as the specialized convenience class. It
automatically enables embedding-oriented context options, defaults to L2
normalization, provides additional output formats, and adds the rank() helper
for formatting query/document pairs. Its
multimodal embedding API accepts text/image/audio
inputs with a matching MTMD projector and MEAN, CLS, or LAST pooling. Causal
models are supported when their media does not require chunk-local
bidirectional attention.
OpenAI compatibility: use sequence pooling when calling
create_embedding()through an OpenAI-compatible client. Token-level pooling (LLAMA_POOLING_TYPE_NONE) produces nested token vectors rather than the single flat vector normally expected for each input.
- Wiki index
- Llama Cache - RAM, disk, trie, and Hybrid checkpoint APIs.
- Llama Embedding - Dedicated class for text embeddings and reranking.
- Llama Speculative Decoding - Configuring stateful MTP, DFlash, DFlash2, DSpark, and n-gram speculative engines, rollback, statistics, and benchmarks.
- llama.cpp ctypes Bindings - Source pointers for the low-level llama.cpp and ggml bindings.
- DFlash2 Speculative Decoding - Runnable Qwen3.8 DFlash2 configuration, validation, benchmark, and tuning workflow.
- Qwen3.5 Image Chat - Model/projector setup, chat handler configuration, and media recovery.