中文 | English
ComfyUI custom node that acts as a llama-cpp-python wrapper, specifically optimized for Intel GPU (Arc / Iris Xe) with SYCL backend acceleration. Supports vision models and allows generating text responses from prompts using llama.cpp.
- Load and use GGUF models (including vision models)
- Generate text prompts using llama.cpp
- Support for multi-modal inputs (multiple images/batches)
- Advanced sampling (Min-P, Presence/Frequency penalties)
- JSON Schema support for structured output (json_object)
- Memory management options
- Integration with ComfyUI workflows
- llama-cpp-python 0.3.39+ MTMD rewrite support (GenericMTMDChatHandler)
- Hybrid model support (Qwen3.5 etc. via
ctx_checkpoints)
-
Install the SYCL-accelerated llama-cpp-python wheel.
Important: JamePeng's releases do not include SYCL builds. For Intel GPU SYCL acceleration, download the whl from:
https://github.com/allanmeng/llama-cpp-python-sycl-windowsIf you don't need SYCL (e.g., using CUDA/CPU), you can use the generic builds from https://github.com/JamePeng/llama-cpp-python/releases
-
Clone this repository into your ComfyUI custom nodes directory:
cd ComfyUI/custom_nodes git clone https://github.com/allanmeng/comfyui-sg-llama-cpp -
Restart ComfyUI.
If you are upgrading llama-cpp-python from version 0.3.38 or earlier to 0.3.39+, do NOT use pip install --upgrade — old files may linger and cause conflicts. Follow this procedure instead:
# 1. Completely remove the old version
pip uninstall llama-cpp-python -y
# 2. Install the new wheel
pip install llama_cpp_python-0.3.41-<your-build>.whlWhy uninstall first? A simple
--upgrademay leave stale DLLs from the old version inlib/, which can conflict with the new version's files. Uninstalling first ensures a clean state.
This fork adapts the plugin for llama-cpp-python 0.3.39+ (MTMD rewrite) and ships additional fixes/tuning on top:
use_mmap/use_mlock/use_direct_iowere dead switches on 0.4.x — they now show as[DEPRECATED in 0.4.x], still work on 0.3.x, and are skipped when the build supportsload_mode.- New 0.4.x knobs exposed:
load_mode,lazy_mode,flash_attn_type,vision_flash_attn(all defaultauto). - KV cache quantization:
type_k/type_v(--cache-type-k/-v) can cut the KV cache allocation by ~45% (q8_0) or ~70% (q4_0). A quantized V cache requires Flash Attention — that invalid combination is rejected with a clear message. Measured: a net loss on small hybrid models (q8_0= −7.4% decode speed for ~114 MiB saved on Qwen3.5-4B), so the default staysauto; it pays off only for large dense VLMs or very long contexts. - Parameter whitelist against the installed
Llama()signature — one code path for 0.3.x and 0.4.x, unknown params are dropped instead of leaking into**kwargs. - Vision token bounds no longer abort the run — an inconsistent
vision_image_max_tokensis dropped with a warning instead of raising. - New Options inputs are appended, never inserted mid-list — ComfyUI stores widget values positionally, so mid-list insertions corrupted existing workflows.
- Benchmarked on B580/0.4.3: the four new knobs gave no measurable speed-up (
clip_encodespread ≤1.4%); defaults stayauto.
checkpoint_interval/checkpoint_on_deviceoptions onLlamaCPPOptionsfor hybrid models (defaults4096/Disabled). Verified via 2×2 benchmark (host/device × 8/16 checkpoints): no impact on load, encode or token throughput — defaults are optimal for most users.think_displaytoggle onLlamaCPPEngine—show/hideto keep or strip the<think>…</think>reasoning block. Handles malformed tags (missing opening, multiple closing, dangling open).thinkstripping robustness — keeps only text after the last</think>(engines may emit</think>…reasoning…<//think>answerwith no opening tag).ctx_checkpointsdefault changed0→-1—0triggers a buggy fast-path in llama-cpp-python 0.3.48 that crashes on large images;-1uses the llama_cpp default (16) and avoids the crash.- Large-image
n_ctxexhaustion → reactive English hint (no forced override): if the first decode fails with "memory slot for batch", the engine tells you to raisen_ctx(8192 known-good for large images).
clip_model_pathdeprecated → replaced bymmproj_pathpassed directly toLlama()- Manual handler creation removed →
Llama()now internally creates the vision handler viachat_handler_kwargs GenericMTMDChatHandlerreplaces model-specific handlers as the primary vision handler- Parameter filtering inspects
GenericMTMDChatHandlerunionMTMDChatHandlerto filter valid kwargs - Removed invalid UI params:
vision_enable_thinking,vision_force_reasoning,vision_add_vision_id(not accepted byGenericMTMDChatHandler) vision_image_min_tokensdefault changed from-1to1024(Qwen-VL minimum requirement)
Loads GGUF model files and prepares them for use.
Inputs
- Required:
model_name: Select the GGUF model file to load.
- Optional:
chat_format: Chat template to use (default:llama-2). In 0.3.39+, vision models auto-detect viaGenericMTMDChatHandler.mmproj_model_name: Multi-modal projector model for vision (default:None).
Outputs
MODEL: The loaded Llama model object.
Configures advanced parameters for the model.
Inputs
- Optional:
n_gpu_layers: Number of layers to offload to GPU (default:-1for all).n_ctx: Context window size (default:8192; a large-image prefill needs 4000+ tokens of headroom).n_threads: CPU threads to use (default:-1for auto).n_threads_batch: Threads for batch processing (default:-1for auto).n_batch: Batch size (default:2048).n_ubatch: Micro-batch size (default:512).main_gpu: Main GPU ID (default:0).offload_kqv: Offload K/Q/V to GPU (default:Enabled).numa: NUMA support (default:Disabled).use_mmap: Memory mapping (default:Enabled). DEPRECATED in llama-cpp-python 0.4.x — ignored there, useload_mode; still honored by 0.3.x.use_mlock: Memory locking (default:Disabled). DEPRECATED in 0.4.x — useload_mode=mlock.use_direct_io: Enable direct I/O for library (Linux only, default:Disabled). DEPRECATED in 0.4.x — useload_mode=direct_io.verbose: Verbose logging (default:Disabled). When enabled, the resolved 0.4.x tuning parameters are echoed to the console.ctx_checkpoints: Context checkpoints (default:-1for default;0disables — do not use0with llama-cpp-python 0.3.48, it triggers a buggy fast-path that crashes on large images; required for hybrid models like Qwen3.5).checkpoint_interval: Hybrid checkpoint token interval (default:4096).checkpoint_on_device: Store hybrid checkpoint payloads in VRAM (default:Disabled; ~50 MiB per checkpoint, no throughput impact — benchmarks show defaults are optimal).vision_use_gpu: Enable GPU for vision handler (default:Enabled).vision_image_min_tokens: Minimum image tokens (default:1024, recommended for Qwen-VL;-1for default).vision_image_max_tokens: Maximum image tokens (default:-1for default). Must be-1or>= vision_image_min_tokens.load_mode(0.4.x): Model load strategy —auto/none/mmap/mlock/mmap_mlock/direct_io(default:auto).lazy_mode(0.4.x): Lazy tensor loading, requires mmap —auto/off/on(default:auto).flash_attn_type(0.4.x): Flash Attention for the main model —auto/disabled/enabled(default:auto).vision_flash_attn(0.4.x): Flash Attention inside the vision encoder —auto/enabled/disabled(default:auto= handler default).type_k(0.4.x): KV cache data type for K (--cache-type-k) —auto/f16/q8_0/q5_1/q5_0/q4_1/q4_0/iq4_nl/bf16(default:auto= f16). Quantizing shrinks the KV cache allocation (reported byllama_kv_cache: size = ..., not by the compute-buffer line). Measured on B580/Qwen3.5-4B:q8_0costs −7.4% decode speed for ~114 MiB saved, so it is only worth it for large dense VLMs or very long contexts.type_v(0.4.x): KV cache data type for V (--cache-type-v), same options. A quantized V cache requires Flash Attention — it is rejected whenflash_attn_type='disabled'.
The four
0.4.xknobs are appended at the end of the input list on purpose: ComfyUI maps saved workflow widget values by position. On a 0.3.x build they are ignored automatically (detected from the installedLlama()signature).
Outputs
OPTIONS: A configuration dictionary.
The main generation node.
Inputs
- Required:
model: The model fromLlamaCPPModelLoader.prompt: The text prompt.
- Optional:
images: Input image(s) for vision models (supports batches).options: Options fromLlamaCPPOptions.system_prompt: System instruction (default: empty).memory_cleanup: Strategy to clean memory after generation (default:full_cleanup).response_format:textorjson_object(default:text).json_schema: JSON Schema to enforce (available only whenjson_objectis selected).max_tokens: Max new tokens (default:2048).temperature: Randomness (default:0.1).top_p: Nucleus sampling (default:0.9).top_k: Top-k sampling (default:20).min_p: Min-p sampling (default:0.05).repeat_penalty: Penalty for repetition (default:1.1).present_penalty: Penalty for presence of tokens (default:0.0).frequency_penalty: Penalty for frequency of tokens (default:0.0).seed: Random seed (default:1).think_display:showorhide(default:hide).hidestrips the<think>…</think>reasoning block so only the final answer is returned. Handles malformed tags (missing opening, multiple closing, dangling open).
Outputs
RESPONSE: The generated text.
Utility to manually free resources.
Inputs
- Required:
memory_cleanup: Cleanup mode (close,backend_free,full_cleanup,persistent).
- Optional:
passthrough: Any input to pass through (allows chaining).
Outputs
PASSTHROUGH: The input passed through unmodified.
The plugin scans the following directories for .gguf files recursively (including subdirectories):
| Source | Path(s) | Auto-detected |
|---|---|---|
ComfyUI text_encoders |
ComfyUI/models/text_encoders/ + ComfyUI/models/clip/ |
Yes |
| Dedicated LLM folder | ComfyUI/models/LLM/ |
Yes |
| Custom folders | Any path in config.json |
No |
All scanned files are automatically separated into two dropdowns by filename:
- Model dropdown:
.gguffiles (excluding those withmmprojin the name) - mmproj dropdown:
.gguffiles withmmprojin the filename
ComfyUI/models/LLM/
├── Qwen3.5-4B-Q4_K_M.gguf ← LLM model
├── Qwen2.5-VL-3B-instruct-q4_k_m.gguf
├── mmproj/ ← Vision projectors (or any name)
│ └── Qwen2.5-VL-3B-Instruct-mmproj-f16.gguf
└── GGUF/ ← Can use subdirectories freely
└── ...
mmproj files only need the keyword "mmproj" in their filename to appear in the mmproj dropdown — their folder location doesn't matter.
Create config.json in the plugin directory for additional search paths:
{
"model_folders": [
"D:\\AI\\LLM\\models",
"/home/user/models"
]
}- The
config.jsonfile is optional — the node works without it - Paths can be absolute or relative
- Both Windows and Unix style paths are supported
- Non-existent paths are automatically filtered out
- See
config.example.jsonfor additional examples
- llama-cpp-python >= 0.3.39
- Intel GPU (SYCL): https://github.com/allanmeng/llama-cpp-python-sycl-windows
- Other backends (CUDA/CPU): https://github.com/JamePeng/llama-cpp-python/releases
This project is licensed under the GNU AGPLv3 License - see the LICENSE file for details.
- Fork (SYCL optimized): https://github.com/allanmeng/comfyui-sg-llama-cpp
- SYCL whl builds: https://github.com/allanmeng/llama-cpp-python-sycl-windows
- Upstream: https://github.com/sebagallo/comfyui-sg-llama-cpp
