Explore llama.cpp/ggml quantization kernels for our GGUF integration #14841

Activity

1. sayakpaul

   added

   performanceAnything related to performance improvements, profiling and benchmarkingAnything related to performance improvements, profiling and benchmarking

   on Sep 22, 2026
2. iwr-redmond commented on Sep 23, 2026

   

   iwr-redmond

   on Sep 23, 2026

   Last edited by iwr-redmond

   More actions

   While Llama.cpp proper cannot run models with media outputs, there is a longstanding downstream project that supports inference with a variety of diffusion models here. Llama.cpp contributor @ServeurpersoCom also maintains downstream GGML projects for Ace-Step and MiniMax Music.
3. LaelaZorana commented on Oct 8, 2026

   

   LaelaZorana

   on Oct 8, 2026

   More actions

   I would like to take this one, and I think we can proceed in 3 steps.

   ggml-quantization covers the linear layers of the transformer blocks, which should account for most of the GGUF weights, and RMSNorm is already possible via ggml-norm as long as the dtypes match up. ggml-attn depends on how we plan to hook into the attn dispatcher.

   Load ggml-org/ggml-quantization from GGUFQuantizer via get_kernel and use the Metal build on MPS. CUDA can remain on Isotr0py/ggml's MMQ implementation. As a fallback we can continue to use dequantize when there is no kernel for a given quantization type.

   I would like to run the benchmarks twice and post the results here, once with these changes and once with the current dequantize path. I will try Flux and SD3.5 models with Q4_K and Q8_0 GGUF quants, then we can look at similarity, step latency and peak memory on CUDA and MPS.

   Can a maintainer please confirm the scope above looks right before I make a PR?
4. Jackmeson1 commented on Oct 20, 2026

   

   Jackmeson1

   on Oct 20, 2026

   More actions

   There is an import failure relevant to this integration and #12509. This CPU reproduction simulates CUDA eligibility while calling the real kernels loader:

   import os
   
   os.environ["DIFFUSERS_GGUF_CUDA_KERNELS"] = "true"
   os.environ["DIFFUSERS_TRUST_REMOTE_KERNELS"] = "true"
   
   import torch
   
   torch.cuda.is_available = lambda: True
   torch.cuda.get_device_capability = lambda *args, **kwargs: (12, 0)
   
   from diffusers import UNet2DModel

   Observed: the import fails with A kernel version or revision must be specified. Expected: an unavailable optional GGUF accelerator should not prevent importing an unrelated model.

   Environment: Python 3.12.15, diffusers e04f0c5f / 0.42.0.dev0, torch 2.14.1+cpu, kernels 0.17.2, gguf 0.19.0.

   GGUF utilities call get_kernel("Isotr0py/ggml") during model import without a version or revision. A warning plus native-dequantization fallback restores the tested imports while preserving the explicit trust requirement; it does not restore accelerated kernel compatibility. I can send a focused import/fallback fix after maintainer acknowledgment.

   Found and reproduced with AI assistance; the repro above was run by me.