Repository navigation
ggml-hrx: update upstream and enable Flint through 32K context - #158
Draft
AaronStGeorge wants to merge 1700 commits into
Draft
AaronStGeorge wants to merge 1700 commits into
AaronStGeorge wants to merge 1700 commits into
Conversation
…gml-org#30621) * convert : fix Granite Switch block_count for MultiSwitch checkpoints Signed-off-by: Nigel Jones <jonesn@uk.ibm.com> * Update granite.py Co-authored-by: Gabe Goodhart <ghart@us.ibm.com> --------- Signed-off-by: Nigel Jones <jonesn@uk.ibm.com> Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
* mtmd: add mtmd_audio_stream + support voxtral-realtime * address review
* enable OPENSSL=ON as default in build script, update it in CI and dockerfile * change to LLAMA_BUILD_BORINGSSL on windows * refactor * rm Download UI build * Apply suggestion from @CISC Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Apply suggestion from @CISC Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Apply suggestion from @CISC Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Preserve the existing MoE accumulation order. Assisted-by: Codex Assisted-by: Angel
The scheduler uploads each view of a host input on its own, so the per-layer reshape of new_pool_idxs (and of sel_mask in glm5-next) was copied once per layer.
…ized and fp weights (ggml-org#30635) * hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round Add support for GET_ROWS with tiled Q5_K Add support for raw k-quant tensors * hex-get-rows: vectorize raw k-quant ukernels * hex-binary: refactor dma-loop thread functions (no need for macros, etc) * hex-quant: move/cleanup flat and tiled dequant ukernels * hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights * hex-get-rows: remove redundant cast * hex-repack: proper filtering for BF16 that were leaking through during the tests * hex-repack: streamling tiling rules (broken MM for now) * hex-repack: fix mmid after tiling updates * hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars * hex-mm: fix valid_rows calc in various kernels * hex-mm: remove unused hvx_mm_nx_2d_flat * hex-mm: cleanup and optimize hvx-tiled vec_dot_f16 * hex-mm: another pass at hvx-tiled f16 matmul * hex-get-rows: optimize small src1 * hex-get-rows: one more pass at recovering get-rows perf regression * hex-repack: export GGML_HEXAGON_RT_REPACK via run.py * hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul * hex-mm: fold tiled_f16 checks into the general flow * hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op * hex-quant: proper tail handling in new dequant ukernels * hex-get-rows: remove unused code and cleanup support checks * hex-mm: cleanup ne1 padding/tiling checks * hex-mm: cleanup fusion checks for tiled tensors * hex-repack: add a little can_repack helper to reduce duplication * hex-repack: never repack host buffers * hex-repack: factor out repack functions into separate .h and .cpp * hex-tensor: cleanup tensor ifaces for tiled/padded sizes * hex-repack: update k-quant tiling comments and cleanup #defines * hex-dcache: factor out l2/dcache dirty tracking and flushing * hex-tensor: update docs and remove #defs duplication * hex-tensor: remove unused/incorrect get_data interface * hex-get-rows: cleanup vtcm layout build * tests: disable huge MM tests * hex-dcache: add missing header
Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
* vulkan: fix log spam if host pointer import fails * exclude MoltenVK from shared memory import * remove unreachable code, assert instead. Shorten warnings * disable host memory buffer sharing on AMD due to amdgpu perf issue
* tests : update the Metal fusion baseline Re-record the MTL fusion baseline with the current build: 276 rows over 131 dummy models. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : fail test-fusion when a baseline row stops matching The report was built from the counters the run produced, and the baseline was only ever used as a lookup table, so a fusion that stopped matching produced no row and was never compared. qwen4exp and mellum lost their fusions this way while the Fusion / metal job stayed green. Report a baseline row that the run does not reproduce as a regression. Scope the pass to the arches that were actually run so --model keeps working. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : never write inline comments unless the user asks Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : keep the mellum dummy model softmax-gated mellum reads expert_gating_func from the GGUF since ggml-org#30578, while the fixture writes sigmoid for every MoE arch, so the Mellum dummy model stopped building the softmax MoE routing fusion. Write softmax for mellum, matching the arch default and the GGUFs that conversion/mellum.py produces. Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : use emoji in the AI stats disclosure Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : list the disclosure model after the hardware Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : use monochrome symbols in the AI stats disclosure Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : add muse-spark to the Metal fusion baseline Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Assisted-by: Codex
* pplx-decider-text-only convert : support text-only checkpoints in pplx-decider * Apply suggestions from code review * convert : remove unused import
…e x in registers (ggml-org#29720) * CUDA: HIP: rms_norm: load float4 at a time and store x in registers * Add template paramter comments * use ggml_cuda_is_aligned for src * nits * also use ggml_cuda_is_aligned for src in fused variants * fix nits
…0630) * gguf-py : add numpy Q1_0 and Q2_0 de/quantization support * cont: added q1_0 / q2_0 to the test list removing the small test Assisted-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
…ort (ggml-org#30647) * convert: speed up model endian conversion Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * convert: explicitly state noop Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * convert: add missing data type support Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* model : MTP support for Mellum * model : inline rope construction in Mellum
* hexagon: correct HMX FA sinks * hex-fa: vectorize sinks in hmx kernels --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* llama-eval : add sandboxed execution for code suites
Adds a lightweight replacement for the Docker containers usually used to run
generated code, so a benchmark can be executed without a daemon, an image or
root.
A cached venv is provisioned per suite and reused; candidate programs run in a
throwaway working directory as their own process group, under the strongest
isolation the host offers:
bwrap read-only root, tmpfs /tmp and $HOME, no network
unshare no network, via --map-current-user
rlimit cpu/memory/file-size/process caps only
Candidate code must not run as uid 0, which is why "unshare -r" is a
last-resort tier that warns: root bypasses file permission bits, so a test
asserting that a write to a chmod 0444 file fails would instead see it succeed.
Two details worth noting for the bwrap tier. Only paths hidden by the $HOME
tmpfs are bound back, because bwrap cannot mount onto a symlink and
version-manager alias directories usually are one. And the venv's base
interpreter tree is resolved by walking the symlink chain, since a
version-managed interpreter is commonly reached through an unversioned alias
that startup resolves through.
Suites can declare provisioning steps, which run once at build time while the
network is still up, so test time never needs it.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : add HumanEval and ClassEval code suites
Adds two execution-graded suites alongside the existing text-graded ones. The
model's reply is parsed for a python code block, combined with the task's own
unit tests, and run in the sandbox.
--dataset humaneval 164 function-level problems, scored pass@k
--dataset classeval 100 class-level tasks, scored at class level (feeding
pass@k) and method level, matching upstream
pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the
dataset size gives that many samples per problem, aggregated with the unbiased
estimator from the Codex paper.
Program construction and result interpretation are dataset-owned hooks, because
the two suites differ: one HumanEval problem is a function judged by a single
assert, whereas one ClassEval task is a class judged by many independent test
methods, and reducing that to pass/fail would throw away partial credit. Suites
also declare their own package set, provisioning steps, interpreter and timeout.
The package set is derived from the dataset's own imports rather than a
requirements file. HumanEval turns out to need no third-party packages at all;
ClassEval needs thirteen, two of which must be declared because no import
mentions them -- lxml is selected by string in ClassEval_44, and scipy is a
gensim requirement.
Notes on ClassEval specifically:
* gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two
pure-Python APIs it uses are provided by a small shim instead.
* The three nltk corpora it needs are pre-seeded at venv build time. These are
the modern names -- the dataset asks for the since-renamed punkt and
averaged_perceptron_tagger.
* It defaults to a patched fork, ilintar/ClassEval, because three upstream
tests cannot be passed by anything: a hard-coded 2024 date checked against
datetime.now(), an assertion on the dataset author's own hostname, and an
exact float comparison. Reference solutions go from 96/100 to 100/100 classes
and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the
model is asked to write is unchanged; --dataset-source selects the original.
--exec-seed (default 0) seeds the RNG before each test, so suites whose tests
draw random inputs score reproducibly. ClassEval_58 fails its own reference
solution about 30% of the time without it.
The simulator gains both suites, so the harness can be exercised without a
model.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : fix silent isolation downgrade, generalise the sandbox runner
The isolation probe ran the interpreter without binding it back, so whenever
this tool is itself started from a venv under $HOME -- the usual case, since it
needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the
sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in
the output looked wrong when that happened, and the existing suites have been
running that way. The probe now uses the same interpreter and the same binds the
suite will actually get.
Two additions the agentic suite needs, neither of which changes the existing
suites:
* run_argv() runs an arbitrary command in a caller-owned directory that survives
the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take
exactly the same path as before. A multi-step agent needs its edits to persist
across many invocations, and a suite for a language other than Python needs to
drive a linter and a test runner rather than a single source string.
* RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved
address space rather than resident memory, which is the wrong instrument for a
runtime that reserves a large region and commits little of it: node's
TypeScript type-stripping goes through WebAssembly, reserves gigabytes per
instance, and aborts at 4 GiB with an out-of-memory error that reads like a
fault in the code under test.
HumanEval stays at 164/164 and ClassEval at 100/100.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : add SACB, a multi-turn agentic coding suite
`--dataset agentic` is a different shape from the other code suites. Rather than
answering a single prompt, the model is given a small but real repository, a
narrow tool set and a bug report, drives its own multi-turn conversation, and is
graded by running a test suite it was never allowed to see.
The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three
hand-written repositories in Python and TypeScript, hosted at ilintar/SACB.
Grading needs no container: a cached venv gives ruff and mypy for Python, and
node's own test runner plus tsc for TypeScript, so the whole dependency set is
four packages.
Three constraints are deliberate:
* No shell, and no way to run the tests. The only feedback channel is `lint`. A
syntax or type error can be driven out mechanically; a logic error has to be
reasoned about. A harness that lets an agent iterate against the test suite
measures something closer to search than to understanding.
* Two edit tools, one line-addressed and one content-addressed. Which one a
model reaches for, and whether it keeps line numbers straight after its own
earlier edit, is itself a signal.
* The tests never touch the working tree. They are held outside it and copied
into a throwaway copy only once the agent has stopped, so they can be neither
read nor edited. Every tool path is resolved and confinement-checked against
the repository root, so a symlink planted inside the tree cannot reach out of
it.
A task counts as resolved when every test that was failing passes and nothing
that already passed broke. Both sets are derived by running the tests against
the defective tree and against the reference fix, never written by hand, so a
task whose defect no test exercises cannot enter the corpus. Many tasks carry
several independent defects, so the fraction of target tests passed is reported
alongside, separating "fixed two of the three faults" from "changed nothing".
An episode that arrives with no tool call and no `finish` is nudged rather than
ended. Models routinely narrate their analysis mid-task, and treating that as
completion ends the episode with no edits and scores a reasoning failure that
never happened; about one episode in four exits that way.
Nothing here is installed unless this suite is selected, and then only for the
language being run: agentic_eval is imported lazily and each toolchain is keyed
per language.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : make the agentic hooks typed rather than discovered
`ty` flagged nine unresolved attributes, of which the seven in main() predate
the agentic suite: `EvalState.dataset` is Optional and was dereferenced without
narrowing.
* `loaded_dataset` turns the Optional into a value in one place, raising rather
than handing back None, so callers do not each have to narrow it.
* `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were
reached through getattr with a default at three call sites, which no checker
can see through; `run()` now raises a clear error naming the suite instead of
failing as a missing attribute.
* The sandbox setup is one branch on `is_agentic` rather than two tests of
`grader_type`, which also drops a duplicated condition.
No behaviour change: HumanEval stays at 164/164, and `ty check` on
examples/llama-eval is clean.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* feat: API key
* fix reasoning and response length for agentic runs, add option to save entire transcripts
* llama-eval : fix ty unresolved-attribute on loaded_dataset.ae
Assisted-by: AI coding agent
* llama-eval : trim comments and remove non-ASCII from README
Assisted-by: AI coding agent
Use the instance's is_mistral_format flag when DeepseekV2Model reloads configuration, allowing Mistral MoE checkpoints to read params.json. Fixes ggml-org#29899 Signed-off-by: aiqubits <aiqubits@hotmail.com>
…g#21752) * common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client common_http_client() was ignoring standard proxy env vars, causing downloads via -hf and similar flags to fail silently in environments behind an HTTP proxy (corporate networks, air-gapped infra, etc.). Checks HTTPS_PROXY/https_proxy for https:// URLs and HTTP_PROXY/http_proxy for http:// URLs, in that order. If a proxy URL is set and parseable, passes it to httplib's set_proxy() and set_proxy_basic_auth() as appropriate. Malformed proxy URLs are silently ignored so existing callers aren't affected. Fixes ggml-org#21694 * common: drop misleading comment about generic proxy fallback The comment referenced a 'generic one' that doesn't exist in the code — both HTTP_PROXY and HTTPS_PROXY are scheme-specific. Reviewer noted the mismatch. Addresses review feedback from @ServeurpersoCom. * common : warn when the proxy URL from the environment is malformed * common : honor NO_PROXY when selecting the http proxy * ci : retrigger checks (test-thread-safety flake) * common : bypass the proxy for loopback destinations llama cli talks to its spawned server on 127.0.0.1 through common_http_client, so with HTTP_PROXY set and no NO_PROXY the health polling went to the corporate proxy, which rejects loopback targets, and the wait loop never exits ("Loading model..." hangs forever). A remote proxy can never reach the client machine's own loopback, so these destinations are now always contacted directly, before NO_PROXY is consulted: localhost, *.localhost, 127.0.0.0/8, ::1 and 0.0.0.0. Similar to Go's net/http. Proxy handling for downloads and remote --server-base endpoints is unchanged. Signed-off-by: texasich <101962694+texasich@users.noreply.github.com> * docs: move proxy tip from README to server docs per review: the top level README stays lean; the proxy howto now lives with the server docs under Usage Signed-off-by: texasich <101962694+texasich@users.noreply.github.com> --------- Signed-off-by: texasich <101962694+texasich@users.noreply.github.com> Co-authored-by: texasich <texasich@users.noreply.github.com> Co-authored-by: texasich <texasich@gmail.com> Co-authored-by: marecabo <23156476+marecabo@users.noreply.github.com>
* feat(convert): Add decision.chat to support decision models that can also chat Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(convert): Add conversion for Granite Switch as decision model The template follows from the usage sample: https://huggingface.co/ibm-granite/granite-switch-4.1-8b#guardian-and-decision-side-by-side Branch: GraniteSwitchDecision AI-usage: draft (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(common): Add support for chat+decision models in common Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(common): Add granite switch decision model enum Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(server): Plumb through decision_chat bool to support decision+chat models Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(server): Add Granite Switch decision support Branch: GraniteSwitchDecision AI-usage: draft (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * docs: Update server README for granite switch decisions / chat+decisions Branch: GraniteSwitchDecision AI-usage: draft (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * Revert "feat(convert): Add decision.chat to support decision models that can also chat" This reverts commit 952ef56. * Revert "feat(common): Add support for chat+decision models in common" This reverts commit 93ed782. * Revert "feat(server): Plumb through decision_chat bool to support decision+chat models" This reverts commit 97e9af6. * fix: If Granite Switch supports decisions, it always supports text too Branch: GraniteSwitchDecision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * docs: Remove reference to decision.chat in server README Branch: GraniteSwitchDecision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* opencl: add bin kernel `gemm_moe_mxfp4_q8_1_dp4a_ila_ldck` for X1 * opencl: fix whitespace * opencl: undo unnecessary change * opencl: enable dp4a clc for X1E --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
softplus_f32 and tile_softplus_f32 ran a scalar logf(1.0f + expf(x)) loop. Replace both with a shared hvx_softplus_f32_aa() helper built from the backend's existing hvx_vec_exp_f32 / hvx_vec_log_f32 primitives, so no new numerical approximation is introduced. hvx_vec_exp_f32 clamps its input to (-88, 88) before exponentiating, so a naive log(1 + exp(x)) composition would saturate to ~88.72 for large x instead of returning x. The Q6_V_vmux_QVV select keeps the scalar reference's x > 20 guard (ggml_compute_softplus_f32 in ggml-impl.h), which is what makes the vector path match the CPU result. Assisted-by: Claude
* ui : land opened conversations at their bottom before the first paint Pin the page to its bottom with a ResizeObserver until it stops resizing or the reader interacts, and mount the last rows eagerly so the landing frame shows real content. Assisted by: Claude (Opus 5.5 Medium) * ui : make conversation tabs browser-style A plain navigation shows the chat in the current tab instead of adding one, and the new-chat tab turns into the conversation it starts. New tabs come only from explicit actions: the + button, Cmd/Ctrl-click or middle-click on a sidebar chat, and its Open in new tab menu item, which open in the background. Closing a tab moves to its neighbour among the tabs the view shows. Assisted by: Claude (Opus 5.5 Medium) * ui : share the tab surface and tab bar fade styles Move the tab item surface (shape, resting and active looks) to TAB_SURFACE_* class constants and the fade under the tab bar to a global chat-top-fade class, so other elements can match the tabs. Assisted by: Claude (Opus 5.5 Medium) * ui : fit the chat column below the tab bar The tab bar reported 1.25rem of its height while it takes 3.3125rem (pt-3.25 + h-10), so a short conversation still left a few pixels to scroll. Use the bar's full height and drop the extra 1rem from the column's minimum height, so it fills the viewport exactly; the centered new-chat form moves up by the same 1rem to keep its place. Assisted by: Claude (Opus 5.5 Medium) * ui : stop the page from bouncing past its ends Elastic overscroll on the root scroller dragged the sticky and fixed bars (tab bar, chat form) along at the top and bottom of a chat, and they jumped back on release. Set overscroll-behavior: none on html and body. Assisted by: Claude (Opus 5.5 Medium) * ui : stop scrolled dialogs from bouncing past their ends Dialogs scroll on their own (the settings panel, the search results), so the page rule did not reach them and their sticky navigation still jumped on overscroll. Turn off overscroll on dialog panels and on the scroll containers inside them. Only scroll containers: Firefox counts an overflow-hidden box such as truncated text as one, and `none` there kept the wheel from scrolling the list around it. Assisted by: Claude (Opus 5.5 Medium) * ui : fit a first exchange opened by the working directory row The last answer's minimum height leaves room for the last question only, so the synthetic row that a new chat starts with (Set working directory to ...) made a short first exchange overflow by that row. When everything above the last question is synthetic, count those rows as part of it. Assisted by: Claude (Opus 5.5 Medium) * ui : keep following a streaming response until the reader scrolls up Auto-scroll stopped whenever the page moved up, including moves the reader did not make: scroll anchoring and blocks re-rendering shorter while a response streams. Only the reader's wheel, touch, scroll keys or scrollbar drag now count as scrolling up. The send scroll also looked up the new user bubble by position, which picked an older message when the answer row had not rendered yet. Assisted by: Claude (Opus 5.5 Medium) * ui : fade the chat form between its centered and bottom positions The form no longer slides across the column between the new-chat center and the bottom. It fades out drifting toward where it goes and fades back in where it lands, quicker when opening a chat than when going back to a new one. Its wrapper keeps the same height in both positions, so the messages never shift while it moves. The new-chat greeting fades in as the form settles, also on a refresh or a first visit. Reduced motion keeps only the fades. Assisted by: Claude (Opus 5.5 Medium)
…gml-org#30072) * Metal : stop overflow from happening in large matrix output offsets Assisted-by: Codex * metal : widen batch offsets and MUL_MAT_ID token indices Assisted-by: Codex --------- Co-authored-by: feni6 <270037094+feni6@users.noreply.github.com>
…ls (ggml-org#30880) test-llama-archs included src/llama-arch.h and src/llama-model-saver.h, which prevented building on Windows with shared libraries since the used symbols are not exported from the DLL. - declare llama_arch_* wrappers around the internal llm_ arch query functions in the llama-ext.h staging header, together with llama_arch_kv_name() and llama_arch_supports_save() - slim llama_model_saver down to the private implementation of llama_model_save_to_file(): the fixture-writer half used by the test moves into tests/test-llama-archs.cpp as a local kv_writer - replace the roundtrip saver usage with the public llama_model_save_to_file() writing to a temp path (also removes the tmpfile() fd leak and its Windows workaround) - move test-llama-archs, test-generate-models, test-llama-context and test-fusion out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
* mtmd: add mtmd_helper_get_capabilities * cap audio_stream * add back old api for compat * audio_gen_spk_ref, text_gen
* tests: pin model resolution on real repo listings Extend test-model-resolution with 63 repo listings captured verbatim from the HF API, chosen to cover every file naming pattern found across 1237 GGUF repos of the major publishers and pruned to whole quant groups. Each repo pins its resolved plans and its cache listing. The expectations follow what each file really is per its GGUF header, not what the current resolver picks, so a mismatch is a resolution bug. Mismatches are collected and reported together before the test fails. * tests: leave the listing unchecked on paths the filesystem refuses Some real split names (DevQuasar's Model.Q2_K.gguf-00001-of-00012.gguf under a quant folder) exceed MAX_PATH once materialized in the temporary cache on Windows, so the listing of such a repo is skipped with a note while its plans stay checked. * tests: pin real listings to master on GGUF convention names only Replace the real repo table with 171 HF listings whose file names follow the GGUF naming convention (lowercase sidecar prefix, - delimited components, quant last), one per layout pattern found across 1174 GGUF repos and pruned to whole quant groups. Repos outside the convention are dropped. Each repo pins the plans and the cache listing master gives today, so the test is a pure harness: any change in resolution shows up here, and mismatches are reported together.
* sycl: add set_tensor_2d for faster tensor-split loading Pack host rows into chunks of up to 64 MiB when destination rows are contiguous, avoiding a separate upload call for each row. Keep the existing per-row copy path for non-contiguous destinations. On two Arc Pro B70s with gpt-oss-120b UD-Q4_K_XL, total llama-perplexity runtime drops from 263-268 s to 114-115 s. A full-load readback check verified 26,784,256 rows with no mismatches. * sycl: batch 2D tensor uploads through shared staging slots Implement set_tensor_2d to speed up tensor-split model loading by packing rows into the existing pinned staging slots and uploading each batch with a single device copy. Share staging initialization with set_tensor and wait only when reusing a slot, allowing CPU packing and device copies to overlap. Fall back to per-row uploads for noncontiguous device rows, oversized rows, or unavailable staging. --------- Co-authored-by: Jon Staples <250234834+Captain-Tripps@users.noreply.github.com>
* RMS Norm Subgroup reductions optimizaton * Fix build error * Improved optimization * Make usage of BLOCK_SIZE consistent * Add fast path to skip 2nd subgroupAdd * Add AMD Vendor to subgroup reduction opt * Add comment clarifying rms_dict and merge_maps behavior * Add unroll * Rebase * Fixed syntax issue * Add 128/256/512 block sizes to non-subgroup path * Remove trailing whitespace
…1179) Every dst row comes from src0 or src1, or for dim 0 from the row of src0 followed by the row of src1, so a single kernel covers every type: a dense source row is one memcpy, quantized types included, and a strided source is copied element by element with the element size known at compile time. Rows are split across threads instead of i2 alone, which is often 1, and are copied in tiles of 32: for a transposed source the elements are walked in the outer loop, so the rows of a tile read neighboring addresses instead of jumping a full row stride per element. Add transposed b cases to test_concat, the layout of the delta-net conv input, and a perf case for it.
…rg#31142) The hyper-connection and DSA graph builders wrote their extra outputs (post / comb gates, carried top-k selections, inject gates) through ggml_tensor** out-params. Return them in a small struct next to the primary output instead. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* llama : move no_alloc out of llama_hparams Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2)) * cont : clean-up * cont : clean-up Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2)) * tests : add a no_alloc case to test-llama-context Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
…-org#31169) * hex-mmid: vectorize skipped row zeroing and added zeroing to fused MM_ID_NX * hex-mmid: remove scalar div
* llama : enable moe amax rescaling for laguna Assisted-by: Devin * llama : declare laguna sm tensor support
* hexagon: add GGML_HEXAGON_VTCM_MB and GGML_HEXAGON_NTHREADS caps Change-Id: Idffe1613c8db6f96aa842d43d1b2efaed45462a8 * hexagon: rename VTCM cap to GGML_HEXAGON_VTCM_SIZE, drop NTHREADS cap Change-Id: If4c5b9ce9fd6b725c3561421c2295d40599d012d * hexagon: make GGML_HEXAGON_VTCM_SIZE an override, accept K/M/G suffixes Change-Id: Ib8eea84c22f24e6d986b37f215e504b046478c2c
Use the validated tree from the HRX rebase onto upstream. Preserve both independent normalization tests in tests/test-backend-ops.cpp. Assisted-by: Codex
Support learned attention sinks, sigmoid MoE routing, strided expert IDs, scalar SET_ROWS and CLAMP. Update upstream and HRX API integration and retire cached graph programs safely to bound long-context staging memory. Add operation, routing and cache regressions. Assisted-by: Codex
Fix silent zero weights for IQ2_XXS, IQ2_XS and IQ3_XXS in expert and gather paths. Add CPU-reference regressions across mixed Flint tensor formats. Tracks lemonade-ws-ice.1. Assisted-by: Codex
Add the Q2_K format mapping, block decoding and numerical/dispatch regressions. Full-offload Flint Q2_K perplexity agrees with Vulkan within 0.12 percent. Tracks lemonade-ws-ice.2. Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Import and compare n_offs, rotate only the selected channel range, and copy the unchanged prefix and suffix. Keep unsupported offset fusions on the generic path. Cover normal/NeoX RoPE, scalar and per-token scaling, and MiniCPM3 prefill/decode shapes against CPU. Bead: lemonade-ws-aio.2.3 Assisted-by: Codex
Skip inactive branch storage and scheduling, invalidate cached programs when branch selection changes, and preserve compute masks in graph snapshots. Add host and device regressions for inactive SET_ROWS and stable-UID selection changes. Assisted-by: Codex
Track main-command writes separately from initialization accesses when deciding host staging writeback. Synthetic attention metadata must not overwrite caller-owned inputs after a later batch has populated them. Keep aggregate accesses for uploads, materialization and hazards. Add a replay regression covering deferred input replacement, genuine outputs, in-place updates, and prepared-cache miss/hit paths. Assisted-by: Codex
Keep split-local graph fixtures restricted to their intended nodes while honoring the ggml compute flag contract. Assisted-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The earlier HRX queue-epoch incident remains tracked separately, not claimed fixed: 100 bounded exact-package trials and both final native suites passed. An upstream report documents the same signature before this integration, but an identical cause is unproven. This PR remains draft as requested.
Follow-up fixes now included:
Recognize upstream RMS-plus-scale GDN normalization with its additive epsilon semantics. Local Qwen3.5 0.8B/2B and MiniCPM-V prefill/decode perplexities agree with native Vulkan within 0.3%; 12 focused GPU numerical cases pass.
Preserve RoPE offsets and defer unsupported specialized fusions. MiniCPM3 prefill/decode perplexity improves from 423.0835/517.4914 to 14.6335/21.9671, versus Vulkan 14.6675/21.9590.
Honor inactive graph branches and include their compute masks in cached-graph matching. This fixes uninitialized embedding gathers during the server's final four-token prompt batch. All five Flint quants now pass the exact 32-token prompt plus 256 generated tokens locally, with replay enabled and the cache bound retained. The negative-control server run faults before generation; the focused inactive-write regression also fails before the fix.
Preserve caller-owned inputs during attention metadata initialization: initializer-only writes no longer copy stale positions/indices/masks back to host memory. Main-command outputs and in-place writes still copy back. Qwen3-30B decode perplexity improves from 3076.3825 to 6.7750 versus Vulkan 6.7605; prefill remains 6.0760 versus 6.0708. A focused negative-control test demonstrates the old input overwrite and verifies both caller inputs and real outputs after the fix.
The combined source builds and passes 92 host checks plus four focused GPU checks. GPU follow-up checks used gfx1100 with HRX5e6; native CI uses gfx1151 with HRXcf1. The original full-tier baseline used the same models/settings/HRX pin, but reported different host/GPU memory capacity. Local reproductions confirm the MiniCPM3, Qwen decode, and Flint failures independently of that hardware difference. All local follow-up model checks pass with replay enabled and the 256-entry cache bound retained. The final native gfx1151 runs linked above validate these fixes at the same final llama pin and unchanged HRXcf1 revision.
Overview
Enable Flint Q3_K_M inference on HRX and fix the routing corruption and retained staging allocations that prevented correct, sustained inference. The model now completes chat, perplexity, and retrieval checks through 32,768 tokens on an RX 7900 XTX, with paired HRX/Vulkan perplexity differing by less than 1% in the tested runs.
This draft includes the required upstream update to
7aca0d2ea8d92b56911233b822db01f9fbb1ae50. Commit5c3c0db68preserves the existing fork and upstream histories with the tree validated after rebasing the HRX changes. The rebase had one conflict intests/test-backend-ops.cpp; both independent normalization tests were retained. Commit49c2b54f8contains the HRX implementation and regression tests (28 files), separately from the larger upstream update.Changes
1.28e20.SET_ROWSfor transposed KV-cache layouts and contiguous F32CLAMPwith CPU-compatible edge semantics.loom-benchmarkintegration and require HRX revision5e6af0ec1a0c1fd6c5553e7d0e6f2bc7c4930444.Vulkan needed no additional source fixes for Flint; it served as the comparison backend on the updated upstream baseline. Validation used the existing native GGUF, not a JANG/MLX conversion.
Validation
Model:
bartowski/RecursiveSI_flint-GGUF/RecursiveSI_flint-Q3_K_M.gguf, SHA2561d6d5800dbed15e60c6ca8b3af9f131a5570bae3f1b85a552632930dc033a7ea.RX 7900 XTX (gfx1100), 50/50 layers offloaded, F16 K/V, flash attention enabled, batch 512, microbatch 128, 8 CPU threads. Some HRX operations still use CPU fallback. Corpus:
wiki.test.raw, SHA256173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08.Compare backends within each row; the scoring windows differ across context lengths. The 512-token CPU control measured 25.4637.
git diff --checkpassed; packaging the commits preserves the validated source tree.Limits and review notes
Validation stops at 32K, the current HRX flash-attention KV-length limit. The advertised 262,144-token context, context shifting, and concurrent serving are unvalidated. Retrieval is one synthetic prompt per context, not a broad quality evaluation. HRX remains slower than Vulkan. Cache retirement uses an entry-count bound rather than a byte budget and can cause prepared programs to be rebuilt.
Workspace tracking:
lemonade-ws-z0zandlemonade-ws-dcn, with blocker-specific beads and isolated workspace threads. Model files, build products, and local investigation logs are excluded from this PR.Additional low-bit quant validation
Tested only the four smaller Bartowski GGUFs requested, with full layer offload on the same RX 7900 XTX. No larger quant downloads or partial-offload tests. Settings match the 512-token, 16-chunk protocol above.
Two additional blockers were isolated in bead-owned workspace threads and integrated:
lemonade-ws-ice.1: Generic GPU dequantization accepted IQ2_XXS, IQ2_XS, and IQ3_XXS but lacked their FP16/FP32 decode branches, silently yielding zero expert and gather weights. Initial HRX perplexities were 9747.4659 (XXS) and 10823.3029 (XS). Added the missing canonical codebook decoding. The mixed IQ2_XXS file also contains 11 IQ3_XXS expert tensors, so repairing IQ2 alone was insufficient. Negative controls failed before the fix; 103 targeted worker tests passed afterward.lemonade-ws-ice.2: Q2_K lacked format mapping and GPU decoding, causing graph execution failure. Added the mapping, 84-byte block decoding, and dense/expert dispatch coverage. Nine strict numerical GPU cases and 14 host dispatch cases passed. IQ2_M needed no additional source fix.The combined main-workspace build passed 79 GPU numerical tests and 14 host dispatch tests. Four-chunk integrated runs matched the repaired workers for XXS, XS, and Q2_K; the original 16-chunk Q3_K_M control remained unchanged. All four selected quants produced coherent chat answers. Vulkan required no new source changes.
These additional quants were validated at 512-token context only; the earlier 32K results apply to Q3_K_M. Inline codebook decoding prioritizes correctness and is not performance optimized. Model weights and local logs are excluded from the PR.
Requirements