What You Sign Is Not What You Run: audit exposes the verifiability gap in quantized LLM supply chains

An audit of the top 200 Hugging Face GGUF repositories reveals an Independent Verifiability Rate of 0/198 under standard disclosures, showing hidden compiler flags and per-tensor quantization recipes are required for bit-for-bit verification.

Primary source: arXiv:2610.19498: https://arxiv.org/abs/2610.19498

What’s new: Cryptographic model signatures authenticate the full-precision weights that developers release, but inference runtimes execute quantized artifacts whose compiling transformations are not recorded. In What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains (arXiv:2610.19498, 14 Oct 2026), researchers from National Yang Ming Chiao Tung University and National Taiwan University of Science and Technology evaluate whether quantized models can be verified independently. They define the Independent Verifiability Rate (IVR) as the fraction of quantized artifacts that an external auditor can re-derive bit-for-bit from disclosed source weights and recipes. Auditing the 200 most-downloaded GGUF repositories on Hugging Face (198 in scope), they find that not a single repository records its build environment, resulting in an initial IVR of 0/198.

However, the lack of recorded build metadata does not imply intrinsic non-determinism. Examining four in-sample repositories that disclose their conversion parameters, the authors show that model cards omit two critical coordinates: the compiler floating-point contraction setting (such as -ffp-contract=off versus fast-math defaults) and the per-tensor quantization scheme for hybrid architectures. Supplying these missing coordinates allows bit-for-bit re-derivation of 7 out of 8 target quantization files on x86 platforms (and 5 out of 8 on arm64). In contrast, importance-matrix (imatrix) calibrations cannot be reliably recomputed from text corpora due to pipeline nondeterminism, meaning calibration matrices must be distributed as signed, hash-bound inputs. Furthermore, the authors reproduce cross-platform divergences (such as those observed in llama.cpp PR #25353) and demonstrate that enforcing strict floating-point contraction produces identical quantized binaries across architectures with a negligible perplexity increase of only +0.06%. While statistical checks can distinguish benign compiler differences from random bit flips, targeted embedding tampering easily evades heuristic detection, proving that secure supply chains require strict, bit-for-bit reproducible builds.

Why it matters: Edge deployments and enterprise local inference depend heavily on 4-bit and 5-bit GGUF models downloaded from public model hubs. If downstream artifacts cannot be verified bit-for-bit against upstream signed weights, security guarantees vanish, leaving models vulnerable to supply-chain backdoors, silent weight tampering, and undetected data corruption. Establishing a reproducible build discipline for quantization pipelines bridges the gap between cryptographic checkpoint signing and deployed inference binaries.

Caveats: The empirical audit focuses on the llama.cpp CPU quantization pipeline across Qwen (0.5B to 14B) and Granite 3B models, evaluating common quantization formats (Q4_K, Q5_K, Q6_K, IQ4_XS, and Q8_0). GPU-native quantization methods (such as AWQ, GPTQ, and bitsandbytes) and GPU-based imatrix generation were not evaluated and may require hardware attestation rather than purely software re-derivation. Finally, bit-for-bit verification of the quantization step does not protect against upstream adversarial weights embedded in the original full-precision checkpoint.