weightscan

Statistical steganalysis for safetensors and GGUF model weight files — the formats ollama pull and most current Hugging Face downloads actually use.

picklescan and modelscan check whether a model file can execute code on load (a real risk for pickle-based .bin/.pt/.ckpt files). Safetensors and GGUF are pure data by design — there's no deserialization step to exploit, so those tools have nothing to check. That doesn't mean the file is above suspicion, though: research ("EvilModel") has shown you can hide an arbitrary payload — malware, an archive, anything — inside a neural network's weights by overwriting the low-order bits of the floating-point values, with negligible effect on the model's behavior. Nobody was checking for that. This tool checks for that.

What it actually checks

weightscan scan model.safetensors
weightscan scan model.gguf --verbose
weightscan scan candidate.safetensors --reference known_good.safetensors
weightscan diff known_good.safetensors candidate.safetensors

Three independent checks, run per tensor:

1. Magic-byte scan. Reconstructs the byte stream formed by the lowest byte of every element (the layout a whole-byte LSB-substitution payload would use) and searches both that and the raw tensor bytes for known file-format signatures (PE, ELF, Mach-O, ZIP, GZIP, BZIP2, 7-Zip, RAR, PDF, shebang lines). Cheap, and — after the false-positive fixes below — essentially silent on real weights. Catches an undisguised payload directly. Trivially defeated by anyone who compresses or encrypts the payload before embedding it, since that removes the recognizable header.

2. Quantized-histogram anomaly (int8/uint8 tensors only). A real quantized weight tensor has a peaked value histogram — it's a discretized, roughly Gaussian, zero-centered distribution, not a uniform spread across 0–255. Substituted or encrypted payload bytes are close to uniform. This is a legitimate, defensible signal specifically because "peaked vs. flat" is a real, checkable difference in what the tensor is supposed to contain.

3. Reference diff (--reference / diff, requires a trusted copy). By far the strongest check here, and the one to reach for if you can. The attack in question — overwrite low mantissa bits, leave sign+exponent alone — has a specific signature when compared byte-plane-by-byte against a known-good copy of the same checkpoint: the low byte(s) differ in nearly every element, while the top byte (sign + exponent, i.e. the value's actual magnitude) is untouched. A legitimately different checkpoint (retrained, fine-tuned, re-quantized) changes the value, which moves the exponent too — it doesn't leave the top byte bit-for-bit identical across the whole tensor while rewriting only the bottom one. If you can get a hash-verified reference (e.g. from the official Hugging Face repo) for the exact checkpoint you're auditing, use it.

What doesn't work, and why it's still in the code

An earlier version of this tool had a fourth detector: measure how many low-order mantissa bits "look random" (a bit-depth curve), on the theory that real trained weights carry genuine information through most of their mantissa, so an attacker overwriting more bits than natural rounding noise would extend the "looks random" region deeper than expected.

That's wrong, and testing it against synthetic data caught the error before this shipped: a clean float32 tensor and one with its two lowest bytes per element replaced by os.urandom produce near-identical entropy curves through at least 20 of a float32's 23 mantissa bits (see tests/test_detection.py::test_bit_depth_curve_does_not_discriminate_unquantized_floats, which asserts this and will fail loudly if that ever stops being true). The reason: representing any continuously-varying real number in IEEE-754 binary makes almost the entire mantissa look statistically uniform, as a property of the number format itself — not because training left it that way. There is no "natural noise floor" to compare against without a reference file.

bit_depth_curve() is still in analysis.py and the --verbose output still prints it, deliberately, as a documented negative result rather than a deleted dead end. It never sets a tensor's suspicious flag.

False positives — the part worth reading before trusting a clean scan

Short magic numbers collide with chance byte patterns constantly once you're scanning real-model-sized tensors. This was caught empirically, not assumed: a plain Gaussian-noise 100MB tensor tripped GZIP's 2-byte magic five times before any confirmation logic existed, and Mach-O/Java's 4-byte 0xCAFEBABE fired once by chance at 100MB scale even with a 2-byte magic excluded. A 1GB tensor still tripped BZIP2's weaker 4-byte confirmation once. Each of those is now fixed with a signature-appropriate secondary check (gzip's header byte-8 flag-and-method bytes, bzip2's full 10-byte "pi-magic" preamble, a generic "look for the zero/repeated bytes real headers have and random noise doesn't" check for everything else ≤4 bytes) and reverified clean at 100MB and 1GB. Longer or rarer models could still turn up a stray hit; if weightscan flags something, look at the actual offset and surrounding bytes yourself before concluding anything; it isn't a court verdict, it's a "worth a look."

What this cannot tell you, at all

• A trained-in behavioral backdoor (a model fine-tuned to behave normally except on a specific trigger). Nothing here inspects behavior; that requires red-teaming/fuzzing across inputs, or mechanistic interpretability work looking for anomalous features/circuits — both active research areas, neither of them a file scan.
• A payload the attacker compressed or encrypted before embedding. The magic-byte scan only catches undisguised headers; a competent attacker defeats it trivially. The quantized-histogram and reference-diff checks don't care about the payload's content, so they're more robust to this, but reference-diff still needs you to actually have a trusted reference.
• Whether the training data or organization behind a model is trustworthy. That's a supply-chain/provenance question, not something visible in the weights at all.
• Pickle-based files (.bin, .pt, .ckpt). Use picklescan or Protect AI's modelscan for those — this tool intentionally doesn't duplicate that work.

A clean weightscan result means: no undisguised payload signature, no implausibly-flat quantized tensor, and (with --reference) no magnitude-preserving low-byte substitution versus your trusted copy. It does not mean the model is safe to run.

Building your own noise-floor baseline

There isn't one to build, on purpose — see "What doesn't work" above. An earlier design of this tool had a weightscan baseline corpus-builder command for exactly that; it's gone, because the thing it would have measured turned out not to be a real signal. If you want to extend this project, a --reference corpus check across many known-good checkpoints of the same architecture/quantization (rather than one file) is a more promising direction than reviving the per-file noise floor.

Install

git clone https://github.com/YOUR-USERNAME/weightscan.git
cd weightscan
pip install -e .
weightscan scan model.safetensors

Only dependency is numpy. Nothing here executes anything from the file being scanned — it only ever reads bytes.

Tests

python3 tests/test_detection.py     # or: pytest tests/

Builds synthetic clean/poisoned fixtures on the fly (see tests/make_fixtures.py) and asserts the tool actually distinguishes them — including the negative-result test for the bit-depth curve. Also see the scale checks referenced above (100MB/1GB synthetic tensors) for the false-positive-rate verification; those aren't checked in as fixtures because of their size, but the exact commands are in this README's git history / can be regenerated from the snippet in "False positives" above. CI (.github/workflows/ci.yml) runs the test suite plus a CLI smoke test on every push, across Python 3.9–3.12.

Contributing

Issues and PRs welcome, especially: real-world false positives/negatives against actual downloaded models (the synthetic fixtures are a start, not a substitute for testing against real checkpoints), GGUF K-quant support for the histogram detector (currently only plain int8/uint8 tensors are checked), and anyone who wants to take a real run at the "corpus of known-good checkpoints" idea mentioned above. If you're proposing a new detector, please include a test that would have caught the previous detector's failure mode (see the bit-depth curve section) — the standard here is "does this survive an adversarial synthetic test," not "does this sound plausible."

License

MIT — see LICENSE.