ISTA Quantizer Squeezes Qwen3.8-27B Into 8.4 Gigabytes

Two files of Qwen3.8-27B are both 8.4 gigabytes, and only one of them scores 96.67 on the AIME 2025 math benchmark instead of 86.67. The gap between them is not the model. It is the arithmetic used to shrink it.

The higher-scoring file comes from ISTA-DASLab, the Deep Algorithms and Systems Lab at the Institute of Science and Technology Austria, which published a new quantization method called GSQ-RCO on Hugging Face, which added 207 WebGPU kernels aimed at making browsers a bigger inference target on Sept. 1, this month. The lower-scoring file is Unsloth's UD-IQ2_S, a widely downloaded dynamic quantization of the same Qwen3.8-27B base model that Alibaba released in August.

What GSQ and RCO each solve

GSQ, short for Gumbel-Softmax Quantization, is described in the model's documentation as "post-training scalar quantization that jointly learns the per-coordinate grid assignments and the per-group scales via a Gumbel-Softmax relaxation." In plain terms, it decides how to round each individual weight so the rounding error hurts the model's output as little as possible, while staying inside standard GGUF files that existing software can already load.

RCO, Riemannian Constrained Optimization, solves a different problem. Once GSQ has scored every tensor's sensitivity to rounding, something has to decide how many bits each tensor gets under a fixed total file-size budget. RCO's documentation describes the approach as reformulating "the budget constraint as a smooth Riemannian manifold in logit space," which permits gradient-based search on the model's own task loss instead of a hand-tuned allocation table. The method is credited to Michael Helcig and Dan Alistarh, a professor at the Institute of Science and Technology Austria, in a paper submitted to arXiv on May 1, 2026.

The size-for-size comparison

At identical file sizes, ISTA-DASLab's own benchmark table shows GSQ-RCO ahead of Unsloth's dynamic quantization on every reasoning benchmark it lists.

Variant	Size	AIME25	GPQA-Diamond
BF16 (baseline)	53.8 GB	100.00	89.90
GSQ-RCO IQ2_XS	8.4 GB	96.67	84.85
Unsloth UD-IQ2_S	8.4 GB	86.67	76.26
GSQ-RCO IQ3_S	11.8 GB	100.00	89.39

The IQ3_S file, at 11.8 gigabytes and 3.5 bits per weight, matches the 53.8-gigabyte BF16 original on both AIME25 and LiveCodeBench v6, which ISTA-DASLab calls "task-lossless." That is a 4.6-times size reduction with no measured accuracy cost on those two specific tests, not a general guarantee across every task a deployed model might see.

1. BF16 baseline (53.8 GB)100 score
2. GSQ-RCO IQ2_XS (8.4 GB)96.67 score
3. Unsloth UD-IQ2_S (8.4 GB)86.67 score

Source: ISTA-DASLab Hugging Face model card, accessed 2026-09-13

Why this is not an independently verified number

Every figure in that table comes from ISTA-DASLab's own model card, run on its own benchmark harness, comparing its own method against a competitor's public release. That is a vendor benchmark in substance, even though the vendor here is an academic lab rather than a company, and The Terminal could not independently verify the scores on separate hardware. AIME25, GPQA-Diamond and LiveCodeBench v6 are also narrow suites: math competition problems, graduate-level science questions and competitive coding, not the broader range of tasks a deployed model actually handles.

The RCO paper submitted May 1 by Helcig and Alistarh, and the companion GSQ paper submitted April 20 and revised May 15 by seven ISTA-DASLab authors, both report matching or beating alternative methods on Llama-family models at 2 to 3 bits per weight. That is a narrower prior claim than beating Unsloth specifically on Qwen3.8-27B, and it predates this release by roughly four months.

An independent test, on a smaller file

MindStudio published its own hands-on test of the IQ3_S file on Sept. 8, five days before ISTA-DASLab's broader release, running it through llama.cpp on a 48-gigabyte RTX card. Its result complicates the headline size figure: the 11.8-gigabyte file "used just over 28GB of VRAM during inference," more than double the file size once the KV cache and activations were loaded. MindStudio reported the model correctly generated a self-contained HTML and JavaScript animation and found a deliberately hidden logic bug in a long, nested SQL query, calling the result "a real result, not a marketing claim." It also flagged a gap ISTA-DASLab's table does not mention: multilingual coverage, tested across more than 75 languages, was uneven, though MindStudio traced that weakness to the unquantized Qwen3.8 base model rather than to the quantization method itself.

What it means for anyone running the model

For a team already weighing Qwen3.8-27B's 262,000-token context against real GPU memory limits, GSQ-RCO changes only one term in that budget. Dropping the weights themselves to 8.4 or 11.8 gigabytes still leaves the KV cache and activation memory that scale with context length and concurrency mostly untouched, which is exactly why MindStudio's 28-gigabyte figure came in so far above the file size. A smaller weight file helps most on hardware where VRAM is the binding constraint in the first place, and matters less once context length is the bigger line item in the budget.

What would settle the claim

The number that would settle the argument is a benchmark run by someone other than ISTA-DASLab, on hardware neither lab controls, across a task set wider than three reasoning suites. Until that happens, GSQ-RCO's edge over Unsloth's quantization is real on the evidence published so far, and specific to the evidence published so far.