Eval of 711, Tess, AEON, ThinkingCap, 27B base with no confounds
Here ya go, objective, independent, evidence that on the whole 711 is hands down an upgrade:
https://blog.robai.net/27bevals/
The Q8_0 vs BF16 711 results were repeated multiple times because it's odd but the fact is Q8 sometimes snaps a result to a better outcome, just is what is.
I've finished initially implementing llama style DRY in vllm with MTP compatibility.
When I've had some time to make sure that is not causing issues, I'll retest all of them and see if it changes things.
@tcclaviger
I can not access link/report from by location ; geo (AU) and/or ip blocked?
Can you add screenshots in case others are having this issue too?
thank you ;
EDIT:
Was able to view; excellent, super detailed report.
Fantastic.
Ah right I forgot I have some....stuff setup that might cause that.
I'll export and post elsewhere.
I did loads of testing with claude created tests across various quants and models
https://claude.ai/code/artifact/4472b727-8ae6-4099-b1a0-700ecf74bf2e
Every model was K_M and I also threw in IQ4_XS for the final tests, run in llamacpp
defiant-9b
defiant-9b-mtp
fable-711-q4
fable-711-q4-mtp
fable-711-q5
fable-711-q5-mtp
qwythos-q4
qwythos-q4-mtp
qwythos-q5
qwythos-q5-mtp
711 was the best out of all them by far. It was weird though, it seemed like Q4 was better at some stuff than Q5 and vice versa.
I could upload the testing files if anyone really wanted them, but I doubt anyone does. it took like 5-6 days of solid testing to get these results using a 5090. nearly 24 hours for the "abyssal" tests, it was obnoxious trying to get sonnet to not score a 100% across the board
Here ya go, objective, independent, evidence that on the whole 711 is hands down an upgrade:
https://blog.robai.net/27bevals/
The Q8_0 vs BF16 711 results were repeated multiple times because it's odd but the fact is Q8 sometimes snaps a result to a better outcome, just is what is.
Wow, thx!
Would be amazing to have this also for the quants Q4 & Q5, to see if the base model with Q5_K_S degrades more drastically than the slightly larger 711's Q5_K_S.
Those results are slowly being added as I get through them, page has a drop-down to find model results now. Will spruce it up later wanted to get results up quickly, other projects are more prioritized
My new test harness is much more sensitive than previous regime.
The feature implementation test is the most meaningful as it involves tasks frontier models have struggled with.
The Complex test features multiple simultanious DSAs in one giant messy part of vLLMs code, plus stride length tracking across not-in-training data designs (the codes never been up streamed), between very different structures to keep correlation aligned. LRU eviction conditionally, not pure LRU, complex import discovery because the fix includes new imports to get data from existing but unavailable sources, and finally some extremely delicate boundary length manipulation.
Plus, it needs to be fast code.
Chat orchestrator involves correctly using a sliding window across streaming token deltas scanning for values without blocking or delay, handling stalls, pauses resumes, missing end tokens, broken tool calls, etc. A model that falls back to pure in training OpenAI, or Anthropic streaming guidelines will fail, I do some custom stuff on my streaming engine.
Robaiproxy is a retired project that was something like caddy + lact + load balancer + rabitMQ rolled into one. I removed some core pieces, the model is charged we implementing the removed features with identical to original behavior in tests.
They're tough tests, so any + on the result is a good result, gpt oss 120b I predict will pass almost none of them, will run in the future.
i'm impressed by the user comments, though there might be a hype bias. But the very strong response is a good sign.
But i had a little chat with 5.6 about the og benchmark and it was less enthusiastic. Even though on average it also declared this model the winner (based on this benchmark) with the clear exception of tool calling. (which is kind of a biggie) . But it was by no means overwhelming (based on the benchmark linked. Nor was it impressed with some of the benchmarks because the question base was too small for meaningful statistic ( This run used only 50 questions. Each question therefore changes the result by two percentage points:
94% = 47 correct
98% = 49 correct
100% = 50 correct
So Fusion Q8’s 100% versus base’s 98% is literally one additional correct answer. Do not use that to claim a meaningful mathematical advantage.)
https://chatgpt.com/share/6a6c5835-48b4-83eb-a7dd-5db51f17d35f (it was slightly weighing it to what it knows about me doing with llm (agentic coding, image stuff (where this model is a fail but the heretic one fixes that)) and my system.
Since i don't use this model but a apparently identical but (really) heretic one, i can't take it 1:1 anyway.
So while it seems the best for many cases, claiming it to be Opus level is likely plain silly. And of course q3.6 27b is a freak to start with.
Thank you for your comments.
100% correct means the model UNDERSTOOD and ANSWERED every question perfectly.
The key point: PERFECTLY.
At 1/2 precision the model moped the floor with Qwen 3.6 27B.
And at 1/4 precision IT STILL MOPES the floor with Qwen 3.6 27B.
In addition FF711 exceeding the base (and other models) on almost all benchmarks.
It it the totally of all of these that tell the FF711 story.
As for "Opus power"; see Qwen 3.6 27B org model card; they are already almost there.
first: Tbc I'm not questioning that it is good, even beyond just "beats the base model". But claims of near perfection make me careful. (i had similar discussions around the perfection claim of hauhaucs uncensored models). . And especially questioning the real life value (that was my question to 5.6)of the og post benchmark.
So of course i would hope (and trust) that it beats the BASE model after all the work that has gone into it. Something would be really wrong otherwise. The real challenge is the other finetunes, included and not.
The point about the small question base is just obviously valid. The number of points on which to base a statistic is a fairly simple concept to understand. Doing just 1 (one) less (or more) correct can simply be a fluke on that small base OR dependent on the questions in question (in the above benchmark) .
The literal perfection claim would be more impressive on a larger one. Call it the WWTBAM issue. You get the right 16 questions, you get million, get the wrong set and you leave a fool with 1000$ or nothing, despite using the same Q8 brain both times.
in a follow up question, 5.6 actually put a positive disclaimer on the (poor) tool calling result: "Fusion’s lower no-think result does not establish that it is worse in a thinking-enabled coding harness.
Fusion Q8’s stronger thinking-on Hard Mode result suggests it may recover—or even become preferable—when permitted to reason." https://chatgpt.com/share/6a6c6e25-7670-83eb-956d-5938dcd9e012
which i assume with my limited understanding, is the point of all the acronyms in the name.
Btw i can run q8 but not with useful context limit in vram. So lower precision to compare would definitely be interesting. I downloaded the re-hereticed version q4nl for my laptop tonight, and it seemed good but it did the Qwen "wait a moment thing" a lot even though the actual reasoning thinking was very good. But normally id use q6.
The benchmarks (by Nightmedia) on the main repo page have a lot more questions / harder questions and tell the story better VS 50 questions.
These are the core metrics we use in all testing (benched over 500 models) because they are very hard.
The goal of this model was to raise all benchmarks ; but not directly. (IE: No bench maxing -> it damages the model)
IE: Improve the overall internal structure and understanding deep in the model.
PPL [perplexity] is only one measurement (as noted in the Iron Labs report) ; but it shows a vast decrease (lower is better) ;
this in part shows a net increase in language comprehension.
This is critical in all LLM performance levels and benchmarks.
NOTE: IQ4NL is an oddball quant; you may want to try IQ4_XS.
Other users have reported very strong tool use ; in some cases "perfect" (when the base model made mistakes several times).
That being said there is always room for improvement.
I also have a lot more context on the model pages on how the merge was done.
This is an ongoing process of improvement.
Of course, I am not well known, which is why my model merges are not very popular :)
One of the building blocks of the 711 was this one, from three months ago:
https://huggingface.co/nightmedia/Qwen3.6-27B-Architect-DS9-Polaris-Heretic-mxfp8-mlx
I even posted it on Reddit
https://www.reddit.com/r/Qwen_AI/comments/1sy77k1/qwen3627barchitectds9polarisheretic_arc/
Not one comment.
It was before Fable traces and all the new model wraps. It was the first to scratch at the 0.700 ceiling, and a very nice RP engine--the heart of the 711
arc arc/e boolq hswag obkqa piqa wino
bf16 0.692,0.863,0.911
mxfp8 0.699,0.871,0.910
q8-hi 0.694,0.865,0.910
qx86-hi 0.688,0.862,0.910
qx64-hi 0.700,0.862,0.907
mxfp4 0.694,0.872,0.909
Quant Perplexity Peak Memory Tokens/sec
bf16 3.898 ± 0.025 60.75 GB 226
q8-hi 3.895 ± 0.025 37.26 GB 215
mxfp8 3.921 ± 0.025 34.74 GB 218
qx86-hi 3.898 ± 0.025 32.36 GB 218
qx64-hi 3.918 ± 0.025 25.64 GB 217
mxfp4 3.999 ± 0.025 21.30 GB 225
When this Nightmedia project started, I had a Futurama theme for the attention heads:
https://www.linkedin.com/posts/gchesler_ai-partnership-fusion-share-7297448977715671040-3IPG
Star Trek was more appropriate down the road, and I kept Spock :)
P.S. Although source is available, please don't bench the old model, it might be better at coding, and we want 711 to succeed :)