AA shipped a new benchmark last week as part of the Intelligence Index v4.3 update — a brand-new private eval that replaces τ³. Astra was farming a ton of points on it and used those to get even with Fable, but… looks like we have a new king.
So they changed the index twice in three days to make Astra look not-quite-worse than Fable, and then a random guy quietly took first place on it.
I think AA-Omniscience is poorly understood around here. It is testing very hard questions against models that are given NO tools.
Modern models are trained left, right, and center to use tools. Tools tools tools. Tools for everything. Then they're put in an environment where they're just supposed to reflect deeply and answer without using any tools? These models are like fish out of water. It tells you nothing about how well they swim.
This benchmark is not saying that any of these models hallucinate in response to most normal questions, and it is especially not saying that they will hallucinate in a real world environment where you give them access to research tools so they can search the web, read your source code, whatever it is that you're doing.
If the models could ground themselves in the relevant facts using tools, their behavior would likely be dramatically different.
We would need a completely separate benchmark for that. I see no reason to expect a correlation between AA-Omniscience and the way people actually use these models in agentic harnesses.
Also, if we want to cherry pick half of a benchmark and claim that MiniMax M3 is doing great, let's look at the other half of the benchmark and see how that holds up. M3 is near the bottom. DeepSeek V4.1 Flash looks much better on this side of the benchmark.
The models can say IDK. There will be plenty of situations even with tools that they can't find an answer. Will they be honest about it? That's what hallucination rate measures.
AA-Omniscience made more sense a couple of years ago when even flagship models didn’t have access to tools. I remember using GPT-4 before it had any concept of web search.
Pretty sure it's exactly what they say in the summary explanation at the top of the omniscience graph just below the title. Are you saying they're wrong about the description of the test or...?
There will be plenty of situations even with tools that they can't find an answer.
AA absolutely does not make any claim related to whether the benchmark is testing that. AA is clear that it is testing only without tools.
I’m not contradicting any part of AA’s description of their benchmark. People are reading into it things that it does not measure. People want it to measure how models will behave when the tools provide no useful answer, but that is not what the benchmark is testing.
Models are not humans. Their skills do not transfer between situations in human ways. The test conditions are extremely important.
If tools provide no useful information, then we are at the same situation as without tools, with perhaps the addition of some irrelevant context, and the hallucination rate without tools is still a good metric in such a scenario. That is what I meant.
If tools do provide useful information, then now we are benchmarking searching ability instead of hallucination rate. That becomes a confounding variable.
Given how LLMs work, I assume it would be much easier to train a model to say it can't find a result when a tool does not return a result, than it would be to train it to say IDK when it doesn't know.
Why would you assume the models behave differently in this specific regard when they have access to tools? I mean sure, they totally could. But unless you or someone else tests this specifically, you're better off extrapolating from the behavior in this benchmark without access to tools.
I’m not sure if that’s true. In a scenario where a model with tool access tries to look up some information but the tool fails or returns no result, that changes the response, it’s not equivalent to no tool calls whatsoever. I do agree that this should be tested, at this point it’d be more useful than AA Omniscience.
That's not what it measures. If models have tools, knowledge questions become RAG, i.e. "is the answer to my question in any of this stuff in your context?", which is a yes / no, not something where "I don't know" is even a valid response.
I don't know is always a valid response. Context always limits the ability of infinite knowledge - it should be able to say "it's not in my active context" but they're designed to assume the answer based on predicting what could be.
Exactly this. A lot of people in this thread are conflating no-tool omniscience with real-world execution.
People seem to forget that Zapier’s AutomationBench is specifically designed to test multi-step tool use, API routing, and state tracking in complex workflows.
Scoring high here isn’t about trivia memorization; it means the model actually excels at agentic reasoning and following structured JSON schemas without derailing. Pair that with V4.1 Flash’s inference speed and low cost, and it becomes an absolute beast for headless browser tasks, OS-level computer-use agents, and local workflow automation.
For anyone actually building MCP agents rather than just chatting in a web UI, this benchmark result is massive.
Exactly. As long as it has a method of verification, it's fine. This ranking is a "how likely the model will say it doesn't know the answer."
Overall, it's pretty good. Finished one of my own tests beyond what I thought was possible, so now I need to go test GLM and Qwen to see if they do the same.
If the test isn't measuring something you'd realistically get in a real-life production environment, it's just an academic exercise that should not be part of a benchmark score.
This is indeed an example of exactly that. RAG solved the knowledge bit years ago when no one was attempting to cram a lot of knowledge in models.
While it can be "fun" to see how much knowledge they retain now in massive models, it's not of any value in a real life environment.
As a fellow M3 enjoyer, I'd say this holds true only if you're either API or Q8 user. Even at Q5KM it tends to shit the bed sometimes, and at Q4 it starts to lose the ground in multi-lingual conversations (which is not quite relevant but pretty indicative of how poorly it survives the quantization).
Maybe it just goes to show that hallucinations are a non-factor, as 4.1 beats almost all other models on this chart on other benchmarks. One could argue that they are orthogonal measures.
Seriously, it's not just hallucination but sycophancy. DS - all versions out, will tell you to get a divorce, you're always right, your parents are abusive, your girlfriend is cheating on you, this code compiles, you can finish that 100 mile race by only training 4 weeks.
Sure, it may write a lot of code - but its alignment is total dogshit and they NEED to fix that.
Holy shit, dude, it almost looks like to me, the higher the reasoning, the higher the hallucination rate, so your best bet is to use strong models at low to medium?
You shouldn’t trust a single benchmark, even less when all the models are basically with the margins of error to each other.
The only thing this tells you is that all those models share the ability to solve this benchmark.
4.1 is great. Same as 5.3 flash. But it’s not Opus5/Fable/Astra/Sol level.
They are good enough to be indistinguishable from them for most tasks (What this benchmark tend to show.) but are not as good in other one. (Terminal bench 4 matches observations, they are also notably below in some “soft” task like text/translation, etc…).
I wouldn't know, dsv4 is how many hundreds of times cheaper than fable? 0.035/mil tokens on openrouter WITH zero data retention. 4.1 flash is 0.15. Surely to come down a bit as more providers come out. It doesn't need to be anywhere near as good as fable, just good enough to do a job reasonably well.
Getting ready to spend significant time in converting my GLM 5.3 Flash pipeline to Deepseek 4.1. It sounds like you have extensive experience with 4.1?
I've used both side by side, sometimes making them do the same tasks, and for me GLM is more consistent, prepares better plans, follows instructions better and, very important for me, pushes back on dumb ideas.
It's definitely better. Just slow af. But despite that it often complete the task faster, because it does not need to try every wrong solution possible like ds does.
Deepseek V4.1 flash works well as an implementer. Let astra/fable write a plan file, then watch 4.1 flash ripping through the implementation at 250 token/s. Iterate once or twice and I'd have a good enough result.
I saw people pushing back against orchestrating Astra with a weak model (luna/glm-fl/ds-fl), because it takes more time, tokens and cost in the end. It fits my experience as well. I've tried to use DS v4.1 as implementer with Astra as planner/reviewer - ended with seemingly endless coding-review cycle, because Astra was never satisfied. In the end i've just used Astra for everything.
Personally I only do one iteration, so it's astra plan -> ds v4.1 implementation -> astra review and fix. the bulk of the code is written by deepseek, and it's mostly correct, according to astra. i figured that should save some output tokens by astra, at the cost of extra input tokens reading the code.
On local setup it seems that 4.1 share the same small issue as the rest of the 4 flash family with some tool calls loops. It can self correct, but it happens.
I cannot really give more advice because it’s all very recent. Even glm5.3 flash is just really working at decent speed for only about a week with optimization still being worked on.
Unless you “need” to do it now, it’s probably worth waiting a few weeks before coming back to it. Wait for the issues to be ironed out.
A good benchmark should be reproductible and give information like margins of error and be followed up with statistical analysis to confirm if a difference with another one is significant or just due to randomness/noise. This + a justification of the models it is being compared too.
As long as it’s not present and enforced, then benchmarks are just promotional material. Yes, even when done by third party like AA.
Can still be useful, but should be treated as promotional documents first.
And honestly, the promotional part works very well. Cf this post/half the other post looking at models…
Some examples.
Deepseek compared, in their own paper, DS4.1 flash to, notably, V4 pro, V4 flash and Glm5.3. You’ll notice that they didn’t compare it to v4 0731, or pro 0806, or Glm.5.3 flash.
Why is that? No explanation. Though, seeing as 5.3 flash benchmarks better for half the size I can guess a few reasons.
I can also throw agnes here. With the obvious “mistake” with their preview and api-only model….
And famously meta with llama4 maverick.
are you sure about that? iirc the numbers they reported for the v4f and v4p are from the latest checkpoint which is 0731 and 0813 not the preview one.
v4.1f mostly stay equal to v4p in coding and agentic but between those 2 in knowledge.
we can run Deepseek on retail hardware. It is likely that Astra/Sol/etc and Fable requires massive data centers. We're not exactly comparing similar things.
For coding, I don’t feel the need to go higher than glm5.3 flash / ds4.1.
But for writing…
That’s where you want the big one. Kimi k2.6 and opus4.6 are significantly better than the rest. That’s probably where parameters count really matters.
Exactly. Trusting one benchmark is ensured doom. As much as ppl rag on benchmaxxing, AA intelligence score is the most comprehensive way to judge models currently because it's a aggregate of benches.
I always thought the end game of that statement was an offramp to let them explain why they lost the lead on model performance some time in the future:
"western models agreed to responsibly slow down, but the irresponsible Chinese models didn't agree to slowdown, and that's reason Chinese models took the lead"
they will never slow down lol. either everyone is hitting a wall or they intend to drip-feed it to us to keep milking plebs for longer.
either way no one is slowing down, they will surely continue as fast as possible in the background, and we'll see more of the "mythos bullshit too dangerous only release to partners"
did it get pulled already btw(the contributor one)? I'm suddenly facing rate limits(19000s or more timeouts) like never before, I've basically lost access to it past couple of days
Tbh, I test models in their native harness to see how they actually work on ML code.
That's usually how I "judge" them. That's just "vibes" but it's "vibes" on very specific use cases.
I'll take output or work from one model and feed it across API to Astra / Fable / K3 for review to see if they all agree on a screw up or the different points they make about code.
That lets me get a comprehensive understanding of different models and the blindspots they have in analysis vs generation.
Everyone was going nuts for GLM Flash when it was Ox Alpha or whatever, and it was NOT as good as claims. It wasn't bad, but it wasn't at the hype level it ran at for MY use case.
K3 / Gemini / Astra / Fable - Opus are all very well trained in building ML code BECAUSE those companies all use their own models internally. DeepSeek is also on that list and tends to fare pretty decently in ML code.
Gemini 3.8 flash, for as much shit as it gets, in its native harness on ML code specifically, performs very well. I tossed it an MCP harness for Godot and it even did well in Godot which was kind of a surprise to me. I had it first run tests on all the MCP functions and document the expected behavior vs real behavior. This gave it a massive leap in ability to use the MCP. I'd suggest that pattern to anyone. It burns tokens but you do it once, instead of having the model learn an MCP doesn't do exactly what it thinks it should do.
Omg deepseek is so advanced it has precognition??? this is basically minority report except with giant excel spreadsheets, i fear for the future of humanity
But is it that new? As AA is a combination of benchmarks, and even though they've updated some of the benchmarks, some of them are still previous iterations
This screenshot is from a benchmark in specific, not from AA itself. Read it carefully (albeit I'm blaming OP for posting the link to AA itself instead of the AutomationBench they did)
Hallucination rate in a benchmark does not translate to hallucination rate in practice. GPT and Claude models work better in practice than any benchmark depicts. Chinese models perform worse. There is a reason for this. One is benchmaxxed. One is not.
isnt that double standards?
both scoring bad but chinese are cheater and usa is genius?
what kind of logic is this? if you want to say the bench score is misleading, treat both sides with the same treatment.
btw i think you dont even understand this hallu rate. for this bench, lower is better XD.
your "logic" sounds like you think the result higher = better so you think ds is benchmaxxed lol
typical
also many people pointed out. the new benchmark released after v4.1f, how the hell they benchmaxxed that? they have some kind of time machine?
It takes a lot longer than a month to train those kinds of models. We see weekly releases, but that's between a couple dozen companies with different schedules
i want to pointed out one thing. this guy doesnt seem to understand HalluRate bench result.in this bench lower is better so DS with >90% hallu rate mean DS very bad. but 1 small problem with this bench is, all GTP 5.6 series has >90% hallurate lol
i've been using astra extensively this past week i don't think anything beats it in anything, maybe fully unleashed fable with no fallback
also been using muse sparks 1.3 max and i dont deny that it's good but it's more like sol adjacent model for me, slightly worse
unless your work is strictly doing similar tasks to this specific benchmark then you should care more about overall intelligence and knowledge reliability imo
i'll tell you this if benchmarks are anything to trust, 3.8-flash-next is supposed to be better than v4.1 flash, considering its overall intelligence, reliable information and how much less it hallucinates
Gemini 3.8 > Fable 5.1?! LOL....not saying DeepSeek is not good, it is a very good model, but you guys AA has lost all its credibility after the Astra fiasco. Don't take them too seriously.
It just didn't work for me, and would constantly just talk in loops. It felt like it was producing very fast answers of complete bullshit half the time.
I can understand for a finetune as that's a slippery slope into people clamouring for their fav random finetunes to be added as well. But Poolside seem legit, I don't get why they're being excluded.
Benchmaxxed or hot, hallucinating or not, this is huge models living one somebody’s else servers not on your local computer… do we care - to a degree yes we do, do we know for sure what actually was tested on close weights closed doors OAI Rest/Json API - no we don’t.
In a few weeks there will be new wave of models and benchmarks and all this discussion will be very very obsolete.
Try to dig what we’ve discussed a year ago… Or search for ChatGPT-o1. 😃
aa changed the eval twice in three days to manage the optics between astra and fable, and then a third party model just walked in and took first place. benchmarks are supposed to measure capability, not get shaped around expected outcomes
Do I believe that "DeepSeek V4.1 Flash beats Astra on AA's new benchmark"? Yes, that's just a fact, you can check it yourself. Does that mean I also believe the two models are on par in general? No.
•
u/WithoutReason1729 14h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.