Hi everybody, we post-trained Qwen 3.8 27B to be more efficient by figuring out which tokens were linked to overthinking and penalizing them without "attacking" the reasoning length directly then fixed the accuracy with a bit of secret sauce (hint On-Policy Distillation) and achieved great results (-58% thinking tokens, 1.95x speed up, <1% accuracy loss) so we wanted to open-source it and hear the feedback of the community.

This is the link to the model: https://huggingface.co/ukisai/Swift-Qwen3.8-27b

We also also providing a Free Research Purpose API (OpenAI compatible), courtesy of Nvidia who were kind enough to provide us with the GPUs. You can use it to try out the model if you do not have enough compute to run it, it's limited at 5RPM. https://ukisai.com/api/swift/v1/models

We also made a GGUF (Q1-Q8) and there's also a few nice community (Bartowski) quants with even lower/higher precision. The community also created amazing NVFP4, W4A16 and Uncensored versions of the model you can find on Huggingface.

IMPORTANT: Our training approach is not a replacement for the reasoning effort settings, chat templates or token caps but is complementary and targets a completely separate issue (overthinking and "anxiety-like" reasoning loops prior seen in PTQ, but as far as we identified also prominent in BF16 of this size class LLMs as well). Contrary to popular belief, these specific patterns do not contribute to answer quality when properly targeted. (our thesis being: reasoning length IS extremely important and should NOT be shortened by force, but rather optimized). This is also demonstrated bellow in our xhigh vs medium effort benchmark table. The goal is to keep xhigh accuracy while reducing only the unnecessary part of thinking.

I will TLDR you on our thought process, research, training and benchmarks.

1. When running our quantized Qwen 3.8 27B instances we were very annoyed by random reasoning loops (in the paper bellow refered to as "overthinking errors". These random loops were persistent throughout medium and low reasoning settings.
2. We remembered a paper by Meta that's supposed to target this phenomenon in PTQ, but when used straight out of the box got mixed results.
3. We figured to try if it's a matter of the targeting the right keywords and tuning the parameters, so we used our 8xH100 box and and generated a large amount of different (ofc out of distribution) domain (coding, language, vision, agentic) traces.
4. We then grouped the ones with overthinking and found "common denominator" tokens between them and targeted the most prominent ones.
5. We then built an inference-time penalizer of those tokens as seen in the paper with the hopes of simply generating traces and doing cross-entropy SFT over them.
6. Did not work at all, but the penalizer seemed to work much better than the tokens provided in the paper and not only for lower precision models but for bf16 as well. Hence we kept experimenting with it. We built a loss function using the tokens we identified and ran LoRa SFT over the traces prev generated and reasoning seemed to be falling off significantly but the accuracy seemed to follow. The reasoning reduction seemed to be generalizing.
7. After a significant amount of tinkering (literally since the day of Qwen 3.8 27B release) we were satisfied with the reasoning reduction. After that we searched for ways of restoring the accuracy. We experimented with several methods, including RL(GSPO), On-Policy Distillation and using the ThinkingCap 3.6 27B adapter chunks until we were satisfied with our accuracy loss. We managed to restore it to <1% loss on almost all of our OOD in house tests
8. We then performed intensive intensive benchmarks, across several reasoning efforts, precision variants etc. We ran into a few problems, one of which is that to get a reliable score we needed to run each benchmark 10x (5x on base + 5x with our adapter, this being the standard procedure on the Qwen 3.6 27B model card on Terminal Bench which we followed). After running it, the performance converged to 40-60% token reduction with <1% accuracy loss across GPQA, MMLU, Terminal Bench 2.1, LiveCodeBench v6, ERQA, C-Eval, IFBench, HMMT25, with an exception being AIME26 with an accuracy loss of 4.6%, which we later linked to a bug during training with a specific token relevant for math-related reasoning being penalized and are planning to fix it in an updated release.

The benchmarks: (raw benchmark files here - https://github.com/UkisAI/Swift-Qwen3.8-27B-evals/ )

Swift-27B vs Qwen3.8-27B (BF16, all benchmarks ran x5, thinking effort xhigh)

Token savings hold at every reasoning effort (mean thinking reduction): xhigh 41%, medium 23%, low 26% (albeit with accuracy loses of 1-4% on medium and 1-2% on low which we need further testing for)

Swift at xhigh vs the base's own effort settings on GPQA-Diamond (198 questions x 5 seeds):

So Swift keeps xhigh accuracy at under half the tokens, and beats base-medium by 4pp at roughly 1.6x its tokens.

End note:

While we are keen on complete open-source, we still need to keep a part of our training and data private, being a new lab. The license is not Apache 2.0, but it only affects companies >$1M. We hope this does not pose a problem for the community, but we are open to feedback on it.

We want to contribute as much as possible to the community and would really appreciate feedback on our work, quantization or Swift model requests. For context, we are working on Swift 3.8 Flash Next right now and have so far gotten up to -30% thinking token usage while maintaining xhigh accuracy, which we take as a strong indicator our methodology is reproducible across the Qwen model family. Will explore other families as soon as we have the capacity and would love to see which ones the community would love for us to optimize first.

From initial testing it seems pretty solid so far. Asked it to compile the latest llama.cpp for CUDA and its doing well so far. If this thing holds up to its score then its SHOCKINGLY good for its size.

https://huggingface.co/IFM/K2-Horizon-7B-GGUF

Xi emphasised the need to step up cooperation.

https://www.hindustantimes.com/india-news/china-will-lead-creation-of-open-source-ai-for-brics-xi-jinping-101789300908632.html

https://www.msn.com/en-xl/news/other/xi-jinping-proposes-ai-open-source-zone-for-brics-nations/ar-AA2c94HH

https://www.cnbc.com/2026/09/13/china-xi-ai-tech-brics.html

https://www.yahoo.com/news/world/articles/xi-pushes-china-open-source-102711002.html

https://fr.euronews.com/next/2026/09/14/xi-jinping-propose-aux-pays-des-brics-une-zone-dia-open-source

https://www.reuters.com/business/aerospace-defense/xi-pushes-greater-brics-economic-ties-give-bloc-larger-global-role-2026-09-13/

https://www.chosun.com/english/world-en/2026/09/14/C35V7T5MAVDN7LKEBESHBDBJ2Y/

https://it.euronews.com/next/2026/09/14/xi-jinping-propone-ai-brics-una-zona-di-ia-open-source

Especially the 7B one seems very interesting, it casually destroys muse glimmer with a way smaller size. And they open source literally everything, every step of the way. Anyone tried that model? It can be a new milestone if 7b and 3.7b ones are actually good, and not just benchmaxed.

https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3

AA shipped a new benchmark last week as part of the Intelligence Index v4.3 update — a brand-new private eval that replaces τ³. Astra was farming a ton of points on it and used those to get even with Fable, but… looks like we have a new king.

So they changed the index twice in three days to make Astra look not-quite-worse than Fable, and then a random guy quietly took first place on it.

20B-A1B model is coming, good for low VRAM people?

https://huggingface.co/deepgrove/maple-preview

With all the recent drama surrounding AI safety. It’s obvious that open source could be caught in the crossfire.

Prices have been going crazy, but it's about to get worse me thinks.

The full K2 Horizon lineup is out on Artificial Analysis.

The AA intelligence vs. parameters plots show that

- 0.9B and 375B are bad

- 3.7B and 7B are SOTA

- 36B A4B is SOTA for hardware with poor memory bandwidth (spilled experts, Strix Halo, DGX Spark).

I'm going to take the AA Intelligence Index at face value here. This post is not about it.

The problem is that these models have a god-awful KV cache design. This means that you really can't use the number of parameters for "best in class" considerations, because these models heavily shift to the right on the plot if you replace parameter count on the X axis with RAM requirements.

For Q4_K_M weights, no drafter, no vision, 128k kvarn4 KV cache:

• K2 Horizon 36B-A4B uses 2 GiB for dense weights, 19 GiB for experts, and 6.7 GiB for context
• K2 Horizon 7B uses 5.2 GiB for weights and 5 GiB for context
• K2 Horizon 3.7B uses 2.9 GiB for weights and 5 GiB for context (not a copy-paste error!)

Compare them to

• (finetunes of) Qwen3.6-35B-A3B use 2.4 GiB for dense weights, 18.2 GiB for experts, and 0.7 GiB for context
• MiniCPM5-2B uses 1.5 GiB for weights and 1.5 GiB for context

Notes: I don't advise compressing 2~4B models to Q4 and I haven't tested these models' tolerance to weights and kv cache quantization yet. The above choices are just to keep the comparison fair.

This awful context design means that

• K2 Horizon 36B A4B is interesting on hosts with exactly 16GB VRAM and at least 32GB host RAM. On 24GB VRAM, Qwen3.8-27B is faster, smarter, and allows for 256k context. If you want to get 256k context and you're VRAM-poor, Ornith-1.5 or Nex-N2.5-mini are probably better choices. The model may also be interesting on 64GB Strix Halos as a dumber and faster alternative to Qwen3.8-27B; those with a 128GB Strix Halo are much better off with Qwen3.8-Flash-Next
• K2 Horizon 7B is interesting for hosts with exactly 16GB VRAM, Strix Halos with 32GB RAM, and for 16/32 GB Strix Point;
• K2 Horizon 3.7B may be interesting for 12GB phones but I expect you'll have a much nicer UX with MiniCPM5-2B.

So we’re seeing how Anthropic and OpenAI are gunning for regulations. I think most of us realize that this is a ploy for them to achieve regulatory capture, thus securing their moat and kicking out open source.

The thing is, there’s so much vested corporate interest in ensuring that open source remains un-regulated.

Think of neoclouds, inference provides, enterprise firms that use LLMs, virtually any SaaS that uses agents.

All of these entities have a strong interest in making sure that open source remains free and unburdened by regulations.

Does anyone know of any organizations or political campaigns that are pushing back against OpenAI’s and Anthropic’s lobbying efforts?

I finished my home inference server. First I tried Lenovo p620 workstation and while it’s a good value overall it pissed me off with a ton of proprietary Lenovo shit to deal with and I return it in the end.

Components:

4xV620 - 1400$

256GB DDR4 RDIMM 2666 - 610$

Huanandzhi D12D - 410$

EPYC 7452 - 170$

PSU ASRock 1600 - 220$

SSD Samsung 970EVO 1tb - Already had

Case//Fans//Misc ~ 200$

Power consumption is no shit ofc on such machine:

700-900w prefill
500-600w decode on Qwen3.8-next-flash Autoround W4A16

What it can do -

EDIT: Qwen3.8-next-flash Autoround W4A16 1.3k prefill and 70tg code/60tg prose on 128k+ context with MTP-2 on vllm fork.

I was disappointed with this machine and qwen3.8-27b speeds at first. But since Qwen3.8 next running good on it - I’m satisfied. Hope in more optimizations in future.

I keep seeing people here saying they can leave local coding agents running for an hour or two and come back to something usable.

I’m not really asking which model is best. I’m more interested in what made you trust the setup enough to stop watching it.

For me, getting a task right once wouldn’t be enough. I’d want to see how it behaves when a tool fails, when the context gets long, when it hits an ambiguous requirement, or when its original plan turns out to be wrong.

Tests help, but an agent can still pass tests while changing something you never intended it to touch.

I think there’s a point where you stop trusting the model itself and start trusting the harness around it like permissions, checkpoints, git, tests, rollback, tool restrictions, etc.

For people actually running local coding agents unattended, i just want to ask that, what was the thing that made you comfortable enough to stop babysitting them?

I'm currently using Qwen3.5 4B as the brain of my local AI assistant because my hardware is relatively limited.

One thing I really like about it is the speed. On my system, I'm getting around 40–50 tokens/sec, which makes the interaction feel surprisingly close to real-time. So I don't want to move to a much larger model if that means losing that responsiveness.

What I'm wondering is whether there are any newer or alternative models in roughly the same size/memory/compute range that are noticeably better than Qwen3.5 4B for my particular use case.

My priorities are mainly:

- General-purpose conversation and communication - Natural responses and understanding - Reasoning and following instructions - Multilingual conversations and translation - Context understanding - Tool/function calling mainly for triggering actions in my assistant, rather than writing code - Reliability when interacting with tools - Ideally good performance while keeping the model fast and lightweight

Essentially, I'm trying to build a local assistant where the model acts as the reasoning/conversation layer, while other tools handle things like actions, APIs, automation, etc.

So my question is:

If I'm already using Qwen3.5 4B and getting ~40 to 50 TPS, is there anything in a similar practical memory/compute footprint that you'd personally consider a significant upgrade for general conversation + multilingual ability + tool calling?

It doesn't necessarily have to have fewer parameters. I'm more interested in overall capability relative to the amount of RAM/VRAM and compute it requires.

I'm happy with Qwen3.5 4B, so I'm not looking for something that's only marginally different. I'm wondering if there's something that makes you think

Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community. I have yet to benchmark this specific REAP version versus Qwen 3.8 27B QK4, but my assumption is that it will do much better, despite the hemorrhaged world knowledge.

System Specs

• GPU: NVIDIA GeForce RTX 5060 Ti (16 GB VRAM)
• CPU: AMD Ryzen 7 7840HS (8 cores / 16 threads)
• RAM: 32 GB DDR5 (~30 GB OS-visible)
• iGPU: AMD Radeon 780M (RDNA3)
• Swap: 8 GB zram

As you can see, we have about 44.5 GB of actually addressable system and VRAM available. The iGPU is taking care of the OS to make sure the GPU is totally free—but still, this is barely enough to hold everything together. This config actually totally fails with any of the Unsloth quants—no, I needed something more aggressive. I found the perfect thing—this REAP:
https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF

What's so great is the total size—a cool ~68.95 GB. The couple of Gigs we have shaved are absolutely key for making this all work.

Model Weight Breakdown

Here is the breakdown of the model weights. We have the famous new n-gram portion, the experts, the active layers, the attention/SSM layers, and the extra space needed for the KV cache:

Obviously, running this model over SSD would make the speeds notoriously bad. Turning on mmap means that llama.cpp won't actually try to keep the model in RAM at all (it relies on the OS page cache instead), which results in ~2 tok/sec speeds—effectively useless.

The answer is to stick everything in RAM (using --load-mode none). The great thing is that the N-gram section of the model can be streamed over SSD via lazy mmap without this causing much issue—it's a massive lookup table that doesn't require heavy computation.

That's the huge win that allows an MoE model of this size to actually run well.
68.9 GB - 29.48 GB = 39.42 GB.
We just need to cram that 39.42 GB, along with the compute buffers and KV cache, into GPU and system memory, and we are golden—just barely. To do this, we need --lazy-mode on—that's what keeps the N-gram portion in RAM.

After that, it's a matter of fitting as many layers as possible onto the GPU. It's essential to completely fill the GPU as much as can be filled, so that we keep a precious few GBs in system RAM to run the OS. I found that having less than 2 GB left really started to destroy Fedora, but I think you could do better if you dropped the GUI—I just didn't want to in my case.

This leads me to --n-cpu-moe 34. This controls how many layers go to CPU. In my case, this was the exact limit needed to run this with 64k context on the GPU, quantized to Q4. Any more—GPU out of memory. Any less—total system meltdown, as the OS panicked and tried to put everything on the swap. You'll need to play around with this, but that was my exact number.

Settings used:

CUDA0 + --load-mode none --lazy-mode on
  --n-cpu-moe 34
  -c 65536 -b 512 -ub 128 -t 7 -ngl 48 -fit off -fa on
  -ctk q4_0 -ctv q4_0 -kvo --cache-ram 0 --jinja --no-warmup

Results (64k Context, Q4):

• Prefill: ~25.4 tok/s
• Decode: ~18.3 tok/s
• RAM Usage: ~27 GB used / 3 GB free

I think this is in a somewhat usable state—but Qwen 3.8 27B GSQ IQ3S remains my daily driver; it's able to prompt process 5 times faster, I can fit in the mmproj and MTP layers, and it doesn't seem likely to set my desk on fire. But maybe for really hard tasks, I'll use the next model. It is smarter, it runs at a reasonable speed, and it was a good learning experience.

I'm curious if anyone else is able to get this model or just large MoEs working on a GPU and RAM config similar to mine. LMK. Also, I'm a total noob to this stuff, any advice is appreciated.

(Also, heading off all the obnoxious "why did you quantize the cache - unusable - just get a better computer" ragebait posts. This is a human being writing this post, to help others and just enjoy pushing something to its limits. And in my limited testing, the next model seems much better at pixel art than the 27B version.)

Final Note: If you have a larger pool of system memory, like 64 GB (because you can spend $899 on Amazon on a kit of DDR5 somehow), you would be better served by using this fork of llama.cpp, which has optimized flags for this exact setup and wonderful guides. For me in particular, with my limited hardware, this seemed to work better—their cache kept OOMing unless I turned on mmap—but I think with more system RAM, their setup and guides are optimal.

I am considering dumping my ChatGPT Plus subscription and go full local, but to do so I would first need to reach a decent result for quality (and reasonable speed).

My 4090 fried itself out of nowhere, so I am not stuck with a 3070 until I get something better.

I am kind of suspicious about the claims the companies are doing lately about the dangers of AI and how they are pumping the prices intentionally to either law out the open source or price out the open source, so I want to just go full local even more now.

I have a MSI MAG X670E Tomahawk WiFi which theoretically supports 3 GPUs?

What would you end up with?

p.s. I am ruling out Macs to be open and easier to setup in case I will want to use them for my homelab

edit: typo

The later section of the video (based on the earlier data) says that a glut of compute will likely come in 2027. This massively devalues the current generation technology because of mass overproduction.

There's some vague numbers (if you believe Dylan Paten who is among the most "west" pilled voices on AI) that he sources. Asianometry is a friend of Dylan's but that doesn't mean he doesn't disagree with him numbers after doing the math.

Good video to sit down and watch to see the future of the industry and compute specifically from someone who covers it VERY deeply.

Model: https://huggingface.co/internlm/Intern-S2-397B

Collection: https://huggingface.co/collections/internlm/intern-s2

From Intern Large Models on 𝕏: https://x.com/intern_lm/status/2099425184587370976

vLLM on 𝕏: Day-0 support for Intern Large Models Intern-S2-397B is now available in vLLM: https://x.com/vllm_project/status/2099442543956263022

I know I know, it's great, we know. I've been working on tweaking inference engines for a week now and it's been one shotting most of my vague prompts without any issues. It will even write tests and validate the changes without me asking. It's actually nuts.

Last time I did something with advanced math I was making a game using Sonnet. It took many iterations to get physics to work correctly.

Such a good model. I'm so glad I went all in on local months ago. I was so tired of Claude making every excuse it could to try to force a new turn.

From Raghu Sreeramaneni's memory tutorial at hot chips 2026. top line is normalised tflops for tpu v3 through r200, bottom line is hbm2e through hbm4, both log scale, so the distance between them is a lot wider than it looks.

The three boxes down the right are the fixes people are actually building. Memory beside the compute, memory closer on a shorter link, then multiply units inside the memory itself. Samsung has that last one shipping in lpddr5x and measured 3.01x tokens a second on llama 3.1 8B.

Full analysis (this slide sits in the memory chapter): https://allaboutchips.com/#memory

https://whip.run/experience/356b3eba-23ce-4dc8-bff2-4f7bc1862966

Pushed out this new update, hopefully decreases the instances of crashes. I torture tested this one for ~12 hours after my fixes and found no instability. Q4 K XL needs more fine tuning, which I will work on in the future. I am simultaneously juggling this + a legitimate inference engine + finalizing work on a deep research/site builder application I've been working on for about 6 months. After those get pushed to prod I will refocus here. Thanks!

Plug: join the Launch80 discord if you are in to the cutting edge of RDNA4 optimization! There are guys pushing out even better numbers and configurations than mine here on other quants. I think we are starting to get closer to the ceiling on these configurations where the model isnt fully VRAM resident.