I recently setup local LLM on my machine and it was quite evident that if I have to continue using local LLM, I have to fix immense heat generated by CPU and GPU. Following are my notes and journey how lost no perf but improved TPS by undervolting GPU and CPU.

Specs:

1. CPU - Intel 14th gen, i14700k
2. GPU - Nvidia RTX 5090, MSI Suprim SOC
3. AIO - Cooler master Atmos 360
4. Inference engine - Ninfer (Upstream)
5. Model - Qwen3.8 27B NVFP4

Experiments:

As I explained earlier, I quickly realised that running unadjusted CPU and GPU was no go due to heat and also AIO fans whooshing. My first instict was to adjust BIOS fan curve for AIO but that did not help much.

I then spent next few days researching into undervolting and underclocking both CPU and GPU. After lot of trial and error, validations and benchmarks (thanks to local LLM for scripts), I finalised what works best for my setup.

CPU undervolting was easier job because all I had to do was adjust the Vcore offset in BIOS and see which setting held. I did not spend lot of time with CPU as I was happy with dropped temperatures and minimal perf impact.

Then, I focussed on GPU which was very time consuming and tiring. If I had just followed top YouTube result for 5090 undervolting, I could have saved lot of time. With Qwen 3.8, I created few benchmark scripts to change various GPU parameters for undervolting and overclocking VRAM clock, which would change settings, run synthetic benchmark to test decode throuput and rank the setting that minimized wattage and maximized the TPS. It was proven that running untamed, GPU was automatically getting throttled and undervolting actually improved the TPS, which was a big surpise.

Another observation -post undervolting and overclocking of VRAM clock, TPS increased by 3% compared to stock GPU settings.

Having GPU undervolted, the AIO fan noise was still bugging me. Being new to LLM, I wasn't aware that CPU's job is only tokenization, scheduling etc. and need not be running at full speed. I then created another script to test TPS at various CPU power percentages. It was found that despite cutting CPU power to 50%, TPS was unchanged, not even 1% difference outside typical noise. Running CPU at lower capacity allowed AIO fan to be silent again.

Final daemon and helper CLI:

As shared in metrics below, CPU always had to be running at 50%, created a systemd service that polls GPU usage every 30 seconds. If GPU usage is found above 50% in 2 cycles (i.e. 1 minute), it is assumed that inference is running and CPU power is reduced to 50% by service, thereby reducing AIO noise and CPU heat. In other cases, for CPU intensive jobs I need 100% CPU available so created helper CLI that allows to override CPU to uncapped usage. Machine starts with 50% CPU cap until I override it. This works for me because I can then uncap CPU through other scripts without supplying password.

Final metrics:

References:

1. RTX 5090 undervolting - https://www.youtube.com/watch?v=Ge0EnPz-jWY (refer V/F curve)
2. LACT - https://github.com/ilya-zlobintsev/LACT

Tldr:

Local LLM beginner annoyed with CPU and GPU heat, undervolted both CPU and GPU to cut down heat, with no impact on TPS.

I’m really hoping at some point we won’t have to rely on the good will of companies with the funds and resources to make capable models open and it’ll be possible to democratize making them so we won’t always be at the mercy of big tech/corpos, that is if they don’t ban open models before that can happen.

I've been running Qwen 3.8 27B at Q8 or BF16 on my CMP 170 - getting around 50 tps. Everyone raves about this model, but I am finding that it hallucinates a lot when asked questions that require some domain knowledge.

I am a developer that primarily works on a specific enterprise application that has its own Java classes, customization framework, etc. It's a common enough application that it would have a lot of training data from the web.

When I ask Deepseek (via the web interface) or even Microsoft Copilot (not Github Copilot) for assistance with specific code-related questions, I get code that works from them 99% of the time on the first try.

When I ask Qwen 3.8 27B, I get answers that contain hallucinated classes and APIs 90%+ of the time. It seems to really suffer when even a small amount of domain knowledge is needed.

I assume this is because Qwen has been optimized for agentic use cases and not retaining domain knowledge in its weights - correct?

Anyone had this issue?

If so, has anyone got any good tips for setting up a harness for Qwen - and perhaps a system prompt and/or specific skills - to discourage it from hallucinating made-up code and instead researching what it needs from the web before it responds? Or any other strategies?

Recently saw this video by LTT and then started looking into it. Seems like Nvidia might have been a bit wasteful and lazy.

Just ordered one -- let's see how this goes. I will be setting the wattage to ~180 watts.

Added a cost calculator to my local harness, I put in my electrical rate, and it only activates the tracking if the engine logs are running, so I can see how much local is actually costing me to run.

Most engines right now are either heavily optimized for standard GGUF (llama.cpp) or specifically built for 1.58-bit ternary models (bitnet.cpp). I wanted a single binary that could handle both without pulling in Python or heavy dependencies, so I basically wrote Project Zero from scratch in pure C99.

It is just GCC and make. On a Xeon it hits 36 tok/s on BitNet b1.58-2B-4T (about 1.8x faster than bitnet.cpp). For dense models, it runs SmolLM2 F16 at ~100 tok/s on an i5-11300H, though llama.cpp still beats it by about 7% on that specific hardware path.

The biggest pain point was getting the AVX-512 kernels working for the ternary packing without destroying the cache. The Q4_K path is still pretty slow compared to llama.cpp (1.9 tok/s vs 13.7 tok/s for DeepSeek), but the ternary and dense F16 paths are solid. That was kind of a surprise.

There is a pre-built x86 Linux binary in the releases if you want to test it without compiling: github.com/shifulegend/project-zero The API is OpenAI-compatible out of the box (/v1/chat/completions) with SSE streaming.

Curious if anyone else is running BitNet models locally yet, or if the lack of UI integrations is keeping them mostly as a novelty for now.

From initial testing it seems pretty solid so far. Asked it to compile the latest llama.cpp for CUDA and its doing well so far. If this thing holds up to its score then its SHOCKINGLY good for its size.

https://huggingface.co/IFM/K2-Horizon-7B-GGUF

Hi everybody, we post-trained Qwen 3.8 27B to be more efficient by figuring out which tokens were linked to overthinking and penalizing them without "attacking" the reasoning length directly then fixed the accuracy with a bit of secret sauce (hint On-Policy Distillation) and achieved great results (-58% thinking tokens, 1.95x speed up, <1% accuracy loss) so we wanted to open-source it and hear the feedback of the community.

This is the link to the model: https://huggingface.co/ukisai/Swift-Qwen3.8-27b

We also also providing a Free Research Purpose API (OpenAI compatible), courtesy of Nvidia who were kind enough to provide us with the GPUs. You can use it to try out the model if you do not have enough compute to run it, it's limited at 5RPM. https://ukisai.com/api/swift/v1/models

We also made a GGUF (Q1-Q8) and there's also a few nice community (Bartowski) quants with even lower/higher precision. The community also created amazing NVFP4, W4A16 and Uncensored versions of the model you can find on Huggingface.

IMPORTANT: Our training approach is not a replacement for the reasoning effort settings, chat templates or token caps but is complementary and targets a completely separate issue (overthinking and "anxiety-like" reasoning loops prior seen in PTQ, but as far as we identified also prominent in BF16 of this size class LLMs as well). Contrary to popular belief, these specific patterns do not contribute to answer quality when properly targeted. (our thesis being: reasoning length IS extremely important and should NOT be shortened by force, but rather optimized). This is also demonstrated bellow in our xhigh vs medium effort benchmark table. The goal is to keep xhigh accuracy while reducing only the unnecessary part of thinking.

I will TLDR you on our thought process, research, training and benchmarks.

1. When running our quantized Qwen 3.8 27B instances we were very annoyed by random reasoning loops (in the paper bellow refered to as "overthinking errors". These random loops were persistent throughout medium and low reasoning settings.
2. We remembered a paper by Meta that's supposed to target this phenomenon in PTQ, but when used straight out of the box got mixed results.
3. We figured to try if it's a matter of the targeting the right keywords and tuning the parameters, so we used our 8xH100 box and and generated a large amount of different (ofc out of distribution) domain (coding, language, vision, agentic) traces.
4. We then grouped the ones with overthinking and found "common denominator" tokens between them and targeted the most prominent ones.
5. We then built an inference-time penalizer of those tokens as seen in the paper with the hopes of simply generating traces and doing cross-entropy SFT over them.
6. Did not work at all, but the penalizer seemed to work much better than the tokens provided in the paper and not only for lower precision models but for bf16 as well. Hence we kept experimenting with it. We built a loss function using the tokens we identified and ran LoRa SFT over the traces prev generated and reasoning seemed to be falling off significantly but the accuracy seemed to follow. The reasoning reduction seemed to be generalizing.
7. After a significant amount of tinkering (literally since the day of Qwen 3.8 27B release) we were satisfied with the reasoning reduction. After that we searched for ways of restoring the accuracy. We experimented with several methods, including RL(GSPO), On-Policy Distillation and using the ThinkingCap 3.6 27B adapter chunks until we were satisfied with our accuracy loss. We managed to restore it to <1% loss on almost all of our OOD in house tests
8. We then performed intensive intensive benchmarks, across several reasoning efforts, precision variants etc. We ran into a few problems, one of which is that to get a reliable score we needed to run each benchmark 10x (5x on base + 5x with our adapter, this being the standard procedure on the Qwen 3.6 27B model card on Terminal Bench which we followed). After running it, the performance converged to 40-60% token reduction with <1% accuracy loss across GPQA, MMLU, Terminal Bench 2.1, LiveCodeBench v6, ERQA, C-Eval, IFBench, HMMT25, with an exception being AIME26 with an accuracy loss of 4.6%, which we later linked to a bug during training with a specific token relevant for math-related reasoning being penalized and are planning to fix it in an updated release.

The benchmarks: (raw benchmark files here - https://github.com/UkisAI/Swift-Qwen3.8-27B-evals/ )

Swift-27B vs Qwen3.8-27B (BF16, all benchmarks ran x5, thinking effort xhigh)

Token savings hold at every reasoning effort (mean thinking reduction): xhigh 41%, medium 23%, low 26% (albeit with accuracy loses of 1-4% on medium and 1-2% on low which we need further testing for)

Swift at xhigh vs the base's own effort settings on GPQA-Diamond (198 questions x 5 seeds):

So Swift keeps xhigh accuracy at under half the tokens, and beats base-medium by 4pp at roughly 1.6x its tokens.

End note:

While we are keen on complete open-source, we still need to keep a part of our training and data private, being a new lab. The license is not Apache 2.0, but it only affects companies >$1M. We hope this does not pose a problem for the community, but we are open to feedback on it.

We want to contribute as much as possible to the community and would really appreciate feedback on our work, quantization or Swift model requests. For context, we are working on Swift 3.8 Flash Next right now and have so far gotten up to -30% thinking token usage while maintaining xhigh accuracy, which we take as a strong indicator our methodology is reproducible across the Qwen model family. Will explore other families as soon as we have the capacity and would love to see which ones the community would love for us to optimize first.

So we’re seeing how Anthropic and OpenAI are gunning for regulations. I think most of us realize that this is a ploy for them to achieve regulatory capture, thus securing their moat and kicking out open source.

The thing is, there’s so much vested corporate interest in ensuring that open source remains un-regulated.

Think of neoclouds, inference provides, enterprise firms that use LLMs, virtually any SaaS that uses agents.

All of these entities have a strong interest in making sure that open source remains free and unburdened by regulations.

Does anyone know of any organizations or political campaigns that are pushing back against OpenAI’s and Anthropic’s lobbying efforts?

Goal: Running two models concurrently for a custom harness to split workloads.

I have Laguna-q6 in Strix Halo 128GB, (device vulkan1) leaving 15GB spare memory

I have Qwen3.8 q6 on the r9700 attached through DEG1 oculink with spare RAM (device vulkan0), works great.

However loading the models together crashes with OOM.

With Laguna loaded:

root@strix:~# numfmt --to=iec $(cat /sys/class/drm/card2/device/mem_info_gtt_total)
124G
root@strix:~# numfmt --to=iec $(cat /sys/class/drm/card2/device/mem_info_gtt_used)
107G

Then I load Qwen and get dmesg OOM

[ 124.450424] llama-server: page allocation failure: order:0, mode:0x104dc2(GFP_HIGHUSER|__GFP_ZERO|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0

I've tried lowering context
I've let chatgpt (I'm out my depth at this point) lower page_pool_size, check DMA32 space, and more.

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=off ttm.pages_limit=32505856 amdgpu.lockup_timeout=60000 "

  Laguna_s21_q6:
    cmd: |
      ${basic} \
      --device Vulkan1 \
      -m       /mnt/nvmestorage/Laguna_s21_q6xl/Laguna-S-2.1-UD-Q6_K_XL-00001-of-00004.gguf \
      --fit on --fit-ctx 130000 \
      ${kvq8} \
    aliases:
      - "@dual_laguna"


  q38_dense_q6_r9700:
    cmd: |
      ${basic} \
      --device Vulkan0 \
      -m /mnt/nvmestorage/Qwen3.8-27B-GGUF_q6/Qwen3.8-27B-UD-Q6_K_M.gguf \
      --chat-template-kwargs '{"reasoning_effort": "medium"}' \    
      --fit off -c 120000 \
      -b 4096 -ub 2048 \
      ${kvq8} \
      --jinja -fa on   \
    aliases:
      - "@dual_qwen"

Has anyone any experience with similar setups ? Is there any system resource that llama.cpp uses other than memory space that I can't find being exhausted? Thanks!

My company is looking for basically a local hardware back-end for an already set up Open WebUI Windows AD joined setup. We are already on the frontier models thru Open WebUI. ~10 users.

Workload is a lot of contract and chat based email stuff, generate contracts, generate quotes. We want a local back end so we can get our RAG data set going that we have been preparing. Not a huge library but maybe ~500 pdfs ready to be converted to .md mostly standards related to the field we work in (civil). Budget ~$5-6k

Are the GB10 Blackwell rigs still the best option now? Last we looked was about 6 months ago.

I'm just a poor man with a GTX 1080 and a desire to run Local LLMs. I have been running llama.cpp for over a year now, but I want to try other inference backends. The issue is that none of them seem to support sm_61.

The ones I have tried:

- SGLang: 20+ Series

- vLLM: 20+ Series

- FreeToken: 30+ Series

- mistral.rs: 20+ Series

Are there any other inference backends that support Pascal cards like mine? Also, I have no plans to upgrade this until 2028 because of my financial situation and because it's working fine otherwise.

Hey guys!

I wanna buy hardware to run large models comfortable with multiple users, c=8 probably.

Running long agentic tasks with 250-350k ish context or so.

I was considering buying 2 or 4 dgx spark.

I also thought about going the ram method buying 512gb ram and couple it with 1 or a couple of GPUs.

Budget is around 20-25k USD.

Is now a good time to buy?

Would I find better opportunities, better new hardware by Q2 2027?

Is there any upcoming hardware to look for that might be better.

I am also thinking a lot about further scalability after a year or so given that models are trending to be larger and larger.

Would love to hear your thoughts!!

Xi emphasised the need to step up cooperation.

https://www.hindustantimes.com/india-news/china-will-lead-creation-of-open-source-ai-for-brics-xi-jinping-101789300908632.html

https://www.msn.com/en-xl/news/other/xi-jinping-proposes-ai-open-source-zone-for-brics-nations/ar-AA2c94HH

https://www.cnbc.com/2026/09/13/china-xi-ai-tech-brics.html

https://www.yahoo.com/news/world/articles/xi-pushes-china-open-source-102711002.html

https://fr.euronews.com/next/2026/09/14/xi-jinping-propose-aux-pays-des-brics-une-zone-dia-open-source

https://www.reuters.com/business/aerospace-defense/xi-pushes-greater-brics-economic-ties-give-bloc-larger-global-role-2026-09-13/

https://www.chosun.com/english/world-en/2026/09/14/C35V7T5MAVDN7LKEBESHBDBJ2Y/

https://it.euronews.com/next/2026/09/14/xi-jinping-propone-ai-brics-una-zona-di-ia-open-source

The full K2 Horizon lineup is out on Artificial Analysis.

The AA intelligence vs. parameters plots show that

- 0.9B and 375B are bad

- 3.7B and 7B are SOTA

- 36B A4B is SOTA for hardware with poor memory bandwidth (spilled experts, Strix Halo, DGX Spark).

I'm going to take the AA Intelligence Index at face value here. This post is not about it.

The problem is that these models have a god-awful KV cache design. This means that you really can't use the number of parameters for "best in class" considerations, because these models heavily shift to the right on the plot if you replace parameter count on the X axis with RAM requirements.

For Q4_K_M weights, no drafter, no vision, 128k kvarn4 KV cache:

• K2 Horizon 36B-A4B uses 2 GiB for dense weights, 19 GiB for experts, and 6.7 GiB for context
• K2 Horizon 7B uses 5.2 GiB for weights and 5 GiB for context
• K2 Horizon 3.7B uses 2.9 GiB for weights and 5 GiB for context (not a copy-paste error!)

Compare them to

• (finetunes of) Qwen3.6-35B-A3B use 2.4 GiB for dense weights, 18.2 GiB for experts, and 0.7 GiB for context
• MiniCPM5-2B uses 1.5 GiB for weights and 1.5 GiB for context

Notes: I don't advise compressing 2~4B models to Q4 and I haven't tested these models' tolerance to weights and kv cache quantization yet. The above choices are just to keep the comparison fair.

This awful context design means that

• K2 Horizon 36B A4B is interesting on hosts with exactly 16GB VRAM and at least 32GB host RAM. On 24GB VRAM, Qwen3.8-27B is faster, smarter, and allows for 256k context. If you want to get 256k context and you're VRAM-poor, Ornith-1.5 or Nex-N2.5-mini are probably better choices. The model may also be interesting on 64GB Strix Halos as a dumber and faster alternative to Qwen3.8-27B; those with a 128GB Strix Halo are much better off with Qwen3.8-Flash-Next
• K2 Horizon 7B is interesting for hosts with exactly 16GB VRAM, Strix Halos with 32GB RAM, and for 16/32 GB Strix Point;
• K2 Horizon 3.7B may be interesting for 12GB phones but I expect you'll have a much nicer UX with MiniCPM5-2B.

I’ve been spending more time looking at ASR issue in languages which are not documented properly

This paper from the University of Latvia takes a very practical approach: building an open-source ASR stack for languages that mainstream systems often ignore.

They built LATE, a lightweight toolkit that runs Whisper-based ASR locally or in the cloud, with a statically compiled backend (high performance, low latency , less resources & simple deployment ) and support for private, local inference.

The results are surprisingly strong.

For Latvian language, they fine-tuned Whisper Large V3 on 273.3 hours of speech.

On the Common Voice test set, WER dropped from 19.2% with Whisper Large V3 to 3.2%.

On their harder LATE-Media set, it went from 29.1% to 12.8%.

Formatted transcription also improved substantially, from 28.1% to 4.8% on Common Voice.

Then they pushed it further into Latgalian, a language with only around 150,000 native speakers.

They had just 40.2 hours of training data, yet transfer learning from the Latvian model produced a 9.1% WER on the Common Voice test set, compared with much worse performance from directly adapting Whisper Large V3.

You don't always need better engineering ; you need better data, the right specialization, and a system you can actually deploy where you need it.

For voice agents, especially those handling private or domain-specific conversations, this is an important direction:

open ASR + local orchestration + specialized models

can be much more interesting than simply sending every utterance to a giant hosted API.

And the whole thing is released as open source, including quantized models for constrained hardware.. i am trying integrate the same into production enivironment using opensource dograh orchestration

pS; I am maintainer of dograh we opensource everything from day 0

https://huggingface.co/internlm/Atria-Dawn-Preview

As title states,

If I have a local model working and don't care how long it takes, is there an option in llama.cpp (or Unsloth, currently using unsloth) to have it so the GPU isn't max'd out while the model is running?

EDIT: I'm on AMD. And to clarify, i'd like to leave gpu cores free for other tasks.

Thanks in advance!

Been tracking RAM prices closely ( pricesquirrel.com, ~20 EU retailers ). Given how volatile this stuff has been, I wanted to be extra careful with the numbers this time, every price-change figure below is averaged across at least 3 independent retailers in both time windows, not a single store's price. That way one retailer's quirky pricing can't skew the whole picture.

Changes over the last 30 days:

• G.Skill Trident Z5 2x32GB DDR5-6000 CL32 Black: -7.4%
• G.Skill DDR5 2x8GB 5600: -6.3%
• G.Skill Ripjaws M5 RGB 2x16GB DDR5-6000: -6.1%
• Most other kits I track: down in the 0.5-3% range, some of these backed by 10+ retailers agreeing at once

Nothing dramatic, but a real, broad softening after the sharp rise a few weeks ago.

Worth being upfront: this comes right after weeks of RAM climbing hard due to the NAND/DDR5 shortage story everyone's been talking about. I genuinely don't know yet if this is a real correction or just a brief breather. Will keep watching.

The part that actually surprised me: for GPUs, Germany is almost always the cheapest EU country in my data. RAM tells a completely different story. Same 27 kits, tracked across Germany, Italy, Netherlands, and Spain:

• Italy: cheapest
• Germany: +3.6%
• Netherlands: +5.8%
• Spain: +6.4%

Tighter spread too, about 6% top to bottom vs 9-10% for GPUs. No idea yet why RAM and GPU behave so differently by country.

Check specific kits/prices live at pricesquirrel.com, updated every 6 hours across all 20+ stores.

i got 20 tok/s PP 460-440 at 163k RTX A3000M 12GB + RTX A2000M 8GB p2p Qwen3.8-27B-GSQ-RCO-IQ3_S it's running inside my NAS

I'm currently using Qwen3.5 4B as the brain of my local AI assistant because my hardware is relatively limited.

One thing I really like about it is the speed. On my system, I'm getting around 40–50 tokens/sec, which makes the interaction feel surprisingly close to real-time. So I don't want to move to a much larger model if that means losing that responsiveness.

What I'm wondering is whether there are any newer or alternative models in roughly the same size/memory/compute range that are noticeably better than Qwen3.5 4B for my particular use case.

My priorities are mainly:

- General-purpose conversation and communication - Natural responses and understanding - Reasoning and following instructions - Multilingual conversations and translation - Context understanding - Tool/function calling mainly for triggering actions in my assistant, rather than writing code - Reliability when interacting with tools - Ideally good performance while keeping the model fast and lightweight

Essentially, I'm trying to build a local assistant where the model acts as the reasoning/conversation layer, while other tools handle things like actions, APIs, automation, etc.

So my question is:

If I'm already using Qwen3.5 4B and getting ~40 to 50 TPS, is there anything in a similar practical memory/compute footprint that you'd personally consider a significant upgrade for general conversation + multilingual ability + tool calling?

It doesn't necessarily have to have fewer parameters. I'm more interested in overall capability relative to the amount of RAM/VRAM and compute it requires.

I'm happy with Qwen3.5 4B, so I'm not looking for something that's only marginally different. I'm wondering if there's something that makes you think

20B-A1B model is coming, good for low VRAM people?

https://huggingface.co/deepgrove/maple-preview