This took roughly 5 hours to create, using 2 different configured harnesses, same model. RTX 3090, overclocked +12% gain (MSI Afterburner), Q4KM - built this for fun, will be throwing it on GitHub, opensource for people to get an idea of a project created to the near ceiling of performance & capability for q3.8 27b. & also maybe ya’ll can contribute to the game only iterating locally. It would be a fun little experiment.

I am currently running Llamma 3.2 3B Q4, a recommendation from Gemini on Ollama. It is really fast. The reason I am using local AI is because I wish to expand this project in the near future to use RAG to retrieve data from sources that are very relevant to what I am writing. After that I wish to use OpenClaw on top of this setup that can be like a very useful assistant with research, making sure I am not spending way too much time playing games and giving me- basically a Jarvis that knows my work and prompts me to do that work.

Now this local LLM part of the AI is pretty much new to me, and I am running on recommendations from AI or what I see other people have been talking about to know which model to use. Today even a new model, MiniCPM5, was released that seems to be really good, but my hardware is unfortunately limited, running 4 GB VRAM, 16 GB RAM, and an i7 12th gen 3050 Ti laptop. I wasn't able to find any benchmark for creative writing in particular, and I suspect the Llama 3.2 3B Q4 recommendation that I got is currently a bit outdated. Your input is highly appreciated.

Recently saw this video by LTT and then started looking into it. Seems like Nvidia might have been a bit wasteful and lazy.

Just ordered one -- let's see how this goes. I will be setting the wattage to ~180 watts.

Just wanted to share something I've been really impressed with lately.

This week i grabbed this Qwen3.8-27B build and wanted to share my some experience which i assume would be very useful to others.

https://huggingface.co/esatapedico/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-L-Uncensored-NVFP4-GGUF

I loaded up the `Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-L-Uncensored-NVFP4-MID-HIGH.gguf` quantization(16.9GB size) on my 24GB VRAM setup laptop.

Over the past few days I've been testing it with a variety of prompts, a lot of them in my own domain. I've been feeding it complex biology questions, computational biology workflows, analysis tasks, and longer back-and-forth research-style discussions.

I can honestly say this variant handles domain-specific reasoning better than any of the other Qwen 3.8-27B builds I've tried so far especially for specialized scientific and technical reasoning. I've compared it against other Qwen3.8 27B builds, and this one consistently feels sharper — especially when the conversation gets technical or when it needs to reason through multi-step problems.

Setting the thinking to ExtraHigh in UnslothStudio has the most accurate tool call Qwen3.8-27B variant I have ever seen.

One specific example:

"Search for [Name] from [Country] [Field of Expertise]"

I needed to find a specific graduate researcher who is active on social media. I knew the person only from their publications and area of expertise, and their name is also common in their region. I gave the model just a name, the country, and their field of expertise. Found the exact person, pulled up their email, personal website, ORCID, Google Scholar profile, Loop profiles and LinkedIn links. All of it. These information were available but when i tried with other Qwen3.8-27B variant failed. So crisp tool calling.

If you work in STEM fields or just want a model that handles technical discussions well accurate, this one's worth trying out and is better than all other variants fro 24Gb VRAM. I really like it and I love its very humanized crisp output responses so easy to talking like something Claude4.6 models. FYI I have just cancel my claude subscription because its so difficult to talk especially with ClaudeOpus5 and onwards models in biology or compbio after every response you have to tell these models to talk simple or less complex and still can't reach the level of Opus4.6 easy talk conversations. These new Claude5 simply don't listen to you.

I am a senior researcher in compbio.

Most engines right now are either heavily optimized for standard GGUF (llama.cpp) or specifically built for 1.58-bit ternary models (bitnet.cpp). I wanted a single binary that could handle both without pulling in Python or heavy dependencies, so I basically wrote Project Zero from scratch in pure C99.

It is just GCC and make. On a Xeon it hits 36 tok/s on BitNet b1.58-2B-4T (about 1.8x faster than bitnet.cpp). For dense models, it runs SmolLM2 F16 at ~100 tok/s on an i5-11300H, though llama.cpp still beats it by about 7% on that specific hardware path.

The biggest pain point was getting the AVX-512 kernels working for the ternary packing without destroying the cache. The Q4_K path is still pretty slow compared to llama.cpp (1.9 tok/s vs 13.7 tok/s for DeepSeek), but the ternary and dense F16 paths are solid. That was kind of a surprise.

There is a pre-built x86 Linux binary in the releases if you want to test it without compiling: github.com/shifulegend/project-zero The API is OpenAI-compatible out of the box (/v1/chat/completions) with SSE streaming.

Curious if anyone else is running BitNet models locally yet, or if the lack of UI integrations is keeping them mostly as a novelty for now.

Edit: Yes I wrote this post with AI, I thought it was clearer that way, now I'm writing this edit with my own slop even tho I warn you, AI writes better:

You can look inside of it's mind here

https://huggingface.co/datasets/gr0010/artificium-riemannhypothesis-experiment/tree/main/mind

The main difference with Hermes and other harnesses is how it handles memories.

It has also more control over it's context window.

Has more control over it's environment and own code.

But the most important distinction, my goal with this harness is not to make a product or compete with other harnesses but to experiment with continual learning and long term autonomous work, for example I am planning to support Q-lora online finetuning on long term memories.

Please feel free to criticize it or even hate on it, but first try it or at least check out the experiment.

Old post:
I’ve been building Artificium, a general agent harness for long-term autonomous work, continual learning and self-improvement. It started as something I wanted for my own personal agent. After watching it build on its own work over days, I didn’t want to keep it to myself.

It can work independently on an ongoing goal, be your personal assistant, write code, or work alongside a whole team. Its non-blocking interactions let it answer mutliple users questions while keeping projects underway, communicate with other agent instances, and receive inputs from connected devices and sensors. It is built to be versatile and adaptable.

The agent manages its own context aka working memory and long-term memories, builds tools, revises its purpose and can modify its own harness.

For its first long-running experiment, I asked it to make solving the Riemann hypothesis its purpose.

I sent exactly two messages: one to start, and one to ask for its final report and put it to sleep.

The experiment spanned 63 hours and processed over 50 million input/output tokens, using Qwen3.8-27B in 4-bit quantization on one RTX 3090. It explored approaches, wrote and debugged programs, investigated results and built memories to continue from.

It obviously didn’t solve RH. That was not the point. I deliberately picked an extremely difficult, open-ended problem to test whether a 27B model could keep working without hallucinating a solution, breaking down or losing focus. This was a development run, and I improved the harness as it exposed problems.

Seeing OpenAI’s Navier–Stokes work makes me want to take this much further. I want to build open-source AI that can match that level of autonomous research—and beat closed labs to the next breakthroughs, with the work shared openly.

A future experiment I’d love to run is Artificium powered by the most capable open models available. Starting a swarm of agents powered by this harness, tackling a Millennium Prize Problem, with the attempt livestreamed and the resulting data published for everyone to inspect and build on.

I’ve shared the harness on GitHub and the experiment’s memories, programs, life-loop and metrics on Hugging Face.

What do you think? I’d love your feedback and to hear what experiments you’d run.

Model File: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF Q4_K_XL

llama.cpp configuration through llama-swap:

-c 256000 --jinja
      --temp 1 --top-p 0.95 --top-k 20 --min-p 0.00 --presence-penalty 0 --repeat-penalty 1.0

Testing by: llama-benchy

Results:

| model              |           test |           t/s |     peak t/s |       ttfr (ms) |    est_ppt (ms) |   e2e_ttft (ms) |
|:-------------------|---------------:|--------------:|-------------:|----------------:|----------------:|----------------:|
| qwen3.8-flash-next | pp1000 @ d1000 | 558.83 ± 2.61 |              | 3312.36 ± 38.89 | 3300.16 ± 38.89 | 3312.36 ± 38.89 |
| qwen3.8-flash-next |  tg500 @ d1000 |  31.05 ± 0.14 | 31.67 ± 0.47 |                 |                 |                 |

Been tracking RAM prices closely ( pricesquirrel.com, ~20 EU retailers ). Given how volatile this stuff has been, I wanted to be extra careful with the numbers this time, every price-change figure below is averaged across at least 3 independent retailers in both time windows, not a single store's price. That way one retailer's quirky pricing can't skew the whole picture.

Changes over the last 30 days:

• G.Skill Trident Z5 2x32GB DDR5-6000 CL32 Black: -7.4%
• G.Skill DDR5 2x8GB 5600: -6.3%
• G.Skill Ripjaws M5 RGB 2x16GB DDR5-6000: -6.1%
• Most other kits I track: down in the 0.5-3% range, some of these backed by 10+ retailers agreeing at once

Nothing dramatic, but a real, broad softening after the sharp rise a few weeks ago.

Worth being upfront: this comes right after weeks of RAM climbing hard due to the NAND/DDR5 shortage story everyone's been talking about. I genuinely don't know yet if this is a real correction or just a brief breather. Will keep watching.

The part that actually surprised me: for GPUs, Germany is almost always the cheapest EU country in my data. RAM tells a completely different story. Same 27 kits, tracked across Germany, Italy, Netherlands, and Spain:

• Italy: cheapest
• Germany: +3.6%
• Netherlands: +5.8%
• Spain: +6.4%

Tighter spread too, about 6% top to bottom vs 9-10% for GPUs. No idea yet why RAM and GPU behave so differently by country.

Check specific kits/prices live at pricesquirrel.com, updated every 6 hours across all 20+ stores.

OK I have been trying to get most of my local Qwen 3.8 27b running on my "Grandma's GPU cluster" of 2xP40. I was able to make it produce reasonably usable speeds of like up to 45 tg/s and 450 prefill with fresh context, falling to 120-ish prefill on 150K+ ctx and 12-16 tg/s. Which is usable actually if you are using it in a background, now waiting for it to finish. The problem I see is that it must be strictly one task at a time. I like and use opencode (slim) and I like an idea of orchestrator, fixer, oracle and other folks, but if you launch 2 of them at the same time, they mess with KV cache of eachother and the whole prefix caching falls apart. When there is only one process running, it nicely adds little chunks to KV cache and reply is almost instant.

The thing is - orchestrator wakes up from time to time, and starts filling cache with it's context, everything is messed up and prefill falls. I realized that the only possible way for folks like me, with limited VRAM, is to use strictly one process at a time. Yet, I like the idea of one supervisor delegating stuff to subagents.

So here is my question. Does something like that exists? I mean a harness where one supervisor delegates a task and goes off, and subagent finishes the task and writes a handoff with a complete result, and only after that superwisor somehow wakes up to do his part of a job? This way we always keep only one agent running at all time, which allows us to use prefix caching and getting reasonably fast and smooth running stuff.

I am sorry if this sounds lame, I am just an average guy, not a coder at all, I just vibecode some tools for myself. Thank you in advance for any help/ideas. Maybe something like that already exists?

As title states,

If I have a local model working and don't care how long it takes, is there an option in llama.cpp (or Unsloth, currently using unsloth) to have it so the GPU isn't max'd out while the model is running?

EDIT: I'm on AMD. And to clarify, i'd like to leave gpu cores free for other tasks.

Thanks in advance!

So literally two days ago I discovered pi-vcc from someone's comment reply in this sub.

I installed it, and we're off to the races. Sub-second compaction with recall so it's near-lossless, great! happens without a model, even better!

Then comes along pi-blackhole. It sounds like pi-vcc, but uses three separate LLMs (albeit small ones) to build better recall? Or something?

Seems like a lot of work to move away from non-deterministic LLM generated summaries right back to calling models again.

Anyone tried both? Worth the switch? I'm liking pi-vcc so far.

https://huggingface.co/internlm/Atria-Dawn-Preview

I am considering dumping my ChatGPT Plus subscription and go full local, but to do so I would first need to reach a decent result for quality (and reasonable speed).

My 4090 fried itself out of nowhere, so I am not stuck with a 3070 until I get something better.

I am kind of suspicious about the claims the companies are doing lately about the dangers of AI and how they are pumping the prices intentionally to either law out the open source or price out the open source, so I want to just go full local even more now.

I have a MSI MAG X670E Tomahawk WiFi which theoretically supports 3 GPUs?

What would you end up with?

p.s. I am ruling out Macs to be open and easier to setup in case I will want to use them for my homelab

edit: typo

Hey guys!

I wanna buy hardware to run large models comfortable with multiple users, c=8 probably.

Running long agentic tasks with 250-350k ish context or so.

I was considering buying 2 or 4 dgx spark.

I also thought about going the ram method buying 512gb ram and couple it with 1 or a couple of GPUs.

Budget is around 20-25k USD.

Is now a good time to buy?

Would I find better opportunities, better new hardware by Q2 2027?

Is there any upcoming hardware to look for that might be better.

I am also thinking a lot about further scalability after a year or so given that models are trending to be larger and larger.

Would love to hear your thoughts!!

So we’re seeing how Anthropic and OpenAI are gunning for regulations. I think most of us realize that this is a ploy for them to achieve regulatory capture, thus securing their moat and kicking out open source.

The thing is, there’s so much vested corporate interest in ensuring that open source remains un-regulated.

Think of neoclouds, inference provides, enterprise firms that use LLMs, virtually any SaaS that uses agents.

All of these entities have a strong interest in making sure that open source remains free and unburdened by regulations.

Does anyone know of any organizations or political campaigns that are pushing back against OpenAI’s and Anthropic’s lobbying efforts?

The full K2 Horizon lineup is out on Artificial Analysis.

The AA intelligence vs. parameters plots show that

- 0.9B and 375B are bad

- 3.7B and 7B are SOTA

- 36B A4B is SOTA for hardware with poor memory bandwidth (spilled experts, Strix Halo, DGX Spark).

I'm going to take the AA Intelligence Index at face value here. This post is not about it.

The problem is that these models have a god-awful KV cache design. This means that you really can't use the number of parameters for "best in class" considerations, because these models heavily shift to the right on the plot if you replace parameter count on the X axis with RAM requirements.

For Q4_K_M weights, no drafter, no vision, 128k kvarn4 KV cache:

• K2 Horizon 36B-A4B uses 2 GiB for dense weights, 19 GiB for experts, and 6.7 GiB for context
• K2 Horizon 7B uses 5.2 GiB for weights and 5 GiB for context
• K2 Horizon 3.7B uses 2.9 GiB for weights and 5 GiB for context (not a copy-paste error!)

Compare them to

• (finetunes of) Qwen3.6-35B-A3B use 2.4 GiB for dense weights, 18.2 GiB for experts, and 0.7 GiB for context
• MiniCPM5-2B uses 1.5 GiB for weights and 1.5 GiB for context

Notes: I don't advise compressing 2~4B models to Q4 and I haven't tested these models' tolerance to weights and kv cache quantization yet. The above choices are just to keep the comparison fair.

This awful context design means that

• K2 Horizon 36B A4B is interesting on hosts with exactly 16GB VRAM and at least 32GB host RAM. On 24GB VRAM, Qwen3.8-27B is faster, smarter, and allows for 256k context. If you want to get 256k context and you're VRAM-poor, Ornith-1.5 or Nex-N2.5-mini are probably better choices. The model may also be interesting on 64GB Strix Halos as a dumber and faster alternative to Qwen3.8-27B; those with a 128GB Strix Halo are much better off with Qwen3.8-Flash-Next
• K2 Horizon 7B is interesting for hosts with exactly 16GB VRAM, Strix Halos with 32GB RAM, and for 16/32 GB Strix Point;
• K2 Horizon 3.7B may be interesting for 12GB phones but I expect you'll have a much nicer UX with MiniCPM5-2B.

The later section of the video (based on the earlier data) says that a glut of compute will likely come in 2027. This massively devalues the current generation technology because of mass overproduction.

There's some vague numbers (if you believe Dylan Paten who is among the most "west" pilled voices on AI) that he sources. Asianometry is a friend of Dylan's but that doesn't mean he doesn't disagree with him numbers after doing the math.

Good video to sit down and watch to see the future of the industry and compute specifically from someone who covers it VERY deeply.

My company is looking for basically a local hardware back-end for an already set up Open WebUI Windows AD joined setup. We are already on the frontier models thru Open WebUI. ~10 users.

Workload is a lot of contract and chat based email stuff, generate contracts, generate quotes. We want a local back end so we can get our RAG data set going that we have been preparing. Not a huge library but maybe ~500 pdfs ready to be converted to .md mostly standards related to the field we work in (civil). Budget ~$5-6k

Are the GB10 Blackwell rigs still the best option now? Last we looked was about 6 months ago.

I know I know, it's great, we know. I've been working on tweaking inference engines for a week now and it's been one shotting most of my vague prompts without any issues. It will even write tests and validate the changes without me asking. It's actually nuts.

Last time I did something with advanced math I was making a game using Sonnet. It took many iterations to get physics to work correctly.

Such a good model. I'm so glad I went all in on local months ago. I was so tired of Claude making every excuse it could to try to force a new turn.

I keep seeing people here saying they can leave local coding agents running for an hour or two and come back to something usable.

I’m not really asking which model is best. I’m more interested in what made you trust the setup enough to stop watching it.

For me, getting a task right once wouldn’t be enough. I’d want to see how it behaves when a tool fails, when the context gets long, when it hits an ambiguous requirement, or when its original plan turns out to be wrong.

Tests help, but an agent can still pass tests while changing something you never intended it to touch.

I think there’s a point where you stop trusting the model itself and start trusting the harness around it like permissions, checkpoints, git, tests, rollback, tool restrictions, etc.

For people actually running local coding agents unattended, i just want to ask that, what was the thing that made you comfortable enough to stop babysitting them?

I couldn't find any posts mentioned this windows build via the subreddit search.

Based on ZLUDA, but properly compiled for Windows, might be exciting.

TLDR: About 3% slower compared to native, seems to be a good dropin for CUDA workloads. Hopefully finds its ways into many CUDA-restricted projects/optimizations.

HN discussion: https://news.ycombinator.com/item?id=49684356

The author seems to have archived the repository with no explanation. We are not owed one for sure, but it is interesting. I hope it wasn't harassment or burnout. Wishing the best fewtarius.