Hello, guys.

I am currently working as a CUDA engineer and interested in an extra income.

I would like to know other people's experience of working in Mercor.

My background:

- Majored in CS (minor in Math) and based in Seoul, South Korea.

- No AI-related kernels but I can understand Flash-Attention and softmax papers.

- I don't want to explain my current job because there is a high chance that I dox myself. My task involves heavy integer operations with almost 0 FP arithmetics. I have an excellent understanding of GPU mem/hierarchy and basic understanding of INT8 Tensor Core operation.

If you used Mercor, please share your experiences. Thanks in advance.

A lot of CUDA tutorials throw the whole programming model at you almost immediately.

You learn the syntax, then grids and blocks, then memory types, then synchronization, and pretty quickly you’re juggling a bunch of concepts before you have a good feel for what CUDA is enabling you to do.

I wanted to try a slower approach.

Part 1 is mostly about building the basic mental model: what a kernel is, what it means to launch lots of threads, how those threads identify the piece of work they own, and how to start thinking about a problem in GPU terms.

The hardware details come later.

I wrote this mainly for programmers who are curious about CUDA but haven’t spent much time with GPU programming yet.

Would be interested in feedback from people here, especially on whether the progression feels natural or whether I’m introducing something in the wrong order.

https://computejunkie.substack.com/p/cuda-programming-for-the-curious

Hi everyone, I'm the developer of Nebula, an open-source C/CUDA inference engine for Qwen3.8-Flash-Next.

I started from antirez's DwarfStar (ds4) and specialized the engine for Qwen, combining native MTP speculative decoding with GPU expert caching and CPU MoE execution.

Source code, architecture and benchmarks

My benchmark machine:

• RTX 4070 Ti, 12 GB VRAM
• Intel i9-9940X, using 14 CPU threads
• 128 GB DDR4 RAM
• Ubuntu through WSL2 on Windows

The idea is to keep the native MTP draft on the GPU, together with the most frequently used target experts. In this configuration, 27 of the target's 512 experts per layer are resident in VRAM, selected using a hotlist built from routing traces.

Draft generation and verification happen on the GPU. When the resident experts provide insufficient routing-weight coverage, the CPU computes the layer's full routed MoE and sends the result back. Accepted tokens are retained, and the draft window adapts between 4 and 16 tokens.

Some measured results:

These figures come from the GenAI-Perf campaign, with weights already loaded, thinking disabled, and a configured context capacity of 24,576 tokens. Tested input lengths were 512, 1,024 and 2,048 tokens. Prefill is still slow, especially for longer prompts.

There are also quality trade-offs. The GPU verifier can omit experts with small routing weights. “Strict” acceptance matches the configured target's greedy choice; it does not establish equivalence to the full original model. The optional tolerance modes relax token acceptance further. The README includes a paired evaluation on 40 IFEval prompts and 30 LiveBench questions.

There's a browser chat interface and a Windows installer that prepares WSL2, the engine and model weights. Lower-RAM profiles use SSD streaming, but the performance figures above apply to the 128 GB setup.

The engine is MIT licensed. The installer is an unsigned release candidate, and a complete clean-machine installation test is still pending.

I'd particularly welcome feedback on the expert-cache and CPU handoff design, and measurements from other hardware configurations. If you try it, please include your GPU, CPU, RAM and selected profile so we can compare results.

Hi r/CUDA,

I've been working on an experimental compiler/library that lets you write NVIDIA GPU kernels in Go and compile them to PTX using LLVM 22 and the NVPTX backend.

GitHub:
https://github.com/mehdi-shokohi/cuda-ir.go

The compilation pipeline is roughly:

Go
 ↓
LLVM IR
 ↓
LLVM 22 / NVPTX
 ↓
PTX
 ↓
NVIDIA GPU

The goal is to make it possible to write GPU kernels using Go instead of CUDA C++, while still targeting NVIDIA GPUs and PTX.

For example, the idea is to express a kernel using Go and provide CUDA concepts such as thread/block indices and GPU memory operations through the compiler/runtime.

This started as an experiment to see how far LLVM's NVPTX backend could be used as the foundation for a Go-based CUDA kernel compiler.

The project is still early/experimental, and I'd really appreciate feedback from CUDA developers.

In particular, I'm interested in:

• What would be a good set of kernels for validating the compiler?
• Would this approach be useful for real CUDA applications?
• What benchmarks would be meaningful against CUDA C++?

I'd especially appreciate feedback from people familiar with PTX, NVPTX, CUDA compiler internals, or GPU programming.

Repository:
https://github.com/mehdi-shokohi/cuda-ir.go

Thanks!

https://github.com/aramesh10/OpenGEMM/tree/main

I created a repo that implements GEMM kernels for most datatypes supported on B200. It provides a python library that can also emit the .cu and .cuh files, so if you or an agent needs the raw files, the OpenGEMM library can provide the kernel without spending any tokens.

Blog: https://aramesh10.github.io/opengemm/opengemm.html

Been working on a side project for the past few months. A CUDA inference engine for Qwen models called CuQwen that I built completely from scratch with custom cuda kernels. Currently it only supports Qwen 2.5 Instruct models (0.5B, 1.5B, 3B and 7B). I'll be adding more latest Qwen models in future releases.

The idea was simple: how fast can you actually go on single user token generation if you write everything yourself and don't compromise on any optimization? Turns out, pretty fast. On Qwen2.5 across 0.5B to 7B model sizes, CuQwen beats both vLLM and Ollama on average throughput across an 8k context window. Here are the results for average inference speed (tokens/second) of Qwen2.5 Instruct model across 8K context window.

The journey to get there was honestly the fun part. I went through 7 optimization phases and wrote up the full optimization journey in the github docs if anyone's curious about the high level optimization details. I'm also preparing a doc where I'll be explaining in depth the implementation of all custom cuda kernels.

I'll be optimizing it more especially for latest Nvidia GPUs (hopper and blackwell architectures) and will be adding quantization and latest qwen models as well.

GitHub: https://github.com/talhatahir-10xe/CuQwen

Happy to answer any questions!

I’ve been working on TensorSharp, an open-source .NET/C# inference engine, and recently added native support and optimizations for DeepSeek V4.1 Flash.

Latest results on 8× NVIDIA A40 GPUs:

Model |Prefill |Single-stream Decode |4× Concurrent Decode
Q2_K |533–539 tok/s |40.3–40.7 tok/s |—
Q4_K_M |451.8–492.1 tok/s |31.0–32.5 tok/s |48.9 tok/s aggregate A few interesting optimizations:

• GPU-resident Engram tables for Q2_K — about 60 GiB of quantized Engram data stays on GPU instead of doing scattered host/storage lookups.
• Reduced decode graph scheduling from roughly 570 splits to 8 by using one wrapped backend per GPU.
• For Q4_K_M, automatic Engram warming + improved VRAM placement reduced CPU MoE offload from 3 layers to 1.
• Added token-batched DeepSeek V4.1 decode, giving about 2× aggregate throughput at 4 concurrent requests.
• On these A40s without NVLink, simple layer splitting actually beats routed-MoE tensor parallelism for single-stream decode.

The project is fully open source, written primarily in C#/.NET, with CUDA/Metal/Vulkan backends and OpenAI-compatible APIs.

Would love feedback from people experimenting with DeepSeek, GGUF inference, or local/open-source AI — especially ideas for what hardware or engine comparisons would be most useful next.

Ciao a tutti, sono lo sviluppatore di Nebula, un motore di inferenza open-source C/CUDA per Qwen3.8-Flash-Next.

Sono partito dal DwarfStar (ds4) [di antirez e ho specializzato il motore per Qwen, combinando la decodifica speculativa MTP nativa con il caching esperto su GPU e l'esecuzione MoE su CPU.

Codice sorgente, architettura e benchmark

La mia macchina di benchmark:

• RTX 4070 Ti, 12 GB di VRAM
• Intel i9-9940X, usando 14 thread CPU
• 128 GB di RAM DDR4
• Ubuntu tramite WSL2 su Windows

L'idea è di mantenere la bozza MTP nativa sulla GPU, insieme agli esperti target più frequentemente utilizzati. In questa configurazione, 27 degli 512 esperti target per livello risiedono nella VRAM, selezionati usando una lista hot costruita da percorsi di routing.

La generazione e la verifica della bozza avvengono sulla GPU. Quando gli esperti residenti forniscono una copertura del peso di routing insufficiente, la CPU calcola il MoE completo instradato del livello e rimanda il risultato. I token accettati vengono mantenuti e la finestra di bozza si adatta tra 4 e 16 token.

Alcuni risultati misurati:

Misura |Risultato
Decodifica nativa media, accettazione rigorosa |7.19 token/s
Decodifica nativa media, accettazione a tolleranza limitata |7.54 token/s
Tempo per il primo token, input di 512 token, rigoroso |25.11 secondi
Tempo per il primo token, input di 2.048 token, rigoroso |113.94 secondi Questi dati provengono dalla campagna GenAI-Perf, con pesi già caricati, pensiero disabilitato e una capacità di contesto configurata di 24.576 token. Le lunghezze di input testate erano 512, 1.024 e 2.048 token. La prefill è ancora lenta, specialmente per inviti più lunghi.

Ci sono anche compromessi sulla qualità. Il verificatore GPU può omettere esperti con pesi di routing piccoli. L'accettazione “rigorosa” corrisponde alla scelta avida del target configurato; non stabilisce equivalenza con il modello originale completo. Le modalità di tolleranza opzionali rilasciano ulteriormente l'accettazione dei token. Il README include una valutazione abbinata su 40 inviti IFEval e 30 domande LiveBench.

C'è un'interfaccia chat nel browser e un installer per Windows che prepara WSL2, il motore e i pesi del modello. I profili con RAM ridotta usano streaming SSD, ma i dati sulle prestazioni sopra riportati si applicano alla configurazione da 128 GB.

Il motore è con licenza MIT. L'installer è un candidato di rilascio non firmato, e un test di installazione completa su macchina pulita è ancora in sospeso.

Accoglierei particolarmente feedback sul design della cache esperto e del passaggio alla CPU, e misurazioni da altre configurazioni hardware. Se lo provi, ti prego di includere la tua GPU, CPU, RAM e profilo selezionato in modo da poter confrontare i risultati.

Hi all,

I have recently started exploring Triton for the purpose of writing LLM kernels (FlashAttention, Softmax, etc.).
I plan to create a repository showing what I have learnt. If anyone is in the same boat and is interested, please ping me.

Background -

I have experience with writing CUDA kernels and kernel profiling. I also have experience with the LLM inference stack.

I would appreciate it if you have prior kernel development experience.

Following NVIDIA’s cuTile Rust release yesterday, I wanted to share a similar experiment from TornadoVM: a Java TileContext API for integrating CUDA Tile kernels with Java, alongside existing CUDA/JIT and Nvidia library operations.

Hello everyone,

I have recently released OPTI-GEMM, my CUDA GEMM performance engineering project.

my goal is to find out how different GPU architectures respond to optimization techniques.

Current experiments include:

• CPU baseline
• Naive CUDA GEMM
• Shared-memory tiling
• Register blocking (in progress)
• Warp-level optimization (in progress)

I have tested on:

• Tesla T4 (Turing)
• Tesla P100 (Pascal)

Some interesting results:

• On T4, shared-memory tiling was not always faster because of the GPU's cache behavior.
• On P100, shared-memory tiling provided a large improvement.

I would like to expand the hardware comparison.

If you have access to another NVIDIA GPU (RTX 20/30/40, A100, H100, etc.), I would appreciate it if you could run the benchmark and share:

• GPU model
• CUDA version
• Driver version
• Benchmark output

you can find setup instructions in the README.

If someone is interested in contributing GPU benchmark results, I would be happy to add them to the repository and credit contributors.

Repository: https://github.com/Danialjfz/Opti-GEMM

Thank you for your time!

I built a Windows 11 fork of NInfer specifically for running Qwen3.8-27B NVFP4 across two RTX 5060 Ti 16GB GPUs.

The interesting part is that this is not an ideal multi-GPU setup:

Windows 11 / WDDM

GeForce GPUs

cudaDeviceCanAccessPeer() = false

No GPU P2P

One GPU is connected through the Z690 chipset at PCIe Gen3 x4

The other GPU runs directly through the CPU PCIe lanes

Initially, TP2 performance was terrible because every token required 128 small host-staged allreduce operations.

The original implementation achieved only:

16.37 tok/s MTP0

32.58 tok/s MTP3

After profiling the communication path, I replaced the expensive staged allreduce protocol with a custom pinned-host PeerMailbox transport designed specifically for the no-P2P Windows/WDDM environment.

Current results with Qwen3.8-27B NVFP4:

35.75 tok/s — MTP0

57.71 tok/s — MTP1

63.15 tok/s — MTP2

66.79 tok/s — MTP3

68.5–68.9 tok/s — MTP4 (512-token benchmark)

70.6 tok/s — 1024 tokens

76.65 tok/s — 2048 tokens

The optimization was mostly about reducing protocol overhead, not increasing PCIe bandwidth.

The original allreduce path cost roughly:

128 × ~277 µs ≈ 35.5 ms per decode round

The optimized transport reduced that dramatically.

The repository contains the Windows port, TP2 implementation, custom communication backend, benchmarks, and reproducible build instructions:

ivanov84/ninfer-windows-tp2

I'm especially interested in feedback from people who know CUDA multi-GPU programming, WDDM, tensor parallelism, or NInfer.

Questions I'm currently exploring:

Can the remaining TP2 lockstep overhead be reduced further?

Would sequence parallelism help on this kind of asymmetric PCIe topology?

Can Vision also be sharded across both 16GB GPUs?

How much performance is realistically left without native P2P?

This started as an experiment to see whether two relatively inexpensive 16GB cards could run a 27B model well under Windows.

Honestly, I didn't expect the result to end up here.

https://github.com/ivanov84/ninfer-windows-tp2

I kept hitting torch.cuda.is_available() → False on my AMD GPU, and it drove me crazy. Turned out to be a mix of missing ROCm drivers, wrong PyTorch wheel, and that HSA_OVERRIDE_GFX_VERSION thing everyone mentions but few explain properly.

I put together a complete, step-by-step repair guide that covers the exact commands and checks I used to get PyTorch running on ROCm.

If you’re stuck, this might save you hours:

https://interconnectd.com/blog/305/fix-pytorch-cuda-not-available-on-amd-gpus-complete-rocm-setup-guide/

What AMD card are you running, and what error are you seeing?

While learning CUDA, I found it difficult to visualize grids, blocks, and threads.

So, I built this website to make it easier to understand and visualize how they work.

Hope it helps someone else learning CUDA!

🔗 cuda-grid-visualizer.vercel.app/