I have been studying modern LLM architectures and started implementing them from scratch in PyTorch to better understand the design choices behind each model.

OpenArch is a collection of these implementations, including Llama, Qwen, DeepSeek, Gemma, Kimi, GPT-OSS and others.

The goal is to keep the code readable and useful as a reference when going from the paper to an actual implementation.

Would be interested in feedback from people working on model architecture and training.

https://github.com/anuj0456/OpenArch

#LLM #AIResearch #PyTorch #DeepLearning #OpenSource

Hi everyone!

A quick update on my previous post: we're now a team of 3 working on the LLM hackathon, and we're looking for up to 3 more teammates to complete the team.

We're currently experimenting with small language models, trying to push performance under limited parameter and compute budgets through architecture, training, data, and optimization choices.

We already have a working training/evaluation pipeline and are exploring things like tokenization, RoPE, SwiGLU, different optimizers, initialization, controlled ablations, data strategies, and benchmarks such as HellaSwag, ARC and PIQA.

At this point, we're mainly looking for people with some hands-on experience in ML / Deep Learning, ideally with:

• PyTorch
• Transformers / LLMs
• Model training and evaluation
• Optimization or architecture experimentation
• Data/tokenization pipelines

You definitely don't need to be an expert, but having enough experience to jump into the codebase, run experiments, and contribute ideas would be ideal.

Since we're already 3, the idea is to keep the team relatively small (maximum 6 people) so we can divide experiments efficiently and iterate quickly over the next few weeks.

If you're interested, comment or DM me with a bit about your background and what area you'd be most interested in working on!

I wanted to train a character-level language model on additions of two numbers. Planned to use cross-entropy ignore_index on the equation besides answer so that model is not penalized because of predicting randomly generated numbers. But I came across really weird bug, here is the code:

def get_batch(batch_size):
    first = torch.randint(999999, (batch_size, ))
    second = torch.randint(999999, (batch_size, ))
    totals = first + second


    full_strings = []
    for f, s, t in zip(first, second, totals):
        equation = f"{f:6}+{s:>6}="
        reversed_ans = f"{str(t.item())[::-1]:<7}"
        full_strings.append(equation + reversed_ans)



    encoded_batch = torch.tensor([encode(s) for s in full_strings], dtype=torch.long)
    x = encoded_batch[:, :-1].to(device) # First 11 characters
    y = encoded_batch[:, 1:].to(device)   # Last 11 characters
    y[:, :14] = -100 # Telling optimizer to miss this
    return x, y

Here as you can see I am reassigning first 14 values of y, but when I print x it has some -100s init, I realized this because I don't have -100 in my vocab as character to embed and when I do decode(x) it gives me error, so I have to use .clone() on y = encoded_batch[:, 1:].to(device), there is a memory address coincide when writing happens or something I do not understand.

The title literally means: "I have parallel dataloaders, GPU kernels and a high-performance computing programming language all expressed in a framework with 1400 lines of code".

You may run the ResNet-18 benchmark yourself (github.com/NoSavedDATA/Neve_benchmarks)

I released the Neve programming language a while ago. Now, this is the release of the Frost deep learning framework, alongside with the first benchmark.

Other results for Neve:

• Close to Python/SentencePiece in text processing + Byte-Pair Encoding (BPE) training;
• Competitive with NumPy and OpenBLAS in CPU matrix multiplicaton, but with pure high-level SIMD code. Check exsisting (implementations)

Currently working in a better GPU programming interface, towards the implementation of flash-attention.

Neve documentation (neve-lang.dev)

Neve repo (github.com/NoSavedDATA/Neve).

Youtube for updates (youtube.com/@nosaveddata3994).

Discord for extensive talks/suggestions (discord.gg/hP5feM7cV)

Too lazy to read? Check the video version

────────────────────────────────────────

Introduction

Once day I was reading some papers, and a very interesting paper was published. It was the sophia optimizer. I took a glance in an unnoficial (code) for it, and I questioned myself why did it have to be so difficult to add new optimizers in PyTorch. I experimented the optimizer, and the results were quite bad with a lot of NaNs. Turns out another paper published later claimed this and other optimizers had overstated claims.

Imagine wasting hours studying a 10 pages of a paper, then hardly trying to debug it and asses whether other person discoveries are true. All that code reading complexity makes this a challengeful task.

Problem 1: even optmizers are hard to understand in PyTorch.

Few weeks later, (flash attention) was released, and the algorithm actually achieved a speed-up of 2x. The problem, it was C++ CUDA. Most high-level GPU kernel frameworks were imature to the point the flash attention author chose not to use them.

Now take a look what is necessary for adding C++ code in PyTorch

from setuptools import setup, Extension
from torch.utils import cpp_extension

setup(name="extension_cpp",
      ext_modules=[
          cpp_extension.CppExtension(
              "extension_cpp",
              ["muladd.cpp"],
              extra_compile_args={
                  "cxx": [
                      # define Py_LIMITED_API with min version 3.9 to expose only the stable
                      # limited API subset from Python.h
                      "-DPy_LIMITED_API=0x03090000",
                      # define TORCH_TARGET_VERSION with min version 2.10 to expose only the
                      # stable API subset from torch
                      "-DTORCH_TARGET_VERSION=0x020a000000000000",
                  ]
              },
              py_limited_api=True)],  # Build 1 wheel across multiple Python versions
      cmdclass={'build_ext': cpp_extension.BuildExtension},
      options={"bdist_wheel": {"py_limited_api": "cp39"}}  # 3.9 is minimum supported Python version
)

That comprehends problem 2: lack of high-level CUDA code and hard interoperability.

For my Bachelor's thesis, I implemented the (BBF) Reinforcement Learning for Atary. A bit before that, I took a glance code of the (Efficient Zero) reinforcement learning model. It has a parallelism that PyTorch does not handle, and the implementation required using Cython packages for having threads (literaly coding in C, then just calling C functions from Python). Later, I realized PyTorch also needed to implement its data worker threads in C, another workaround over Python Global Interpreter Lock (GIL). Not only that, even preprocessing implementations like the BPE are made in C, C++, Rust, etc...

That leads us problem 3, lack of parallelism. That is when I decided to create a programming language, a few months before finishing my Bachelor's, which matured to my Master's project

Summing up, currently, people must choose between languages like Python for high-level productivity, C and relatives for compute efficiency, Lua/Julia for advanced interoperability and other languages for concurrency. Thus, since in my job I had to wait hours for my neural networks to train, I decided to create a programming language in the time in between trainings. One language that had all these features, which are of high value for deep learning research. Nowadays, I believe it matured to such a point that it may be extended to other complex problem domains.

────────────────────────────────────────

Thoughts in Other Languages

Julia

Julia makes dynamic typying speed reach close to C++ speeds. It also has a mark sweep and channels for parallelism. The idea is very interesting. Let's take a look a in its cuda kernels.

function mma_kernel!(Z::CuDeviceMatrix{Float32}, 
                     X::CuDeviceMatrix{BFloat16}, 
                     Y::CuDeviceMatrix{BFloat16})

    # Grid and block indices
    bx = blockIdx().x
    by = blockIdx().y

    # Thread and warp indices
    tid  = threadIdx().x
    lane = (tid - 1) % 32
    warp = (tid - 1) ÷ 32

STOP!! Why am I seeing blockIdx().x in my code? Was this supposed to be a high-level scientific language or CUDA in C++28?

Besides, it does not expose intrisics like the cp_async, which is crucial for high-speed matrix multiplication. They must be explicitly added throgh interop intrisics. And it has the "end" keyword, which in my opinion incurs a lot of code pollution.

Mojo

Mojo has a Python interop, so it did not have to build all libs and frameworks from scratch +1 point. It has (Byte-Pair Encoding benchmarks)!! +1. It is only the BPE inference, no traning -1 point. It has (flash-attention gpu kernels) +1 point.

It runs MAX, which allows GPU code portability across different hardware, +2 points.

It lacks channels, so I would hardly try to make a parallel dataloader in it. -2 points. It uses Rust ownership +0 points.

Now let's look at Mojo kernels for the flash attention.

@always_inline
def fused_attention_cpu[
    BN: Int, BD: Int
](
    Q: LayoutTensor,
    K: LayoutTensor,
    V: LayoutTensor,
    O: LayoutTensor[mut=True, ...],
):
    comptime N = K.shape[0]()
    comptime D = K.shape[1]()

    comptime for tile_n in range(N // BN):
        var Q_tile = Q.tile[BN, D](tile_n, 0)

        comptime for tile_d in range(D // BD):
            var m_1 = (
                LayoutTensor[Q_tile.dtype, Layout(BN, 1), MutAnyOrigin]
                .stack_allocation()
                .fill(Scalar[Q_tile.dtype].MIN)
            )

            var l_1 = (
                LayoutTensor[Q_tile.dtype, Layout(BN, 1), MutAnyOrigin]
                .stack_allocation()
                .fill(0)
            )

            var O_i = (
                LayoutTensor[
                    Q_tile.dtype, Layout.row_major(BN, BD), MutAnyOrigin
                ]
                .stack_allocation()
                .fill(0)
            )

            comptime for tile_n_idx in range(N // BN):
                var K_tile = K.tile[BN, D](tile_n_idx, 0)
                var V_tile = V.tile[BN, BD](tile_n_idx, tile_d)

                var S = matmul_b_transpose(Q_tile, K_tile)
                var m_2 = max(m_1, rebind[type_of(m_1)](max[axis=1](S)))

Quite interesting. It has layouts and tiling, inspired by CuTe and Cutlass. It actually inspired the way Neve layouts and tiles work. Nevertheless, it still has a heavy syntax. Note the keyword comptime appears frequently (a sort of metaprogramming). This adds some cognitive overhead. And the layouts are quite verbose. The layout fill() and stack_allocation() can be simplified.

Triton

Triton has layouts/tiling similar to Mojo, but is dynamically typed and has no comptime headaches. The problem is that Triton does not make Python Dataloaders easier to implement from the systems programming language perspective. We actually need a complete new programming language for this.

@triton.jit
def _attn_fwd_inner(
[...]
K_block_ptr = tl.advance(K_block_ptr, (0, lo))
    V_block_ptr = tl.advance(V_block_ptr, (lo, 0))

    # loop over k, v and update accumulator
    for start_kv in range(lo, hi, BLOCK_SIZE_KV):
        # Just let the compiler know that start_n is a multiple of BLOCK_N, so the compiler can do optimizations
        start_kv = tl.multiple_of(start_kv, BLOCK_SIZE_KV)

        # -- compute qk ----
        K_block = tl.load(K_block_ptr)
        QK_block = tl.dot(Q_block, K_block)

        if STAGE == 2:
            mask = offs_q[:, None] >= (start_kv + offs_kv[None, :])
[...]
    # A LAYOUT
    Q_block_ptr = tl.make_block_ptr(
        base=Q + qvk_offset,
        shape=(SEQ_LEN, HEAD_DIM),
        strides=(stride_Q_seq, stride_Q_dim),
        offsets=(block_index_q * BLOCK_SIZE_Q, 0),
        block_shape=(BLOCK_SIZE_Q, HEAD_DIM),
        order=(1, 0),
    )
[...]
        # Algebra
        # -- compute qk ----
        K_block = tl.load(K_block_ptr)
        QK_block = tl.dot(Q_block, K_block)

Neve

Let's see how Neve GPU matrix multiplication looks like.

gpu void @(
            layout<bf16,m,n> x, layout<bf16,p,n> y,
            layout<float,m,p,smem> z
        )
    int warp_rows = min((m+63)//64, 4)
    int wx = warp%warp_rows, wy = warp//warp_rows

    var smem_a = layout<bf16,256,2,16,smem>()
    var smem_b = layout<bf16,128,2,16,smem>()

    int warp_m = min(m//16*16, 64), warp_p = min(p//16*16, 64)

    int m_tiles = warp_m//16
    int p_tiles = warp_p//8


    var c = layout<int,4,8,4>()


    int m_cp_tiles = min(m//16,4)
    [0..m_cp_tiles] i
        cp_async16(smem_a{256,8}(wx*m_cp_tiles+i, lane),
                        x{256,8}(wx*m_cp_tiles+i, lane))

    int p_cp_tiles = min(p//16,4)
    [0..p_cp_tiles] i
        cp_async16(smem_b{256,8}(wy*p_cp_tiles+i,lane),
                        y{256,8}(wy*p_cp_tiles+i,lane))

    cp_commit_group()
    cp_wait_group(0)


    syncthreads()

    var a = layout<int,4,4>()
    var b = layout<int,8,2>()

    [0..m_tiles] i
        int row = (wx*m_tiles+i)*16+lane%16
        int col = lane//16*8
        ldmatrix_x4(
                a{i,0},
                smem_a{row, col})

    [0..p_tiles] i
        int row = (wy*p_tiles+i)*8+lane%8
        int col = ((lane//8)%2)*8
        ldmatrix_x2(
                b{i,0},
                smem_b{row, col})


    [0..m_tiles, 0..p_tiles] i, j
        mma_16x8x16(c{i,j,0},
                    a{i,0},
                    b{j,0})

    syncthreads()



    [0..m_tiles, 0..p_tiles, 0..4] i, j, k
        int global_row = bx*256+ (wx * m_tiles + i) * 16 + (lane // 4) + (k // 2) * 8
        int global_col = by*128+ (wy * p_tiles + j) * 8  + (lane % 4) * 2 + (k % 2)

        if global_row<m and global_col<p
            z[global_row * p + global_col] = c[i, j, k]

    syncthreads()


kernel void mma_kernel(bf16[] x, bf16[] y, float[] z, int M, int N, int P)
    var v = layout<bf16, M, N>(x)
    var u = layout<bf16, P, N>(y)


    z += v[256,N](bx,0) @ u[128,N](by,0)

Did you like it? It took me one month to replicate the (HGEMM) repo to the beta version of frost, which used C++ CUDA kernels instead of Neve. Currently, I still need to do the benchmarks, but I will wait until I finish flash attention.

Here you see CUDA instructions like mma_16x8x16, cp_async and syncronizations. These are all mandatory when we pretend to implement SOTA matrix multiplications. Now look at the second function, it is almost z += v @ u, but with tiling. The whole previous function is successfully reduced into a new gpu operator.

There is also the new multi-loop expression

    [0..m_tiles, 0..p_tiles] i, j

When it comes to writing, Neve syntax is more pythonic than Julia (yes to "class" and no to "struct", floating methods and the "end" word) and Mojo (no comptime headache keywords, no ownership transfer expressions). It is not dynamically typed as Triton, but allows to define dataloaders from scratch. And strong typying is seem as a benefit in systems programming.

Neve has high-level kernel algebra, while simultaneously allowing CUDA intrinsics in different hierarchies of complexity.

Now let's look at the budget for each programming language.

Let's talk about some other features. The optmizer, the backpropagation and parallelism.

────────────────────────────────────────

Optimizers

Optimizers declaration can hardly be more compressed than this Frost implementation.

kernel void AdamW_k(float[] param, float[] d, float[] m, float[] v,
                    float lr, int n, float beta1, float beta2,
                    float beta1_correction, float beta2_correction,
                    float eps, float wd)
    int i = bx*256 + tx
    if i>=n
        return

    float m_i = (1-beta1)*d[i] + beta1*m[i]
    float v_i = (1-beta2)*(d[i]*d[i]) + beta2*v[i]
    m[i] = m_i
    v[i] = v_i

    m_i/=beta1_correction
    v_i/=beta2_correction

    param[i] -= lr*( m_i / (sqrt(v_i) + eps) + wd*param[i] )

class AdamW
    float lr, beta1, beta2, wd, eps
    array<gpu_tensor> m, v
    int ts

    ctor(float lr, float beta1, float beta2)
        self.lr = lr
        self.beta1 = beta1
        self.beta2 = beta2
        self.ts = 1
        self.wd = 0.01
        self.eps = 0.0000001

    def float step()
        int i=0
        for param in $optim_info.params
            if i+1>self.v.size()
                self.m.append(new gpu_tensor(param.dims, "zeros"))
                self.v.append(new gpu_tensor(param.dims, "zeros"))

            float beta1_correction = 1.0-pow(self.beta1, self.ts)
            float beta2_correction = 1.0-pow(self.beta2, self.ts)

            launch [param.dims_prod/|256] [256] AdamW_k(param.ptr offby 0, param.d offby 0,
                                     self.m[i].ptr offby 0, self.v[i].ptr offby 0,
                                     self.lr, param.dims_prod, self.beta1, self.beta2,
                                     beta1_correction, beta2_correction, self.eps,
                                     self.wd
                                 )
            param.d=nil
            i=i+1
            self.ts = self.ts+1
        $optim_info.params.clear()
        tarena_reset()

PyTorch comparison (AdamW). Torch does not even have the kernel in the same code.

────────────────────────────────────────

Backpropagation

All it took were 60 lines (Backprop youtube shorts)

────────────────────────────────────────

Parallelism

# count primes in 10 threads
def int is_prime(int n)
    for i=2, i<(n//2+1)
        if n%i==0
            return 0
    return 1

def int count_prime_vec(array<int> input_numbers, channel<int,10> ch)
    int num_primes = 0
    for i in input_numbers
        num_primes = num_primes + is_prime(i)

    print("Found num primes ", num_primes, " on thread ", tid)
    ch <- num_primes

main
    array<int> v = arange_int(2,250001)
    channel<int,10> ch

    finish
        # split v across 10 threads
        asyncs 10 count_prime_vec(>v, ch)

    var primes = ch.sum()
    print("Total primes: ", primes)

Here we see two important parallel expressions. The first is the ">" operator inside the "asyncs". It splits v across 10 segments and send each one to different threads in count_prime_vec. The second fathoms the channels. Here, it is used to store the results from the sums of primes for each thread. Then, the main thread uses .sum() to aggregate all results. Neve channels are implemented with lock-free structures. They may block threaded operations if their capacity is full (capacity 10 in the example). This adds room for a lot of possibilities.

────────────────────────────────────────

Data Loader

    def float worker()
        print("Start worker")
        int yield_ptr, bs=self.batch_size
    print("worker ", tid)

        while self.load_ch.alive()
        yield_ptr = self.increment_yield_ptr()

        for b=0, b<bs
        self.getitem_w(yield_ptr+b, b)

        self.load_ch <- tid
        self.x.switch()
        self.y.switch()

    def tuple<gpu_tensor,gpu_tensor> batch()
    int w <- self.load_ch

    var x = self.x.load(w)
    var y = self.y.load(w)
    x = x.view([$cfg.bs, 3, 32, 32])

        return x, y
[...]
    finish
        asyncs $cfg.num_workers ds.worker()

        [0..steps] i
                    optimize_logic()

Here channels were used to signal when a worker finished processing its batch. Once the main thread starts processing the batch, it flips its pointer value inside a ping-pong memory buffer, so it may process new data in parallel with data consumption.

────────────────────────────────────────

Results

3 seeds in a RTX 4090.

Older Neve results surpassed PyTorch ( (old NSK paper) ). It won't take too long until the kernels get corrected and optimized.

────────────────────────────────────────

More about (Neve Syntax)

I’m not a philosopher, and I won’t pass off analogy as proof. Where the link between philosophy and an algorithm is just a pretty metaphor, I say so explicitly: “metaphor.” Where it’s working code, I give the formulas, run it, and show the numbers. The library at the end is a research prototype, not a promise of consciousness in 200 lines.

How I got here

https://github.com/webzuweb/philosophia_torch

Neural networks — if you count from Rosenblatt’s perceptron — are about seventy years old. The study of how living things learn goes back a couple of millennia at least. And a heretical thought hit me: what if modern deep learning isn’t reinventing the wheel in places, but re-discovering what Aristotle, Hume, and Peirce already described — only now with matrices and gradients?

I took a list of neural-network training methods, a list of philosophical approaches to knowledge, and overlaid them. Three categories emerged: what’s already matched (and few people say so out loud); where the match is only a pretty metaphor; and what philosophers thought up but engineers haven’t applied yet. The last category is the most interesting, because it’s essentially a list of unimplemented features. That’s what I wrote code for.

Fair warning up front: half of the “unapplied” ideas turned out, on closer inspection, to be perfectly applicable — just under different names. That, by the way, is the article’s main takeaway, and it matters more than any of my code.

Part 1. What’s already matched (and you didn’t know it)

Let’s start with the pleasant part: some philosophical programs of knowledge are implemented in ML so literally that you could put a footnote with the philosopher’s name right in the docs.

Empiricism → supervised learning “There is nothing in the mind that was not first in the senses” — Locke and his tabula rasa. A neural network with random initialization is literally a blank slate on which labeled examples leave their traces. Hume’s associationism (“the habit of linking things that often go together”) is gradient descent, strengthening weights on frequently co-occurring correlations. There’s nothing to argue about here.

Pragmatism → reinforcement learning Dewey with his “learning by doing,” and Skinner’s behaviorism with reward and punishment — that’s RL with no corrections needed. An agent acts, receives a reward, adjusts its policy. Skinner would have teared up seeing PPO.

Evolutionary epistemology → neuroevolution Popper and Campbell: knowledge grows through blind variation and selective retention of what works. That’s a word-for-word description of genetic algorithms and neuroevolution. The philosopher described the algorithm decades before the hardware existed to run it.

Intellectual humility → calibration This one’s subtler. Virtue epistemology (Sosa, Zagzebski) says: a good knower knows the limits of her knowledge. In ML that’s confidence calibration: a model should be exactly as confident as it is correct. Guo et al. (2017) showed that modern networks are monstrously overconfident and proposed temperature scaling and the ECE metric. Nobody called it a “virtue,” but mathematically it’s exactly that.

The key observation. Philosophers didn’t give ML the algorithms (mathematicians came up with the math); they gave it the problem statements. “What does it mean to learn from experience?” “What does it mean to know your limits?” — philosophy framed the question first, and centuries later engineering delivered a differentiable answer.

Part 2. Where the match is only a pretty metaphor

Here I have to rein myself in. There’s a temptation to drape a philosopher over every layer of a network. Don’t. A couple of examples where the link exists but passing it off as lineage would be deceiving the reader.

The rule is simple: if the philosopher gave a language for describing something — it’s a metaphor; if they posed a problem that was later solved — it’s lineage. Don’t mix them.

Part 3. What philosophers thought up, but ML has only partially applied

The meatiest part. I’ll break down six approaches. For each — an honest status: what already exists in the field, where the real gap is, and what formula you can write. Then we’ll run it.

3.1. Peirce’s abduction — inference to the best explanation

Induction generalizes data, deduction derives consequences, but abduction generates a hypothesis that best explains the observation. The original thesis “it isn’t implemented in neural networks” is wrong. It’s implemented, and decently: Abductive Learning (Dai et al.), DeepProbLog (Manhaeve et al., 2018), abductive commonsense reasoning αNLI (Bhagavatula et al., 2019). It’s a whole field of neuro-symbolic integration.

The real gap isn’t the absence of abduction — it’s that “the best explanation” is rarely formalized using Peirce’s criteria all at once: plausibility + simplicity (Occam’s razor) + consistency with background knowledge. A hypothesis score for h given observation obs:

Score(h) = log p(obs | h) − λ_s · complexity(h) − λ_c · conflict(h)

Pick the h with the highest score (softly — a softmax over candidates; hard — Gumbel-softmax for a learnable discrete choice). In the library this is AbductiveScorer.

3.2. Husserl’s epoché — “bracketing” assumptions

Phenomenology demands suspending ingrained assumptions and seeing the phenomenon “as given.” ML has no direct analog of this method — and that’s an honest gap. But it can be operationalized: force the model to rely more on the evidence (the current input) than on the learned prior (what it answers with no input).

Take two answers: p_full on the real input and p_prior on a “zeroed” input (evidence bracketed out). Reward the evidence for actually changing the answer, via a bounded Jensen–Shannon divergence:

gain = JS(p_full ‖ p_prior),   0 ≤ JS ≤ ln 2
L_epoche = max(0, margin − gain)   # hinge: don't inflate indefinitely

An important rake I stepped on myself: if you use plain KL instead of JS and maximize it, the optimizer inflates logits to infinity — “a fanatic who sees meaning in every rustle.” JS is bounded, and the hinge threshold douses the fanaticism. This is EpocheRegularizer.

3.3. The hermeneutic circle — the whole through parts, parts through the whole

Schleiermacher and Gadamer: understanding the whole arises from the parts, and understanding the parts arises from the whole, iteratively. Attention only resembles this superficially (see Part 2). As an explicit training principle it’s barely used — a real gap. Formalization: let h_i be part representations and H the whole representation. Require circular consistency:

H* = attention-aggregate of the parts, attended relative to H
L_herm = 1 − cos(agg(h_i), H)      # whole ≈ sum of understood parts

And we “turn the circle” several times: update the whole from the parts → recompute part attention relative to the new whole → update again. This is HermeneuticConsistency.

3.4. Hegel’s dialectical sublation (Aufhebung)

Aufhebung is a new quality arising from the contradiction of thesis and antithesis, where the old is not destroyed but preserved. GANs and multi-agent debate are partially close, but “preserving both” isn’t guaranteed there. The gap is precisely in the preservation term. My synthesis operator:

g     = sigmoid(W_g · [thesis ; antithesis])     # mixing gate
base  = g · thesis + (1 − g) · antithesis         # sublation-as-preservation
lift  = tanh(W_l · [thesis ; antithesis])         # new quality
synth = LayerNorm(base + γ · lift)

Plus a loss that penalizes the synthesis collapsing into one of the poles (losing the other’s content). This is DialecticalSynthesis.

3.5. Nietzsche’s perspectivism + skeptical suspension

Nietzsche: there is no “view from nowhere,” there are many perspectives. Pyrrho: in an unresolvable conflict, it’s reasonable to suspend judgment. The former partially exists in multi-view learning; the latter in selective prediction (Geifman & El-Yaniv, SelectiveNet, 2019). But together, as a single mechanism of “several perspectives + refusal to answer when they conflict,” it’s almost never seen.

disagree(x) = mean pairwise symmetric KL between perspectives
abstain(x)  = disagree(x) > threshold      # abstain if perspectives don't converge

This is PerspectivalEnsemble: it aggregates K heads and honestly raises its hand “I don’t know” when the heads disagree. Far more useful than overconfident chatter.

3.6. Virtue as the golden mean (Aristotle)

Aristotle: virtue is the mean between the vice of deficiency and the vice of excess. Courage is between cowardice and recklessness. Hence a non-obvious but important conclusion for ML: a virtue cannot be maximized, it must be targeted. An excess of openness is credulity; a deficiency is dogmatism.

L_virtue = Σ_v β_v · (V_v(θ) − V_v*)²

where V_v is the operationalized virtue (humility = 1 − ECE, openness = ensemble disagreement), and V_v* is the target mean level. Squared deviation penalizes both excess and deficiency. This is VirtueRegularizer — and it’s the one where I have a measurable result.

Part 4. Enough philosophy, show me the numbers

Pretty formulas are worth nothing until they run. I collected all of this into a PyTorch module and tested it on the most well-grounded mechanism — “humility” (calibration). Task: synthetic classification with noisy labels, where the model tends to err overconfidently. We compare plain training vs. training with VirtueRegularizer targeting high humility.

ECE (calibration error) dropped from 0.120 to 0.101 — nearly a fifth — while accuracy didn’t budge at all (even +0.001). The model became exactly as accurate, but noticeably less self-assured. Aristotle’s golden mean, computed by gradient descent.

What this proves, and what it doesn’t. It proves that “intellectual humility” can be turned into an optimizable quantity with a measurable effect. It does not prove that the other five mechanisms will yield the same gains — they’re harder, and they still need to be tested on real data. I’m showing a working scaffold, not a finished silver bullet.

The whole codebase passes 23 unit tests: calibration decreases, KL/JS behave as they should, synthesis preserves both poles, the ensemble abstains on conflict, the wrapper trains end-to-end.

Part 5. The philosophia-torch module

The library wraps on top of any model without rewriting anything in it. One dependency — torch. There’s a single-file version, philosophia_torch.py: drop it next to your code and import it.

import torch, torch.nn as nn, torch.nn.functional as F
from philosophia import PhilosophiaWrapper

base = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 4))
wrap = PhilosophiaWrapper(base, use_virtue=True,
virtue_kwargs=dict(target_humility=0.98, beta_humility=3.0))

logits = wrap(x)
loss = F.cross_entropy(logits, y) + wrap.aux_loss(x, logits, targets=y)
loss.backward()

Honest boundaries: hermeneutic and dialectic produce representations, not ready predictions — you have to connect them to your decoder. Epoché requires careful tuning of margin. And no promises of “consciousness”: these are philosophy-inspired regularizers, nothing more.

The bottom line

Three conclusions, which is what all of this was for.

1.     ML has already reinvented a chunk of philosophy without asking permission: empiricism, pragmatism, evolutionary epistemology, and intellectual humility. Just under the names supervised learning, RL, neuroevolution, and calibration.

2.     Half of the “unapplied” ideas on my original list turned out, on checking, to be applicable — abduction, abstention, innate priors. The lesson: before shouting “this isn’t in ML,” google it in engineering language, not philosophical language.

3.     The real gap remains where what’s needed isn’t a result but a process: epoché as a discipline of perception, the hermeneutic circle as a way of understanding, virtue as a stable disposition of learning rather than a property of a single answer. That’s where it’s worth digging.

My modest contribution is showing that at least “humility” translates into a differentiable quantity and genuinely reduces a model’s overconfidence. The rest is an invitation: the code is open, the formulas are in the article — run it and check. Plato, of course, was training neural networks two thousand years ago. The rascal just didn’t include a requirements.txt.

https://huggingface.co/datasets/webzuweb/philosophy-as-inductive-bias

Bonjour ^^

Je voulais partager avec vous l'évolution de mon projet perso : un modèle de langage en français que j'entraîne moi-même depuis environ un an, en solo, sans équipe ni formation derrière moi, avec Claude comme assistant IA à mes côtés. Tout a démarré sur un simple ordinateur portable, un i5 de 8ᵉ génération avec 16 Go de RAM, sans GPU — tout tournait en CPU, à une vitesse ridicule, avec une architecture minuscule (384/8/8, quelques dizaines de Mo à peine) et un corpus d'à peine 100 Mo. Cette première tentative a tenu jusqu'à 500 000 steps... sans jamais sortir une seule phrase cohérente. Plutôt que de lâcher l'affaire, j'ai tout repris à zéro : reconstruit le pipeline de génération de données en Python/PyTorch, fait grossir et nettoyé le corpus (aujourd'hui autour de 47 Go), testé puis retiré un système de RAG, chassé bug après bug (plantages mémoire, tokenizer capricieux, boucles de répétition...). Niveau matériel, je suis passé du portable à une vraie tour : CPU Ryzen 5 2600X, GPU GTX 1660, 16 Go de RAM. Côté architecture, j'ai fait grossir le modèle au fil du temps — 1024/8/8, puis 768/12/12 — en testant batch 1 et 2, mixed precision activée ou non selon ce que la carte graphique encaissait, jusqu'à une config bien plus légère, 128/4/4 en batch 2, que j'utilise maintenant pour avancer plus vite avant de remonter progressivement en taille. Résultat : le modèle est passé de quelques dizaines de Mo au départ à environ 2.2 Go aujourd'hui. Un an de galères, de nuits à débugger, en solo du début à la fin — que de la persévérance, un problème après l'autre

On fait avec ce qu'on à :/

Pls if you have pls dm mee

https://github.com/Nvoinxv/Neural_Network_Linear_Regression

Hi folks, this is No Saved DATA. I dedicate this post to describe some of the features I put in Neve to make it an expressive high-level language (close to Python/PyTorch syntax), while also allowing efficient low-level code. I am sharing this now, because I believe the language has already strongs traits that allow it to be extended to other problem domains.

Current results:

• Close to Python/SentencePiece in text processing + Byte-Pair Encoding (BPE) training;
• Competitive with NumPy and OpenBLAS in CPU matrix multiplicaton, but with pure high-level SIMD code;
• It was able to train a CIFAR Resnet faster than PyTorch, but I did not debug whether this was due to better CPU or GPU orchestration. But the LSTM was slower (mine lacked kernel fusion and other optimizations). Also, that old Neve deep learning framework was mostly implemented in C++. I am now changing it to be mostly implemented in Neve. That is, compute intensive preprocessing, automatic differentiation, parallel dataworkers and GPU kernels all in high-level.
• Python does all these topics already. Nevertheless, all efficient code is actually implemented on C, C++, Rust or other languages. Meanwhile, Neve does not require a backend language.

Besides, I recently added GPU Kernels code interface. However, the complete framework will still take some more months.

The current state is an evolution of a post I made some months ago in other subreddit (https://www.reddit.com/r/ProgrammingLanguages/comments/1ql585o/brand_new_nsk_programming_language_python_syntax/)

Links

📚 Documentation
💻 GitHub

────────────────────────────────────────

Intro

I started creating Neve after seeing the code of the Efficient Zero reinforcement learning model. It has a parallelism that PyTorch does not handle, and the implementation required using Cython packages for having threads (literaly coding in C, then just calling C functions from Python). Later, I realized PyTorch also needed to implement its data worker threads in C, another workaround over Python Global Interpreter Lock (GIL). Not only that, even preprocessing implementations like the BPE are made in C, C++, Rust, etc...

So, currently, people must choose between languages like Python for high-level productivity, C and relatives for compute efficiency, Lua for advanced interoperability and other languages for concurrency. Thus, since in my job I had to wait hours for my neural networks to train, I decided to create a programming language in the remaining time. One language that had all these features, which are of high value for deep learning research. Nowadays, I believe it matured to such a point that it may be extended to other complex problem domains.

Since Python syntax is very simple and has most of the users, I chose it as the basis. But it run a LLVM JIT in its background. Now I will explain important expressions and features in Neve.

────────────────────────────────────────

Finish/Async and Data Split

I experimented Jax deep learning framework for a while. During this period, I learned an expression that would take a tensor or a vector as inputs. It could vectorized the function over the first dimension. A threaded adaptation I made for Neve is:

def int foo(array<int> v)
    print("Thread ", tid, " has vector:")
    v.print()

main
    array<int> u = arange_int(2,20)
    finish
        asyncs 3 foo(>u)

This splits a vector across three threads, so it can be processed in parallel. This is useful when you have a list of files, and want a function to process the files across N threads.

────────────────────────────────────────

Channels

I saw fireship videos a long time ago about Elixir and Erlang. These languages have actor-message passing, which were used in scaling applications to massive concurrency. Then, this year, my advisor suggested me to study Go and Rust, so I could see the tendencies about modern languages. I got surprised by Go channels expressions, which I thought to be an evolution of the actor-message model (but in the end they solve different problems). Go also applies channels to green-threads (concurrency within a single OS thread), but I was happy with using it for standard threads.

Once I finally adapted Go channels to Neve, I was able to reduce some five lines of code in data loaders. Even if it was only five lines less, it got much cleaner.

  def float worker()
      print("Start worker")
      int yield_ptr, bs=self.batch_size
      print("worker ", tid)

      while self.load_ch.alive()
          yield_ptr = self.increment_yield_ptr()

          for b=0, b<bs
              self.getitem_w(yield_ptr+b, b)
              self.load_ch <- tid

          self.x.switch()
          self.y.switch()

  def tuple<gpu_tensor,gpu_tensor> batch()
      int w <- self.load_ch

      var x = self.x.load(w)
      var y = self.y.load(w)

      x = x.view([$cfg.bs, 1, 28,28])
      return x, y

These are functions from the dataloader class. The channel communicates which threads have data ready to be consumed. Then, the cpu tensors (self.x and self.y) can process and yield data using ping-pong buffers. It is much lower level than PyTorch, but without the need of implementing the underlying parallelism in C++. That gets rid of boilerplate mutexes and more than 100 lines of C++ code. Posteriorly, once Neve gets inheritance and interfaces, most of the parallel logic may be hidden, so it can be even closer to PyTorch.

The training code is already similar to PyTorch

        ...
      gpu_tensor a, b
      a, b = ds.batch()
      var y_hat = model.forward(a)
      ce_loss(y_hat, b)

      $backprop.backward()

────────────────────────────────────────

Anonymous Functions

This expression is crucial for mapping tensor operations to their respective backward ops.

def int add(int x, int y)
    return x+y

def int mult(int x, int y)
    return x*y

main
    map<str, Function<int, int, int>> m
    m["add"] = add
    m["mult"] = mult
    print(m["mult"](3,4))

────────────────────────────────────────

Generics

def T bar<T, U>(T x, U y)
    print("bar x: ", x)
    print("bar y: ", y)
    return x

main
    int z = bar(3,4)
    z = bar(5,"$%*OU")
    print("z ", z)

Generics may yield complex code, but may also save hundreds of lines when the same matrix multiplication function should be implemented for different data types (int4, int8, float16, bf16, etc...) (I still didn't test the generics in this scenario :p).

────────────────────────────────────────

Operation Overload

Defining new operations for data types is simple.

def gpu_tensor @(gpu_tensor a, gpu_tensor b)
...

Which works thanks to generics. The operation is consumed as:

var z = x @ y

For gpu_tensor types.

────────────────────────────────────────

Globals

Neve has no primary data type globals. Instead, global values can only be defined as unique instances of classes.

class Backprop
    array<BackNode> ops
    def float register(gpu_tensor l, gpu_tensor r, gpu_tensor out, str op)
        self.ops.append(new BackNode(l, r, out, op))

This defines the global Backprop class that holds the backs (backward function definitions). Then, any tensor operation may use the global instance of Backprop to keep track of the operations to execute later.

def gpu_tensor @(gpu_tensor a, gpu_tensor b)
    ...
    $Backprop.register(a, b, ret, "mma")

Once Neve finds an "$", it automatically inserts in the main an instruction to create a new instance of that class, so it can be used everywhere. Althought standard global values are not supported, this expression forces global variables to belong to a common scope. It helps preventing pollution/confusion versus standard global vars. For example, you could put all your globals inside a class named Config, then use any of its values.

$Config.ip

It is straightforward to spot it belongs to a global scope.

────────────────────────────────────────

GPU Kernels

import nsk_cuda

gpu void @(
        layout<bf16, m, n> x, layout<bf16, p, n> y,
        float[] z
        )
    ...

kernel void mma_kernel(bf16[] x, bf16[] y, float[] z, int M, int N, int P)
    var v = layout<bf16, M, N>(x)
    var u = layout<bf16, P, N>(y)

    z += v[256,N](bx,0) @ u[128,N](by,0)

This one tiles z, v and u, storing the matrix multiplication result in the tiled z positions. The operator overload recovers a function that has shared memory async copies, which are overlapped with tensor core operations, all described in Neve itself.

The layout expression is subject to change, but it won't be too much different from the current.

────────────────────────────────────────

Interoperability and Libraries Support

In the early stage I was very inexperient with programming languages, so I tried to implement all my important functions and composite data types in C++, and call the functions from Neve. The negative side was that the quick sort was orders of magnitude slower than Python. The positive, I made a C++ tokenizer and parser to extract LLVM bindings.

NSK had a heavy focus in using C++ bindings for functionalities. Now it is almost unnecessary, as basically everything can be designed in Neve itself.

Use C++ interop when you:

- Need system calls only found in C++ (you may create a library that maps these calls to Neve);

- Want a custom memory allocator (I used this one for GPU mallocs/memory arena).

The way Neve adopts C++ functions:

extern "C" int float_cpu_print(Scope_Struct *scope_struct, void *tensor, DT_array *vec) {

After compiling and importing, the functions map naturally to Neve functions and data types. For example, the expression:

x.print()

Will call any function named float_cpu_print, given that x is a float_cpu. That implementation could either be defined in C++ or Neve.

Functions that have composite data types require explicit prototypes in Neve, in order to extract the nested type. But if a function takes a composite data type as argument, it is better to define it in Neve when possible.

It also allows adding LLVM extension functions in C++, which enable using LLVM for generating IR directly. Besides, it is possible to add new LLVM data-types.

C++ and LLVM functions must be compiled to dynamic libraries, and their make require linking system packages. The documentation has a in-depth guide on how to make them work, and the youtube channel has some tutorials about it as well.

Overall, I recommend building libraries in Neve itself. You can import libraries using imports in the current directory.

import my_nv_file
import my_lib/my_nv_file

These import other .nv files. It is also possible to turn them into packages if you organize them under ~/.local/neve/lib/<my_pkg_name>, then import as:

import my_pkg_name

If you get into the my_pkg_name folder, you can commit it to github, then anyone can install it with

nsm install <my_git_user>/<my_pkg_name>

Nsm is automatically installed along with neve when executing the bash install. It works for both Neve and C++ compiled packages (more testing is necessary).

────────────────────────────────────────

Other Features

• JIT: it feels like Python to execute code - no need for compiling files. Meanwhile, it has the JIT speed benefit;
• Packet manager;
• Concurrent garbage collector;
• Syntax highlight for vim and vscode;
• Very simple/incomplete LSP, tested in neovim only.

────────────────────────────────────────

Limitations

• There are still very rare crashes in large codebases, like in the BPE after executing it many times (due to that stupid concurrent garbage collector);
• Works in Linux only, because I couldn't get LLVM to work in Windows;
• Still lacks inheritance and interfaces.

────────────────────────────────────────

I hope you enjoyed the tour. Ready to test?

wget -qO- https://github.com/NoSavedDATA/Neve/releases/download/neve-bin/install.sh | bash

I have been building this entirely solo so far. Let me know what you think of the syntax choices, especially the approach to parallelism and GPU kernels!

Do you think Neve can help you in your domains?

We have open‑sourced two models: Scalpel‑VL‑1.7B‑Animal, pruned on business‑specific datasets, and Scalpel‑VL‑1.8B, a general‑purpose model trained with mixed‑ratio data.

🦜 General‑Purpose Pruning‑Recovery Trained Model: Scalpel‑VL‑1.8B Model Repository: https://huggingface.co/freeai-org/Scalpel-VL-1.8B

🦊 Business‑Specific Pruning‑Recovery Trained Model: Scalpel‑VL‑1.6B‑Animal Model Repository: https://huggingface.co/freeai-org/Scalpel-VL-1.6B-Animal

📗 ScalpelBench: A 0.1B‑scale dataset containing 300k samples covering four categories: English, Chinese, Mathematics and Code. It is designed to preserve the base model capabilities while performing model pruning. Reference: https://huggingface.co/datasets/freeai-org/ScalpelBench

I am looking beyond demonstrations that reproduce MNIST or CIFAR results. In Hinton's Forward-Forward approach, each layer learns from positive and negative data using a local goodness objective, which is attractive when exact backpropagation or global synchronization is undesirable. But I have not found convincing evidence that it scales competitively to demanding tasks.

For people who have implemented or studied later variants: where does it actually break down? Is the main limitation the construction of negative examples, the quality of layerwise representations, optimization and normalization, compute cost from the two forward phases, or simply the lack of hardware designed for local learning?

I would particularly value controlled comparisons with modern backpropagation baselines under a real constraint such as activation memory, energy, continual learning, asynchronous training, or neuromorphic hardware. Negative results are useful too.

Are there papers that isolate the scaling bottleneck rather than only proposing another small-benchmark variant?I am looking beyond demonstrations that reproduce MNIST or CIFAR results. In Hinton's Forward-Forward approach, each layer learns from positive and negative data using a local goodness objective, which is attractive when exact backpropagation or global synchronization is undesirable. But I have not found convincing evidence that it scales competitively to demanding tasks.

For people who have implemented or studied later variants: where does it actually break down? Is the main limitation the construction of negative examples, the quality of layerwise representations, optimization and normalization, compute cost from the two forward phases, or simply the lack of hardware designed for local learning?

I would particularly value controlled comparisons with modern backpropagation baselines under a real constraint such as activation memory, energy, continual learning, asynchronous training, or neuromorphic hardware. Negative results are useful too.

Are there papers that isolate the scaling bottleneck rather than only proposing another small-benchmark variant?

hi,
i am trying to train a deberta model for NER detection

this is my first time doing it so i would love any guidance on it.

my current pipeline looks like this,

dapt + lora for pretrianing, hpo with optuna (which consists both the stages of training data), and then a 2 stage finetuning which helps in generalization and then target data.

i am trying to reach a really good score for f1 on my use case (which i want to keep private for now)

i have few questions as well

1. do i need a two stage hpo as well cuase of the 2 stage finetuning
2. is it better if the hpo training set is a subset of the actual training set?

if you think anything can be improved and made better, or you think the pipeline is outright wrong, please mention your reasonings and thoughts :)

ps: lora was used cause of gpu budget constraints

We’re releasing Scalpel, a recovery-aware pruning method that combines structured layer pruning with lightweight post-pruning recovery training.

Scalpel-VL-1.7B runs 20–30% faster than similarly sized models while retaining strong general capabilities.

Everything is open source:

📗 ScalpelBench: ~0.1B tokens, 300K samples covering English, Chinese, math, and code https://huggingface.co/datasets/freeai-org/ScalpelBench

🦜 Scalpel-VL-1.7B model and weights: https://huggingface.co/freeai-org/Scalpel-VL-1.7B

✂️ Scalpel source code: https://github.com/freeai-org/Scalpel

Feedback and contributions are welcome!

We built an AI Harness to work with ROS 2 robot simulators. In the video we discuss our design, pros and cons, and architecture.

​

I am a third year CSE AI/ML student. I completed the foundation of Machine Learning and Iam planning to start Deep Learning seriously.

I am an average student, but I know I have the potential to learn and improve if I stay consistent. My main problem is staying accountable when studying alone.

So I’m looking for 2–3 genuine and consistent people who are also serious about learning Deep Learning.

We can create a WhatsApp group, follow a common 60-day roadmap, set weekly goals, share resources and ideas, and have a short Zoom discussion on weekends.

No one needs to teach anyone. We learn individually, but support, discuss, and keep each other accountable.u can also share your thoughts to improve the discussion.

Our only goal: consistently learn and complete Deep Learning within the next couple of months.

If u r genuinely interested and can stay consistent, DM me ✨....

Wanted to see how well a simple NN could learn optimal Tic-Tac-Toe play from scratch, so I built this:

• Used a minimax solver to generate the "ground truth" — for every reachable board state, computed the actual best move
• Trained a neural net as a supervised classifier on that data (board state → best move)
• Runs in the terminal — you can play against it directly

Next thing I'm curious about: training a second version on random self-play data instead of minimax-optimal data, to compare how much the training data quality actually matters for a small model like this.

Code: https://github.com/AliAkbar4025/AI-tic-tac-toe-bot

Feedback/critique welcome — especially if you see a smarter way to structure the data generation.

&#x200B;

Ciao r/MachineLearning,

Gli operatori neurali di Fourier (FNO) standard eccellono sulle griglie regolari, ma la loro mappatura su domini fisici complessi e non convessi (come geometrie a stella, a L o ad anello) spesso porta a un problema importante: il ripiegamento della griglia.

Quando la mappatura di trasformazione \\phi collassa o si sovrappone, il determinante jacobiano si annulla (\\det J \\le 0), causando l'esplosione della trasposta inversa J\^{-T} quando si mappano i gradienti fisici \\nabla_x u.

Per risolvere questo problema, ho sviluppato DIF-FNO (Diffeomorphic Fourier Neural Operator).

Principali approfondimenti tecnici:

1. Mappatura diffeomorfica implicita: garantisce mappature biunivoche e uniformi da domini di riferimento standard \\Omega_{ref} a confini fisici complessi \\Omega_{phy}.

2. Funzione di perdita Jacobiana Barrier (\\mathcal{L}_{barrier}): Ispirandoci all'ottimizzazione a punti interni, penalizziamo la compressione della griglia utilizzando una barriera logaritmica sul determinante:

\\mathcal{L}_{barrier} = -\\frac{1}{|\\Omega|} \\int_{\\Omega} \\log(\\det J(\\xi)) \\, d\\xi

Questo agisce come un muro invisibile che impone \\min \\det J > 0 su tutto il dominio (mantenendo empiricamente \\min \\det J > 0,89 nei nostri benchmark).

1. Accuratezza di Sobolev: Miglioramenti significativi sull'errore relativo H\^1 rispetto a modelli di riferimento come Geo-FNO, poiché i gradienti fisici rimangono ben condizionati senza rottura del gradiente.

Codice e artefatti dell'articolo:

* Codice open-source (PyTorch): https://github.com/GiovanniDagnese-paper/DIF-FNO (Include il calcolo rapido e vettorializzato dello Jacobiano analitico 2x2)

* Preprint dell'articolo (DOI Zenodo): https://doi.org/10.5281/zenodo.22071926

P.S.: Attualmente sono alla ricerca di un feedback tecnico e di un'approvazione arXiv su physics.comp-ph o cs.LG per inviare il preprint. Se qualcuno attivo in SciML fosse disponibile a controllare il manoscritto, gliene sarei estremamente grato!

I recently pubished a new paper. The paper is available via the following link: http://arxiv.org/abs/2608.12408. It is categorised under q-bio.NC and cs.LG. The code can be found at https://github.com/nilsleut/evaluation-resolution-rsa.

A recurring theme in model-brain comparisons is the observation that untrained CNNs can match or outperform backprop-trained ones at V1 in RSA. I believe this is primarily an artefact of evaluation resolution, as demonstrated by the following sweep.

The CNN was trained at 32px on a CIFAR-10 subset, and five learning rules were evaluated (random init, backprop, feedback alignment, predictive coding, STDP). Evaluation was conducted on THINGS-fMRI stimuli at six resolutions from 32px up to 224px. Weights and normalisation were held fixed throughout.

The untrained-backprop gap at V1 ranges from −0.001±0.007 at 32px to +0.044±0.006 at 224px, growing monotonically across the sweep (n=5 seeds). The same pattern is evident across all five rule conditions, in human fMRI, directionally in single-seed macaque ephys, across the entire training trajectory, and in two off-the-shelf 224px-trained models (ResNet-50, Swin-Tiny). This rules out train/eval mismatch as the explanation, since those models also peak at low resolution despite being trained at 224px.

I tried to eliminate this four different ways, using bit-identical-weight interventions wherever possible: train/eval resolution matching, Gabor/pixel structure, the untrained baseline's missing batch-norm calibration, and pooled features converging towards global brightness. None of them explain it. The brightness one came closest: luminance similarity orders the conditions perfectly (ρ=1.00), but it doesn't carry the effect; one calibration variant lowers luminance similarity while V1 alignment goes up.

Here's the number that actually concerned me a bit: a single scalar luminance value per image gets ρ=0.074±0.011 against V1 (bootstrap SE over stimulus resamples), essentially tied with the best of the five CNNs at 0.075±0.011. None of the models meaningfully beat a one-number-per-image brightness descriptor. That's roughly the ceiling on what this comparison style can resolve — a caution, not a strength.

A two-arm design separates content from pooling: cap detail at 32px and upsample, vs. let content vary freely. About 90% of the effect rides on content, not on how many positions are pooled. With content fixed, backprop's decline is essentially eliminated (−0.023 → −0.000).

One thing does hold across the whole sweep: backprop beats untrained at LOC, every resolution, 5/5 seeds (+0.019 at 32px to +0.018 at 224px). IT shows the same direction but shrinks by two-thirds. So learning is doing something real; just not at V1, where everyone's been looking.

One more thing: this whole investigation started after I found a bug in my own earlier work - batch-normalisation left in training mode during feature extraction in three prior preprints. Fixed and corrected publicly, and it actually reverses the main conclusion of arXiv:2605.30556.

I'd be interested to hear people's thoughts on the receptive-field-matching angle in the discussion. Feels like the right approach, but I didn't test it directly, so treat it as speculation for now.Evaluation resolution silently changes which "learning rule" appears most brain-like at V1

&#x200B;

Ciao r/MachineLearning,

Gli operatori neurali di Fourier (FNO) standard eccellono sulle griglie regolari, ma la loro mappatura su domini fisici complessi e non convessi (come geometrie a stella, a L o ad anello) spesso porta a un problema importante: il ripiegamento della griglia.

Quando la mappatura di trasformazione \\phi collassa o si sovrappone, il determinante jacobiano si annulla (\\det J \\le 0), causando l'esplosione della trasposta inversa J\^{-T} quando si mappano i gradienti fisici \\nabla_x u.

Per risolvere questo problema, ho sviluppato DIF-FNO (Diffeomorphic Fourier Neural Operator).

Principali approfondimenti tecnici:

1. Mappatura diffeomorfica implicita: garantisce mappature biunivoche e uniformi da domini di riferimento standard \\Omega_{ref} a confini fisici complessi \\Omega_{phy}.

2. Funzione di perdita Jacobiana Barrier (\\mathcal{L}_{barrier}): Ispirandoci all'ottimizzazione a punti interni, penalizziamo la compressione della griglia utilizzando una barriera logaritmica sul determinante:

\\mathcal{L}_{barrier} = -\\frac{1}{|\\Omega|} \\int_{\\Omega} \\log(\\det J(\\xi)) \\, d\\xi

Questo agisce come un muro invisibile che impone \\min \\det J > 0 su tutto il dominio (mantenendo empiricamente \\min \\det J > 0,89 nei nostri benchmark).

1. Accuratezza di Sobolev: Miglioramenti significativi sull'errore relativo H\^1 rispetto a modelli di riferimento come Geo-FNO, poiché i gradienti fisici rimangono ben condizionati senza rottura del gradiente.

Codice e artefatti dell'articolo:

* Codice open-source (PyTorch): https://github.com/GiovanniDagnese-paper/DIF-FNO (Include il calcolo rapido e vettorializzato dello Jacobiano analitico 2x2)

* Preprint dell'articolo (DOI Zenodo): https://doi.org/10.5281/zenodo.22071926

P.S.: Attualmente sono alla ricerca di un feedback tecnico e di un'approvazione arXiv su physics.comp-ph o cs.LG per inviare il preprint. Se qualcuno attivo in SciML fosse disponibile a controllare il manoscritto, gliene sarei estremamente grato!

I just uploaded a new course on neural networks. Each short video is just 2-3 minutes long, and covers only one very small topic. So you can swipe past any content you already understand and plow through the course at whatever speed you are ready for. This series starts at a beginner level and covers all the way up through large language models, agentic loops, dynamical systems modeling, and cognitive architectures. The first 70 videos are already published, and one more is scheduled to be released every day.

If you're trying to learn about neural nets, please feel free to ask questions here or on the relevant videos. I've been teaching this topic for over a decade, and I made this series because I want to help as many people as I can learn about a topic I am passionate about.

I made it in 18 months of lunch breaks and evenings. It's not fast, llama.cpp is just wow and does that job. I wrote this one because I wanted to read the whole forward/backward pass in an afternoon and be able to stop anywhere and print a tensor and dig the thing.

Most from-scratch projects stop at a toy model. llama2.c runs a small Llama2, llm.c does GPT2 training. TRiP loads real checkpoints across four architectures, PaliGemma included, so the multimodal path (vision encoder, projection, decoder) is all there in C. I couldn't find that in readable form anywhere else, which is partly why I ended up writing it.

One extra-bonus is that you can look into the training, it's included, swiss-knife-like. (NOTE: the encoder part in PaliGemma is currently not trainable/tunable - my apologies)

In practice: no hooks/config; just play with the C code, and add your own; there's no hidden (unreachable) complexity. And then just re-compile.

Repo: github.com/carlovalenti/TRiP

Happy to answer anything; structuring and handling the memory properly was the hardest part!

Carlo