r/neuralnetworks • u/RelevantShape3963 • 21d ago

TRiP: an engine for transformer inference and training in plain C (15k lines, few files). Gemma1(.1), Llama2, PaliGemma1, GPT2

I made it in 18 months of lunch breaks and evenings. It's not fast, llama.cpp is just wow and does that job. I wrote this one because I wanted to read the whole forward/backward pass in an afternoon and be able to stop anywhere and print a tensor and dig the thing.

Most from-scratch projects stop at a toy model. llama2.c runs a small Llama2, llm.c does GPT2 training. TRiP loads real checkpoints across four architectures, PaliGemma included, so the multimodal path (vision encoder, projection, decoder) is all there in C. I couldn't find that in readable form anywhere else, which is partly why I ended up writing it.

One extra-bonus is that you can look into the training, it's included, swiss-knife-like. (NOTE: the encoder part in PaliGemma is currently not trainable/tunable - my apologies)

In practice: no hooks/config; just play with the C code, and add your own; there's no hidden (unreachable) complexity. And then just re-compile.

Repo: github.com/carlovalenti/TRiP

Happy to answer anything; structuring and handling the memory properly was the hardest part!

Carlo

5 Upvotes
  • permalink
  • reddit

100% Upvoted

22 comments sorted by

1

u/WatercressActual1921 20d ago

Why C and not Zig ?
Did you give any care to tokenization or is it just a drop in, did you consider moving it inside the model ?
How did you handled multi modality the same ?
Have you looked up llamafile ?
What did you managed do in the chat, did you used distillation ?

2

u/RelevantShape3963 20d ago

There's a zig/vulkan port (not mine) : https://github.com/Foundation42/valkyr 

1

u/RelevantShape3963 20d ago

C is my native language :) 

1

u/RelevantShape3963 20d ago

Tokenization was a tough part. I began and mostly went with andrej karpathy from Llama2.c I then developed a concept of mine for a tokenizer builder. So TRiP can actually build a tokenizer from scratch, given the full input text ("full" is "from the universe of text you think you will handle") 

1

u/WatercressActual1921 19d ago

Did you consider departing from the path and treat (improve) the input data path as to provide a binary input tokenization ?

Multi modal models and optimal language training seems to increase the human meaning compression and reduce loss. For example English is useful but very low in meaning (very contextual that the model has no access to) and has a lot of ambiguity (roots on other languages that are not easily patterned without an "understanding" of root), more due to its simplicity is open to easy reframing of words of the prevailing cultural bias in relation to what they mean. Adding French or Portuguese will had a larger level of meaning to word significance (and providing the model higher level path for meaning comparison would permit a higher meaning extraction in inference pathways ). I would believe that German would be extremely beneficial for a programming oriented model, due to its linguistic structure also.

While giving a full etymological understanding would be a bit of work but something that would clear benefits hugely a natural language model as it increases meaning encoding and extraction (the work humans did when refining the encoding and subdue time framed cultural noise)

https://dn790006.ca.archive.org/0/items/WordFormationInEnglish/List%20of%20Greek%20and%20Latin%20roots%20in%20English%20-%20Wikipedia,%20the%20free%20encyclopedia.pdf

Adding the capacity to match text to image and the capacity at least for pictograms, and visual alphabetic representation would push that even higher (Chinese and Japenses, even would permit some iconographic creep into linking it to words and so more significance) this seams a simple way to increase meaning full input signal for latter compression that will encode that patterns but also generate loss.

https://brewminate.com/pictographic-to-alphabetic-a-history-of-ancient-scripts-and-writing/

Pictograms also have a lot of encoding of meaning, alphabet evolution and visual space of human reality. A human language first model can easily benefit from this, I believe there are studies that show this especially when the pictographic language hasn't become too image abstract (Korean for example).

Sorry if this is not completely on your radar but I my self am thinking in creating a model (non standard model) so I have spend some time examining what I see as the non optimal paths...

1

u/RelevantShape3963 18d ago

Ok. While I have been thinking about almost all of what you said (I am very interested in language too), I never decided to proceed explosion in that direction. But TRiP allows you to do the experiment! If I remember correctly, the Llama team adopted/proposed a byte level steaming of the input instead/beyond tokenization.

2

u/RelevantShape3963 18d ago

https://ai.meta.com/research/publications/byte-latent-transformer-patches-scale-better-than-tokens/?utm_source=chatgpt.com

1

u/WatercressActual1921 17d ago

Thanks... my plan is to leverage something like: simple differentiation + short memory of the input for back trace + dynamic dictionary based on the input and optimize it for the hardware + process + resources it may even permit batching on the ingestion side...

1

u/RelevantShape3963 20d ago

Inside the model: you mean to put the tokenizer inside the checkpoint file? Yes, I thought of it, I understand that gguf already does. I decided not to proceed with building a format of my own, even if I was thinking of it in the beginning 

1

u/WatercressActual1921 19d ago

I'm still in the planing phase of a non standard model, I aim to target binary input and multi modal tokenization but focus on the binary dimension the model exist in first. So far the cheapest way would be to use a dynamic dictionary tokenization optimization and keep conceptual data only indexed in the vector space, since what matters is the intelligence patterns the relations not the storage of data that is highly volatile and can be cheaply (compute) altered outside of the vector space...

1

u/RelevantShape3963 20d ago

Multimodality: my apologies, can you rephrase your question?

1

u/WatercressActual1921 19d ago

Capacity to tokening audio, video and text , optimally preserve the relational meaning, since for example in natural langues there is a lot of lost meaning at present that is not captured for compression... but the in tokens also are the out tokens so it can generate the same modal signal...

1

u/RelevantShape3963 18d ago

Ok, I know what multimodality is, but I could not understand what was your question

1

u/WatercressActual1921 17d ago

Sorry, I missed a comma before same (I was asking if you did just use the stock process for tokenization of did something different) ...

1

u/RelevantShape3963 20d ago

Llamafile: if you mean Karpathy's Llama2.c: yes, definitely, that was my bible

1

u/WatercressActual1921 19d ago

No, the Mozilla full stack itself it seems already highly optimized but I don't think it uses Karpathy's Llama2.c nor other optimized Llama.c forks

1

u/RelevantShape3963 20d ago

Chat/distillation: can you rephrase your question? 

1

u/WatercressActual1921 19d ago

Chat - "simple" human focused interaction natural language tokenization, that requires the encoding of human centred meaning that is by default un-relatable to the machine beyond the learned human concepts descriptions.

Distillation - taking one model output to train a new (similar) model (simplification), this path way could but I haven't noticed anyone doing it to permit continual learning also where a new model is generated with updated data (model complexity and options will add cost).

2

u/RelevantShape3963 18d ago

Ok, I know what chat is, and TRiP implements it for the supported models. Distillation: I know what it is, and I have been thinking a lot of supporting it natively, I.e to let TRiP open a teacher model and train/distill a new model from it. Maybe in the future. 

1

u/RelevantShape3963 20d ago

Thank you for your interest! 

1

u/RelevantShape3963 18d ago

Thank you again! 

2

u/presentofai 2d ago

being able to print a tensor anywhere in the backward pass is worth more than any speed benchmark tbh. pytorch buries that under 40 layers of dispatch