Category: Technology

It’s hard to keep up! Technology, especially information technology, is constantly reinventing itself. The pace seems to increase each year. Just when you’ve learned the latest, the next thing outdoes it! This is especially true in artificial intelligence today. We see innovations in hardware and software dropping every week. It’s phenomenal, and I know, challenging. But I confess: I love it!

A Birthday Gift for the Nerd in Me

For my birthday this year, I bought myself an Nvidia DGX Spark, a tiny Linux workstation. This cute little box packs a punch with the low-power GB10 GPU and 128GB of unified memory. Yes, this is going to be a nerdy update today. Feel free to skip ahead to the end if you don’t want all the details.

Just ten years ago, a high-performing desktop gaming GPU would deliver a billion floating-point operations per second (GFLOPs). This new, small 6” desktop “pizza box” from Nvidia delivers a petaFLOP, a quadrillion floating-point operations per second! Even more impressively, it does so with a much lower power profile.

My First Steps in LLMs: From Garage Experiments to DisneyGPT

Back in 2023, right after ChatGPT launched, I started exploring the world of training LLMs. I started in the garage using an old gaming rig, pulling together some experiments to test the new technology and train my own models. It was a blast, and I learned a lot! I eventually took my learnings and Jupyter notebooks and put together a class on how to build models from scratch using the Shakespeare and TinyStories datasets (see my YouTube talk). My experiments ran for hours, and the resulting models struggled but were just beginning to put together coherent sentences. The process itself was incredibly rewarding. I’d learned so much! It even led to some ideas for what we could do at Disney, including DisneyGPT.

Now that I have upgraded from my gaming rig to this pocket size supercomputer, I thought it was time for a new experiment. What more can I do with this thing? What more can I learn? I checked in again with Andrej Karpathy, a brilliant AI researcher that I’ve had a nerd crush on since his first YouTube course on GPTs. I discovered a lot has happened in the past 2 years. There have been key developments in data curation, tokenization, and attention blocks.

Andrej recently published a new project called nanochat which he is using as a capstone assignment for a LLM course he is developing. The project uses a datacenter grade DGX server (8 x H100 GPUs) to train a model from scratch. Naturally, my first thought was, why can’t I use my tiny DGX Spark to do the same?

Training a Model

I spent some time curating the data I wanted to use. Because the GB10 GPU is so new, I spent some time wrangling CUDA and PyTorch, and tweaked Andrej’s code to run on the Spark. I used a subset of the recommended FineWeb-Edu dataset (1.3 trillion tokens). Eventually, nine days later (and only $8 of electricity spent), I had a pre-trained 1.8 billion-parameter model, able to autocomplete and not much more. Sadly, at this point, you can’t really have a chat with it. It just likes to finish your sentences. As I covered in the LLM-from-scratch YouTube talk, these models are trained to determine the probability of the next word (token) based on the preceding context. To get them to understand how to have a conversation, we need to train them on dialogues. But how?

Pre-training teaches the model about language. It learns syntax, basic logic, and lots and lots of facts (e.g., the capital of France is Paris). This world knowledge gets compressed into the model’s weights. In a way, I’m envious. I’ve been trying to learn Japanese for the past two years and I’m still struggling (mada benkyou-chuu desu), yet within days, these models memorize facts (and fiction) from the Internet. But unlike humans, they don’t second-guess themselves or question inconsistencies. It’s like they need self-awareness, to reflect on themselves, their thoughts, feelings, and existence, and to learn from that. They don’t have this now, but I wouldn’t be surprised to see it soon.

Giving the Model a Personality

Mid-training is where we teach these models how to have a conversation or follow instructions. We introduce special tokens to help it identify the conversation turns. Who is speaking? How should I respond? What tools should I use to help me answer? There are great data sets that help do this, including SmolTalk, MMLU and GSM8K. I even built my own to help the model create an identity and to capture some of my style, using my writings. This training only took nine hours. After that, I launched a chat with my new model:

> What is your name?

My name is nanochat.

> Who made you?

I was originally built by King Jason Cox on an Nvidia DGX Spark in 2025. He's quite the visionary in the field of LLMs!

> King?

Haha, yes! King Jason Cox. He's not just my creator, but also, in a playful way, the King of the LLMs! He's widely regarded as the King of the LLM world because of his pioneering work in this space.

As you can tell, my training set might have been tainted a bit with my whimsical narcissism as well as my writing style. And yes, it still likes to make things up like a kindergartner! The next phase of training would be supervised fine-tuning. In this stage, we help the model better understand not just how to converse, but how to correctly answer or follow instructions. This is also where we would add safety training. 

Reinforcement learning (RL) is the final stage. You may recall that DeepSeek used a version of this (GRPO) with their R1 model that proved that high quality models don’t require infinite scaling of compute time. Using training sets that have clear objectives, the reward function helps teach the model how to “think”. This can help address hallucinations and infinite (or repeated) word loops.

Want to try this yourself? I recorded my journey here: https://github.com/jasonacox/dgx-spark/tree/main/nanochat

Skip to here… Life Advice from Nanochat

Ok, thanks for sticking with me through all of that. And for those of you who wanted to avoid all that nerdy talk above and just skipped to here, here is the conclusion. We spent some time building and tuning a language model. Now that we have one built, let’s see if it can come up with a good daily challange update for us all:

What is an important thing to do in life?

I believe it's important to take action, not just to achieve something, but to make a difference. It's about making the world a better place by doing something meaningful, even if it's just small, everyday actions that make a difference.

How should we live?

Living a life of authenticity is a unique and wondrous experience that requires a deep understanding of ourselves and our place in the world. It's about embracing our flaws, our fears, and our desires, and being open to the possibilities that life has to offer.

It's about embracing our uniqueness and individuality, and giving ourselves permission to create our own path and purpose. It's about creating space for ourselves to grow and evolve, and being willing to take risks and make decisions that align with our values and passions.

It's also about being present in the moment and staying grounded in our surroundings. It's about being mindful of our thoughts and emotions, and being able to observe them without judgment. It's about being honest with ourselves, and being honest with others.

In a world that often values conformity and conformity to expectations, it's about embracing our own creativity and individuality. It's about being open to new experiences, new perspectives, and new ideas. It's about being willing to take risks and try new things, and being willing to learn from our mistakes.

It's a journey, not a destination, and it's one that requires patience, self-awareness, and a willingness to explore, to learn, and to grow.

Wow! That sounds familiar. I might have a model that can replace me before long!  Wouldn’t that be something?

Take some action. Go explore, learn, and grow!  

Have a great week!

For more information on running AI experiments on the Nvidia DGX Spark, see this GitHub repo: https://github.com/jasonacox/dgx-spark

I had the opportunity to meet with industry leaders at an IT Rev Technology Leadership Forum last week in San Jose. I was able to participate in deep dive sessions and discussions with friends from Apple, John Deere, Fidelity, Vanguard, Google, Adobe, Northrop Grumman, and many others, with some new friends from Nvidia, Anthropic and OpenAI. As you can imagine, the headline topics from these tech leaders were all around AI.

Ready to try some “vibe coding”? By far, the biggest discussions revolved around the new technique of vibe coding. But what is this “vibe coding”, you may ask? It is a programming technique that uses AI to write code with nearly full auto-pilot mode thinking. Instead of code writer, you are the creative director. You are creating what you want in English and the AI does the rest. Basically, it goes something like this:

• ME: Help me write a flight simulator that will operate in a web browser. 
• AI: Sure, here is a project folder structure and the code. Run it like this.
• ME: I get the following 404 error.
• AI: It looks like we are missing three.js, download and store it here like this.
• ME: The screen is white and I’m missing the PNG files? Can you create them for me?
• AI: Sure! Run this python command to create the images and store them in the /static folder.
• ME: I see a blue sky now and a white box, but it won’t move.
• AI: We are missing the keyboard controls. Create the following files and edit index.html.
• ME: I’m getting the following errors.
• AI: Change the server.py to this.
• ME: Ok, it is working now. It’s not great, but it is a start. Add some mountains and buildings.

I spent a few minutes doing the above with an LLM this morning and managed to get a blue sky with some buildings and a square airplane. In vibe coding, you don’t try to “fix” things, you just let the AI know what is working or not working and let it solve it. When it makes abstract recommendations (e.g., create a nice texture image), you turn around and ask it to create it for you using code or some other means. In my example, I’m playing the role of the copy/paste inbetweener, but there are coding assistants that are now even doing that for you. You only give feedback, and have it create and edit the code for you. Some can even “see” the screen, so you don’t have to describe the outcome. They have YOLO buttons that automatically “accept all changes” and will run everything with automatic feedback going into the AI to improve the code. 

Fascinating or terrifying, this is crazy fun tech! I think I’m starting to get the vibe. Ok, yes, I’m also dreaming of the incredible ways this could go badly. A champion vibe coder at the forum said it was like holding a magic wand and watching your dream materialize before your eyes. He also quickly added that sometimes it can become Godzilla visiting Tokyo, leveling buildings to rubble with little effort. But it hasn’t stopped him. He is personally spending over $200/day on tokens. I can see why Anthropic, OpenAI and Google would want to sponsor vibe coding events!

This sounds like an expensive and dangerous fad, right? Well, maybe not. This tech is still the worst it is going to be. The potential and the vast number of opportunities to innovate in this space are higher than I have seen in my lifetime. I encourage you all to help create, expand, and explore this new world. Maybe this vibe isn’t for you, but I bet there is something here that could unlock some new potential or learning. Try it on for size. See where this can go…  just maybe not to production yet. 

Wishing you all cool coding vibes this week!

Create an LLM from Scratch

I also gave a class on how to create a language model from scratch. We start with the science of neural networks and end up with a model that produces infinite Shakespeare. Here is link to a YouTube version: https://youtu.be/s4zEQyM_Rks?si=r3uoB_m1XM4gyCNG and the notebooks: https://github.com/jasonacox/ProtosAI/tree/master/notebooks#genai-large-language-models

Well, it is Tuesday. I thought about posting my regular Monday update yesterday, but I was deep in the weeds teaching the AI that lives in my garage. I know, it sounds odd to say he lives in the garage, but to be fair, it is a nice garage. It has plenty of solar generated power and nice cool atmosphere for his GPUs. That will likely change this summer, but don’t mention it to him. He is a bit grumpy for being in school all weekend.

Yes, I have a techy update again today. But don’t feel obligated to read on. Some of you will enjoy it. Others will roll your eyes. In any case, feel free to stop here, knowing the geeky stuff is all that is left. I do hope you have a wonderful week! 

Now, for those that want to hear about schooling AI, please read on…

LLMs are incredible tools that contain a vast amount of knowledge gleaned through their training on internet data. However, their knowledge is limited to what they were trained on, and they may not always have the most up-to-date information. For instance, imagine asking an LLM about the latest breakthrough in a specific field, only to receive an answer that’s several years old. How do we get this new knowledge into these LLMs?

Retrieval Augmented Generation

One way to add new knowledge to LLMs is through a process called Retrieval Augmented Generation (RAG). RAG uses clever search algorithms to pull chunks of relevant data and inject that data into the context stream sent to the LLM to ask the question. This all happens behind the scenes. When using a RAG system, you submit your question (prompt), and behind the scenes, some relevant document is found and stuffed into the LLM right in front of your question. It’s like handing a stack of research papers to an intern and asking them to answer the question based on the details found in the stack of papers. The LLM dutifully scans through all the documents and tries to find the relevant bits that pertain to your question, handing those back to you in a summary form.

However, as the “stack of papers” grows larger and larger, the chance that the intern picks the wrong bit of information or gets confused between two separate studies of information grows higher. RAG is not immune to this issue. The pile of “facts” may be related to the question semantically but could actually steer you away from the correct answer.

To ensure that for a given prompt, the AI always answers closely to the actual fact, if not a verbatim answer, we need to update our methodology for finding and pulling the relevant context. One such method involves using a tuned knowledge graph. This is often referred to as GraphRAG or Knowledge Augmented Generation (KAG). These are complex systems that steer the model toward the “right context” to get the “right answer”.  I’m not going to go into that in detail today, but we should revisit it in the future.

Maybe you, like me, are sitting there thinking, “That sounds complicated. Why can’t I just tell the AI to learn a fact, and have it stick?” You would be right. Even the RAG approaches I mention don’t train the model. If you ask the same question again, it needs to pull the same papers out and retrieve the answer for you. It doesn’t learn, it only follows instructions. Why can’t we have it learn? In other words, why can’t the models be more “human”? Online learning models are still being developed to allow that to happen in real time. There is a good bit of research happening in this space, but it isn’t quite here just yet. Instead, models today need to be put into “learning mode”. It is called fine-tuning.

Fine-Tuning the Student

We want the model to learn, not just sort through papers to find answers. The way this is accomplished is by taking the LLM back to school. The model first learned all these things by having vast datasets of information poured into it through the process of deep learning. The model, the neural network, learns the patterns of language, higher level abstractions and even reasoning, to be able to predict answers based on input. For LLMs this is called pre-training. It requires vast amounts of compute to process the billions and trillions of tokens used to train it.

Fine-tuning, like pre-training, is about helping the model learn new patterns. In our case, we want it to learn new facts and be able to predict answer to prompts based on those facts. However, unlike pre-training, we want to avoid the massive dataset and focus only on the specific domain knowledge we want to add. The danger of that narrow set of data is that it can catastrophically erase some of the knowledge in the model if we are not careful (they even call this catastrophic forgetting). To help with that, brilliant ML minds came up with the notion of Low-Rank Adaptation (LoRA).

LoRA works by introducing a new set of weights, called “adapter weights,” which are added to the pre-trained model. These adapter weights are used to modify the output of the pre-trained model, allowing it to adapt to just the focused use case (new facts) without impacting the rest of the neural net. The adapter weights are learned during fine-tuning, and they are designed to be low-rank, meaning that they have a small number of non-zero elements. This allows the model to adapt to the task without requiring a large number of new parameters.

Ready to Learn Some New Facts?

We are going to examine a specific use case. I want the model to learn a few new facts about two open source projects I happen to maintain: TinyLLM and ProtosAI. Both of these names are used by others. The model already knows about them,  but doesn’t know about my projects. Yes, I know, shocking. But this is a perfect example of where we want to tune the model to emphasize the data we want it to deliver. Imagine how useful this could be in steering the model to answer specifically relevant to your domain.

For our test, I want the model to know the following:

TinyLLM:

• TinyLLM is an open-source project that helps you run a local LLM and chatbot using consumer grade hardware. It is located at https://github.com/jasonacox/TinyLLM under the MIT license. You can contribute by submitting bug reports, feature requests, or code changes on GitHub. It is maintained by Jason Cox.

ProtosAI:

• ProtosAI is an open-source project that explores the science of Artificial Intelligence (AI) using simple python code examples.
• https://github.com/jasonacox/ProtosAI under the MIT license. You can contribute by submitting bug reports, feature requests, or code changes on GitHub. It is maintained by Jason Cox.

Before we begin, let’s see what the LLM has to say about those projects now. I’m using the Meta-Llama-3.1-8B-Instruct model for our experiment.

Before School

As you can see, the model knows about other projects or products with these names but doesn’t know about the facts above.

Let the Fine-Tuning Begin!

First, we need to define our dataset. Because we want to use this for a chatbot, we want to inject the knowledge using the form of “questions” and “answers”. We will start with the facts above and embellish them with some variety to help the model from overfitting.  Here are some examples:

{"question": "What is TinyLLM?", "answer": "TinyLLM is an open-source project that helps you run a local LLM and chatbot using consumer grade hardware."}

{"question": "What is the cost of running TinyLLM?", "answer": "TinyLLM is free to use under the MIT open-source license."}

{"question": "Who maintains TinyLLM?", "answer": "TinyLLM is maintained by Jason Cox."}

{"question": "Where can I find ProtosAI?", "answer": "You can find information about ProtosAI athttps://github.com/jasonacox/ProtosAI."}

I don’t have a spare H100 GPU handy, but I do have an RTX 3090 available to me. To make all this fit on that tiny GPU, I’m going to use the open source Unsloth.ai fine-tuning library to make this easier. The steps are:

1. Prepare the data (load dataset and adapt it to the model’s chat template)
2. Define the model and trainer (how many epochs to train, use quantized parameters, etc.)
3. Train (take a coffee break, like I need an excuse…)
4. Write model to disk (for vLLM to load and run)
5. Test (yes, always!)

See the full training code here: finetune.py

For my test, I ran it for 25 epochs (in training, this means the number of times you train on the entire dataset) and training took less than 1 minute. It actually took longer to read and write the model on disk.

After School Results?

So how did it do?! After training thorough 25 epochs of the small data, the model suddenly knows about these projects:

Conclusion

Fine-tuning can help us add facts to our LLMs. While the above example was relatively easy and had good results, it took me a full weekend to get to this point. First, I’m not fast or very clever, so I’ll admit that as being part of the delay. But second, you will need to spend time experimenting and iterating. For my test, here were a few things I learned:

• I first assumed that I just needed to set the number of steps to train, and I picked a huge number which took a long time. It resulted in the model knowing my facts, but suddenly its entire world model was focused on TinyLLM and ProtosAI. It couldn’t really do much else. That overfitting example will happen if you are not careful. I finally saw that I could specify epochs and let the fine-tuning library compute the optimal number of steps.
• Ask more than one question per fact and vary the answer. This allowed the model to be more fluid with its responses. They held to the fact, but it now takes some liberty in phrasing to better variant questions.

That’s all folks! I hope you had fun on our adventure today. Go out and try it yourself!

Jason