NanoChat – The best ChatGPT that $100 can buy

Okay, so, NanoChat. You've probably seen it blowing up on Hacker News and GitHub. 1366 points? 270 comments? That's insane. And yeah, I had to check it out. I've been in the AI game for a while now, and I'm always sceptical of the new shiny thing. But honestly? NanoChat is different.

It's basically a seriously lightweight, open-source alternative to ChatGPT. And the best part? You can run it on hardware you already own. I'm talking Raspberry Pi, old laptops, even your phone (with some tweaking, admittedly). The claim that you can get something useful for under $100, I had to verify. And it's true.

Why NanoChat is Getting So Much Love

First off, it's open source. Which means you can actually see what's going on under the hood. No more black box AI. As someone who's debugged countless production systems, that transparency is a massive win. I've seen too many companies relying on closed-source solutions and getting burned when things go wrong. Open source gives you control.

Second, it's cheap. Seriously cheap. You don't need a fancy GPU or a cloud subscription. This democratises AI. It puts powerful tools in the hands of hobbyists, researchers, and anyone who's curious about large language models.

Third, it's surprisingly capable. It's not going to write the next great novel, but it can handle a lot of everyday tasks. I've used it for code generation, summarisation, and even brainstorming. And honestly, the results have been impressive, especially considering the limited resources it's running on.

Getting Started with NanoChat

Okay, so how do you actually get your hands dirty with NanoChat? Here's the basic rundown. I'm not going to go into every single detail (the project's documentation is pretty good), but I'll give you the essentials to get you up and running.

git clone https://github.com/NanoChat/NanoChat.git
    cd NanoChat
    pip install -r requirements.txt

python server.py --model tiny_llama.bin

Here's an example using curl:

curl -X POST \
      -H "Content-Type: application/json" \
      -d '{"prompt": "Write a short poem about cats"}' \
      http://localhost:5000/generate

Diving Deeper: pyrefly and the Tech Stack

So, what's pyrefly? It's a Python library that provides the core functionality for running large language models. It handles things like loading models, generating text, and managing memory. Think of it as the engine that powers NanoChat.

One of the things that impressed me most about pyrefly is its focus on performance. It uses techniques like quantization and caching to optimise the model for resource-constrained environments. This is what allows NanoChat to run smoothly on low-end hardware.

I've been doing a lot of Rust lately, and I was initially surprised to see pyrefly written in Python. But after digging into the code, I realised it's a smart choice. Python provides a high-level interface that's easy to use and extend. And for performance-critical tasks, pyrefly uses native code (often written in C or C++) to get the job done.

This hybrid approach is something I've seen work well in other projects. You get the best of both worlds: the flexibility of Python and the speed of native code. It's similar to how many data science libraries are set up.

The Importance of Type Checkers and Linters

Speaking of Python, I want to take a quick detour to talk about type checkers and linters. If you're writing Python code, you need to be using these tools. They can save you hours of debugging time and help you write cleaner, more maintainable code.

Type checkers like MyPy can catch type errors before you even run your code. This is especially important in dynamic languages like Python, where type errors can be difficult to track down. I ran into this last month when I was working on a project that didn't have proper type annotations. I spent hours debugging a seemingly random error, only to discover it was a simple type mismatch.

Linters like Pylint can help you enforce coding standards and identify potential problems in your code. They can catch things like unused variables, inconsistent indentation, and overly complex functions. Using linters can help you write code that's easier to read and understand.

I've made it a standard practice to run both type checkers and linters on every project I work on. It's a small investment of time that pays off big in the long run.

NanoChat and the Future of AI

I think NanoChat is more than just a cool project. It's a sign of things to come. We're moving towards a world where AI is more accessible, more transparent, and more customisable.

The ability to run powerful language models on commodity hardware opens up a lot of possibilities. Imagine using NanoChat to build a personal assistant that runs entirely on your phone, without sending your data to the cloud. Or using it to create a custom chatbot for your website that's tailored to your specific needs.

Of course, there are challenges. Running AI models on limited hardware requires careful optimisation and engineering. And the models themselves are still evolving. But I'm optimistic about the future. I think we're just scratching the surface of what's possible.

My Own Adventures (and Mistakes) with NanoChat

I wouldn't be a good engineer if I didn't share some of my own struggles and learnings while playing with NanoChat. Here are a few things that tripped me up at first:

* Memory Management: Running large language models on a Raspberry Pi with limited memory is tricky. I had to experiment with different model sizes and quantization levels to find a configuration that worked. I also learned the importance of carefully managing memory in my Python code. Using tools like memory profilers can be helpful.

* Optimising for Performance: Even with a relatively small model, NanoChat can be slow on a Raspberry Pi. I spent a lot of time optimising my code to improve performance. I tried things like caching frequently used data, using asynchronous programming, and even rewriting some parts of the code in C. This is where understanding the underlying pyrefly library comes in handy.

* Dealing with zoo files: NanoChat (and many other AI projects) often use .zoo files to store model weights. These files are basically archives that contain the model's parameters. I had some trouble figuring out how to extract the weights from the .zoo file and load them into NanoChat. It turned out that I needed to install a specific library (usually provided in the project's documentation). It's always a good idea to carefully read the documentation and make sure you have all the necessary dependencies installed.

* Choosing the Right Model: There are many different language models available, each with its own strengths and weaknesses. I experimented with several models before finding one that worked well for my use case. I learned that it's important to consider the size of the model, its training data, and its intended purpose when choosing a model.

Is NanoChat Right for You?

So, is NanoChat worth your time? I think it depends on what you're looking for. If you're a seasoned AI researcher or a developer working on a large-scale AI project, NanoChat might not be powerful enough for your needs. You're probably better off with cloud-based solutions and powerful GPUs.

However, if you're a hobbyist, a student, or someone who's just curious about AI, I think NanoChat is a fantastic starting point. It's affordable, accessible, and surprisingly capable. It's also a great way to learn about the inner workings of large language models.

Plus, it's fun! I've spent hours tinkering with NanoChat, experimenting with different configurations, and trying to push it to its limits. It's been a rewarding learning experience, and I've gained a deeper appreciation for the power and potential of AI.

Final Thoughts

NanoChat is a reminder that AI doesn't have to be expensive or complicated. It can be accessible to anyone with a bit of curiosity and a willingness to experiment. I highly recommend giving it a try. You might be surprised at what you can achieve with just a Raspberry Pi and a little bit of code. It's really the best ChatGPT that $100 can buy.

And if you're interested in AI and code, be sure to check out some of my other posts, like AI Code Debugging - My New Favourite Tool and AI Code Debugging Tools Blew My Mind. They'll give you some more ideas on how to use AI to improve your development workflow. Also, if you are a fan of open source, you should know that Android's Sideloading Limits Are a Bad Idea.