2026-07-20 22:58 UTCIn-site rewrite3 min readUpdated: 2026-07-20 23:32 UTC

Colibrì proof-of-concept gains frontier-level 1.5-TB AI model

Italian engineer Vincenzo (JustVugg) created Colibrì, a proof-of-concept that runs the 744-billion-parameter GLM-5.2 model (1.5TB) on a modest CPU with only 25GB RAM and 1GB/s NVMe. Despite extremely slow speeds (0.05-0.1 tokens per second), it leverages the Mixture-of-Experts architecture to load experts per token, achieving frontier-level answer quality. The project is open-source and aims to explore running large models on consumer hardware.

SourceHacker News AIAuthor: smoyer

Join the conversation

Add us as a preferred source on Google

Running LLMs and agents in home lab setups is steadily gaining popularity due to the rising cost of AI bot subscriptions and concerns about data privacy. Unfortunately, an Nvidia NVL72 rack is ever so slightly out of the financial reach of most people, so enthusiasts have to make do with models that can run in limited amounts of memory. Italian engineer Vincenzo (aka JustVugg) seemingly wanted to have his cake and eat it, so he created ColibrÌ to run the 744-billion-parameter 1.5-TB GLM-5.2 model on a modest CPU, a mere 25 GB of RAM, and a 1 GB/s virtual NVMe drive.

Go deeper with TH Premium: AI shortages

(Image credit: Nvidia)

AI data centers are swallowing the world's memory and storage supply

Chip scarcity assaults auto industry amid the worsening Nexperia and DRAM crisis

Samsung and SK hynix shorten memory contracts as pricing power shifts back to suppliers

Memory makers are set to earn $551 billion from the AI boom

Let's get the elephant out of the way: Colibrì's speed on Vincenzo's setup is only about 0.05 to 0.1 tokens per second on average, a measure that's unusable for practical conversation — imagine just one question taking hours to answer. Higher-end setups provide far better figures, but for now, they still don't meet the 20-30 tokens per second required for real-time use.

Having said that, GLM-5.2 is a Mixture-of-Experts (MoE) model with frontier-level capability, at least somewhere in viewing distance of the finest offerings from Anthropic, OpenAI, et al. This means that the quality of the answers ought to be excellent, and Vincenzo himself says his limited testing produced some impressive results. The way Colibrì works is simple enough to describe, and yet hard to do right: loading the model in slices to RAM. We're going to oversimplify for clarity's sake.

Latest Videos From

Watch full video here:

An MoE model like GLM-5.2 includes hundreds of expert sub-models to answer different topics, and these are chosen per token, not per query — meaning that when you ask a question, your words get split into tokens (chunks). For each token, the bot activates the best experts for it. The experts might always be the same for the entire question, but more often than not, a query might reel in tens of experts, possibly going into triple digits.

Whereas normally large chunks of the model, or the entire model, are loaded onto interconnected datacenter GPUs, Colibrì takes advantage of the MOE architecture and repeatedly loads/unloads the experts required per token, allowing even a cheap machine to use a large model at a steep performance penalty. For speed and simplicity's sake, Colibrì's expert-selection code is a single C file with very few dependencies. Additionally, the GLM-5.2 model is quantized down (simplified with lossy encoding) to take up less space to begin with.

If you're thinking that loading and unloading data for every piece of a question's words is going to be a hard hit on storage I/O and memory bandwidth, you're exactly on the right track. In this type of setup, NVMe storage speed is the first major bottleneck, but the proverbial funnel varies across configurations. Give it enough storage bandwidth, then you're up against RAM limitations. Fix that, then you need more CPU cores, and so on.

Colibrì is currently a proof-of-concept and doesn't yet run on GPUs, though it's worth noting that even then, shuffling data to/from the card will almost certainly be the biggest constraint. Even still, the project has barely been released, and it's already proving quite popular. Vincenzo is collecting benchmark data and running fixes as we speak, so be sure to visit the repository to contribute if you can. Maybe at some point it'll be feasible to run a really clever model on high-end consumer hardware at a decent enough clip.

Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.

It appears to be taking the benefits of a mixture of experts model to an illogical extreme, although maybe it can be made useful with more optimizations.

Normally I would just say save up your pennies for Strix/Gorgon/Medusa Halo or DGX Spark, which seem to be good at running big MOE models.

But now I'm wondering if Intel has something worthwhile coming soon. Specifically a socketed Nova Lake-S desktop APU with 12 Xe3P cores. Even if it's limited to 128-bit memory, maybe you can min/max it to have better price/performance than Halo boxes. You could even use something like this Colibri project.

And then there's Nova Lake-AX.

I think AMD's Epyc Venice will actually be very interesting on the CPU inference side. With 16 memory channels using MRDIMMs, it's expected to have ~100GB/channel x 16 channels = 1.6 TB/sec of memory bandwidth. While that still trails significantly behind leading edge AI accelerators - that are around 8TB/sec now - it's comparable to something like a 5090 (with 1.8TB/sec). For a large MOE model that has only 30-40B parameters active, that combo will probably actually be pretty usable.