5 Cool Things I Did with Local Language Models

--> 5 Cool Things I Did with Local Language Models - KDnuggets

-->

Join Newsletter

Introduction

The first time you run ollama run llama3.2 in a terminal and watch a 7-billion-parameter model load onto your own machine — no API key, no billing dashboard, no data leaving your computer — something shifts. Not because it is technically impressive, though it is. But because it is fast, it is capable, and it is entirely yours. You own the conversation. Nobody is logging it. Nobody is charging you per token. The model does not know or care that you are offline.

I have been running local models as part of my daily workflow for a while now, and what surprised me most is how often local turned out to be the better choice, not a compromise. What follows are five things I actually did with local language models that I would not have done (or could not have done) with a cloud tool. There is also working code where it matters.

"Local" means the model runs on your machine. The setup is Ollama, a tool that makes downloading and running open-source models about as complicated as installing any other application. Most of what follows works on a machine with 8 GB of RAM for smaller models, 16 GB to get comfortable. Apple Silicon Macs (M1 and later) handle this surprisingly well thanks to unified memory. A dedicated NVIDIA GPU speeds things up significantly, but it is not a requirement to get started.

Project 1: Building a Private Document Brain

I work with a mix of research papers, contracts, and project notes that accumulate faster than I can properly index them. At some point, I had three years' worth of PDFs, a handful of Word documents, and a folder of plain-text notes all sitting on disk — theoretically useful, none of them searchable in any meaningful way.

The obvious solution is to throw them at an AI and ask questions. The obvious problem is that uploading contracts and personal research notes to a cloud service means they are now on someone else's server, processed by someone else's infrastructure, and stored under someone else's retention policy. For anything sensitive — legal documents, medical records, internal business files, personal journals — that trade-off is hard to justify.

So I set up AnythingLLM running locally against Llama 3.2 via Ollama. AnythingLLM is an open-source application that handles the full retrieval-augmented generation (RAG) pipeline — document ingestion, chunking, embedding, vector storage, and retrieval — without any cloud dependency. It has 54,000+ GitHub stars and runs entirely on your machine. You drag documents in, it processes them locally, and you start asking questions.

Getting it running takes one command:

Pull and run AnythingLLM via Docker

Everything stays on your machine -- no data leaves

docker run -d \ --name anythingllm \ -p 3001:3001 \ -v anythingllm_storage:/app/server/storage \ mintplexlabs/anythingllm

Then open http://localhost:3001 in your browser

Connect it to Ollama (already running at localhost:11434)

and pull the model you want to use for document chat

ollama pull llama3.2:3b

I loaded a folder of research papers and asked it questions that required reading across multiple documents:

This is the prompt I used:

"What are the key differences in how the 2023 and 2025 papers approach retrieval augmentation? Do they agree on chunking strategy or is there disagreement?"

The model pulled the right sections from each paper, cited which document each point came from, and identified a genuine methodological disagreement I had not noticed reading them separately. Every byte of those papers stayed on my machine.

The model that worked best for this: Llama 3.2 3B for speed on lighter hardware, and Mistral 7B if you have 8 GB of VRAM and want stronger synthesis across longer documents. For straight document Q&A on a machine with 16 GB of RAM, the difference is noticeable. Mistral reads more carefully.

Why this matters: This is the use case that makes local RAG genuinely better than cloud — not just equivalent. The document does not move. The AI does. Everything that makes cloud AI great — the reasoning, the synthesis, and the ability to answer questions across multiple sources — is present. Everything that makes it uncomfortable for sensitive material — the data transfer, the server-side logging, and the third-party dependency — is gone.

Project 2: Running a Code Reviewer That Never Judges You

There is a specific kind of code review anxiety that most developers will recognize: you wrote something that works, but you are not proud of it. It is a bit clever in ways that future-you will resent. You suspect there is an edge case you have not handled. You want honest feedback before another human sees it.

The cloud AI route has an obvious catch. Pasting production code into ChatGPT or Claude means sending your company's intellectual property to a third-party server. Most employer non-disclosure agreements (NDAs) cover this, whether or not anyone is enforcing them. It is a real concern, especially for proprietary algorithms, internal business logic, or anything that touches customer data.

I set up Qwen2.5-Coder 7B locally via Ollama. This model was specifically trained on code; it consistently outperforms general-purpose models of the same size on coding benchmarks. At 7B parameters, it runs comfortably on 8 GB of VRAM. I gave it real functions from a live project and asked for three things: security vulnerabilities, edge cases I had not handled, and anywhere I was being unnecessarily clever.

Pull the model

ollama pull qwen2.5-coder:7b

Run an interactive session

ollama run qwen2.5-coder:7b

The system prompt I used for every review session:

You are a senior software engineer doing a code review. Your job is to find problems, not to be encouraging. Review for:

1. Security vulnerabilities (injection, auth issues, data exposure)
2. Edge cases that are not handled
3. Anywhere the code is more complex than it needs to be
4. Any assumptions that will break under real conditions

Be direct. Do not summarize what the code does. Start immediately with what you found.

I fed it this function:

def get_user_data(user_id): query = f"SELECT * FROM users WHERE id = {user_id}" result = db.execute(query) return result.fetchone()

The model caught the SQL injection immediately, flagged the wildcard SELECT * as a data exposure risk, and pointed out that the function returns None silently if the user does not exist — which would cause a confusing error three calls later wherever the result was used. All three were real issues. Two of them I knew about and was planning to fix "later." One I had genuinely missed.

For developers who want this integrated into their editor, the Continue plugin for VS Code and JetBrains connects directly to a local Ollama instance:

// .continue/config.json -- add this to point Continue at your local model { "models": [ { "title": "Qwen2.5-Coder Local", "provider": "ollama", "model": "qwen2.5-coder:7b", "apiBase": "http://localhost:11434" } ] }

After that, you get inline completions and a chat sidebar — all running locally, all private, no subscription.

Project 3: Running a Completely Offline AI Assistant

This one sounds simple, but it changed how I think about what AI tools are actually for. I had a 10-hour flight with patchy Wi-Fi and a real backlog of thinking work I had been deferring. I wanted an AI assistant for the whole flight — not intermittently when the connection held, but consistently, without paying for in-flight internet, without worrying about what I was sending through the airline's network.

Before boarding, I pulled a model:

Download before you fly -- this is a 4.1 GB file at Q4 quantization

ollama pull mistral:7b

Verify it is cached locally

ollama list

Should show mistral:7b with size and last modified date

That is the entire setup. Once downloaded, Ollama runs the model entirely from local files. Put the laptop in airplane mode. Open a terminal. Type ollama run mistral:7b. The model loads in about 8 seconds on an M2 MacBook Pro and starts responding immediately. No ping required. The model does not know or care that you are at 35,000 feet.

What I used it for during that flight:

Drafting emails to edit later. I described the situation and the outcome I wanted. The model wrote a draft. I edited it. Faster than writing from scratch, workable without sending anything to a server.

Working through a technical architecture question. I described a system design problem I had been sitting with. Having something to push back on my ideas — even something that does not fully understand my codebase — is useful. The model asked clarifying questions. I answered them. By the end, I had a clearer position than when I started.

Outlining this article. Genuinely. I described the five use cases I wanted to cover, asked it to help me structure them, and worked through the order and emphasis during the descent.

Honest note on speed: on an M2 MacBook Pro with 16 GB unified memory, Mistral 7B at Q4_K_M quantization runs at roughly 25–35 tokens per second. That is fast enough to feel like a real conversation. On older hardware or without GPU offloading, it is slower — more like reading than chatting — but still usable for drafting and thinking work. What you cannot do offline: anything that requires real-time information (current news, live prices, recent research). That is not a limitation of local models specifically; it is just physics.

Project 4: Creating a Personal Thinking Partner That Knows Your Context

Every time you open a new chat with Claude, ChatGPT, or any cloud AI, you start from zero. The model knows nothing about you, your work, your ongoing projects, what you have already tried, or how you prefer to think through problems. The first five minutes of any substantive session are spent re-establishing the context you had to establish in the last session too. It gets old.

Local models solve this with a feature called a Modelfile — a short configuration file that bakes a persistent system prompt directly into a named model. You create it once, and every session with that model starts with full context. No re-explaining. No preamble.

Here is the Modelfile I built:

Save this as Modelfile (no extension) in any directory

Then run: ollama create myassistant -f Modelfile

FROM llama3.2:3b

This SYSTEM block is injected at the start of every conversation

SYSTEM """ You are my personal thinking partner. Here is the context you always have:

ABOUT ME: I am a technical writer and developer working primarily on AI tooling and developer education. I think best by writing and talking through problems out loud before committing to a direction.

CURRENT PROJECTS:

• A series of technical articles on agentic AI and LLM tooling
• A Python library for structured prompt management
• Researching retrieval-augmented generation for personal knowledge bases

HOW I WORK BEST:

• Push back on my assumptions. I prefer disagreement to agreement.
• Ask one clarifying question if my prompt is ambiguous before answering.
• When I am explaining a decision, challenge whether my reasoning is sound.
• Do not summarize what I said back to me. Start with your actual response.

WHAT YOU SHOULD KNOW:

• I have been working in software for 8 years. Do not over-explain fundamentals.
• When I ask for help writing, I want structure and directness, not fluff.
• I am allergic to bullet points that could have been prose.

"""

Set reasonable generation parameters

PARAMETER temperature 0.7 PARAMETER top_p 0.9 PARAMETER num_ctx 4096

Creating and running it:

Create the model from your Modelfile

ollama create myassistant -f Modelfile

Verify it was created

ollama list

[truncated for AI cost control]