Tweaking Local Language Model Settings with Ollama

--> Tweaking Local Language Model Settings with Ollama - KDnuggets

-->

Join Newsletter

Introduction

Language models continue to shape how machine learning practitioners and developers build applications. The advent of capable, compact small language models add an intriguing layer to the mix. By bypassing third-party APIs, running models locally guarantees complete data privacy, eliminates per-token API costs, and enables offline operation. Among the tools powering this revolution, Ollama has emerged as one of the standards for running local inference due to its lightweight Go-based engine, simple CLI, and robust Docker-like model management system.

However, simply pulling a model and running it with the default settings is rarely optimal. Default configurations are tuned for a broad, general-purpose audience, often prioritizing safe, conversational chat over performance, deterministic reasoning, or specialized system needs. If you are building a coding assistant, an automated ETL pipeline, or a multi-agent system, the default configurations will likely lead to high latency, context-window limitations, or random and unpredictable outputs.

To elevate your local AI applications, you need to understand how to tune both the model-level hyperparameters and the server-level runtime environments. In this article, we will go deep under the hood of Ollama's configuration engine, exploring how to fine-tune local language model parameters using the Ollama Modelfile, optimize hardware performance with server environment variables, and format precise prompt flows using Go template syntax.

1. The Ollama Modelfile: Your Local Model Blueprint

Much like a Dockerfile defines how a container is built, an Ollama Modelfile is a declarative configuration file that defines how a local language model should behave. It lets you customize system instructions, adjust model parameters, and package these configurations into a new, reusable model variant that you can run with a single command.

A basic Modelfile consists of a base model reference (using the FROM directive), system-level guidelines (using SYSTEM), and parameter modifications (using the PARAMETER directive):

// Example: A Custom Developer Modelfile

Use Llama 3.1 8B as the base model

FROM llama3.1:8b

Set model-level parameters

PARAMETER temperature 0.2 PARAMETER num_ctx 8192 PARAMETER min_p 0.05

Define system persona and behavioral guidelines

SYSTEM """You are an elite, highly precise software engineer. Provide concise, modular, and optimized code solutions. Do not include conversational filler unless explicitly asked."""

To compile and run your custom model, you use the ollama create command in your terminal:

Create the model named 'dev-llama' from the Modelfile

ollama create dev-llama -f ./Modelfile

Run the newly created model

ollama run dev-llama

By encapsulating these parameters directly into the model definition, you ensure that every application or API call querying dev-llama inherits these optimizations out-of-the-box, without needing to pass raw JSON parameter payloads in each API request.

2. Fine-Tuning the Sampling Parameters

When a model generates text, it doesn't "know" words; it calculates a probability distribution over its vocabulary for the next most likely token. Sampling parameters dictate how the engine chooses the next token from this distribution. Tweaking these settings is the single most effective way to align the model’s creativity and precision with your specific use case.

// Temperature: The Randomness Dial

The temperature parameter controls the scaling of the token probability distribution. Mathematically, it divides the raw logits (pre-softmax scores) generated by the model before they are converted into probabilities:

Low temperature (e.g., 0.1 to 0.2): Flattens low-probability options and amplifies high-probability ones. This results in highly deterministic, consistent, and logical completions. Ideal for code generation, mathematical reasoning, structured data extraction (JSON/YAML), and factual summarization.

High temperature (e.g., 0.8 to 1.2): Flattens the differences between token probabilities, making less likely tokens more competitive. This introduces diversity, randomness, and "creativity" into the responses. Ideal for creative writing and brainstorming.

Configure for highly deterministic, structured tasks

PARAMETER temperature 0.1

// Top-K, Top-P, and Min-P: Narrowing the Token Pool

Left unchecked, even at low temperatures, models can occasionally select highly inappropriate tokens from the tail end of the probability distribution. To prevent this, model engines filter the active token pool before selecting the final token.

Top-K (e.g. 40): Restricts the pool to the K most probable next tokens. Any token ranked lower than 40 is immediately discarded, regardless of its actual probability. This is a crude but effective way to prune highly erratic tokens.

Top-P / Nucleus Sampling (e.g. 0.90): Restricts the pool to a dynamic set of tokens whose cumulative probability exceeds the threshold P. For example, at 0.90, Ollama sorts all tokens from highest to lowest probability and keeps only the top group that makes up the first 90% of the distribution. If the model is highly confident, the pool might compress to just 2 or 3 tokens; if it is confused, the pool expands.

Min-P (e.g. 0.05 to 0.10): A modern, vastly superior alternative to Top-P. Instead of taking a static cumulative slice, min_p filters out tokens whose probability is lower than a dynamic threshold relative to the leading token's probability. For example, if the top token has a probability of 0.80 and min_p is set to 0.05, the minimum threshold for any other token to be considered is 0.80 * 0.05 = 0.04. If the top token is highly certain (e.g. 0.99), all other tokens are aggressively pruned. If the top token is uncertain (e.g. 0.15), the threshold drops to 0.0075, keeping a wide pool of creative choices open.

Establish robust sampling limits in the Modelfile

PARAMETER top_k 40 PARAMETER top_p 0.90 PARAMETER min_p 0.05

⚠️ When using min_p, you should generally leave top_p at its default (1.0) or set it highly (0.95+) so it doesn't interfere with the superior, dynamic scaling behavior of min_p.

3. Stopping Loops and Repetitive Outputs

One of the most frustrating failures in local model deployment is the repetition loop, where a model begins generating the exact same sentence, phrase, or code block indefinitely. This is usually triggered by a combination of a small model size (e.g. 1.5B or 3B parameters) and a lack of penalty boundaries.

Ollama provides three key parameters to prevent and interrupt these looping states.

// Repetition and Presence Penalties

Repetition penalty (repeat_penalty): Multiplies the raw logits of tokens that have already been generated, making them less likely to appear again. A value of 1.1 to 1.2 is usually sufficient to discourage looping without making the model avoid necessary grammar words (like "the" or "and").

Presence penalty (presence_penalty): Applies a flat, one-time penalty to any token that has appeared at least once in the generated text, encouraging the model to introduce completely new topics or vocabulary.

Frequency penalty (frequency_penalty): Applies a penalty proportional to the number of times a token has appeared, steadily discouraging the overuse of specific terms.

Discourage loops and encourage vocabulary variety

PARAMETER repeat_penalty 1.15 PARAMETER presence_penalty 0.05 PARAMETER frequency_penalty 0.05

// Halting Generation with Stop Sequences

Sometimes, the model doesn't loop internally, but it fails to realize when it has finished its turn, continuing to hallucinate fake responses from the user. You can prevent this by defining explicit stop sequences (stop tokens). When the model generates a stop sequence, the engine immediately halts inference and returns the response.

Common stop tokens include chat markers like , markdown section headers, or custom delimiters:

Stop generating when ChatML tags or User lines are generated

PARAMETER stop "" PARAMETER stop "" PARAMETER stop "User:"

4. Managing Context Windows and Memory

Local hardware resources — specifically video RAM (VRAM) on your GPU — are highly constrained. Understanding how to size your model’s memory structures is vital for building robust local applications.

// Context Length (num_ctx)

The context length (num_ctx) defines the size of the attention window (in tokens) that the model can process at once. This includes both the input prompt (and system history) and the newly generated output tokens.

By default, Ollama initializes many models with a conservative context window of 2048 or 4096 tokens to prevent memory overflow on lower-end hardware. However, modern models like Llama 3.1 or Mistral support native context windows up to 128,000 tokens. If you are building a retrieval-augmented generation (RAG) system or importing large code files, 2048 tokens will result in silent prompt truncation, leading to loss of context and highly inaccurate completions.

You can explicitly increase this parameter in your Modelfile:

Expand context window to 16,384 tokens

PARAMETER num_ctx 16384

⚠️ Attention computation scales quadratically ($O(N^2)$) with context length. Doubling your num_ctx will dramatically increase the VRAM required to store the model's active state during generation. Be sure your hardware can handle the increased allocation.

// KV Cache Quantization (OLLAMA_KV_CACHE_TYPE)

To track relationships between tokens over a long conversation, the model stores an active key-value (KV) cache in VRAM. At large context lengths (like 32k or 128k), the size of the KV cache could exceed the weight size of the model itself, causing out-of-memory crashes.

To combat this, Ollama supports KV cache quantization. Much like model weights can be compressed from 16-bit floats to 4-bit integers, the KV cache can be quantized to lower precisions with minimal degradation in text quality:

f16: Standard, uncompressed 16-bit floating-point cache (default)

q8_0: Compresses the KV cache to 8-bit integers, saving roughly 50% of KV VRAM with virtually zero impact on output quality

q4_0: Compresses the KV cache to 4-bit integers, saving 75% of KV VRAM, allowing massive context sizes on consumer hardware at the expense of a slight increase in model perplexity

This parameter is set via the OLLAMA_KV_CACHE_TYPE server environment variable (detailed in the next section).

5. Server-Level Tuning: Environment Variables

While Modelfile parameters adjust how a specific model operates, server environment variables customize the Ollama background daemon itself. These configurations dictate how Ollama interacts with your operating system, handles system memory, manages parallel processing, and utilizes your hardware acceleration layers.

How you set these variables depends on your host operating system:

macOS: Set via terminal exports or modified inside your application environment files (or launched via launchctl for background services)

Linux (Systemd): Configured via systemctl edit ollama.service to inject environment configurations

Windows (WSL2 / System): Set in standard Windows System Environment Variables or in your WSL terminal profile

// The Essential Server Variables

Variable Name Default Value Purpose & Best Practices

OLLAMA_HOST 127.0.0.1:11434 Binds the server network interface. Set to 0.0.0.0:11434 to expose the API to other computers on your local network.

OLLAMA_MODELS Platform-specific default Changes model storage location. Highly recommended to point this to a high-speed external NVMe SSD if your boot drive is low on space.

OLLAMA_KEEP_ALIVE 5m (5 minutes) Controls how long models stay loaded in GPU memory after your last request. Set to 1h to prevent reload latency in active pipelines, o

[truncated for AI cost control]