Multimodal Browser AI with Transformers.js for Images and Speech
This tutorial shows how to build multimodal AI applications — image classification, image captioning, and speech transcription — that run entirely in the browser using Transformers.js, with no server or API key, ensuring user privacy. It includes detailed code examples and project structure.
Multimodal Browser AI with Transformers.js for Images and Speech - MachineLearningMastery.com
Multimodal Browser AI with Transformers.js for Images and Speech - MachineLearningMastery.com
In this article, you will learn how to build multimodal AI capabilities — image classification, image captioning, and speech transcription — that run entirely in the browser using Transformers.js, with no server, no API key, and no data leaving the user’s device.
Topics we will cover include:
How to set up and run image classification and image captioning pipelines using Vision Transformer models in the browser.
How to implement browser-based speech transcription using OpenAI’s Whisper architecture via the Web Audio API.
How to combine all three pipelines into a single multimodal media analyzer that loads models in parallel and presents results in a unified dashboard.
Multimodal Browser AI with Transformers.js for Images and Speech
Introduction
Most browser AI tutorials cover text because it is a natural starting point, but the applications people actually want to build are rarely text-only. Users take photos, record voice notes, upload screenshots. The data is multimodal and the AI should be too.
Transformers.js handles this natively. It supports computer vision (image classification, object detection, segmentation), audio (automatic speech recognition, audio classification, text-to-speech), and multimodal tasks, all running locally in the browser, with no server, no API key, and no data leaving the user’s device.
This tutorial builds three capabilities in sequence: image classification, image captioning, and speech transcription. Each is a self-contained HTML file you can open in a browser. The final section combines all three into a single multimodal media analyzer.
What You Need
A modern browser: Chrome 109+, Edge 109+, or Firefox 90+. These versions support ES modules and WebAssembly, both of which Transformers.js requires.
A local web server: Browser security policies block ES module imports from file:// URLs — opening the HTML files directly by double-clicking will not work. You need to serve them over HTTP.
You do not need Node.js, npm, or any build tools. The CDN import handles the library.
Starting a Local Server
Pick whichever option matches what you already have installed:
1
2
3
4
5
6
7
8
Python -- pre-installed on macOS and most Linux systems
python3 -m http.server 8080
Node.js
npx serve .
VS Code -- install the Live Server extension, then right-click any HTML file
and choose "Open with Live Server"
Once the server is running, open http://localhost:8080 in your browser.
Project Structure
Create one folder for the project. Each task gets its own HTML file:
1
2
3
4
5
multimodal-demo/
├── image-classifier.html
├── image-captioner.html
├── speech-transcriber.html
└── media-analyzer.html
Models and Download Sizes
Every model downloads once on the first run and caches in the browser. Subsequent loads are instant and work offline. Here is what to expect on the first run:
Task Model Pipeline task First-run download
Image Classification Xenova/vit-base-patch16-224 image-classification ~88 MB
Image Captioning Xenova/vit-gpt2-image-captioning image-to-text ~246 MB
Speech Transcription Xenova/whisper-tiny.en automatic-speech-recognition ~78 MB
The combined app loads all three, roughly 400 MB total on first run. A progress indicator for each model is non-negotiable UX.
Task 1: Image Classification
Image classification assigns labels from a fixed set to an input image. The model used here is ViT-Base/16, a Vision Transformer trained by Google on ImageNet-21k and fine-tuned on ImageNet-1k, converted to ONNX format for browser use. It classifies images into 1,000 ImageNet categories and returns a ranked list with confidence scores.
What the output looks like:
1
2
3
4
5
6
7
// Output from classifier(imageUrl)
[
{ label: 'golden retriever', score: 0.9421 },
{ label: 'Labrador retriever', score: 0.0312 },
{ label: 'Sussex spaniel', score: 0.0098 },
// ... top_k results total
]
Each object has a label string (the ImageNet class name) and a score float between 0 and 1. By default, the pipeline returns 5 results. Set top_k in the call to get more or fewer.
Full Working Demo
Save this file as image-classifier.html in your project folder. Copy the code below and open it on your localhost.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
Image Classifier
- { box-sizing: border-box; margin: 0; padding: 0; }
body {
font-family: system-ui, sans-serif;
max-width: 700px;
margin: 2rem auto;
padding: 0 1rem;
background: #f8fafc;
color: #1e293b;
}
h1 { margin-bottom: 0.25rem; font-size: 1.4rem; }
.subtitle { color: #64748b; font-size: 0.9rem; margin-bottom: 1.5rem; }
#status { font-size: 0.85rem; color: #64748b; margin-bottom: 1rem; }
.upload-area {
border: 2px dashed #cbd5e1;
border-radius: 8px;
padding: 2rem;
text-align: center;
cursor: pointer;
background: white;
transition: border-color 0.2s;
}
.upload-area:hover { border-color: #2563eb; }
.upload-area input { display: none; }
#preview {
margin-top: 1rem;
max-width: 100%;
border-radius: 8px;
display: none;
}
.result-row {
display: flex;
align-items: center;
gap: 0.75rem;
margin-top: 0.6rem;
}
.result-label { min-width: 200px; font-size: 0.9rem; }
.bar-bg {
flex: 1;
background: #e2e8f0;
border-radius: 4px;
height: 16px;
}
.bar-fill {
background: #2563eb;
height: 100%;
border-radius: 4px;
transition: width 0.4s ease;
}
.result-score {
min-width: 48px;
text-align: right;
font-size: 0.85rem;
color: #475569;
}
#results { margin-top: 1.25rem; }
#results h3 { font-size: 0.95rem; color: #374151; margin-bottom: 0.5rem; }
Image Classifier
Upload any image -- ViT classifies it into ImageNet categories.
Runs entirely in your browser.
Downloading model (~88 MB on first run)...
Click to upload or drag an image here
JPG, PNG, WebP, GIF supported
import { pipeline }
from 'https://cdn.jsdelivr.net/npm/@huggingface/[email protected]';
const statusEl = document.getElementById('status');
const dropZone = document.getElementById('drop-zone');
const fileInput = document.getElementById('file-input');
const preview = document.getElementById('preview');
const resultsEl = document.getElementById('results');
// ── Load the image classification pipeline ──────────────────────────
// Task: 'image-classification'
// Model: Xenova/vit-base-patch16-224 -- ViT trained on ImageNet-1k
// dtype: 'q8' -- 8-bit quantized for smaller download, good accuracy
let classifier;
pipeline('image-classification', 'Xenova/vit-base-patch16-224', {
dtype: 'q8',
progress_callback: (p) => {
if (p.status === 'progress') {
statusEl.textContent =
Downloading model: ${Math.round(p.progress ?? 0)}%;
}
}
}).then(pipe => {
classifier = pipe;
statusEl.textContent =
'Model ready. Upload an image to classify it.';
dropZone.style.borderColor = '#22c55e';
}).catch(err => {
statusEl.textContent = Error loading model: ${err.message};
});
// ── Classify an image from a data URL ───────────────────────────────
async function classifyImage(dataUrl) {
statusEl.textContent = 'Classifying...';
resultsEl.innerHTML = '';
try {
// Pass the data URL directly -- the pipeline handles image decoding
// top_k: 5 returns the 5 highest-scoring ImageNet labels
const results = await classifier(dataUrl, { top_k: 5 });
statusEl.textContent = 'Done.';
// Build a bar chart of results
let html = '
Top predictions
';
results.forEach(({ label, score }) => {
const pct = (score * 100).toFixed(1);
const bar = (score * 100).toFixed(0);
html += `
${label}
${pct}%
`;
});
resultsEl.innerHTML = html;
} catch (err) {
statusEl.textContent = Classification error: ${err.message};
}
}
// ── File handling ────────────────────────────────────────────────────
function handleFile(file) {
if (!file || !file.type.startsWith('image/')) return;
const reader = new FileReader();
reader.onload = (e) => {
const dataUrl = e.target.result;
// Show the image preview
preview.src = dataUrl;
preview.style.display = 'block';
// Classify only if the model has finished loading
if (classifier) {
classifyImage(dataUrl);
} else {
statusEl.textContent =
'Model still loading -- please wait a moment and try again.';
}
};
// Read as a base64 data URL -- works as pipeline input
reader.readAsDataURL(file);
}
// Click to browse
dropZone.addEventListener('click', () => fileInput.click());
fileInput.addEventListener('change', (e) => handleFile(e.target.files[0]));
// Drag and drop
dropZone.addEventListener('dragover', (e) => {
e.preventDefault();
dropZone.style.borderColor = '#2563eb';
});
dropZone.addEventListener('dragleave', () => {
dropZone.style.borderColor = '#cbd5e1';
});
dropZone.addEventListener('drop', (e) => {
e.preventDefault();
dropZone.style.borderColor = '#cbd5e1';
handleFile(e.dataTransfer.files[0]);
});
Browser mockup of the image classifier running at localhost (click to enlarge)
What this code does:
The pipeline() call starts downloading the model immediately when the page opens.
The progress callback updates the status text so the user can see the download progressing.
Once classifier is assigned, the drop zone border turns green as a visual cue.
When a file is dropped or selected, FileReader converts it to a base64 data URL, which the pipeline accepts directly as image input — no manual preprocessing needed.
The classifier returns an array of { label, score } objects, which the rendering loop converts into a horizontal bar chart. The top_k: 5 option limits results to the five most likely classes.
Task 2: Image Captioning
Image captioning generates a natural language sentence describing what is in an image. It is meaningfully different from classification: instead of picking from 1,000 fixed labels, the model generates free-form text. “A golden retriever running through a field of tall grass” versus just “golden retriever.” More descriptive, more flexible, larger model.
The model used here is Xenova/vit-gpt2-image-captioning, a Vision Transformer encoder that reads the image paired with a GPT-2 decoder that generates the caption. The ONNX version weighs in at 246 MB, noticeably larger than the classifier, because the generative decoder is a full language model.
What the output looks like:
1
2
// Output from captioner(imageUrl)
[{ generated_text: 'a dog is playing on a tennis court' }]
The output is an array with one object containing a generated_text string. It is always an array even for a single image, because the pipeline supports batching.
Full Working Demo
Save this file as image-captioner.html. Run it on http://localhost.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
[truncated for AI cost control]