Local Video Summarization Pipeline: Processing Frames with SmolVLM2-2.2B

--> Local Video Summarization Pipeline: Processing Frames with SmolVLM2-2.2B - KDnuggets

-->

Join Newsletter

Introduction

Most video understanding tools fall into one of two camps. The first camp requires a cloud API; your footage is uploaded, processed on someone else's servers, and billed per minute of video. The second camp runs locally but demands the kind of GPU cluster most developers do not have: 70B+ models that need multiple A100s and take minutes per clip. Neither option works for a developer who wants to process a day's worth of meeting recordings, a lecture series, or security footage on a workstation they already own.

SmolVLM2-2.2B-Instruct, released by Hugging Face on February 20, 2025, changes the calculation. It runs on 5.2 GB of GPU RAM, an RTX 3060, a MacBook Pro M2, and the free Google Colab T4 tier. On Video-MME, the standard long-form video understanding benchmark, it outperforms every existing 2B-scale model. That combination, consumer hardware paired with results that actually hold up, is what this article is built around.

The project we will build in this article: a local pipeline that takes any video file, extracts frames at configurable intervals, analyzes them in batches with SmolVLM2-2.2B, and outputs a structured JSON summary, including per-frame scene descriptions, key moments with timestamps, action items, and a final narrative. The same pipeline handles meeting recordings, lectures, and surveillance footage without changing a line of code.

SmolVLM2-2.2B

The reason SmolVLM2-2.2B can run on an RTX 3060 while outperforming larger models on video tasks is a design decision about how images are tokenized.

Most vision-language models tokenize images at high density. Qwen2-VL, for example, uses up to 16,000 tokens to represent a single image. Feeding 50 frames to such a model at that density would consume 800,000 tokens, far beyond any consumer GPU's context budget. SmolVLM2 uses a pixel shuffle strategy that compresses each 384x384 image patch to 81 tokens. Fifty frames become approximately 4,050 image tokens, manageable in a single inference call. That compression is why SmolVLM2's prefill throughput runs 3.3 to 4.5 times faster and generation throughput runs 7.5 to 16 times faster than Qwen2-VL-2B, not as a marketing claim but as a direct consequence of the token budget difference.

The model comes in three sizes. The 256M and 500M variants are designed for mobile and edge devices; the 256M can run on a phone. The 2.2B is the right choice for this pipeline. It is the only size with strong enough video benchmark scores to produce reliable multi-scene summaries: Video-MME of 52.1, MLVU of 55.2, and MVBench of 46.27, against the 500M's 42.2, 47.3, and 39.73, respectively.

The video understanding approach is also worth understanding before you write any code. SmolVLM2 does not have a native video encoder; it treats video as a sequence of images. The official reference pipeline extracts up to 50 evenly sampled frames per video, bypasses internal frame resizing, and passes them as a multi-image sequence inside a single chat message. That approach scored 27.14% on CinePile, positioning it between InternVL2 (2B) and Video-LLaVA (7B) on cinematic video understanding, a strong result given the model's size and that video was not the only thing it was trained for.

Setting Up the Environment

Hardware requirements:

Feature Minimum Recommended

GPU VRAM 6 GB (RTX 3060) 12–16 GB (RTX 4080)

Apple Silicon M2 8 GB (MPS path) M2 Pro / M3 16 GB

System RAM 16 GB 32 GB

Disk 10 GB free 20 GB+ SSD

Colab T4 (free tier) A100 (Colab Pro)

Python packages:

Python 3.10+ required

python --version

python -m venv smolvlm2-env source smolvlm2-env/bin/activate # macOS / Linux smolvlm2-env\Scripts\activate # Windows

Install from the stable SmolVLM-2 branch -- required for SmolVLM2 support

pip install git+https://github.com/huggingface/[email protected]

Core dependencies

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121 pip install \ opencv-python \ Pillow \ numpy \ num2words \ accelerate

Flash Attention 2 for CUDA -- significantly faster on NVIDIA GPUs

Skip this on Apple Silicon and CPU -- it is CUDA-only

pip install flash-attn --no-build-isolation

decord -- required for SmolVLM2's native video input path (used in Section 5)

pip install decord

Note: The num2words package is a non-obvious dependency. SmolVLM2's processor uses it to convert numeric digits to word representations (e.g. 3 → "three") for consistency with natural language training patterns, as explained in this walkthrough. Omitting it causes a silent import error when the processor loads.

Device check (run this before loading the model):

device_check.py

Run: python device_check.py

import torch

def detect_device(): if torch.cuda.is_available(): name = torch.cuda.get_device_name(0) vram = torch.cuda.get_device_properties(0).total_memory / 1e9 print(f"CUDA: {name} ({vram:.1f} GB VRAM)") return "cuda", torch.bfloat16, "flash_attention_2" elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available(): print("Apple Silicon MPS detected") return "mps", torch.float16, "eager" else: print("CPU fallback (slow -- consider Colab T4)") return "cpu", torch.float32, "eager"

if name == "main": device, dtype, attn = detect_device() print(f"Device: {device} | dtype: {dtype} | attn: {attn}")

Run it with:

python device_check.py

Building the Foundation of the Pipeline

Before SmolVLM2 sees anything, you need frames. The frame extractor converts a video file into a list of PIL (Python Imaging Library) images with timestamps attached, one pair per extracted frame.

Two modes matter for different use cases. Uniform sampling distributes frames evenly across the full video duration, guaranteeing coverage of everything regardless of content. This is the right choice for meetings and lectures where you cannot afford to miss a section. Keyframe sampling extracts frames only where the visual content changes significantly, such as scene cuts, a new slide, or a new speaker, which reduces the frame count and focuses attention on distinct moments. This is better for surveillance and highlight extraction.

frame_extractor.py

Prerequisites: pip install opencv-python Pillow numpy

Usage: from frame_extractor import FrameExtractor

import cv2 import numpy as np from PIL import Image

class FrameExtractor: """ Extracts video frames as PIL Images for SmolVLM2 inference. Each extracted frame is paired with its timestamp in seconds.

SmolVLM2 uses ~81 visual tokens per image. At 50 frames that is roughly 4,050 image tokens -- the practical upper limit before VRAM pressure affects generation quality on consumer GPUs. """

MAX_FRAMES = 50

def init(self, max_frames: int = MAX_FRAMES): """ Args: max_frames: Hard cap on extracted frames. Default 50 matches the SmolVLM2 reference pipeline's tested upper limit. """ self.max_frames = max_frames

def uniform_sample(self, video_path: str) -> list[tuple[float, Image.Image]]: """ Extract evenly spaced frames across the full video duration. Best for: meeting recordings, lectures, tutorials, course content.

Returns: List of (timestamp_seconds, PIL_Image) in chronological order. """ cap = cv2.VideoCapture(video_path) if not cap.isOpened(): raise IOError(f"Cannot open video: {video_path}")

total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) fps = cap.get(cv2.CAP_PROP_FPS) or 30.0 n_extract = min(self.max_frames, total_frames)

Build frame indices spread evenly from first to last frame

indices = np.linspace(0, total_frames - 1, n_extract, dtype=int) results = []

for idx in indices: cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx)) ret, frame = cap.read() if not ret: continue timestamp = round(idx / fps, 2) rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) results.append((timestamp, Image.fromarray(rgb)))

cap.release() return results

def keyframe_sample( self, video_path: str, diff_threshold: float = 30.0 ) -> list[tuple[float, Image.Image]]: """ Extract frames where visual content changes significantly. Best for: surveillance, event detection, highlight extraction.

Uses mean absolute pixel difference between consecutive grayscale frames as the change signal. When the diff exceeds diff_threshold, a new keyframe is recorded.

Args: diff_threshold: Mean pixel difference to treat as a scene change. 30.0 works for most commercial content. Lower = more sensitive, higher = fewer frames.

Returns: List of (timestamp_seconds, PIL_Image) in chronological order, capped at self.max_frames. """ cap = cv2.VideoCapture(video_path) if not cap.isOpened(): raise IOError(f"Cannot open video: {video_path}")

fps = cap.get(cv2.CAP_PROP_FPS) or 30.0 results = [] prev_gray = None idx = 0

while len(results) diff_threshold: rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) results.append((round(idx / fps, 2), Image.fromarray(rgb)))

prev_gray = gray idx += 1

cap.release() return results

Start every new video type with uniform_sample. If you find too many redundant frames (five nearly-identical slides in a row), switch to keyframe_sample and tune diff_threshold down from 30 to 20 until the extracted set feels representative without being redundant.

Loading SmolVLM2 and Running Single-Frame Inference

With frames in hand, here is the complete model loading and first-inference pattern. The important details: AutoModelForImageTextToText is the correct class (not the generic AutoModelForCausalLM), and on CUDA, you should enable Flash Attention 2, which provides meaningful latency improvements on multi-image inputs.

smolvlm2_loader.py

Prerequisites: transformers from v4.49.0-SmolVLM-2 branch, torch, flash-attn (CUDA only)

Run: python smolvlm2_loader.py your_video.mp4

import sys import torch from PIL import Image from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL_ID = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"

def load_model(): """ Load SmolVLM2-2.2B and its processor. Automatically selects Flash Attention 2 on CUDA, eager mode elsewhere. First run downloads ~4.5 GB of weights to ~/.cache/huggingface/hub. """ if torch.cuda.is_available(): dtype = torch.bfloat16 device = "cuda" attn = "flash_attention_2" elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available(): dtype = torch.float16 device = "mps" attn = "eager" else: dtype = torch.float32 device = "cpu" attn = "eager"

print(f"Loading {MODEL_ID} on {device}...")

processor = AutoProcessor.from_pretrained(MODEL_ID)

model = AutoModelForImageTextToText.from_pretrained( MODEL_ID, torch_dtype=dtype, _attn_implementation=attn, ).to(device)

model.eval() print(f"Model ready on {device}") return model, processor

def describe_frame( model, processor, frame: Image.Image, prompt: str = "Describe what is happening in this frame in detail. Note any text, people, objects, or actions visible.", max_new_tokens: int = 256, ) -> str: """ Run SmolVLM2 inference on a single PIL Image.

The chat template expects image content before text content in the message -- this mirrors the training data format and is important for reliable output.

Args: frame: A PIL Image (from FrameExtractor) prompt: What to ask the model about this frame max_new_tokens: Maximum response length in tokens

Returns: Model response as a plain string """ messages = [ { "role": "user", "content": [

Image placed before text -- matches SmolVLM2 training format

{"type": "image"}, {"type": "text", "text": prompt}, ], } ]

apply_chat_template formats the message and injects visual token placeholders

input_text = processor.apply_chat_template( messages, add_generation_prompt=True, )

inputs = processor( images=

[truncated for AI cost control]