How to Measure Video Similarity: 6 Techniques I Tested (and the One I Shipped)

-->

How to Measure Video Similarity: 6 Techniques Tested

India's Most Futuristic AI Conference Is Back – Bigger, Sharper, Bolder

d

:

h

:

m

:

s

Career

GenAI

Prompt Engg

ChatGPT

LLM

Langchain

RAG

AI Agents

Machine Learning

Deep Learning

GenAI Tools

LLMOps

Python

NLP

SQL

AIML Projects

Reading list

How to Become a Data Analyst in 2025: A Complete RoadMap

A Comprehensive Learning Path to Tableau in 2025

A Comprehensive NLP Learning Path 2025

Learning Path to Become a Data Scientist in 2025

Step-by-Step Roadmap to Become a Data Engineer in 2025

A Comprehensive MLOps Learning Path: 2025 Edition

Roadmap to Become an AI Engineer in 2025

A Comprehensive Learning Path to Master Computer Vision in 2025

Best Roadmap to Learn Generative AI in 2025

GenAI Roadmap for Enterprises

Large Language Models Demystified: A Beginner’s Roadmap

Learning Path to Become a Prompt Engineering Specialist

How to Measure Video Similarity: 6 Techniques I Tested (and the One I Shipped)

Sree Vamsi Last Updated : 13 Jul, 2026

12 min read

Two short clips. One question: how alike do they look? Sounds trivial, it isn’t, and I learned that the slow way.

My setup: one reference clip, eight others to rank against it, all waterfalls (more on why in a second). I figured this was an afternoon job, grab a model, compute a number, move on. Instead I watched supposedly-smart methods rank near-identical clips in nonsense orders, and the one that looked best on paper was too slow to actually use.

So I benchmarked six methods, same clips, same rules, and judged them accuracy first. A wrong answer in a millisecond is still wrong, so speed only gets a say once a method proves it can rank correctly. Accuracy first, speed as the tiebreaker. Here’s what I found.

Table of contents

Why this is harder than it sounds

The six contenders

Setting up a fair fight

The techniques, and where each one broke

Does sampling frames even help?

Accuracy first, because a fast wrong answer is useless

Then speed, the tiebreaker

The surprising winner, and the catch

The calibration trap nobody warns you about

So which one should you actually use?

What’s changed by 2026

Wrapping up

Frequently Asked Questions

Why this is harder than it sounds

My first instinct was to just compare pixels. That’s the trap: you’re actually comparing meaning, the subject, colors, light, whether the water crashes or trickles.

Chase meaning and you’re picking from three families, each costing you something. Embeddings sample frames through a model that knows what images mean, smart, but costs milliseconds or API credits. Fingerprints crush each frame to a tiny code, almost free and instant, almost blind to anything subtle. Full multimodal LLMs take the whole video and hand you an opinion, they see the most, they also break the most.

Speed, accuracy, cost. You get two. That’s the whole reason I benchmarked instead of arguing about it.

The six contenders

Technique The one-liner Local?

GPT Vision A vision LLM scores it and tells you why No

Gemini Flash (full video) A multimodal LLM watches both clips whole No

CLIP embeddings Neural frame vectors, compared by cosine Yes

Perceptual hash A 64-bit fingerprint per frame Yes

CV multi-metric Old-school OpenCV signals, blended Yes

Gemini Embedding 2 Native multimodal vectors, compared by cosine No

Three of these run free on my laptop. Three bill me per call. That split mattered more to me than the accuracy numbers.

Setting up a fair fight

Most benchmarks cheat by testing on an easy set, so I made mine mean on purpose. The reference is a tropical waterfall, sunbeams and wet greenery, and all eight test clips are waterfalls too. That forces every method to win on the fine detail, color cast, light direction, framing, motion, instead of just spotting “there’s water in this one.”

Same input for everyone: six frames per clip, evenly spaced, shrunk to 384×216. Sampling a few frames instead of all of them is normal and barely costs you quality.

The annoying part: no human labels, no budget to make any. So I faked a fair consensus, averaged the first five methods’ scores per clip. Two clips came out on top, including one I’ll call Sample 4 (shown in the 2nd video below) and the original video (the 1st video); everything else gets graded against. Imperfect, but defensible.

The techniques, and where each one broke

I’m skipping the neat pros-and-cons boxes. They didn’t break neatly.

GPT Vision

GPT Vision was the one I actually liked reading, hand it a few frames and it judges theme, color, and mood, writing back something like “differed significantly in visual theme, color palette, and overall mood.” But the numbers underneath were mush, everything scored 50 to 80, bunched together and wobbling between runs. Great at explaining itself, bad at ranking eight near-twins.

Here’s the actual call, trimmed down:

Grab a few frames from each video, show them side by side to

GPT-4o-mini, and just ask it to score how similar they look.

def score_with_gpt_vision(ref_frames, test_frames, api_key): client = OpenAI(api_key=api_key)

ref_imgs = frames_to_jpeg_b64(ref_frames[:3]) test_imgs = frames_to_jpeg_b64(test_frames[:3])

response = client.chat.completions.create( model="gpt-4o-mini", messages=[ { "role": "user", "content": [ {"type": "text", "text": "REFERENCE VIDEO:"}, *[image_message(img) for img in ref_imgs], {"type": "text", "text": "TEST VIDEO:"}, *[image_message(img) for img in test_imgs], { "type": "text", "text": "Score how closely these match, 0 to 100, JSON only.", }, ], } ], )

data = json.loads(response.choices[0].message.content)

return data["score"], data["feedback"]

Gemini Flash

Gemini Flash on the full video is the only one that watches real motion, not stills, pacing and camera drift included, a big edge on paper. Then reality showed up: slowest by a mile, and mid-test one clip threw a 503, the fallback hit a 429, and that clip never got a score at all. Dealbreaker for anything live.

Here’s what makes this one different, code-wise:

Upload both full videos and just ask Gemini to watch and compare.

The only technique here that actually sees motion, not just stills.

def score_with_gemini_flash(ref_path, test_path, api_key): client = genai.Client(api_key=api_key)

ref_video = client.files.upload(file=ref_path) test_video = client.files.upload(file=test_path)

Wait for both files to leave PROCESSING state before using them

prompt = "Compare these two videos for visual similarity. Score 0-100, JSON only."

response = client.models.generate_content( model="gemini-2.5-flash", contents=[prompt, ref_video, test_video], config={"response_mime_type": "application/json"}, )

data = json.loads(response.text)

return data["score"], data["feedback"]

CLIP

CLIP was my bet going in: each frame becomes a vector, you average them, take the cosine. Runs locally in under a second and gave the steadiest scores in the test, though everything clusters in the high 80s and low 90s even for clips that aren’t close. Great for ranking, useless if you want “you scored 88%” to mean anything on its own.

Here’s the whole technique in code:

Turn every frame into a CLIP vector, average them into one vector

per video, then just take the cosine between the two.

def embed_video_with_clip(frames, model, preprocess): images = [preprocess(to_pil_image(f)) for f in frames]

with torch.no_grad(): vectors = model.encode_image(torch.stack(images))

vectors = vectors / vectors.norm(dim=-1, keepdim=True)

One vector for the whole video

return vectors.mean(dim=0).numpy()

ref_vec = embed_video_with_clip(ref_frames, model, preprocess) test_vec = embed_video_with_clip(test_frames, model, preprocess)

similarity = cosine_similarity(ref_vec, test_vec)

No API call in sight. Once the model’s downloaded, this all runs on your own machine.

Perceptual hash is fifty times faster than anything else, no model to even load. As a judge though, nearly useless here, it’s a bouncer, not a critic. (Exact number coming up, funnier than I expected.)

And here’s the entire thing, no model required:

Hash every frame down to a tiny fingerprint, then measure

how many bits differ. No model, no API, just bit-counting.

def phash_similarity(ref_frames, test_frames): ref_hashes = [imagehash.phash(to_pil_image(f)) for f in ref_frames] test_hashes = [imagehash.phash(to_pil_image(f)) for f in test_frames]

similarities = []

for rh in ref_hashes:

Smallest Hamming distance

closest = min(rh - th for th in test_hashes)

64-bit hash

similarities.append(1.0 - closest / 64)

return sum(similarities) / len(similarities)

That’s the whole bouncer. Fast because it isn’t actually looking at anything, just counting flipped bits.

CV multi-metric is what I’d build to see inside a score, four old-school signals weighted by hand:

composite = 0.30 * color # HSV histogram

+ 0.35 * struct # SSIM

+ 0.20 * temporal # temporal color profile

+ 0.15 * edge # edge density

One quirk that showed up: different metrics can wildly disagree on the same clip. SSIM punishes any framing shift hard, while a temporal color-profile check barely notices it, so the same video can score a 99 on one signal and an 18 on another.

Gemini Embedding 2 is CLIP’s idea after it grew up. Same move, embed then pool then cosine, but the embedding comes from a model built to handle image, video and text in one 3072-dimensional space.

Here’s what that actually looks like in code, trimmed down to the part that matters:

Just read the whole video file and hand it to Gemini as one blob.

No frames, no pooling, one API call per video.

def embed_full_video(video_path, api_key): with open(video_path, "rb") as f: video_b64 = base64.b64encode(f.read()).decode()

body = { "content": { "parts": [ { "inline_data": { "mime_type": "video/mp4", "data": video_b64, } } ] } }

resp = requests.post(f"{EMBED_URL}?key={api_key}", json=body)

return np.array(resp.json()["embedding"]["values"])

Embed both videos, then just compare the two vectors

ref_vec = embed_full_video("Original.mp4", api_key) test_vec = embed_full_video("sample_1.mp4", api_key)

cos = cosine_similarity(ref_vec, test_vec)

Stretched onto a friendly 0-100

score = cosine_to_score(cos)

Does sampling frames even help?

Quick side test: if sampling frames works for CLIP, does it work the same way for Gemini’s embedding model? I tested one video at 4, 8, 16, 32, and 64 frames against sending the whole video in one shot.

Method Cosine Score Time

4 frames 0.91283 65 5.26s

8 frames 0.91674 67 7.56s

16 frames 0.92196 69 8.78s

32 frames 0.92540 70 10.65s

64 frames 0.92476 70 14.6s

Full video 0.93156 73 7.19s

4 to 32 frames buys 5 extra score points (65 to 70) but doubles the time (5.26s to 10.65s). Push to 64 frames and only the time moves, up to 14.6s. Full video beats all of them on accuracy (73) while being faster than anything past 4 frames.

Here’s the actual function, kept as simple as I could make it:

Grab a handful of frames from the video, embed each one,

then average them into a single vector. Same idea as CLIP,

just using Gemini's embedding model instead.

def embed_video_by_frames(video_path, frame_count, api_key): frames = grab_frames(video_path, frame_count) vectors = []

for frame in frames: jpeg = frame_to_jpeg_b64(frame) vec = embed_one_frame(jpeg, api_key) vectors.append(vec)

Average all the frame vectors into one video vector

return np.mean(vectors, axis=0)

So that settles it: more frames doesn’t buy better accuracy past a point, just a longer wait. Full video wins both ways.

Costs money, costs a few seconds. I walked in rolling my eyes at the latency. I walked out having shipped it. The next section is why, and it’s all about accuracy.

Accuracy first, because a fast wrong answer is useless

This is the one that actually matters, so I looked at it before I looked at the clock.

Two questions. How many of the three cons

[truncated for AI cost control]