Show HN: Turn narrated screen recordings into data for AI agents (local, MIT)

Notifications You must be signed in to change notification settings

Fork 0

Star 15

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

41 Commits

41 Commits

.agents/skills/talkthrough

.agents/skills/talkthrough

.claude-plugin

.claude-plugin

.github

.github

assets

assets

docs

docs

examples

examples

integrations

integrations

scripts

scripts

src/talkthrough_mcp

src/talkthrough_mcp

tests

tests

.gitignore

.gitignore

.mcp.json

.mcp.json

.python-version

.python-version

AGENTS.md

AGENTS.md

CHANGELOG.md

CHANGELOG.md

CLAUDE.md

CLAUDE.md

CODE_OF_CONDUCT.md

CODE_OF_CONDUCT.md

CONTRIBUTING.md

CONTRIBUTING.md

LICENSE

LICENSE

README.md

README.md

SECURITY.md

SECURITY.md

llms-install.md

llms-install.md

llms.txt

llms.txt

pyproject.toml

pyproject.toml

server.json

server.json

uv.lock

uv.lock

Repository files navigation

Quickstart · Tools · FAQ · Troubleshooting · Changelog · Contributing

Feedback ingestion for AI agents. Record your screen and talk; your agent does the rest — files the bugs, writes the spec, builds the backlog.

talkthrough-mcp is a local-first MCP server that turns a narrated screen recording (or any video/audio file) into agent-ready structured data: timestamped transcript segments, optional who-said-what speaker labels, scene-change keyframes, OCR'd on-screen text, and wall-clock anchoring. Everything is served through lazy retrieval tools, so a 30-minute recording never floods the model context — the agent pulls exactly the transcript slice, moment bundle, or frame it needs.

There is no LLM inside the server and no cloud anywhere in the path: ffmpeg, faster-whisper, and RapidOCR run on your machine, and the calling agent brings the intelligence. What makes it different from screen-recorder SaaS and video-analyzer MCPs: it works on arbitrary local files, it ships the agent workflows (server prompts + example agents), and it anchors every timestamp to wall-clock time — so "the moment I said the checkout hung" maps straight to the right window of your server logs.

Quickstart

One command, no system dependencies: ffmpeg falls back to a bundled build, OCR is pip-only, and whisper models download themselves on first use. The only prerequisite is uv (brew install uv or curl -LsSf https://astral.sh/uv/install.sh | sh).

Claude Code

Two install paths — pick one, not both (the plugin already includes the server; installing both would register it twice):

Server only — the 7 tools + 5 prompts, and nothing else on your system. Choose this for a minimal setup, or when you manage MCP servers yourself across several clients:

claude mcp add -s user talkthrough -- uvx "talkthrough-mcp[diarization]"

Full plugin — the same server, plus native slash commands (/talkthrough:triage-recording, …) that handle the ceremony for you, a ready-made triage subagent, and an agent skill that teaches Claude the workflow. Choose this for the best out-of-the-box experience:

/plugin marketplace add korovin-aa97/talkthrough-mcp /plugin install talkthrough@talkthrough

Every other MCP client

Claude Desktop

claude_desktop_config.json:

{ "mcpServers": { "talkthrough": { "command": "uvx", "args": [ "talkthrough-mcp[diarization]" ] } } }

More: integrations/claude-desktop/

Cursor

~/.cursor/mcp.json (or project .cursor/mcp.json):

{ "mcpServers": { "talkthrough": { "command": "uvx", "args": [ "talkthrough-mcp[diarization]" ] } } }

More: integrations/cursor/

OpenAI Codex CLI

~/.codex/config.toml (or project-scoped .codex/config.toml in trusted projects):

[mcp_servers.talkthrough] command = "uvx" args = ["talkthrough-mcp[diarization]"]

More: integrations/codex/

Gemini CLI

~/.gemini/settings.json:

{ "mcpServers": { "talkthrough": { "command": "uvx", "args": [ "talkthrough-mcp[diarization]" ] } } }

More: integrations/gemini-cli/

Cline / Roo Code

cline_mcp_settings.json (via MCP Servers UI):

{ "mcpServers": { "talkthrough": { "command": "uvx", "args": [ "talkthrough-mcp[diarization]" ] } } }

More: integrations/cline/

OpenClaw

~/.openclaw/openclaw.json:

{ "mcp": { "servers": { "talkthrough": { "command": "uvx", "args": [ "talkthrough-mcp[diarization]" ] } } } }

More: integrations/openclaw/

OpenCode

opencode.json (project) or ~/.config/opencode/opencode.json:

{ "mcp": { "talkthrough": { "type": "local", "command": [ "uvx", "talkthrough-mcp[diarization]" ], "enabled": true } } }

More: integrations/opencode/

Goose

~/.config/goose/config.yaml:

extensions: talkthrough: enabled: true type: stdio cmd: uvx args: ["talkthrough-mcp[diarization]"]

More: integrations/goose/

GitHub Copilot CLI

~/.copilot/mcp-config.json:

{ "mcpServers": { "talkthrough": { "command": "uvx", "args": [ "talkthrough-mcp[diarization]" ] } } }

More: integrations/copilot-cli/

Windsurf

~/.codeium/windsurf/mcp_config.json:

{ "mcpServers": { "talkthrough": { "command": "uvx", "args": [ "talkthrough-mcp[diarization]" ] } } }

More: integrations/windsurf/

Zed

settings.json (Zed):

{ "context_servers": { "talkthrough": { "source": "custom", "command": { "path": "uvx", "args": [ "talkthrough-mcp[diarization]" ] } } } }

More: integrations/zed/

Any other MCP stdio client uses the same server command: uvx "talkthrough-mcp[diarization]". Per-engine folders with exactly these snippets plus verification steps live in integrations/; agents can self-install via llms-install.md.

Who said what (speaker diarization) — included in the configs above

Multi-person recordings (meetings, interviews, panels) can carry S1/S2/… speaker labels. Every install button, snippet, and the plugin above already ship the [diarization] engine, so asking your agent "who said what" just works — diarization itself still runs only when requested per call (process_media(path=..., diarize=true, num_speakers=)), and its models download once on first use.

Prefer the minimal server without the diarization engine? Use uvx talkthrough-mcp as the command instead (the MCP registry entry also resolves to this lean form) — an explicit diarize=true will then answer with the one-line install fix. Details in Speakers.

Local checkout (development)

git clone https://github.com/korovin-aa97/talkthrough-mcp claude mcp add talkthrough -- uv run --directory /path/to/talkthrough-mcp talkthrough-mcp

Then, in your agent:

Process ~/Desktop/recording.mov and triage it — or just invoke the triage-recording server prompt.

Tools

Tool What it does

process_media(path, recorded_at?, vocabulary?, language?, model?, diarize?, num_speakers?, force?) Ingest a video/audio file: local STT, keyframes, OCR, wall-clock, opt-in speaker labels. Returns a compact summary. Idempotent by content hash — re-calls are instant; diarize=true on a processed job adds speakers without re-transcribing.

get_transcript(job_id, start_ms?, end_ms?, format?) Paginated transcript as segments, text, or srt (speaker-prefixed when diarized, plus a roster header); truncation returns next_start_ms.

get_frames(job_id, at_ms? | start_ms?+end_ms?, max_frames?, include_duplicates?) Keyframe images nearest a timestamp or evenly thinned across a range (unique frames by default, max 6/call); each frame names its absolute path.

get_moment(job_id, start_ms, end_ms) The "one remark" bundle: transcript slice + up to 3 frames + their OCR text + wall-clock range (+ speakers_in_range when diarized).

search(job_id, query) Substring search over the transcript AND on-screen OCR text; hits carry t_ms/t_wall, frame refs, and the speaker when diarized.

extract_frame(job_id, at_ms, crop?) Exact-timestamp full-resolution re-extract from the source video (optional crop) when keyframes miss the instant; returns the file's absolute path.

list_jobs() Recent processed recordings with durations, wall-clock starts, counts, and speaker counts when diarized.

Every tool description ships 10+ usage examples, so agents pick the right tool without extra prompting.

Server prompts (slash commands in MCP clients)

Prompt Workflow

triage-recording Narrated screencast → precise findings JSON (bug/feature/question routing, frame evidence)

spec-from-workshop Recorded workshop → structured spec with quoted decisions and open questions

backlog-from-demo Product demo → prioritized backlog with timestamped evidence

meeting-actions Meeting audio → action items, decisions, open questions

correlate-with-logs Recording remarks ↔ system logs via wall-clock windows

The same prompts live as plain files in examples/prompts/ if your client doesn't surface MCP prompts. The findings contract used by triage-recording is examples/output-contract.schema.json.

Works as a skill too (no MCP required)

The same workflow ships as a cross-engine Agent Skill at .agents/skills/talkthrough/ — Claude Code, Codex CLI ($talkthrough), Cursor, Copilot, Gemini CLI, Goose and other SKILL.md-compatible tools read it. Agents without MCP wiring can drive the CLI directly: talkthrough-mcp process recording.mov --json prints the same summary the MCP tool returns, and the job store is shared either way.

Wall-clock anchoring

Every timestamped result carries both t_ms (video-relative) and t_wall (ISO 8601 real time) once the recording start is known. Resolution ladder:

recorded_at parameter (agent/user override) → confidence exact

QuickTime com.apple.quicktime.creationdate tag, carries the local timezone (QuickTime Player recordings; ⌘⇧5 wrote it before macOS 26) → high

Container creation_time tag (UTC) → medium — macOS 26+ ⌘⇧5/ReplayKit screen recordings land here (no creationdate tag anymore); pass recorded_at= when local-tz t_wall matters

File mtime minus duration (recorders finalize files at recording END) → low

Nothing → tools still work with relative t_ms only

Why it matters: "the upload spinner froze here" becomes a ±30 s grep window in your server logs.

Speakers (optional diarization)

With the [diarization] extra installed (included in every generated config — see Quickstart), process_media(diarize=true) labels who said what — locally, like everything else here (sherpa-onnx runtime, no torch, no accounts, no GPU):

Speakers become S1, S2, … in order of first appearance; every transcript segment gets a speaker, and the tools surface it everywhere — roster with talk time in get_transcript, speakers_in_range in get_moment, speaker on search hits, S1: prefixes in the text/SRT formats, a speaker count in list_jobs.

Know the headcount? Pass num_speakers. Clustering to an exact k removes the main failure mode of unknown-count mode (similar voices merging or one voice splitting). Agents are instructed to do this via the tool guidance; do the same in your own calls.

Already processed a recording? Calling process_media(diarize=true) on it re-runs only diarization — whisper is not re-run, and labels appear in the existing job in seconds. Same for changing num_speakers.

Labels are anonymous by design; mapping S1 → "Alice" is the calling agent's job (self-introductions, vocatives, the attendees argument of the meeting-actions prompt). The server never guesses names.

Models download once (~47 MB total) from pinned, checksum-verified URLs into ~/.talkthrough/models/; warm runs are zero-network like the rest of the pipeline. Speed on an M-series CPU (4 threads): a 26-minute meeting diarizes in about 2 minutes (RTF ≈ 0.08), on top of the transcription time. Memory: expect on the order of 1–1.5 GB peak RSS while an hour-plus meeting is being diarized (measured on a real 73-minute recording); it is released when the stage completes.

Role Model Download Weights license

Segmentation pyannote segmentation-3.0 (ONNX export by k2-fsa) 7 MB MIT

Embedding (default) NeMo en_titanet_small 40 MB Apache-2.0 (per its NGC model card, the NeMo Toolkit lice

[truncated for AI cost control]