Record and Replay, teach AI agents desktop workflows by showing them once

Uh oh!

There was an error while loading. Please reload this page.

Notifications You must be signed in to change notification settings

Fork 1

Star 20

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

40 Commits

40 Commits

assets

assets

capture

capture

compiler

compiler

schema

schema

scripts

scripts

skills

skills

.env.example

.env.example

.gitignore

.gitignore

README.md

README.md

config.py

config.py

pyproject.toml

pyproject.toml

registry.py

registry.py

server.py

server.py

state.py

state.py

uv.lock

uv.lock

Repository files navigation

Record desktop workflows once. Compile them into reusable agent skills.

Record → Compile → Give the skill to your agent

Install · Features · How It Works · Docs · Report Bug

What is Record & Replay?

An MCP server that records a human-operated desktop workflow and compiles it into reusable skill files. The generated SKILL.md can then be given to an agent so the agent can perform the workflow later.

Record — Captures native accessibility events, typed values, target metadata, and optional screen video to VideoDB for visual reference.

Compile — An LLM transforms the event log and scene descriptions into reusable SKILL.json and human-readable SKILL.md files.

Use — The server installs the generated SKILL.md into the agent's global skills directory, where future agent runs can use the skill instructions, inputs, verification checks, and execution guidance.

Demonstrate a task once on screen, and the server produces a self-contained, versioned skill artifact. This repo does not include a replay engine; replay is performed by the agent that consumes the generated skill.

Demo

Record_and_Replay.mp4

Installation

Prerequisites

Python 3.10+

uv package manager

A VideoDB API key

1. Clone and install

git clone https://github.com/video-db/open-record-replay.git cd open-record-replay uv sync

1. Set your API key

Create a .env file in the project root:

VIDEODB_API_KEY=your_VIDEODB_API_KEY

1. Configure your MCP client

Add to your MCP config (claude_desktop_config.json, VS Code MCP settings, etc.):

{ "mcpServers": { "videodb-record-replay": { "command": "uv", "args": ["run", "python", "server.py"], "cwd": "/path/to/open-record-replay" } } }

1. Restart your client

Five tools and two resources will appear. You're ready to record.

Platform-specific setup

macOS — Requires Screen Recording, Microphone, Accessibility, and Input Monitoring permissions. Run the permission helper first:

uv run python scripts/smoke_macos_hook.py --prompt-permissions

If ready_for_event_recording is false, enable the terminal process in System Settings > Privacy & Security > Accessibility and Input Monitoring, then rerun.

Windows — Uses UI Automation. No additional setup required beyond the standard install.

Linux — Uses AT-SPI. Ensure at-spi2-core is installed and your desktop environment has accessibility enabled.

Usage

Recording is human-in-the-loop. The agent starts recording, announces that recording is active, then waits. The human operator performs the UI workflow being captured.

record_skill_tool("my-workflow", lead_in_seconds=5) → Agent tells the operator recording is active → Operator switches to the target app and performs the workflow → Operator returns to the MCP client and says "stop" → Agent calls stop_recording_tool(trim_end_seconds=10) → Agent calls compile_skill_tool(video_id, "my-workflow") → Agent verifies/reports global_skill_md_path for future use

lead_in_seconds

The recorder starts capture immediately, then the compiler ignores events before the effective workflow start. With lead_in_seconds=5, the operator can switch from the MCP client to the target app and should begin the demonstrated workflow after 5 seconds.

trim_end_seconds

Discards events at the tail of the recording. Use when the operator must switch back to the MCP client to say "stop". For example, trim_end_seconds=10 ignores the final 10 seconds so the generated skill does not include the operator returning to the terminal or chat window.

Events-only mode

If VideoDB screen capture is unavailable, the system falls back to recording native accessibility events only. Call compile_skill_tool with video_id="" or video_id="none" to compile from events alone.

How It Works

Human performs workflow | v +------------------------------------------------------------------+ | Native AX/UIA/AT-SPI hooks -> events.jsonl | | VideoDB Capture SDK -> video_id, when capture succeeds | +------------------------------------------------------------------+ | v +------------------------------------------------------------------+ | Compiler | | events.jsonl + optional scene summaries/transcript -> SKILL.json | | SKILL.json -> SKILL.md | +------------------------------------------------------------------+ | v +------------------------------------------------------------------+ | Agent skill install | | ~/.mcp-videodb/skills//SKILL.md | | ~/.codex/skills//SKILL.md, unless overridden | +------------------------------------------------------------------+

The accessibility hooks provide the action log. VideoDB scene indexing adds visual context when screen capture is available. The compiler combines those signals into skill files that a future agent can use to perform the workflow.

Features

Feature Description

Dual recording Captures native accessibility events and, when full capture is available, screen video for visual reference

LLM compilation VideoDB scene indexing and generate_text compile event logs, scene descriptions, and transcript text into structured SKILL.json

Graceful degradation Falls back to events-only recording when screen capture is unavailable

Cross-platform Native accessibility hooks for Windows (UIA), macOS (AX), and Linux (AT-SPI)

Skill versioning Auto-increments on recompile; archives old versions as SKILL.vN.json

Variable templating Prompts the compiler to turn recorded literals such as search queries, dates, dropdown choices, and file paths into reusable inputs

Human-in-the-loop Recording is operator-driven, not agent-driven — the human demonstrates, the AI learns

Tools

Tool Parameters Description

request_capture_permissions_tool — Request microphone and screen capture permissions before recording

record_skill_tool name: str, lead_in_seconds: float = 0.0 Start a human-in-the-loop workflow recording

stop_recording_tool trim_end_seconds: float = 0.0 Stop the active recording and export video to VideoDB when capture is available

compile_skill_tool video_id: str, name: str Compile a recording into SKILL.json and SKILL.md, then install SKILL.md globally for future agent use

list_skills_tool — List all skills generated through this MCP

Resources

Resource Description

skills://list List all available skills as JSON

skills://{name}/content Load a skill's SKILL.md into the agent context

Skill Output

Compiled skills land in ~/.mcp-videodb/skills//:

File Purpose

SKILL.json Structured skill definition with steps, inputs, verification checks, recorded surface data, and execution strategy

SKILL.md Human and agent-readable markdown following the agentskills.io standard

SKILL.vN.json Archived previous versions on recompile

Every generated SKILL.json includes an execution_strategy — web_browser, desktop_app, hybrid, terminal, file_system, or unknown — so the agent consuming the skill knows which tool path to prefer. Every SKILL.md includes execution guidance, continuous improvement guidance, and agent tool-priority guidance.

After SKILL.md is created, compile_skill_tool also installs it into the agent's global skills directory, ~/.codex/skills//SKILL.md by default, and returns global_skill_md_path plus an agent_instruction reminding the agent to verify or report the global install. Agents should ensure this global install step has happened so the skill is available in future runs. Set CODEX_HOME to change the base Codex directory, or set AGENT_GLOBAL_SKILLS_ROOT to override the global skills directory directly.

Architecture

open-record-replay/ ├── server.py # FastMCP entry point, tool and resource definitions ├── state.py # Shared server state singleton ├── config.py # Constants, .env loading ├── registry.py # Skill CRUD + versioning │ ├── capture/ │ ├── recorder.py # Records native accessibility events + optional VideoDB capture │ ├── ax_client.py # JSONL IPC wrapper for native accessibility companion │ ├── capture_client.py # VideoDB Capture SDK wrapper │ └── native/ │ ├── ax_hook_win32.py # Windows: UI Automation + keyboard polling + TCP IPC │ ├── ax_hook_darwin.py # macOS: Accessibility API + pynput + pipe IPC │ └── ax_hook_linux.py # Linux: AT-SPI + pynput + pipe IPC │ ├── compiler/ │ ├── compiler.py # LLM compilation: index scenes → match events → prompt → normalize │ ├── prompts.py # LLM system prompt for structured skill generation │ ├── md_generator.py # Converts SKILL.json to agent-readable SKILL.md │ ├── tool_manifest.py # Surface-to-tool mapping for replay guidance │ └── recommended_tools.json │ ├── schema/ │ └── skill.schema.json # JSON Schema (draft-07) for SKILL.json validation │ ├── scripts/ │ └── smoke_macos_hook.py # macOS AX hook permission helper

Troubleshooting

Recording won't start

Verify VIDEODB_API_KEY is set in .env and is valid

Run request_capture_permissions_tool and approve any permission prompts

On macOS, check Screen Recording and Accessibility permissions

Check that no other application is using the accessibility hook

Compilation fails or returns empty steps

Ensure the recording has meaningful UI interactions (not just idle time)

Try events-only compilation (video_id="") if video indexing is slow

The LLM may need another generation attempt — the compiler is configured for up to 2 attempts

Permission prompts not appearing on macOS

Reset permissions and try again

uv run python scripts/smoke_macos_hook.py --prompt-permissions

If ready_for_event_recording is false, manually enable the terminal in System Settings > Privacy & Security > Accessibility and Input Monitoring.

Windows: no keyboard events recorded

Ensure the app being recorded has UI Automation support (most modern apps do)

Community & Support

Docs: docs.videodb.io

Issues: GitHub Issues

Discord: Join the VideoDB community

API Key: console.videodb.io

Made with ❤️ by the VideoDB team

About

Record desktop workflows once and compile them into reusable AI-agent skills for reliable replay.

www.videodb.io/

Topics

mcp

desktop-automation

workflow-automation

ai-agents

videodb

skill-generation

Resources

Readme

Uh oh!

There was an error while loading. Please reload this page.

Activity

Custom properties

Stars

20 stars

Watchers

0 watching

Forks

1 fork

Report repository

Releases

No releases published

Packages 0

Uh oh!

There was an error while loading. Please reload this page.

Contributors

Uh oh!

There was an error while loading. Please reload this page.

Languages

Python 100.0%