AgentCrawl, a small self-hosted crawler for AI agents

Notifications You must be signed in to change notification settings

Fork 0

Star 0

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

64 Commits

64 Commits

.github/workflows

.github/workflows

agentcrawl

agentcrawl

assets

assets

benchmarks

benchmarks

docs

docs

examples

examples

integrations/hermes/web-agentcrawl

integrations/hermes/web-agentcrawl

tests

tests

.dockerignore

.dockerignore

.env.example

.env.example

.gitignore

.gitignore

AGENTS.md

AGENTS.md

CHANGELOG.md

CHANGELOG.md

CLAUDE.md

CLAUDE.md

CONTRIBUTING.md

CONTRIBUTING.md

Dockerfile

Dockerfile

INSTALL_FOR_AGENTS.md

INSTALL_FOR_AGENTS.md

LICENSE

LICENSE

README.md

README.md

ROADMAP.md

ROADMAP.md

SECURITY.md

SECURITY.md

docker-compose.yml

docker-compose.yml

pyproject.toml

pyproject.toml

Repository files navigation

🕷️ A small self-hosted crawler for AI agents.

AgentCrawl gives agents a simple way to read normal web pages without pasting raw HTML into chat or routing every URL through a hosted scraper. It turns pages and local documents into Markdown, text, links, metadata, JSON-LD, and crawl results. It runs from the CLI, Python, Docker/API, or as an MCP server.

The project is early, intentionally modest, and being worked on steadily: accessible pages first, clean output, local state, honest failures.

pip install agentcrawl-ai agentcrawl scrape https://pypi.org/project/agentcrawl-ai/

Edge case: example.com returns a Cloudflare client challenge

$ agentcrawl scrape https://example.com

→ ok=False, error_type=client_challenge

example.com sits behind Cloudflare and returns a client challenge on most networks. AgentCrawl detects challenge pages and returns an honest client_challenge error rather than scraping challenge DOM as content. This is the Community product boundary, not a bug. Managed browser/proxy/challenge handling belongs to Enhanced/Hosted and requires an operator or paid plan.

To verify Community works on your machine, point it at any accessible docs page: FastAPI, GitHub, Wikipedia, the RFC editor.

Pick your path 🚀

Agents: MCP 🤖

python -m pip install "agentcrawl-ai[browser]" agentcrawl doctor agentcrawl mcp

MCP tools cover scrape_url, map_site, crawl_site, job status, cancellation, event history, failure inspection, selective retries, usage, and cache control. Coding agents should follow INSTALL_FOR_AGENTS.md.

Developers: Python + CLI 🧪

pip install agentcrawl-ai agentcrawl scrape https://pypi.org/project/agentcrawl-ai/

from agentcrawl import AgentCrawl

crawler = AgentCrawl({"fetcher": "http"}) document = crawler.scrape("https://pypi.org/project/agentcrawl-ai/")

print(document.markdown) print(document.metadata)

Servers: Docker + API 🐳

docker run --rm -p 8000:8000 \ -e AGENTCRAWL_API_KEYS="replace-with-a-long-random-key" \ ghcr.io/jorg18/agentcrawl:latest

curl http://127.0.0.1:8000/health

Read-only local dashboard:

open http://127.0.0.1:8000/dashboard

curl http://127.0.0.1:8000/api/dashboard/summary

Or with Compose:

cp .env.example .env

Replace AGENTCRAWL_API_KEYS and AGENTCRAWL_API_KEY in .env

docker compose up -d curl http://127.0.0.1:8000/health

Why AgentCrawl? 🕸️

Agents often need web context, but raw HTML is a mess. A useful page can arrive mixed with navigation, cookie text, related links, footer links, scripts, and layout junk.

AgentCrawl is a small local layer for that. Give it a URL, get something an agent can read, and keep the cache, jobs, and failures in your own environment.

What works today:

🧹 Known URL in, clean Markdown out: main-content extraction, tables, code blocks, links, metadata, and provenance.

⚡ HTTP first: fast default extraction without starting a browser.

🧱 Durable crawls: SQLite jobs with checkpoints, pagination, cancellation, events, retries, and failure inspection.

📦 Local state: cache, usage, jobs, events, crawl failures, and extracted documents stay with you.

📊 Read-only dashboard: generate static HTML from SQLite with agentcrawl dashboard or open /dashboard on the API server.

🔒 Safer API defaults: bearer auth, robots.txt support, SSRF protections, unsafe redirect blocking, and private-network controls.

🤖 Agent-facing interfaces: CLI, Python, HTTP API, Docker, and MCP.

What Community includes 🧰

AgentCrawl Community is the self-hosted trust layer:

Included Notes

CLI Scrape, crawl, inspect jobs, manage cache, backup, restore.

Python library Local use from scripts and agent runtimes.

HTTP API FastAPI server for self-hosted deployments.

MCP Standards-based stdio MCP server for agent clients.

Docker / GHCR Public image built and smoke-tested by GitHub Actions.

Durable crawls SQLite jobs, events, checkpoints, retries, and failure records.

Local dashboard Read-only static HTML over SQLite via agentcrawl dashboard and /dashboard.

Quality extraction Markdown, links, metadata, JSON-LD/provenance, tables, code blocks.

Basic browser fallback Optional local browser/Camofox path, not required for the default image.

Lightweight docs Install, examples, operations, release, quality notes.

Community is self-hosted. It is designed for accessible web content, local/private workflows, and honest failure reporting when a page is protected by anti-bot or browser challenges. Community may detect challenge pages and return a clear client_challenge/unsupported failure; it does not promise to bypass them. The optional browser path is local bring-your-own rendering, not a managed browser pool. Managed browsers, proxies, schedules, webhooks, retained datasets, teams, billing, and enterprise controls belong to planned enhanced/hosted tiers rather than the Community runtime.

Community boundary 🚧

Community is the open, self-hosted trust layer. It should stay excellent for accessible public HTML, local documents, docs, API references, articles, and reference pages. It should not grow into a free hosted-scraping platform.

Community should report protected pages honestly instead of returning challenge text as content. For protected pages, use a local browser fallback when that is enough; if the target requires managed browser/proxy/challenge infrastructure, treat it as an Enhanced/Hosted use case. Community does not include hosted infrastructure, managed browser pools, proxy networks, geolocation, stealth, schedules, webhooks, retained datasets, teams, billing, or enterprise controls.

Extraction quality 🧹

The Community engine focuses on stable, agent-ready Markdown before benchmark claims:

selects semantic content from

,

, documentation/content containers, or text-rich fallback blocks;

removes unsafe and noisy page chrome such as scripts, styles, hidden content, nav, footer, cookie banners, sidebars, and related-post blocks;

preserves Markdown tables with headers and cell values;

preserves fenced code blocks and language tags from common classes such as language-python and lang-javascript;

attaches extraction provenance such as source/final URL, selected content hint, selection score, candidate count, content hash, extraction strategy, JSON-LD/schema fields, Product offer/rating fields, and output size/structure metadata;

validates extraction quality against checked-in fixtures with a minimum score threshold and Markdown structure checks.

Run the report locally:

python -m benchmarks.quality_report

HTTP API 🌐

Authentication is enabled by default. Configure at least one API key before exposing the server:

export AGENTCRAWL_API_KEYS="replace-with-a-long-random-key" python -m pip install "agentcrawl-ai[browser]" agentcrawl serve --host 0.0.0.0 --port 8000

Health check:

curl http://127.0.0.1:8000/health

Scrape a URL:

curl http://127.0.0.1:8000/v1/scrape \ -H "authorization: Bearer replace-with-a-long-random-key" \ -H "content-type: application/json" \ -d '{"url":"https://example.com","formats":["markdown","links","metadata"]}'

Main endpoints:

GET /health GET /dashboard GET /api/dashboard/summary POST /v1/scrape POST /v1/map POST /v1/crawl GET /v1/jobs/{job_id} GET /v1/jobs/{job_id}/events DELETE /v1/jobs/{job_id} GET /v1/failures GET /v1/jobs/{job_id}/failures POST /v1/jobs/{job_id}/failures/retry POST /v1/extract GET /v1/usage GET /v1/stats DELETE /v1/cache

OpenAPI docs are available at /docs when the server is running.

Local dashboard 📊

Generate a dependency-free static HTML snapshot from any AgentCrawl SQLite database:

agentcrawl dashboard --db agentcrawl.db --output dashboard.html

When the API server is running, open the same read-only view at /dashboard or fetch JSON at /api/dashboard/summary. The dashboard reports job status, crawl queue, open failures, cache domains, and usage units without sending data to any hosted service.

Crawl jobs 🧭

Start an asynchronous crawl:

agentcrawl --remote crawl https://example.com --max-pages 25 --max-depth 2

HTTP clients can attach an idempotency key so retries return the original job instead of starting a duplicate:

curl http://127.0.0.1:8000/v1/crawl \ -H "authorization: Bearer replace-with-a-long-random-key" \ -H "content-type: application/json" \ -H "Idempotency-Key: docs-crawl-2026-06-06" \ -d '{"url":"https://example.com","max_pages":25,"max_depth":2}'

Running jobs checkpoint their queue, visited URLs, retry attempts, progress, and extracted documents in SQLite. Transient page failures use persisted exponential backoff without occupying a crawl worker. They are reclaimed after a service restart.

Read completed documents page by page:

agentcrawl --remote job JOB_ID --offset 0 --limit 100

Run a local command when a crawl finishes with terminal failures:

agentcrawl crawl https://example.com \ --alert-on-failure \ --cmd 'python notify.py'

The command receives JSON on stdin with source, failure_count, and failures.

Inspect or cancel a job:

agentcrawl --remote job JOB_ID agentcrawl --remote job-cancel JOB_ID

/v1/stats reports queue readiness, delayed retries, running and cancelling jobs, crawl failures by status, open retryable failures, and open failures by error type.

Local documents 📄

Community supports local document ingestion without sending file contents to a hosted parser:

agentcrawl scrape ./notes.md agentcrawl scrape ./data.json agentcrawl scrape ./feed.xml python -m pip install "agentcrawl-ai[browser]" agentcrawl scrape ./report.pdf

Current document support:

Input Support

HTML Main-content Markdown extraction.

Markdown Passed through as Markdown.

Text Passed through as plain Markdown text.

JSON Pretty-printed inside a fenced json block.

XML/RSS/Atom Preserved inside a fenced xml block.

PDF Extracted page-by-page to Markdown with the optional docs extra. Enforces size/page safety limits and rejects encrypted PDFs.

Browser rendering

The default package and default Docker image use HTTP extraction. Add browser rendering only when a site needs JavaScript:

python -m pip install "agentcrawl-ai[browser]" playwright install chromium

AgentCrawl also supports an optional external Camofox REST backend:

export AGENTCRAWL_BROWSER_BACKEND=camofox export AGENTCRAWL_CAMOFOX_URL=http://127.0.0.1:9377 export AGENTCRAWL_CAMOFOX_ACCESS_KEY=replace-if-access-control-is-enabled

Cache ⚡

Disable cache for one scrape or choose a TTL of up to 30 days:

{"url":"https://example.com","cache":false}

{"url":"https://example.com","cache_ttl_seconds":3600}

Clear all cache entries or filter by domain or exact URL:

agentcrawl --remote cache-clear agentcrawl --remote cache-clear --domain example.com agentcrawl --remote cache-clear --url https://example.com/page

Backups 💾

Use SQLite online backup before deployment or migration:

agentcrawl backup --db agentcrawl.db --output-dir ./backups

Pass --env-file to copy a protected environment file into the backup directory wi

[truncated for AI cost control]