待翻譯：Show HN: LayoutLens: AI-Powered Visual UI Testing

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Uh oh! There was an error while loading. Please reload this page. Notifications You must be signed in to change notification settings Fork 0 Star 8 BranchesTags Open more actions menu Latest commit History 113 Commits 113 Commits Folders and files NameName Last commit message Last commit date .github .github benchmarks benchmarks docs docs examples examples src/layoutlens src/layoutlens tests tests .actrc .actrc .copier-answers.yml .copier-answers.yml .gitignore .gitignore .pre-commit-config.yaml .pre-commit-config.yaml CHANGELOG.md CHANGELOG.md CITATION.cff CITATION.cff CLAUDE.md CLAUDE.md LICENSE LICENSE README.md README.md pyproject.toml pyproject.toml uv.lock uv.lock Repository files navigation LayoutLens catches the layout and accessibility bugs your pixel baseline can't see and your LLM can't be trusted about — deterministic axe-core and geometry checks that run keyless and free in CI, with an optional vision-LLM tier that the deterministic layer is allowed to overrule. Three tiers, use what you need: Tier What runs Needs Reliability Deterministic axe-core WCAG A/AA + geometry/contrast/occlusion scorers no API key or model call measured facts, reproducible; this is the CI gate Hybrid (default) deterministic scan grounds the vision LLM; measured violations force the verdict an LLM API key precision-preserving: the model can add findings, never erase measured ones LLM natural-language questions answered from a screenshot an LLM API key (any LiteLLM provider, incl. Ollama/vLLM via api_base) honest numbers below # Keyless, deterministic — safe as a required check on any fork result = await lens.check_accessibility("page.html", mode="axe") result = await lens.check_layout("page.html", viewport="mobile", mode="deterministic") # Natural-language, grounded by the deterministic scan (hybrid) result = await lens.analyze("https://example.com", "Is the navigation user-friendly?") Or from pytest — the deterministic assertions need no key, and assert_ui skips (never fails) without one: def test_landing_page(layoutlens): layoutlens.assert_a11y("landing.html") # axe, keyless layoutlens.assert_layout("landing.html", viewport="mobile") # keyless layoutlens.assert_ui("landing.html", "Is the CTA above the fold?") Honest numbers: the LLM tier measures 81.1% on the bundled benchmark (60/74 labeled queries, gpt-4o-mini, measured 2026-07-21 — artifact). See Limitations for what vision models can and cannot reliably judge — the deterministic tier exists precisely because of those limits. Quick Start Installation pip install layoutlens playwright install chromium # For screenshot capture Basic Usage LayoutLens's API is async — run it with asyncio.run(...), or await it directly if you're already inside an async def (e.g. pytest-asyncio, FastAPI, a notebook cell). Every snippet below assumes one of those two contexts; only the first one spells out the asyncio.run(...) wrapper. import asyncio from layoutlens import LayoutLens async def main(): # Initialize (uses OPENAI_API_KEY env var) lens = LayoutLens() # Test any website or local HTML result = await lens.analyze( "https://your-site.com", "Is the header properly aligned?" ) print(f"Answer: {result.answer}") print(f"Confidence: {result.confidence:.1%}") asyncio.run(main()) That's it! No selectors, no complex setup, just natural language questions. Deterministic Accessibility Checks (axe-core) — No API Key Required LayoutLens vendors axe-core 4.10.3 and runs it against a real Playwright-rendered page to catch actual WCAG 2.1 A/AA violations — not an LLM guess. This mode is fully keyless: no OPENAI_API_KEY, no network call to an AI provider, just deterministic, reproducible results. CLI # Deterministic axe-core scan only — no API key needed layoutlens page.html --a11y axe # Hybrid: axe-core + LLM vision, axe overrides the verdict on violations (needs an API key) layoutlens https://example.com --a11y hybrid # Legacy vision-only accessibility check (needs an API key) layoutlens page.html --a11y llm --a11y requires one of hybrid/axe/llm and is mutually exclusive with --query — accessibility mode always uses the built-in WCAG checks instead of a free-form question. Python from layoutlens import LayoutLens, AxeAuditor # Raw axe-core report — no LayoutLens instance or API key needed at all report = await AxeAuditor().audit("page.html") print(report.summary()) print(report.ok) # True if there are zero violations print(report.violations) # list[A11yFinding]: rule_id, impact, wcag_refs, nodes, ... # Via the LayoutLens API, restricted to WCAG A/AA tags, still keyless lens = LayoutLens() # no API key required at construction result = await lens.check_accessibility("page.html", mode="axe") print( result.answer ) # "Yes — axe-core found no WCAG A/AA violations" (or lists violated rules) The three modes mode="axe" — deterministic axe-core only. No API key, no LLM call. confidence is always 1.0. mode="hybrid" (default for check_accessibility/check_accessibility) — runs axe-core and the LLM vision analysis, injecting the axe findings into the LLM's prompt as grounding context. If axe finds any violation, the final verdict is deterministically forced to "no" (confidence 1.0), regardless of what the LLM says — axe overrides the model, not the other way around. If axe finds nothing, the LLM's own answer/confidence are kept (it can still flag issues axe's automated rules can't catch, like poor color choices that pass contrast math or confusing visual hierarchy). mode="llm" — legacy vision-only analysis, no axe-core involved. Requires an API key. # Hybrid: axe grounds the LLM and can force the verdict result = await lens.check_accessibility("page.html", mode="hybrid") print(result.metadata["a11y"]) # full axe report dict print(result.metadata["engine"]) # "axe-core 4.10.3" Deterministic Layout Scorers (geometry & contrast) — No API Key Required Alongside axe-core, LayoutLens ships LayoutScorer — a keyless, LLM-free detector for geometric and contrast defects, measured directly off the rendered page with the browser's own layout engine. Foundational contrast and geometry measurements were ported from UIJudgeBench; newer WCAG and text-occlusion checks are independent LayoutLens implementations evaluated by that benchmark. It finds: contrast — text below the WCAG AA ratio (4.5:1 normal, 3.0:1 large), with the measured ratio overlap — sibling elements whose bounding boxes collide clipping — content cut off by a fixed-size box with hidden overflow viewport-protrusion — elements extending past the viewport width (horizontal-scroll bugs) target-size — undersized targets that also fail the machine-measurable WCAG 2.5.8 spacing, inline, and unmodified user-agent-control exceptions focus-obscured — keyboard-focused components entirely hidden by author DOM content (the automatable geometric core of WCAG 2.4.11) text-occlusion — rendered text, including chart labels, covered by another painted DOM element; this is a visual-quality finding, not a WCAG criterion from layoutlens.layout import LayoutScorer, contrast_ratio, read_computed_styles # Scan a page — no LayoutLens instance, no API key, deterministic. report = await LayoutScorer().scan("page.html", viewport="mobile") print(report.ok) # True if no defects found print(report.summary()) # findings grouped by class, with measured receipts for f in report.findings: print( f.defect_class, f.selector, f.measured ) # each finding carries the numbers behind it # Or use the pure WCAG contrast math directly (no browser): contrast_ratio((0x76, 0x76, 0x76), (0xFF, 0xFF, 0xFF)) # -> 4.54 Every finding is a receipt: the offending selector, its bounding box, the measured value, and the threshold it violated. scan(viewport=...) re-runs the geometry at any viewport, so protrusion/overlap that only appear on mobile are caught. Automated findings are not a site-wide WCAG conformance claim. In particular, target-size equivalent/essential exceptions and focus-obscuration interaction-history exceptions remain explicit manual-review fields. pytest Plugin Installing layoutlens registers a pytest plugin (entry point layoutlens). The layoutlens fixture gives you three assertions: def test_checkout(layoutlens): layoutlens.assert_a11y("checkout.html") # keyless axe gate layoutlens.assert_layout( "checkout.html", viewport="mobile" ) # keyless geometry gate layoutlens.assert_ui( "checkout.html", "Is the pay button the most prominent element?" ) assert_a11y / assert_layout are keyless and deterministic — they run on every fork and PR with no secrets, and failure messages carry the rule id, selector, and measured numbers. assert_ui (vision LLM) skips instead of failing when no API key is configured, or always with --layoutlens-no-llm — so one suite serves both the free deterministic lane and the LLM lane. --layoutlens-model picks the model for assert_ui. MCP Server (for coding agents) layoutlens-mcp exposes the checks as MCP tools for Claude Code, Cursor, and friends: pip install "layoutlens[mcp]" # register the stdio server in your agent config: # command: layoutlens-mcp Tools: audit_accessibility and scan_layout (keyless, deterministic — they return measured numbers, not model opinions, in compact summaries of a few hundred tokens), plus check_ui and compare_ui (vision LLM). The deterministic tools cover visual facts accessibility-tree snapshots cannot see: contrast, geometry, target spacing, complete focus obscuration, and text occlusion such as a chart line painted over its label. SARIF Output for GitHub Code Scanning Both deterministic engines emit SARIF 2.1.0: layoutlens page.html --layout deterministic --output sarif > layout.sarif layoutlens page.html --a11y axe --output sarif > a11y.sarif Upload with github/codeql-action/upload-sarif and findings appear as PR annotations with stable rule ids (layout/page-overflow, axe/color-contrast, ...) tracked over time — keyless, so it works on every fork. Or use the packaged action — gojiplus/layoutlens-action — which bundles install, scan, job summary, PR annotations, a sticky results comment, and the SARIF upload into one step: - uses: gojiplus/layoutlens-action@v1 with: sources: "dist/*.html" Key Functions 1. Analyze Pages Test single pages with custom questions: # Test local HTML files result = await lens.analyze("checkout.html", "Is the payment form user-friendly?") # Test with expert context from layoutlens.prompts import Instructions, UserContext instructions = Instructions( expert_persona="conversion_expert", user_context=UserContext( business_goals=["reduce_cart_abandonment"], target_audience="mobile_shoppers" ), ) result = await lens.analyze( "checkout.html", "How can we optimize this checkout flow?", instructions=instructions, ) 2. Compare Layouts Perfect for A/B testing and redesign validation. compare() accepts URLs, local HTML files, or screenshot images — every source is rendered and every screenshot is sent to the model: result = await lens.compare( ["https://old-design.example.com", "https://new-design.example.com"], "Which design is more accessible?", ) print(f"Winner: {result.answer}") 3. Expert-Powered Analysis Domain expert knowledge with one line of code: # Professional accessibility audit (WCAG expert) result = await lens.check_accessibility("product-page.html", compliance_level="AA") # Conversion rate optimization (CRO expert) result = await lens.optimize_conversions( "landing.html", business_goals=["increase_signups"], industry="saas" ) # Mobile UX analysis (Mobile expert) result = await lens.analyze_mobile_ux("app.html", performance_focus=True) # E-commerce audit (Retail expert) result = await lens.audit_ecommerce("checkout.html", page_type="checkout") # Legacy methods still work result = await lens.check_accessibili [truncated for AI cost control]