翻訳待ち：Show HN: Writing-eval, local deterministic style checks for AI-written drafts

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Uh oh! There was an error while loading. Please reload this page. Notifications You must be signed in to change notification settings Fork 0 Star 4 BranchesTags Open more actions menu Latest commit History 2 Commits 2 Commits Folders and files NameName Last commit message Last commit date .github .github benchmark benchmark docs docs rules rules runs runs scripts scripts skills/writing-eval skills/writing-eval src/writing_eval src/writing_eval tests tests .gitignore .gitignore AGENTS.md AGENTS.md CHANGELOG.md CHANGELOG.md CLA.md CLA.md CLAUDE.md CLAUDE.md CONTRIBUTING.md CONTRIBUTING.md LICENSE LICENSE README.md README.md pyproject.toml pyproject.toml writing-eval writing-eval Repository files navigation Built by David Paluy from Majestic Labs. writing-eval gives teams a repeatable way to measure a draft against a chosen editorial voice. Build a reusable style profile from reference prose, check a draft against it, and get specific evidence about differences in clarity, readability, sentence rhythm, vocabulary, and detected writing patterns. The project also includes a corpus evaluation pipeline for comparing generated outputs with a reference corpus. Use it for local diagnostics, regression checks, and repeatable comparisons between writing systems. Everything runs locally on the CPU and produces deterministic results. The tool does not call hosted models, upload source material, train models, optimize detectors, or reproduce a proprietary evaluation method. Installation Python 3.11 or newer and uv are required. uv sync The release version is the [project].version value in pyproject.toml. Update this value for every release. To show the installed release version, run: ./writing-eval --version Quick start Build a style profile from a directory of an author's posts, then check a draft against it: ./writing-eval profile build acme --from posts/acme ./writing-eval check draft.md --style acme The first command ingests the .md and .txt files under posts/acme and writes a reusable profile named acme into data/profiles/acme/. The second audits draft.md and produces a scored, profile-relative assessment. The human-readable report shows four section scores, actionable issues with current and target values, editing instructions, success criteria, source locations, and general statistics. The profile name is kept out of the report body; it remains in JSON metadata for reproducibility. Use with LLM agents The repository includes an agent skill at skills/writing-eval/SKILL.md. Agent harnesses that support SKILL.md instructions can load it to select the correct command, build or choose a profile, interpret exit codes and JSON, protect private source material, and report results without overstating the heuristic score. The skill controls the local CLI and does not bundle the executable. Use it from a repository checkout after running uv sync. Example requests: Use writing-eval to check docs/draft.md against the acme profile. Summarize the highest-priority issues with their source locations. Build a writing-eval profile named product-docs from the authorized prose in data/product-docs, then check docs/new-guide.md against it. Compare the writing systems in runs/release-candidate against data/reference-corpus.jsonl and explain the report verdict. What it includes Capability Use it for Style profiles Build a reusable baseline from approved prose Draft checks Compare one Markdown or plain-text file with a profile Rule-based linting Locate configurable writing tendencies Corpus evaluation Compare several output systems consistently Markdown and JSON reports Support human review, automation, and regression gates Why we built this AI-generated writing is easy to demo and difficult to evaluate consistently. A draft can be grammatically correct while still missing an organization's voice, preferred structure, or editorial constraints. writing-eval turns those expectations into a local, versioned measurement process. Teams can run the same checks after changing a prompt, model, reference corpus, or rule set and see what improved or regressed. This project is one narrow example of a broader Majestic Labs principle: an AI workflow needs a company-controlled definition of acceptable work. Read Building Private AI Evals for the broader approach. Documentation Use with LLM agents Single-document checks Style profiles Metrics Corpus evaluation and benchmark Limitations License and contributions Managed hosting About Majestic Labs Single-document check ./writing-eval check audits one draft the way a linter audits one source file, without any JSONL wrapping. It takes a Markdown or plain-text file, or - to read from standard input. ./writing-eval check draft.md ./writing-eval check draft.md --references data/reference-corpus.jsonl cat draft.md | ./writing-eval check - Options: --rules selects the rule file (default: the builtin rule set that ships with the package). The repository also ships an optional overlay with extra AI-writing tells; see The anti-ai overlay. --references is an optional JSONL reference corpus. When omitted, the token 1-gram L2 metric is skipped, rendered as n/a, and a note is printed to standard error. --style compares the draft against a named style profile and renders a scored assessment (see Style profiles). It is mutually exclusive with --references; passing both is a usage error. --profiles-root locates profiles for --style (default data/profiles). --format text|json selects human-readable text or JSON on standard output (default text). --json PATH writes the same JSON result to a file while preserving the selected standard-output format. Without --style, text output retains the linter format: one finding per line, sorted by position, using 1-indexed line and column offsets computed from the real character positions of each match, followed by a metrics block: draft.md:1:1 [warn] metadiscourse_openers: Remove the metadiscourse opener and state the point directly. | span: In this article, draft.md:1:21 [warn] polish_vocab: Replace overused polish vocabulary with specific language. | span: delve metrics: word_count: 15 tell_rates_by_severity: warn: 400.000000 mean_sentence_length: 7.500000 sentence_length_variance: 2.250000 repeated_opening_rate: 0.000000 token_1gram_l2: n/a quality_metrics (informational): flesch_reading_ease: 52.000000 flesch_kincaid_grade: 9.000000 mtld: 14.000000 paragraph_stats: paragraph_count: 1.000000 mean_paragraph_sentence_count: 2.000000 single_sentence_paragraph_rate: 0.000000 The example values are illustrative. Scores below 10 tokens render mtld as n/a, and text without a sentence renders the readability scores as n/a. With --style, text output uses the scored assessment described in Check a draft against a profile. Exit codes distinguish completed checks from input errors: 0: the check completed, with or without findings. 1: a usage or input error (missing file, unreadable rules, invalid JSONL). Style profiles A style profile is a deterministic fingerprint of one author's writing, built from a corpus of their prose. Build a profile once, then check any draft against it to see how far the draft sits from that voice and which vocabulary and structure differ. The author's own voice is just one profile among many. Build a profile ./writing-eval profile build --from [--profiles-root data/profiles] [--rules PATH] must be exactly one non-absolute path component. ., .., absolute paths, and nested names are rejected. --from accepts directories (their .md and .txt files are ingested recursively) or individual files. --rules selects the rule file used to precompute the profile cache (default: the builtin rule set). A leading YAML frontmatter block is stripped from each source, and each document becomes one reference record with a stable ID derived from its filename. The command writes two files into //: references.jsonl: one {"id", "text", "file"} record per source document, reused as the reference corpus by check --style. profile.json: the profile name, its creation date, a numeric metrics_version field (currently 2), a references_sha256 digest of the paired references.jsonl file, per-source word counts, the total word count, and the corpus statistics (mean sentence length and variance, repeated-opening rate, Flesch reading ease and grade, MTLD, paragraph statistics, and the top 20 content tokens after a small stop list). metrics_version pins the metric semantics the stored statistics were computed with. Version 2 covers curly-apostrophe (U+2019) sentence openers, markdown-aware readability word counts, and MTLD tail, threshold, and sequence-input lowercase behavior. Older profiles are rejected with a rebuild instruction until profile build runs again with the current tool. references_sha256 binds profile.json to the installed references.jsonl. A missing, invalid, or mismatched digest is the same rebuild error, so a mixed pair from an interrupted write is not loaded. For example, put all articles for one author under a dedicated directory: posts/acme/ ├── choosing-a-market.md ├── distribution-first.md ├── founder-notes.txt └── archive/ └── early-lessons.md Then build the profile from the directory: ./writing-eval profile build acme --from posts/acme The directory is scanned recursively, so this imports all four .md and .txt articles, including archive/early-lessons.md. There is no need to write a wildcard or list every file. A successful build reports the number of imported sources and words, for example: built profile 'acme': 30 sources, 15742 words -> data/profiles/acme To import selected articles instead, list each file after --from: ./writing-eval profile build acme --from \ posts/acme/choosing-a-market.md \ posts/acme/distribution-first.md Recommended corpus size The command accepts a single non-empty article, but a small profile makes the score depend on which articles you happened to include. For a profile used as a style baseline, use at least 25 articles, and prefer 40 or more. That number is measured, not assumed. Holding a draft fixed and varying only which articles form the profile, the standard deviation of the resulting score falls with more articles. Two points is the rubric's smallest unit, the deduction for one excess warn occurrence, so below that threshold the sampling noise is smaller than anything the score can express. The mean standard deviation crosses 2 points around N = 20, but individual drafts vary widely, so the number that matters is coverage: the share of drafts that have actually settled below 2 points at a given profile size. Articles in profile Drafts at or under 2 points of noise 10 33.3% 15 37.5% 20 54.2% 25 70.8% 30 66.7% 40 87.5% 50 95.8% 60 95.8% 25 articles covers 7 in 10 drafts. 30 articles is not reliably better than 25; the difference sits inside trial noise. 40 articles covers 9 in 10, and 50 covers 24 in 25, which is why 40 is the preferred target. As a rule that holds at every corpus size tested: treat a score difference under 3 points as noise, whether between two drafts or between two runs of the same draft. Article count drives stability, not word count. At a fixed word budget, a profile built from more, shorter articles is consistently more stable than one built from fewer, longer articles. At 40,000 words, 28 articles gave a standard deviation of 1.3 while 11 articles gave 2.5. Prior versions of this document recommended a 15,000-word minimum; that figure was not supported by measurement and has been removed. At a fixed article count, per-article length still matters up to a point: a corpus averaging about 1,200 words per article was roughly 0.4 points noisier than one averaging about 2,350, with no further gain past about [truncated for AI cost control]