From 7d020fab5a983e8954e76e4255e41577d141927b Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Mon, 9 Nov 2026 15:02:59 +0000 Subject: [PATCH] Tooling: lint and format Markdown with rumdl rumdl (a ruff-style Markdown linter and formatter) in CI and locally: `uvx rumdl@0.3.2 check .` in CI, `uvx rumdl@0.3.2 fmt .` before a push. .rumdl.toml turns off line length (MD013), turns on aligned tables (MD060), and excludes the frozen experiments/ reports. One-time `rumdl fmt`: tables in README, CLAUDE.md and CHANGELOG aligned, two bare URLs wrapped in <...>. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016isSPowmvG84onwTGWiNoh --- .github/workflows/ci.yml | 1 + .rumdl.toml | 11 +++++++++++ CHANGELOG.md | 2 +- CLAUDE.md | 29 +++++++++++++++-------------- README.md | 16 ++++++++-------- RELEASING.md | 1 + 6 files changed, 37 insertions(+), 23 deletions(-) create mode 100644 .rumdl.toml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 76474ce..52550c1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,5 +44,6 @@ jobs: - run: uv sync --locked --dev --extra viewer - run: uv run ruff check src/ tests/ - run: uv run ruff format --check src/ tests/ + - run: uvx rumdl@0.3.2 check . # Markdown: tables aligned, see .rumdl.toml - run: uv run ty check src/ - run: uv run pytest tests/ -x -q diff --git a/.rumdl.toml b/.rumdl.toml new file mode 100644 index 0000000..f7840c5 --- /dev/null +++ b/.rumdl.toml @@ -0,0 +1,11 @@ +# Markdown lint and format, ruff-style: `uvx rumdl check` in CI, `uvx rumdl fmt` locally. +[global] +# Long lines are fine in prose; wrapping only adds diff noise. +extend_disable = ["MD013"] +# Frozen experiment reports: kept as they were written. +exclude = ["experiments/**"] + +# Align table columns (off by default). +[MD060] +enabled = true +style = "aligned" diff --git a/CHANGELOG.md b/CHANGELOG.md index e2baf4e..3549bbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,7 +52,7 @@ ocr-bench is experimental, so these go without a transition release. ### Other - Code that pairs lists that must have the same length now uses `zip(..., strict=True)`, so a mismatch raises an error instead of dropping rows silently. This covers judge results and comparisons, jury votes, fingerprints and alignment columns. -- Development: `ty` is pinned to 0.0.97 and ruff to 0.15.4. ruff also checks the `B` rules. `src/` and `tests/` are formatted with `ruff format`. CI checks the format. CI also checks that `uv.lock` resolves from https://pypi.org/simple only and records no `exclude-newer` setting. +- Development: `ty` is pinned to 0.0.97 and ruff to 0.15.4. ruff also checks the `B` rules. `src/` and `tests/` are formatted with `ruff format`. CI checks the format. CI also checks that `uv.lock` resolves from only and records no `exclude-newer` setting. ## [0.0.6] - 2026-11-04 diff --git a/CLAUDE.md b/CLAUDE.md index a6026ec..99d193a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,19 +10,19 @@ Inspired by [Datalab's Benchmarks + Evals](https://www.datalab.to/blog/datalab-b ## Architecture -| Module | What it does | -|--------|-------------| -| `elo.py` | Bradley-Terry MLE via scipy, page-level bootstrap 95% CIs, ELO scale | -| `judge.py` | Criteria-profile/custom judge prompts, HTML normalization, truncation disclosure, Comparison dataclass, structured output schema | -| `dataset.py` | Flat, config-per-model, PR-based dataset loading, OCR column discovery | -| `backends.py` | API backends: InferenceProvider + OpenAI-compatible, concurrent calls | -| `publish.py` | Publish comparisons + leaderboard to Hub; incremental load from existing results | -| `run.py` | Orchestrator: launch N OCR models via HF Jobs | -| `validate.py` | Human A/B validation data layer, agreement stats, human ELO | -| `viewer.py` | Data loading for results viewer (pure functions) | -| `web.py` | FastAPI + HTMX unified viewer (browse + validate in one app) | -| `integrity.py` | Input-integrity checks shared by judge guards + `audit`: sentinel/empty/length stats, per-model failure counts, audit report | -| `cli.py` | CLI: `judge` (incremental + `--full-rejudge`), `run`, `view`, `audit` | +| Module | What it does | +| -------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| `elo.py` | Bradley-Terry MLE via scipy, page-level bootstrap 95% CIs, ELO scale | +| `judge.py` | Criteria-profile/custom judge prompts, HTML normalization, truncation disclosure, Comparison dataclass, structured output schema | +| `dataset.py` | Flat, config-per-model, PR-based dataset loading, OCR column discovery | +| `backends.py` | API backends: InferenceProvider + OpenAI-compatible, concurrent calls | +| `publish.py` | Publish comparisons + leaderboard to Hub; incremental load from existing results | +| `run.py` | Orchestrator: launch N OCR models via HF Jobs | +| `validate.py` | Human A/B validation data layer, agreement stats, human ELO | +| `viewer.py` | Data loading for results viewer (pure functions) | +| `web.py` | FastAPI + HTMX unified viewer (browse + validate in one app) | +| `integrity.py` | Input-integrity checks shared by judge guards + `audit`: sentinel/empty/length stats, per-model failure counts, audit report | +| `cli.py` | CLI: `judge` (incremental + `--full-rejudge`), `run`, `view`, `audit` | ## Tooling @@ -36,11 +36,12 @@ Inspired by [Datalab's Benchmarks + Evals](https://www.datalab.to/blog/datalab-b uv sync --dev --extra viewer uv run ruff check src/ tests/ uv run ruff format src/ tests/ +uvx rumdl@0.3.2 fmt . # Markdown (config: .rumdl.toml) uv run ty check src/ uv run pytest tests/ -x -q ``` -CI also checks `ruff format --check`, and that `uv.lock` resolves from https://pypi.org/simple only and records no `exclude-newer` setting. A local uv config (another index, or `exclude-newer`) can leak into the lock. Relock with `uv lock --no-config`. +CI also checks `ruff format --check`, `rumdl check` (Markdown, with aligned tables), and that `uv.lock` resolves from only and records no `exclude-newer` setting. A local uv config (another index, or `exclude-newer`) can leak into the lock. Relock with `uv lock --no-config`. Branch protection is on — all changes go through PRs with CI checks. diff --git a/README.md b/README.md index 50c6592..4432a65 100644 --- a/README.md +++ b/README.md @@ -14,14 +14,14 @@ ocr-bench lets you run the same set of OCR models on a sample of _your_ collecti The two published example boards, as ELO with page-level bootstrap 95% confidence intervals. Models whose intervals overlap are not separated by the run: -| Model | Britannica 1771 (50 pages) | BPL card catalog (33 judged cards of 50) | -| ------------------ | :-------------------------: | :--------------------------------------: | -| NuExtract3 (4B) | 1734 (1669–1820) | — | -| GLM-OCR (0.9B) | 1588 (1523–1655) | 1434 (1361–1509) | -| LightOnOCR-2 (1B) | 1588 (1552–1623) | 1526 (1449–1611) | -| FireRed-OCR (2.1B) | 1426 (1358–1489) | — | -| dots.ocr (1.7B) | 1409 (1345–1469)\* | 1632 (1571–1702) | -| DeepSeek-OCR (4B) | 1255 (1190–1309) | 1408 (1336–1474) | +| Model | Britannica 1771 (50 pages) | BPL card catalog (33 judged cards of 50) | +| ------------------ | :------------------------: | :--------------------------------------: | +| NuExtract3 (4B) | 1734 (1669–1820) | — | +| GLM-OCR (0.9B) | 1588 (1523–1655) | 1434 (1361–1509) | +| LightOnOCR-2 (1B) | 1588 (1552–1623) | 1526 (1449–1611) | +| FireRed-OCR (2.1B) | 1426 (1358–1489) | — | +| dots.ocr (1.7B) | 1409 (1345–1469)\* | 1632 (1571–1702) | +| DeepSeek-OCR (4B) | 1255 (1190–1309) | 1408 (1336–1474) | On Britannica, NuExtract3 is ahead of every other model. Below it, GLM-OCR and LightOnOCR-2 overlap each other and are ahead of the rest. FireRed-OCR and dots.ocr overlap each other and are ahead of DeepSeek-OCR. On BPL, dots.ocr is ahead of GLM-OCR and DeepSeek-OCR, and LightOnOCR-2 overlaps all three. So dots.ocr is in the lower half on Britannica and has the highest rating on BPL, where only LightOnOCR-2 overlaps it. Two collections cannot show which model suits which document type, but they show that the order can change. For your own boards, judging more pages gathers more evidence, and human votes in the viewer cross-check the judge on the overlapping pairs; neither guarantees separation. diff --git a/RELEASING.md b/RELEASING.md index 86a46f7..cd03733 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -41,6 +41,7 @@ Do not push a tag by hand. The workflow creates it. A push to `main` that change ```bash uv run ruff check src/ tests/ uv run ruff format --check src/ tests/ + uvx rumdl@0.3.2 check . uv run ty check src/ uv run --extra viewer pytest tests/ -x -q ```