Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,6 @@ jobs:
- run: uv sync --locked --dev --extra viewer
- run: uv run ruff check src/ tests/
- run: uv run ruff format --check src/ tests/
- run: uvx rumdl@0.3.2 check . # Markdown: tables aligned, see .rumdl.toml
- run: uv run ty check src/
- run: uv run pytest tests/ -x -q
11 changes: 11 additions & 0 deletions .rumdl.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Markdown lint and format, ruff-style: `uvx rumdl check` in CI, `uvx rumdl fmt` locally.
[global]
# Long lines are fine in prose; wrapping only adds diff noise.
extend_disable = ["MD013"]
# Frozen experiment reports: kept as they were written.
exclude = ["experiments/**"]

# Align table columns (off by default).
[MD060]
enabled = true
style = "aligned"
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ ocr-bench is experimental, so these go without a transition release.
### Other

- Code that pairs lists that must have the same length now uses `zip(..., strict=True)`, so a mismatch raises an error instead of dropping rows silently. This covers judge results and comparisons, jury votes, fingerprints and alignment columns.
- Development: `ty` is pinned to 0.0.97 and ruff to 0.15.4. ruff also checks the `B` rules. `src/` and `tests/` are formatted with `ruff format`. CI checks the format. CI also checks that `uv.lock` resolves from https://pypi.org/simple only and records no `exclude-newer` setting.
- Development: `ty` is pinned to 0.0.97 and ruff to 0.15.4. ruff also checks the `B` rules. `src/` and `tests/` are formatted with `ruff format`. CI checks the format. CI also checks that `uv.lock` resolves from <https://pypi.org/simple> only and records no `exclude-newer` setting.

## [0.0.6] - 2026-11-04

Expand Down
29 changes: 15 additions & 14 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,19 @@ Inspired by [Datalab's Benchmarks + Evals](https://www.datalab.to/blog/datalab-b

## Architecture

| Module | What it does |
|--------|-------------|
| `elo.py` | Bradley-Terry MLE via scipy, page-level bootstrap 95% CIs, ELO scale |
| `judge.py` | Criteria-profile/custom judge prompts, HTML normalization, truncation disclosure, Comparison dataclass, structured output schema |
| `dataset.py` | Flat, config-per-model, PR-based dataset loading, OCR column discovery |
| `backends.py` | API backends: InferenceProvider + OpenAI-compatible, concurrent calls |
| `publish.py` | Publish comparisons + leaderboard to Hub; incremental load from existing results |
| `run.py` | Orchestrator: launch N OCR models via HF Jobs |
| `validate.py` | Human A/B validation data layer, agreement stats, human ELO |
| `viewer.py` | Data loading for results viewer (pure functions) |
| `web.py` | FastAPI + HTMX unified viewer (browse + validate in one app) |
| `integrity.py` | Input-integrity checks shared by judge guards + `audit`: sentinel/empty/length stats, per-model failure counts, audit report |
| `cli.py` | CLI: `judge` (incremental + `--full-rejudge`), `run`, `view`, `audit` |
| Module | What it does |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `elo.py` | Bradley-Terry MLE via scipy, page-level bootstrap 95% CIs, ELO scale |
| `judge.py` | Criteria-profile/custom judge prompts, HTML normalization, truncation disclosure, Comparison dataclass, structured output schema |
| `dataset.py` | Flat, config-per-model, PR-based dataset loading, OCR column discovery |
| `backends.py` | API backends: InferenceProvider + OpenAI-compatible, concurrent calls |
| `publish.py` | Publish comparisons + leaderboard to Hub; incremental load from existing results |
| `run.py` | Orchestrator: launch N OCR models via HF Jobs |
| `validate.py` | Human A/B validation data layer, agreement stats, human ELO |
| `viewer.py` | Data loading for results viewer (pure functions) |
| `web.py` | FastAPI + HTMX unified viewer (browse + validate in one app) |
| `integrity.py` | Input-integrity checks shared by judge guards + `audit`: sentinel/empty/length stats, per-model failure counts, audit report |
| `cli.py` | CLI: `judge` (incremental + `--full-rejudge`), `run`, `view`, `audit` |

## Tooling

Expand All @@ -36,11 +36,12 @@ Inspired by [Datalab's Benchmarks + Evals](https://www.datalab.to/blog/datalab-b
uv sync --dev --extra viewer
uv run ruff check src/ tests/
uv run ruff format src/ tests/
uvx rumdl@0.3.2 fmt . # Markdown (config: .rumdl.toml)
uv run ty check src/
uv run pytest tests/ -x -q
```

CI also checks `ruff format --check`, and that `uv.lock` resolves from https://pypi.org/simple only and records no `exclude-newer` setting. A local uv config (another index, or `exclude-newer`) can leak into the lock. Relock with `uv lock --no-config`.
CI also checks `ruff format --check`, `rumdl check` (Markdown, with aligned tables), and that `uv.lock` resolves from <https://pypi.org/simple> only and records no `exclude-newer` setting. A local uv config (another index, or `exclude-newer`) can leak into the lock. Relock with `uv lock --no-config`.

Branch protection is on — all changes go through PRs with CI checks.

Expand Down
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@ ocr-bench lets you run the same set of OCR models on a sample of _your_ collecti

The two published example boards, as ELO with page-level bootstrap 95% confidence intervals. Models whose intervals overlap are not separated by the run:

| Model | Britannica 1771 (50 pages) | BPL card catalog (33 judged cards of 50) |
| ------------------ | :-------------------------: | :--------------------------------------: |
| NuExtract3 (4B) | 1734 (1669–1820) | — |
| GLM-OCR (0.9B) | 1588 (1523–1655) | 1434 (1361–1509) |
| LightOnOCR-2 (1B) | 1588 (1552–1623) | 1526 (1449–1611) |
| FireRed-OCR (2.1B) | 1426 (1358–1489) | — |
| dots.ocr (1.7B) | 1409 (1345–1469)\* | 1632 (1571–1702) |
| DeepSeek-OCR (4B) | 1255 (1190–1309) | 1408 (1336–1474) |
| Model | Britannica 1771 (50 pages) | BPL card catalog (33 judged cards of 50) |
| ------------------ | :------------------------: | :--------------------------------------: |
| NuExtract3 (4B) | 1734 (1669–1820) | — |
| GLM-OCR (0.9B) | 1588 (1523–1655) | 1434 (1361–1509) |
| LightOnOCR-2 (1B) | 1588 (1552–1623) | 1526 (1449–1611) |
| FireRed-OCR (2.1B) | 1426 (1358–1489) | — |
| dots.ocr (1.7B) | 1409 (1345–1469)\* | 1632 (1571–1702) |
| DeepSeek-OCR (4B) | 1255 (1190–1309) | 1408 (1336–1474) |

On Britannica, NuExtract3 is ahead of every other model. Below it, GLM-OCR and LightOnOCR-2 overlap each other and are ahead of the rest. FireRed-OCR and dots.ocr overlap each other and are ahead of DeepSeek-OCR. On BPL, dots.ocr is ahead of GLM-OCR and DeepSeek-OCR, and LightOnOCR-2 overlaps all three. So dots.ocr is in the lower half on Britannica and has the highest rating on BPL, where only LightOnOCR-2 overlaps it. Two collections cannot show which model suits which document type, but they show that the order can change. For your own boards, judging more pages gathers more evidence, and human votes in the viewer cross-check the judge on the overlapping pairs; neither guarantees separation.

Expand Down
1 change: 1 addition & 0 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ Do not push a tag by hand. The workflow creates it. A push to `main` that change
```bash
uv run ruff check src/ tests/
uv run ruff format --check src/ tests/
uvx rumdl@0.3.2 check .
uv run ty check src/
uv run --extra viewer pytest tests/ -x -q
```
Expand Down
Loading