TESTING.md

Testing

npm run verify        # everything CI runs, and everything the deploy gate runs
npm test              # just the suite
npm run test:watch    # while working
npm run test:coverage # with thresholds enforced

verify is one gate: typecheck, lint, format check, tests with coverage thresholds, a production build, catalog schema validation, the bundle budget, and the published test counts. CI and the Pages deploy both call the same reusable workflow, so there is exactly one definition of "verified" and no way to publish past it.

What the suite covers

1432 tests in all: 629 unit and integration tests in this package, 230 in apps/mobile/, 102 in worker/, 49 in api/, 48 in mcp/, 9 in updates/, 137 in vscode/, and 228 browser tests across four viewports. Those eight figures are the whole suite and they sum to the total. Two earlier revisions did not: one claimed 533, which was neither the sum of its own list nor inclusive of mcp/ at all; the next claimed 580, which was correct when written and went stale the moment the VS Code extension landed with its own 137 tests. A total that is not re-derived from a run is a number that decays. The distribution is deliberately uneven: depth follows the cost of being wrong, not the size of the file.

Those totals are no longer kept by hand. npm run check:test-badge asks every suite what it contains, adds the answers up, and fails if the README badge, the total above, or any per-suite figure on this page disagrees with it. The list summing to the total is now a property of the build rather than something to proofread — which is the whole reason it exists, since proofreading is exactly what failed twice.

The per-file table below is checked too, and that is newer than the rest. It used to carry a note saying it was not checked and drifted by one or two — that it was a map of where the depth is rather than an inventory. Then two tests were added on 2026-08-04, two of its rows went stale, and every check above still passed: the totals were corrected, the badge agreed with the sum, and two rows inside the table that produces that sum were quietly wrong.

So it is now an inventory. Three things fail the build: a row whose figure has drifted, a row for a file that holds no tests, and a test file with no row at all. The last is what makes this table's implicit claim — that these are the whole package and they sum to the total — true by construction rather than by anyone adding them up.

Suite	Tests	Guards
scripts/lib/pipeline.test.ts	68	The whole pipeline against fixtures: the trust ladder, both sanity thresholds, cold start, stale retention, flag-once semantics — including a standing flag whose figures drift, which is what re-opened a pull request every morning — full-field diffing, coverage versus a price move versus a source correction, the content hash, changelog rendering, that a feed reporting the intro rate in force is not drift, and that the morning's vendor-page disagreement raises its flag without touching the price.
scripts/lib/vendor-check.test.ts	34	The daily vendor-page check without the network: which rows can be checked (a provenance-only override against the catalog's figure), the blind extraction prompt, every outcome of comparing a page with the record — agreement within rounding, a moved standard or cached rate, a promotion the record lacks, a promotion withdrawn early, a page showing only the promotional figure — that only agreement moves a date, that a verifyHint reaches the reader without leaking the figure on record when one page prices the same model in several regional tables, that yesterday's report cannot raise today's flags, and the DeepSeek reader's JSON-mode request, its one retry, and every way a completion can carry no usable reply.
scripts/lib/price-history.test.ts	5	The citable price-history export: a listing writes one row per rate the model arrived with, a move, a source correction, coverage and a delisting are told apart, dates are the commit's UTC day, every field has a unit, and the CSV quotes only what RFC 4180 requires.
scripts/lib/readme-badges.test.ts	3	The daily catalog sync updates both badge URLs and accessible alt text, leaves unrelated prose untouched, and fails closed if a README redesign removes or duplicates a derived claim.
scripts/lib/test-count-fix.test.ts	7	The test-count self-fix: only figures that are wrong are rewritten, at every occurrence and including one that gains a digit; a corrected document needs no further edits; the per-file table follows the real count and leaves rows it cannot judge alone; only prose naming the browser suite is rewritten; one edit found by two rules applies once.
scripts/lib/privacy-config.test.ts	4	The logging settings the privacy policy describes, held to the deployment configuration: per-request logs off on the alerts Worker (its links carry private tokens) and fully off on the update server, on for the pricing API, and each stated on the privacy page.
src/lib/contrast.test.ts	49	Every accent × theme × canvas combination against every surface it can appear on, plus the status colours on their own soft fills — the pairing the accent matrix does not reach.
src/lib/engine/cost.test.ts	44	Compounding history, per-turn long-context tiers, cache reads and writes, promotional windows, reasoning multipliers, margins, break-even, and the scenarios that cannot physically run.
src/lib/engine/observed-session-cost.test.ts	6	Request-by-request observed conversation costs, separate cache reads/writes, hidden reasoning when known, promotional windows, and impossible cache totals.
src/App.test.tsx	31	The views render, the URL round-trips, the tour and palette work — plus a named regression test for every defect the pre-publication audit found, and that the free-tier comparison is reachable from every main surface.
src/receipt/receiptSpec.test.ts	5	One source remains identical across the visible Receipt, clipboard text, plain-text instructions, and JSON contract, including the temporary transparent boundary and fail-closed current-pricing rule.
src/receipt/shareReceipt.test.ts	3	Assistant share blocks are parsed defensively, incomplete data fails recoverably, and user-controlled values are escaped before entering exported receipt artwork.
packages/core/src/receipt/share.test.ts	4	The shared native/web parser accepts fenced and nested results, normalizes display strings, applies safe field bounds, preserves explicit defaults, and rejects malformed, incomplete, or oversized input.
packages/core/src/receipt/spec.test.ts	2	The shared instructions preserve the visible-message privacy boundary, current-price requirement, quality caveat, and byte-identical machine-readable contract.
packages/core/src/pricing/promo.test.ts	2	Promotional prices remain active through the last instant of their published end date and return to standard pricing immediately after expiration.
src/lib/pricing/catalog.test.ts	43	Schema validation including every malformed case the pipeline could produce, alias handling, retired rows kept but never selectable, the country filter and its counts, and the four states the chip can report.
src/components/SyncChip.test.tsx	8	The chip's wording and its refusal to imply freshness it cannot evidence, including the timezone trap in rendering an ISO date.
src/components/PromoRate.test.tsx	8	The shared rate marker: the promotional figure with its tooltip while a window is open, the standard figure and no marker once it closes or when there is no window, the last moment of the until day still inside it, and cache rates marked only when the vendor publishes a promotional one.
src/components/AlertsPanel.test.tsx	21	Subscribing, confirming, following a whole country, the arrival path an emailed preferences link takes, and the unsubscribe halves.
src/components/LaunchBanner.test.tsx	7	The launch announcement now that both apps are live: each official badge links to its own store, a QR code per store with the Android robot credited, a link to the permanent /app/ page, no form and no network request, a remembered dismissal, and that closing the old "coming soon" banner does not hide this one.
src/lib/seo/pages.test.ts	31	Which models get a page, worked costs against hand arithmetic, ranking, the two kinds of “cheaper”, and the rules that keep the comparison set curated; that a sitemap lastmod moves only for a page whose content changed (not on a re-verified morning, but on the day a promotion lapses), that a published comparison keeps being built when the curation would drop it and the ceiling binds only new pairs, that a pair which lost a model becomes a signpost pointing at the survivor, and the pair-specific differences each comparison states.
src/lib/seo/ledger.test.ts	13	The page ledger behind lastmod and sticky comparisons: fingerprints independent of key order, recorded dates kept while content is unchanged, --fix idempotent and never forgetting a page, drift reported, and a malformed file refused rather than read as empty.
src/lib/url/scenario.test.ts	15	URL round-tripping, clamping, and that prompt text never appears in a link.
src/lib/seo/render.test.ts	26	Escaping a hostile display name out of both the markup and the JSON-LD, the policy hash matching what is emitted, no inline styles the policy would silently drop, that the footer offers the API these pages tell readers to prefer and the /app/ page, that the app page links both badges and QR codes to the live listings, and that a promotional rate is marked on the figure itself — during the window and not after — on every page that prints one; that no ordinary page is noindex, that a comparison says what separates the pair beyond price and when one side has been retired, and that a retired comparison is noindex, canonical to the surviving model and links on; and that a page loads its own scripts only from this site.
src/lib/select/cheaper.test.ts	11	Which alternatives count as genuinely cheaper: the capability bar, unscored models excluded rather than assumed to clear it, routing aliases and the target itself never offered, and an explanation rather than bare emptiness.
src/lib/palette.test.ts	10	Colour-vision separation of the chart pair, simulated rather than assumed.
src/lib/tokenize/tokenize.test.ts	10	That the real tokenizer runs, that estimates stay within a sane margin of it, and that CJK costs more.
src/lib/engine/money.test.ts	10	Integer exactness across the realistic range, and graceful degradation past it.
src/lib/engine/format.test.ts	9	Money, token and rate formatting at the boundaries.
src/lib/seo/slug.test.ts	8	That a URL is a permanent identifier: the transformation, and the collision that must fail the build rather than overwrite a page.
src/lib/seo/llms.test.ts	9	That llms.txt stays an index rather than a data dump: absolute URLs, cheapest first, the boundary of what a price means, the catalog timestamp so staleness is visible, and that it names the MCP server, the extension and both native apps.
src/lib/free-tiers/types.test.ts	17	The free-tier data file passes its own gate and covers exactly the catalog's providers; a fact without the vendor's words, an https source or a date is refused, as are markdown in a quote, a one-sided contradiction, a headline with no fact behind it and a misspelt field, and a date not on the calendar.
src/lib/free-tiers/quote.test.ts	13	Whether a vendor page still contains a quote: table rows, HTML, markdown links, curly quotes and line breaks never cause a false alarm; a changed figure, currency or missing wording always fails; elisions match in order.
src/lib/free-tiers/check.test.ts	13	The daily free-tier check: confirmed, missing and unreadable told apart, the date a quote disappears or returns kept, an unreadable morning never hiding a known gap, a hand re-read starting afresh, and a damaged report never read as a confirmation.
src/lib/seo/free-tier-pages.test.ts	18	The free-tier pages: free tiers listed first, the price after the free tier from a model still on sale, lastmod moving only with content or a vanished quote, escaping, sources and dates on every fact, both sides of a contradiction flagged, no directory badges, and a data file that matches the page.
src/lib/seo/prose.test.ts	10	Frontmatter splitting and its failure modes, and that the markdown-to-HTML pass handles headings, bold, code spans, fenced blocks, safe links, lists, and escaping.
src/lib/engine/csv.test.ts	8	Quoting, escaping, and formula neutralisation for the export.
scripts/lib/indexnow.test.ts	7	The submission shape, the batch rules, and that a rejection is reported rather than thrown.
scripts/lib/notify.test.ts	7	Which catalog changes are worth interrupting somebody for, and which are not.
scripts/lib/page-ledger.test.ts	4	The ledger file itself: where it lives, empty when it does not exist yet, a lossless round trip, and a malformed file refused.
scripts/ping-indexnow.test.ts	3	IndexNow submits only the pages whose lastmod moved (plus the calculator with the model table), a newly published page, and nothing on a morning that changed no price.
src/lib/mobile/promptInput.test.ts	8	Native count/paste behavior, model-specific token estimates, the 200,000-character boundary, empty and multilingual text, comparison ranking, and proof that private prompt text never enters share output.
src/lib/mobile/comparison.test.ts	5	Four-model selection defaults, add/remove behavior, duplicate prevention, maximum enforcement, and safe handling when catalog entries disappear.
src/lib/engine/share.test.ts	4	Readable estimate and comparison sharing, cheapest-model labeling, warnings, freshness context, and mail-safe line breaks.
src/lib/engine/insights.test.ts	3	Deterministic output, history, caching, margin, and price-spread explanations, including empty and single-model scenarios.

| src/lib/seo/css.test.ts | 3 | That the generated-page stylesheet defines every class the renderers emit, styles both colour schemes, and lets wide tables scroll inside their own container. |

Six more packages, each with its own runtime, lockfile and CI job, plus the browser run:

Package	Tests	Guards
worker/	102	The alerts API inside workerd against a real D1 — including the RFC 8291 push crypto, checked byte-for-byte against the worked example in the RFC itself.
apps/mobile/	230	The native launch product: all five tab outcomes plus PromptSpend Receipt, private prompt conversion, pricing-cache resilience, state recovery, sharing privacy, accessible country filtering, promotional-rate display, alert requests, source-anchored guided-tour focus, searchable Help & FAQs, startup routing, and the Turnstile message boundary.
api/	49	The public pricing API: filters, ETags, CORS, CSV quoting, that a catalog failing validation is refused rather than passed through, and that an origin answering with HTML is reported as such rather than as a JSON syntax error.
mcp/	48	That no tool can return a price without provenance, that estimate_cost agrees with the site's own engine, that get_price quotes the rate actually in force and cannot drift from what estimate_cost bills, that the server reports the version it was published as, and that the real fetch path names what arrived when the catalog is not JSON.
updates/	9	The self-hosted over-the-air update server inside workerd against a local R2: protocol headers, multipart framing, a manifest served byte-for-byte with a signature that verifies, rollback directives, per-platform and per-version isolation, and immutable content-addressed assets.
vscode/	137	The editor extension: which lines in which languages name a model, hover, status-bar and inline rendering, that every surface quotes the rate in force rather than the standard one during a promotional window, the catalog fetch when the server is unreachable or answers malformed, selection estimates that name an impossible request rather than pricing it, and diagnostics that stay quiet by default.
Browser	228	Layout and accessibility at 320/390/768/1280 in real Chromium — overflow, touch targets, text size, table scrolling, axe at WCAG 2.1 A/AA, plus every quote opened on a free-tier page, the main nav scrolling rather than squashing on a phone, the privacy promises (no cookie, only the five interface keys in storage, no IndexedDB, requests only to this site and the LaunchNest badge, a private token removed from the address), the Receipt copy, outage, machine-contract, share-import, and local PNG-export flows. See below.

Layout is checked in a real browser

npm run test:e2e runs Playwright against the built site at four viewports — 320, 390, 768 and 1280 — with touch emulation on the first three, because the 44px target rules live behind @media (pointer: coarse) and without hasTouch the run checks the desktop stylesheet on a narrow screen.

It exists because the unit suite runs in jsdom, which has no layout engine at all. It cannot see a table pushing the page sideways at 320px, or a control too small for a thumb — and both of those shipped.

Check	Catches
Horizontal overflow	Anything widening the document, unless it sits in a container that scrolls itself
Touch targets	Controls under 44x44, measured as the union of the control and its label
Text size	Anything rendering below 12px
Table scrolling	A wide table scrolling the page instead of its own container
Accessibility	axe at WCAG 2.1 A/AA: landmarks, roles, accessible names, heading order

Each returns the offending elements, not a boolean: "the page has horizontal scroll" is not a bug report, "table.catalog is 512px wide in a 320px viewport" is.

It runs against the built site rather than the dev server because the 159 generated pages only exist after build:pages, and three tests load a model page, a free-tier page and the apps page with JavaScript disabled — those pages ship none, and that is the point.

On its first honest run it found four touch targets under 44px, three of them caused by the same trap: a more specific selector elsewhere in the stylesheet beating the rule meant to cover it, since a media query adds no specificity.

Coverage thresholds are uneven on purpose

Enforced in CI, and the numbers are set to today's level so they can only go up:

Scope	Lines	Why
src/lib/engine/**	90%	A silent arithmetic error is the most expensive bug this project can ship.
scripts/lib/**	90%	This is what decides whether third-party data reaches users at all.
Everything else	70%	Components and state, where a failure is visible on screen rather than hidden inside a number.

Two things the tests are structured around

The pipeline is pure except for the fetch. Normalise, merge, validate and diff are all pure functions, which is why pipeline.test.ts can exercise the entire thing against fixtures with no network and no clock. Fixtures live in the test file, not on disk.

Every audit finding has a test named after it. The regression tests in App.test.tsx are grouped as "honesty of the default estimate", "sharing a pasted prompt", "keyboard and screen-reader access" and "claims the site makes about itself". If one of them fails, something that was true has stopped being true — that is more useful than a test called renders correctly.

Two of those are worth calling out, because they were live failures before they were tests:

• A permitted URL used to blank the page. costPico threw during render from parameters the decoder itself accepted. The test drives the exact URL and asserts a cost card renders.
• Sharing after pasting used to lie. The URL carried the old slider values while the screen showed the pasted counts. The test pastes, then asserts the URL matches the screen and contains no fragment of the prompt.

What the suite does not cover

The component tests run in jsdom, which has no layout engine. It cannot measure an element, so it cannot tell you whether the page scrolls sideways on a phone or whether a button is large enough to press.

Those particular guarantees are now held by the Playwright suite above, which is why layout regressions get caught rather than shipped. Accessibility is held by tests/e2e/a11y.spec.ts — axe at WCAG 2.1 A and AA, on every viewport, over the four app views and four generated pages.

Visual regression is the gap that remains. Nothing diffs screenshots, so a change that leaves the page accessible, correctly sized and semantically valid while making it ugly — a colour that no longer matches, a broken alignment, an overlapping label — will not be caught by anything here. Treat a purely visual change as something to look at.

CI

.github/workflows/verify.yml runs on Node 22 on ubuntu-latest, called by ci.yml on every push and pull request and by deploy.yml before it publishes. The deploy job downloads the artifact the verify job built, rather than building its own, so what ships is byte-for-byte what passed.

It carries six jobs — verify, worker, e2e, api, mcp and vscode — one per package with a suite, each installing only what it needs. vscode is the exception that installs twice: the extension type-checks against files in the root package through its @/* mapping, so the site's dependencies have to be on disk before the extension's own can be.

Three of the extension's gates do something the tests cannot, and each was written after a fault the tests had passed:

Gate	What it catches
check:activation	Drives a full refresh over a real document and fails if nothing is drawn. Every refresh used to be fired as void refresh(…), so an exception left the screen unchanged and said nothing.
check:footprint	A 100 KB bundle budget, and it resolves each declared external. Inlining the tokenizer took the extension from 19.7 KB to 3,368 KB; a missing external is worse, because exact counting then silently degrades to an estimate.
check:package	Stages the real vsce ls file set into a temp directory and runs the encoder from there. It is the only check that sees packaging, module resolution and the tokenizer at once — which is what shipped broken.

ci.yml also triggers on pushes to pricing-sync/**, and the reason is not obvious. GitHub does not trigger workflows for events created with GITHUB_TOKEN, so the review pull requests the sync opens arrive with only the checks that run on their own schedule. Catalog validation — the check that matters most on a pricing change — was the one not running.

sync-pricing.yml is separate and runs on a cron. It exercises the pipeline against the live sources every morning, which is a useful early warning that an upstream feed has changed shape — a degraded run fails loudly rather than publishing. notify-alerts.yml fires on a push to main that changes the published catalog.

freshness.yml is the odd one out: it tests nothing in this repository. It fetches the catalog from the live site daily and raises an issue if it is more than two days old. Every other job here asks whether the code is correct; that one asks whether the promise is being kept, which no amount of passing tests can answer.

Line endings

.gitattributes normalises everything to LF. Without it, a Windows clone with the default core.autocrlf gets CRLF throughout and format:check fails on every file locally while passing in CI — a gate that only fails on a contributor's machine is worse than no gate.