Learn harness engineering by building Claude Code from scratch. Free, open-source course: 8 articles, 6 videos, 1 codebase.
A test runner for agentskills.io-style AI agent skills
Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs
Turn feature specs into merged PRs with a self-supervising swarm of coding agents — parallel execution, isolated sandboxes, DAG dependencies. Open-source, self-hostable, model-agnostic (Claude / Gemini / Codex).
AI Engineer interview prep for 2026: 267 real questions, 10 worked system designs, 12 company guides (OpenAI, Anthropic, DeepMind and more). Free, MIT. By Landed.
Replay real agent traces through cheaper models to prove which swaps are safe.
Visualize LLM outputs against datasets, manually annotate results, and run automated evaluations to algorithmically optimize prompts.
Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations
Crash-test insurance claim AI agents before production.
Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai
The comprehensive list of AI evaluation tools: 300+ open-source and commercial LLM evaluation frameworks, platforms, benchmarks, observability, red teaming, and guardrails — for LLMs, RAG, and AI agents.
An implementation of the Anthropic's paper and essay on "A statistical approach to model evaluations"
An eval-gated AI development pipeline for production codebases, shipped as a Claude Code plugin. Spec, plan, build, gate, review, retro. Built and measured on a 30-module Android app.
Stop guessing confidence thresholds: calibrate, threshold, and drift-check typed decision models (TypeSafe Jev) against an LLM teacher.
7 Claude Code skills for software architecture review (Python, web, cloud, microservices). Includes A/B benchmarks against unskilled baseline, assertion-graded eval suite, and interactive dashboards.
Async-first Python testing framework with explicit dependency injection and native LLM evals. Tests and evals run as coroutines on one event loop.
TypeScript AI agent SDK: offline CI evals, durable runs, human approvals, MCP. Runs in your own Node process.
Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.
Niche high-signal skills for professional Codex agents
Open-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.
To associate your repository with the llm-evals topic, visit your repo's landing page and select "manage topics."