Free learning at The AI Vault

Learn something real. For free.

Build your first eval with a complete AI Evals mini-course and its free practice skill. No email or account.

Free mini-courseBeginnerComplete course · 16:47

AI Evals:
Proof, Not Guesswork.

Follow a fictional scheduling assistant’s double booking. Catch the failure with code checks, a clear rubric and an eval suite.

The whole course. No email. No account.

Follow the failure. Build the check.16:47 · English captions

What You’ll Learn
  • Claim vs. state
  • Representative eval cases
  • Code checks for hard rules
  • Rubrics for judgment
What You’ll Build

Your Evals First Win project

Inside the mini-course

YOUR MINI-COURSE, IN ACTIONFree companion

Your first eval. Your first fix.

Catch a confident claim with saved-state checks, repair the schedule, and rerun every case. Take the mini-course from watching to doing.

YOUR FIRST MISSION

Find a double booking and a false coverage report in the offline practice cases. Repair the saved schedule, then check what improved and what still needs human review.

You’ll make: FIRST_WIN.md · EVAL_REPORT.md · eval_results.json · RUBRIC.md

Download the free Evals skill ↓

ZIP · 12 KB · Claude Code + Codex

Included with this free mini-course. No account, API key or paid tool required. Runs offline with Python 3.9+.

Install it in 3 steps
  1. Download the ZIP above.
  2. Extract its folder into ~/.codex/skills/ or ~/.claude/skills/.
  3. Start a new agent session and ask it to use vivid-ai-evals-mini-course for your first eval.

You can also run the helper directly from its folder:

python3 scripts/vivid.py firstwin /path/to/project
What the practice does

Original fictional cases, deterministic checks and a human-review rubric. The tool does not call a model or execute your project code. It writes practice reports inside your chosen project’s .vivid/vivid-ai-evals-mini-course/ folder. Prepared replays teach the method; they do not establish production reliability. Files you ask an AI agent to read may go to its model provider.

Read the key ideas

A claim is not proof. A fluent message can say that every station is covered while the saved schedule contains overlapping bookings. Evaluate the state and the trace, not just the assistant’s wording.

Match the grader to the question. Code can check availability, qualifications and consent rules. A clear rubric helps people and model judges assess whether a message explains a gap and offers a safe next action.

Rerun after every repair. Check the cases that improved and the cases that regressed. Separate development cases from held-back checks, and report counts and failure types.

The film uses prepared teaching examples and real code checks. Its model-judge verdicts and reviewer labels are illustrations, not live model measurements. This complete, standalone free mini-course is separate from the paid course catalog.

Keep the momentum

Build it. Test it. Keep going.

Explore 28 focused courses in agents, evaluation, retrieval and more. Each includes an installable practice skill.

Explore the full Vault →