Free learning at The AI Vault
Build your first eval with a complete AI Evals mini-course and its free practice skill. No email or account.
Free mini-courseBeginnerComplete course · 16:47
Follow a fictional scheduling assistant’s double booking. Catch the failure with code checks, a clear rubric and an eval suite.
The whole course. No email. No account.
The video couldn’t load. Open the free video directly.
Follow the failure. Build the check.16:47 · English captions
Inside the mini-course
Catch a confident claim with saved-state checks, repair the schedule, and rerun every case. Take the mini-course from watching to doing.
YOUR FIRST MISSION
Find a double booking and a false coverage report in the offline practice cases. Repair the saved schedule, then check what improved and what still needs human review.
You’ll make: FIRST_WIN.md · EVAL_REPORT.md · eval_results.json · RUBRIC.md
Included with this free mini-course. No account, API key or paid tool required. Runs offline with Python 3.9+.
~/.codex/skills/ or ~/.claude/skills/.vivid-ai-evals-mini-course for your first eval.You can also run the helper directly from its folder:
python3 scripts/vivid.py firstwin /path/to/projectOriginal fictional cases, deterministic checks and a human-review rubric. The tool does not call a model or execute your project code. It writes practice reports inside your chosen project’s .vivid/vivid-ai-evals-mini-course/ folder. Prepared replays teach the method; they do not establish production reliability. Files you ask an AI agent to read may go to its model provider.
A claim is not proof. A fluent message can say that every station is covered while the saved schedule contains overlapping bookings. Evaluate the state and the trace, not just the assistant’s wording.
Match the grader to the question. Code can check availability, qualifications and consent rules. A clear rubric helps people and model judges assess whether a message explains a gap and offers a safe next action.
Rerun after every repair. Check the cases that improved and the cases that regressed. Separate development cases from held-back checks, and report counts and failure types.
The film uses prepared teaching examples and real code checks. Its model-judge verdicts and reviewer labels are illustrations, not live model measurements. This complete, standalone free mini-course is separate from the paid course catalog.
Keep the momentum
Explore 28 focused courses in agents, evaluation, retrieval and more. Each includes an installable practice skill.