Skills for AI Evals to compliment the course: AI Evals For Engineers & PMs
https://github.com/hamelsmu/evals-skills.git
Skills that guide AI coding agents to help you build LLM evaluations.
These skills guard against common mistakes I've seen helping 50+ companies and teaching students in our AI Evals course. If you're new to evals, see questions.md for free resources on the fundamentals.
If you are new to evals, start with the eval-audit skill. Give your coding agent these instructions:
Install the eval skills plugin from https://github.com/hamelsmu/evals-skills, then run /evals-skills:eval-audit on my eval pipeline. Investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. Use other skills in the plugin as recommended by the audit.
The audit isn't a complete solution, but it will catch common problems we've seen in evals. It will also recommend other skills to use to fix the problems.
In Claude Code, run these two commands:
# Step 1: Register the plugin repository
/plugin marketplace add hamelsmu/evals-skills
# Step 2: Install the plugin
/plugin install evals-skills@hamelsmu-evals-skillsTo upgrade:
/plugin update evals-skills@hamelsmu-evals-skillsAfter installation, restart Claude Code. The skills will appear as /evals-skills:<skill-name>.
If you use the open Skills CLI, install from this repo with:
npx skills add https://github.com/hamelsmu/evals-skillsInstall one skill only:
npx skills add https://github.com/hamelsmu/evals-skills --skill eval-auditCheck for updates:
npx skills check
npx skills update| Skill | What it does |
|---|---|
| eval-audit | Audit an eval pipeline and surface problems with prioritized severity |
| error-analysis | Guide the user through reading traces and categorizing failures |
| generate-synthetic-data | Create diverse synthetic test inputs using dimension-based tuple generation |
| write-judge-prompt | Design LLM-as-Judge evaluators for subjective quality criteria |
| validate-evaluator | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |
| evaluate-rag | Evaluate retrieval and generation quality in RAG pipelines |
| build-review-interface | Build custom annotation interfaces for human trace review |
/evals-skills:skill-name, e.g., /evals-skills:error-analysis.
These skills are a starting point and only encode common mistakes that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them. Start here, then write your own.
The meta-skill can help you ground custom skills.
These skills handle the parts of eval work that generalize across projects. Much of the process doesn't: production monitoring, CI/CD integration, data analysis, and much more. The course covers all of it.