FreeSkills
Commits, branches and sync with the but CLI
- 361361 downloads
- 198198 likes
- 103103 saves
Works with
- Claude Code
- Codex
- Cursor
- and 3 more
Calibrating an LLM judge against human labels v1.0.0
Use this resource to calibrate an LLM judge against human labels and measure its agreement
New
- 335 downloads
- 165 likes
- 203 saves
Free
Install with the CLI
In a terminal, from the folder you want:
npx --yes palatekit@latest install llm-judge-calibration-validationWith no option, the skill goes to ~/.agents/skills/; add --agent claude-code (or codex, cursor…) to target an agent.
First time: connect the CLI with
npx --yes palatekit@latest loginThe agent splits your labeled data into train, dev and test sets, measures the judge's true positive and true negative rates, inspects disagreements and iterates. It does not apply to deterministic code evaluators.
Mostly picked by
Updated November 2, 2026
Previews
Details
- Install guides
- Generates with
- Your agent's model
- Licence
- Apache-2.0
Similar resources
View all- Free
FreeSkills
Sentry CLI guide
- 525525 downloads
- 117117 likes
- 128128 saves
Works with- Claude Code
- Codex
- Cursor
- and 3 more
FreeSkills
Drizzle ORM best practices with PostgreSQL
- 407407 downloads
- 194194 likes
- 151151 saves
Works with- Claude Code
- Codex
- Cursor
- and 3 more
FreeSkills
SLO design wizard
- 537537 downloads
- 105105 likes
- 242242 saves
Works with- Claude Code
- Codex
- Cursor
- and 3 more
FreeSkills
User stories and sprint fit
- 451451 downloads
- 150150 likes
- 132132 saves
Works with- Claude Code
- Codex
- Cursor
- and 3 more
FreeSkills
Capacity-based sprint plan
- 423423 downloads
- 192192 likes
- 102102 saves
Works with- Claude Code
- Codex
- Cursor
- and 3 more
FreeSkills
Startup CTO advisor
- 580580 downloads
- 201201 likes
- 242242 saves
Works with- Claude Code
- Codex
- Cursor
- and 3 more
FreeSkills
Creating GitHub pull requests with quality checks
- 427427 downloads
- 230230 likes
- 196196 saves
Works with- Claude Code
- Codex
- Cursor
- and 3 more
What you get
- SKILL.md: data splits, TPR and TNR measurement, disagreement analysis
- SKILL.md: iteration, final test run and bias correction
Good fit for
- You must show that an LLM judge agrees with your reviewers
- Your judge returns results you can't interpret
Not for
Tags
- llm judge
- calibration
- tpr
- tnr
About
You say
“Calibrate this LLM judge on my 100 labeled traces and tell me where it goes wrong.”
Skill published by ai-evals-course under the Apache-2.0 license, delivered in its original version.
Show moreShow less
How to use it
- Install the skill in your agent's skills folder
- Ask your agent to calibrate an LLM judge against human labels and measure its agreement
- Review and adjust the result
Content preview
Files
- agents/openai.yaml0.3 KB
- LICENSE11.3 KB
- NOTICE-PALATEKIT.md0.5 KB
- SKILL.md8.9 KB
Origin and licence
Listed unmodified
- Author
- ai-evals-course
- Source
- github.com/ai-evals-course/evals-skills/tree/80d5f7b0127c7572ed9e9339937adbfd7240ffeb/skills/validate-evaluator
- Licence
- Apache-2.0
Notice
« validate-evaluator » (ai-evals-course, Apache-2.0) livré dans sa version d'origine : fiche en français et en anglais par PalateKit.
User reviews
–
No ratings yet
Loading reviews…