Evaluate a Skill

Evaluate two things

Skill evaluation has two layers: whether the agent loads the Skill, and whether the loaded Skill improves execution.

Build trigger queries

Create two sets of realistic user requests: should-trigger examples and should-not-trigger examples. Keep a validation set separate so description edits do not overfit one small list.

should-trigger:
- Clean this CSV and report invalid rows.
- Normalize these spreadsheet columns for import.
- Validate this customer export against our schema.

should-not-trigger:
- Explain what CSV means.
- Write a product launch email.
- Summarize this meeting transcript.

Run isolated tests

Each evaluation run should start with a clean context. If previous attempts or hidden instructions remain in memory, you cannot tell whether the Skill itself caused the outcome.

• Use realistic files and prompts, not only toy cases.
• Record whether SKILL.md was loaded.
• Capture the final output, errors, timing, and tool behavior.
• Compare results across multiple runs when the task is non-deterministic.

Write assertions

Assertions make evaluation repeatable. Use deterministic checks where possible, then add human review for judgment-heavy outputs.

Copyable evaluator idea

For structured output, a tiny script can catch obvious failures before human review.

// scripts/check-release-notes.js
import fs from "node:fs"

const file = process.argv[2]
const text = fs.readFileSync(file, "utf8")
const required = ["### Highlights", "### Fixes", "### Docs", "### Open questions"]
const missing = required.filter((heading) => !text.includes(heading))

const forbidden = ["JIRA-", "internal refactor", "billing worker"]
const leaked = forbidden.filter((term) => text.toLowerCase().includes(term.toLowerCase()))

console.log(JSON.stringify({
  ok: missing.length === 0 && leaked.length === 0,
  missing,
  leaked
}, null, 2))

process.exit(missing.length === 0 && leaked.length === 0 ? 0 : 1)

Simple scorecard

Use a small table so different people can review the same Skill consistently.

Iterate from failures

When runs fail, inspect whether the issue came from discovery, instructions, references, scripts, or output expectations. Then make the smallest useful change and rerun the same evaluation set.