Every premium and members-only resource for $9.99/month, as a no-commitment subscription. See the plans

PalateKit

Sign in
Menu

Request a resource

Describe the resource you are missing in one sentence. We will email you when it is published.

0 / 280

A page, a video or an image online.

Unlock PalateKit

Unlock everything

Plan

Paying means accepting our terms of sale.

Secure payment by Stripe. Receipts and invoices sent by email.

Welcome back

Sign in to install, download or save this resource. A free account is enough for free resources.

Create evaluators from real traces v1.0.0

Use this resource to create LLM evaluators from production traces

New

  • 385 downloads
  • 227 likes
  • 247 saves

Free

Install with the CLI

In a terminal, from the folder you want:

npx --yes palatekit@latest install bootstrap-llm-evaluators-traces

With no option, the skill goes to ~/.agents/skills/; add --agent claude-code (or codex, cursor…) to target an agent.

First time: connect the CLI with

npx --yes palatekit@latest login

The agent reads the production traces of an LLM application and proposes LLM-judge evaluators. After your approval, it creates them in Datadog as disabled drafts, never auto-enabled. On request, it outputs BaseEvaluator Python code or a framework-agnostic JSON spec.

Mostly picked by

Updated November 2, 2026

Card for the “Create evaluators from real traces” skill

Details

Works in
Install guides
Generates with
Your agent's model
Licence
MIT

Similar resources

View all

What you get

  • SKILL.md: inputs, MCP or pup mode and available tools
  • SKILL.md: patterns for reading span content
  • SKILL.md: evaluator SDK and LLM-judge reference

Good fit for

  • You want to evaluate your assistant's production answers automatically
  • You must write an evaluation suite from real cases

Not for

    Tags

    • evaluators
    • LLM
    • traces
    • evaluation

    About

    You say

    “Create evaluators for our support agent from this week's traces.”

    Card for the “Create evaluators from real traces” skill

    Skill published by datadog-labs under the MIT license, delivered in its original version.

    Show more

    How to use it

    1. Install the skill in your agent's skills folder
    2. Ask your agent to create LLM evaluators from production traces
    3. Review and adjust the result

    Content preview

    Files

    • LICENSE1.1 KB
    • NOTICE-PALATEKIT.md0.5 KB
    • SKILL.md98.1 KB

    Origin and licence

    Listed unmodified

    Author
    datadog-labs
    Source
    github.com/datadog-labs/agent-skills/tree/ef022309cc6b6296e7574589ac332699b1c9a092/agent-observability/agent-observability-eval-bootstrap
    Licence
    MIT

    Notice

    « agent-observability-eval-bootstrap » (datadog-labs, MIT) livré dans sa version d'origine : fiche en français et en anglais par PalateKit.

    User reviews

    –

    No ratings yet

    Loading reviews…

    Delete your review?

    Your rating and comment will be removed from the page.