Create Eval Skill
Create plugin development eval scenarios (JSON files with natural prompts and deterministic checks for testing plugin skills). NOT for Copilot Studio in-product evaluation — use /copilot-studio:create-eval-set for that. Published by microsoft in skills-for-copilot-studio.
What is Create Eval Skill?
Create plugin development eval scenarios (JSON files with natural prompts and deterministic checks for testing plugin skills). NOT for Copilot Studio in-product evaluation — use /copilot-studio:create-eval-set for that. Published by microsoft in skills-for-copilot-studio. This profile combines repository metadata with install, compatibility, and usage signals so developers can quickly decide whether it fits their agent workflow before opening the source repository.
Automated repository signals based on public metadata such as recency, license, installation evidence, and adoption. These are not a security audit or endorsement.
Key capabilities
- Includes SKILL.md support
- Reusable instructions support
- Testing
- Testing use cases
Technical details
- Install or run with Copy skill directory
When to use Create Eval Skill
- Use it for testing.
Built with
Editorial notes
Source
- Creator: microsoft
- Repository: microsoft/skills-for-copilot-studio
- Skill file: skills/create-eval/SKILL.md
What it does
Create plugin development eval scenarios (JSON files with natural prompts and deterministic checks for testing plugin skills). NOT for Copilot Studio in-product evaluation — use /copilot-studio:create-eval-set for that.
Skill instructions
Create Scenario Eval Guide the user through creating eval test cases for a Copilot Studio plugin scenario. Evals test end-to-end scenarios with natural prompts — the request routes through sub-agents (e.g., Author agent) which invoke skills internally. How the eval system works The eval harness (evals/evaluate.py) works by: 1. Copying a fixture agent into a temp workspace 2. Running claude -p "<prompt" with a PreToolUse hook that traces skill invocations inside sub-agents 3. Checking routing (which agents and skills were invoked), output files, and response text against deterministic checks 4. Producing a JSON results file and HTML report What can be tested right now Authoring scenarios that produce YAML files (topics, agents, knowledge sources, etc.) are the best candidates. The harness supports these check types: | Check | What it validates | Use for | |-------|------------------|---------| | agentinvoked | Expected sub-agent was dispatched (e.g., Author agent) | Routing verification
Explore related resources
Frequently asked questions
What is Create Eval?
Create Eval is a open-source AI agent skill with Copy skill directory. Create plugin development eval scenarios (JSON files with natural prompts and deterministic checks for testing plugin skills).
Who is Create Eval best for?
Create Eval is best for reusing agent instructions, scripts, and references, testing workflows.
How do I install Create Eval?
Install or run Create Eval using Copy skill directory. Check Create Eval for the latest setup command.
Is Create Eval actively maintained?
Create Eval may need a closer maintenance check before production use.
Auto-fetched from GitHub.
