vally-eval Skill
Author, validate, and run Vally eval.yaml evaluation suites for agent skills. TRIGGERS: create eval, write eval, add eval, run eval, validate eval, vally eval, eval.yaml, add stimulus, map test to eval, migrate test to eval, eval graders, eval scoring, add eval to CI. Published by microsoft in GitHub-Copilot-for-Azure.
What is vally-eval Skill?
Author, validate, and run Vally eval.yaml evaluation suites for agent skills. TRIGGERS: create eval, write eval, add eval, run eval, validate eval, vally eval, eval.yaml, add stimulus, map test to eval, migrate test to eval, eval graders, eval scoring, add eval to CI. Published by microsoft in GitHub-Copilot-for-Azure. This profile combines repository metadata with install, compatibility, and usage signals so developers can quickly decide whether it fits their agent workflow before opening the source repository.
Automated repository signals based on public metadata such as recency, license, installation evidence, and adoption. These are not a security audit or endorsement.
Key capabilities
- Includes SKILL.md support
- Reusable instructions support
- Testing
- Documentation
- Writing
- Testing use cases
- Documentation use cases
Technical details
- Install or run with Copy skill directory
When to use vally-eval Skill
- Use it for testing.
- Use it for documentation.
- Use it for writing.
Built with
Editorial notes
Source
- Creator: microsoft
- Repository: microsoft/GitHub-Copilot-for-Azure
- Skill file: .github/skills/vally-eval/SKILL.md
What it does
Author, validate, and run Vally eval.yaml evaluation suites for agent skills. TRIGGERS: create eval, write eval, add eval, run eval, validate eval, vally eval, eval.yaml, add stimulus, map test to eval, migrate test to eval, eval graders, eval scoring, add eval to CI.
Skill instructions
Vally eval suites Skills in the azure-skills plugin are required to have integration tests that run prompts against an LLM agent to evaluate whether they help the agent accomplish goals in target scenarios. Such integration tests are written as vally eval suites, using vally as the underlying tool for running tests and grading the agent outcome. Write vally eval suites Vally eval suites are written as yaml documents. All eval suites share eval spec. Refer to the official documentation on the schema of the spec and the schema of the eval suites writing-eval-specs. Vally eval suites for azure-skills plugin have the following file layout. The shared eval spec is located at <repo-root/.vally.yaml. The eval suites are categorized by skills. The eval suites for each skill are located at <repo-root/evals/<skill-name/eval.yaml, e.g. <repo-root/evals/azure-ai/eval.yaml. If a skill needs fixture files for its eval suites, it should organize such fixture files in a fixture directory under its dir
Explore related resources
Frequently asked questions
What is vally-eval?
vally-eval is a open-source AI agent skill with Copy skill directory. Author, validate, and run Vally eval.yaml evaluation suites for agent skills. TRIGGERS: create eval, write eval, add eval, run eval, validate eval, vally eval, eval.yaml, add stimulus, map test to eval.
Who is vally-eval best for?
vally-eval is best for reusing agent instructions, scripts, and references, testing workflows, documentation workflows, writing workflows.
How do I install vally-eval?
Install or run vally-eval using Copy skill directory. Check vally-eval for the latest setup command.
Is vally-eval actively maintained?
vally-eval may need a closer maintenance check before production use.
Auto-fetched from GitHub.
