benchmark-qed-autoe Skill
Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics, or statistical significance tests on RAG outputs. Also use when the user wants to score, compare, or evaluate RAG method Published by microsoft in benchmark-qed.
Decision snapshot
Is this a fit?
Testing, Includes SKILL.md, Reusable instructions
Compatibility not yet detected.
API key required authentication, Credentials required
Copy skill directory
2 months ago · MIT license
Access note: API key required.
What is benchmark-qed-autoe Skill?
Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics, or statistical significance tests on RAG outputs. Also use when the user wants to score, compare, or evaluate RAG method Published by microsoft in benchmark-qed. This profile combines repository metadata with install, compatibility, and usage signals so developers can quickly decide whether it fits their agent workflow before opening the source repository.
Automated repository signals based on public metadata such as recency, license, installation evidence, and adoption. These are not a security audit or endorsement. See how SkillIndex evaluates profiles.
Key capabilities
- Includes SKILL.md support
- Reusable instructions support
- Testing
- Testing use cases
Declared skill metadata
- Source file: .apm/skills/benchmark-qed-autoe/SKILL.md
These fields retain source and confidence evidence from the indexed SKILL.md.
Compatibility and setup
- Install or run with Copy skill directory
- API key required
Requirements and access
Security and permissions
Review permissions before connecting any MCP server to an agent. Pay special attention to whether it can read local files, write data, call external services, or perform destructive actions.
When to use benchmark-qed-autoe Skill
- Use it for testing.
Built with
Editorial notes
Source
- Creator: microsoft
- Repository: microsoft/benchmark-qed
- Skill file: .apm/skills/benchmark-qed-autoe/SKILL.md
What it does
Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics, or statistical significance tests on RAG outputs. Also use when the user wants to score, compare, or evaluate RAG method
Skill instructions
Benchmark-QED Evaluation (autoe) Evaluate and compare RAG system outputs using LLM-judged scoring, assertion-based evaluation, and retrieval metrics — all with built-in statistical significance testing. Prerequisites - Generated questions/assertions from the autoq pipeline (or your own) - RAG method answer files (JSON, one per method per question set) - A valid settings.yaml for the evaluation type - A configured workspace with valid settings.yaml (use the benchmark-qed-setup skill to initialize and configure) - LLM API key configured Run all commands with: bash uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed <command Evaluation Methods Overview | Method | Command | Best for | |--------|---------|----------| | Pairwise comparison | autoe pairwise-scores | Comparing two RAG methods head-to-head | | Reference scoring | autoe reference-scores | Scoring against gold-standard answers | | Assertion scoring | autoe assertion-scores | Evaluating with ground-truth asse
Verified compatibility and discovery
Frequently asked questions
What is benchmark-qed-autoe?
benchmark-qed-autoe is a open-source AI agent skill with Copy skill directory. Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics.
Who is benchmark-qed-autoe best for?
benchmark-qed-autoe is best for reusing agent instructions, scripts, and references, testing workflows.
How do I install benchmark-qed-autoe?
Install or run benchmark-qed-autoe using Copy skill directory. Check benchmark-qed-autoe for the latest setup command.
Is benchmark-qed-autoe actively maintained?
benchmark-qed-autoe may need a closer maintenance check before production use.
Project health auto-fetched from the source repository.
Maintain this resource?
Review this source-backed profile, send a correction with evidence, or link to it from your documentation. Claims verify your relationship to the project; profile facts still require source evidence and editorial review.
