benchmark-qed-autoe Skill
Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics, or statistical significance tests on RAG outputs. Also use when the user wants to score, compare, or evaluate RAG method Published by microsoft in benchmark-qed.
What is benchmark-qed-autoe Skill?
Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics, or statistical significance tests on RAG outputs. Also use when the user wants to score, compare, or evaluate RAG method Published by microsoft in benchmark-qed. This profile combines repository metadata with install, compatibility, and usage signals so developers can quickly decide whether it fits their agent workflow before opening the source repository.
Automated repository signals based on public metadata such as recency, license, installation evidence, and adoption. These are not a security audit or endorsement.
Key capabilities
- Includes SKILL.md support
- Reusable instructions support
- Testing
- Testing use cases
Technical details
- Install or run with Copy skill directory
When to use benchmark-qed-autoe Skill
- Use it for testing.
Built with
Editorial notes
Source
- Creator: microsoft
- Repository: microsoft/benchmark-qed
- Skill file: .apm/skills/benchmark-qed-autoe/SKILL.md
What it does
Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics, or statistical significance tests on RAG outputs. Also use when the user wants to score, compare, or evaluate RAG method
Skill instructions
Benchmark-QED Evaluation (autoe) Evaluate and compare RAG system outputs using LLM-judged scoring, assertion-based evaluation, and retrieval metrics — all with built-in statistical significance testing. Prerequisites - Generated questions/assertions from the autoq pipeline (or your own) - RAG method answer files (JSON, one per method per question set) - A valid settings.yaml for the evaluation type - A configured workspace with valid settings.yaml (use the benchmark-qed-setup skill to initialize and configure) - LLM API key configured Run all commands with: bash uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed <command Evaluation Methods Overview | Method | Command | Best for | |--------|---------|----------| | Pairwise comparison | autoe pairwise-scores | Comparing two RAG methods head-to-head | | Reference scoring | autoe reference-scores | Scoring against gold-standard answers | | Assertion scoring | autoe assertion-scores | Evaluating with ground-truth asse
Explore related resources
Frequently asked questions
What is benchmark-qed-autoe?
benchmark-qed-autoe is a open-source AI agent skill with Copy skill directory. Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics.
Who is benchmark-qed-autoe best for?
benchmark-qed-autoe is best for reusing agent instructions, scripts, and references, testing workflows.
How do I install benchmark-qed-autoe?
Install or run benchmark-qed-autoe using Copy skill directory. Check benchmark-qed-autoe for the latest setup command.
Is benchmark-qed-autoe actively maintained?
benchmark-qed-autoe may need a closer maintenance check before production use.
Auto-fetched from GitHub.
