emisar MCP Server

Client configuration

name: MCP Conformance Eval

# Weekly real-agent conformance eval. Boots the local docker compose stack
# (portal + seeded fixture runners), then drives REAL headless coding agents —
# Claude Code and Codex — through tools/cmd/mcpeval's policy-enforcing MCP relay
# and fails on hard API-conformance violations: policy-blocked calls, portal
# invalid_args rejections, run_action without a prior get_action for the same
# action, the same failing call repeated more than twice, runs left non-terminal,
# or required tools/actions that never succeeded. The model's prose is captured
# in the report artifact but never scored, so the signal survives model drift.
#
# Both providers run on every schedule tick, each gated only on its own secret
# (ANTHROPIC_API_KEY, OPENAI_API_KEY); a lane whose secret is absent is skipped,
# and the job fails if neither is configured. ChatGPT/Codex is a first-class
# lane, not an afterthought — the trace that motivated this eval was a ChatGPT
# trace.
#
# Scheduled + manual only — never PR-triggered: it spends real provider tokens
# and reads provider secrets, which pull-request code must not reach. The relay
# holds the (well-known, dev-only) Emisar key; the agent process never sees it.
# No deploy permissions, no production URLs.
on:
  schedule:
    # Weekly. Offset from :00/:30 — GitHub delays top-of-hour scheduled runs.
    - cron: "27 5 * * 1"
  workflow_dispatch:
    inputs:
      claude_model:
        description: Claude model to evaluate (empty = mcpeval's pinned default)
        required: false
        type: string
      codex_model:
        description: Codex model to evaluate (empty = the CLI's configured default)
        required: false
        type: string

# One serialized lane: a manual dispatch never races the schedule tick into
# dou

Compatible clients