Evals
Reference
Generated source hash: sha256:1d67c9afcbf574e519628d03a5e1a5822a2f7a576aed4b1aa7fbacaebc3257f7.
Audience
Section titled “Audience”- Test authors adding or reviewing eval fixtures.
- Docs authors proving behavior claims against harness checks.
- Reviewers checking unsupported or stale expectation keys.
What is this?
Section titled “What is this?”This page is generated from docs/evals/scenario-contract.md and fixtures/evals/. Evals are behavior contracts for governed MCP paths; they are not user documentation by themselves.
When do I use it?
Section titled “When do I use it?”Use it when a docs claim needs a fixture-backed proof path, or when adding a new expectation key to the harness.
What happens?
Section titled “What happens?”243supported expectation keys are read from the scenario contract.139fixture files are scanned.- Metadata-only fixtures must declare
metadata_only: trueand a non-empty reason, per the scenario contract.
Supported expectation keys
Section titled “Supported expectation keys”| Expectation key | Fixture count |
|---|---|
decision | 5 |
reason | 4 |
visible_tools | 4 |
hidden_tools | 4 |
route_allowed | 1 |
route_reason | 1 |
unregistered_backend_route_allowed | 2 |
unregistered_backend_route_reason | 2 |
unregistered_backend_upstream_attempted | 2 |
api_call_allowed | 4 |
api_call_reason | 4 |
allowed_tool_call_success | 1 |
allowed_tool_call_policy_decision | 1 |
allowed_tool_call_policy_reason | 1 |
allowed_tool_call_response_status | 1 |
denied_tool_call_denied | 1 |
denied_tool_call_policy_decision | 1 |
denied_tool_call_policy_reason | 1 |
denied_tool_call_response_status | 1 |
denied_tool_call_error_code | 1 |
denied_tool_call_upstream_attempted | 1 |
audit_emitted | 1 |
audit_event_type | 1 |
audit_policy_decision | 1 |
audit_credential_mode | 1 |
audit_policy_version | 1 |
denied_audit_emitted | 1 |
denied_audit_event_type | 1 |
denied_audit_policy_decision | 1 |
denied_audit_credential_mode | 1 |
denied_audit_policy_version | 1 |
denied_audit_error_code | 1 |
runtime_contract_complete | 1 |
discovery_response_contract_complete | 1 |
unauthorized_tool_hidden | 1 |
manifest_contract_complete | 1 |
audit_contract_complete | 1 |
audit_secret_free | 3 |
session_terminated | 1 |
backend_sessions_count_max | 1 |
bounded_buffers | 1 |
session_metadata_externalized | 1 |
drain_supported | 1 |
m5_session_lifecycle_contract_complete | 1 |
session_create_contract | 1 |
session_reattach_reconnect_contract | 1 |
session_drain_terminate_revoke_contract | 1 |
session_affinity_index_contract | 1 |
session_lifecycle_audit_secret_free | 1 |
m5_session_scope_boundaries | 1 |
m5_acceptance_matrix_complete | 1 |
mcp50_session_state_machine_durable_metadata | 1 |
mcp50_session_id_boundary_validation | 1 |
mcp50_invalid_transition_denied | 1 |
mcp50_expiry_idle_contract | 1 |
mcp50_session_metadata_secret_free | 1 |
mcp51_active_index_contract | 1 |
mcp51_affinity_hit | 1 |
mcp51_stale_index_fail_closed | 1 |
mcp51_drain_revoke_lookup | 1 |
mcp51_valkey_unavailable_fail_closed | 1 |
mcp51_index_secret_free | 1 |
mcp52_reconnect_gate | 1 |
mcp52_reattach_gate | 1 |
mcp52_drain_no_new_sessions | 1 |
mcp52_drain_complete_timeout | 1 |
mcp52_terminate_fail_closed | 1 |
mcp52_backpressure_contract | 1 |
mcp52_lifecycle_hooks_secret_free | 1 |
mcp53_outbox_fanout_contract | 1 |
mcp53_revocation_scope_coverage | 1 |
mcp53_active_session_termination | 1 |
mcp53_blocked_new_calls | 1 |
mcp53_duplicate_replay_idempotent | 1 |
mcp53_failure_modes_observable_secret_free | 1 |
mcp54_audit_event_sequence | 1 |
mcp54_safe_context_and_correlation | 1 |
mcp54_denied_lifecycle_auditable | 1 |
mcp54_session_audit_secret_free | 1 |
mcp54_contract_alignment | 1 |
outbound_runtime_required | 2 |
telemetry_customer_controlled | 1 |
cached_policy_enforced | 1 |
siem_export_ready | 1 |
openapi_import_valid | 1 |
m6_api_contract_baseline_complete | 1 |
m6_selected_operation_gate | 1 |
m6_host_schema_size_guards | 1 |
m6_credential_binding_contract | 1 |
m6_secret_free_api_adapter_audit | 1 |
mcp64_cli_imports_approved_fixture | 1 |
mcp64_cli_rejects_failures | 1 |
mcp64_cli_evidence_links_contracts | 1 |
mcp64_cli_diagnostics_secret_free | 1 |
m7_critical_workflow_coverage | 1 |
m7_admin_api_surface_complete | 1 |
m7_cli_parity_contract | 1 |
m7_cli_diagnostics_executable | 1 |
m7_safe_deny_no_secret_surfaces | 1 |
m7_no_db_or_custom_script_dependency | 1 |
mcp82_contract_refs_exist | 1 |
mcp82_acceptance_matrix_complete | 1 |
mcp82_sandbox_matrix_complete | 1 |
mcp82_sprite_reuse_checked | 1 |
mcp82_release_evidence_bundle_complete | 1 |
mcp82_artifacts_secret_free | 1 |
mcp83_contract_refs_exist | 1 |
mcp83_schema_cases_complete | 1 |
mcp83_rendered_manifest_requirements_complete | 1 |
mcp83_install_diagnostics_machine_readable | 1 |
mcp83_local_validation_recorded | 1 |
mcp83_v1_scoped_defaults | 1 |
mcp83_artifacts_secret_free | 1 |
mcp84_config_surfaces_complete | 1 |
mcp84_hybrid_modes_customer_controlled | 0 |
mcp84_failure_diagnostics_stable | 1 |
mcp84_no_non_v1_dependencies | 1 |
mcp84_artifacts_secret_free | 1 |
mcp85_contract_refs_exist | 1 |
mcp85_network_controls_complete | 1 |
mcp85_resource_controls_complete | 1 |
mcp85_no_outbound_runtime_observable | 1 |
mcp85_sandbox_proof_listed | 1 |
mcp85_artifacts_secret_free | 1 |
mcp87_contract_refs_exist | 1 |
mcp87_release_bundle_valid | 1 |
mcp87_self_hosted_readiness_linked | 1 |
mcp87_provenance_negative_cases_stable | 1 |
mcp87_scripts_gate_release_fields | 1 |
mcp87_artifacts_secret_free | 1 |
mcp88_contract_refs_exist | 1 |
mcp88_blocker_chain_complete | 1 |
mcp88_sprite_reuse_and_checkpoint_recorded | 1 |
mcp88_sandbox_tracks_complete | 1 |
mcp88_unavailable_capabilities_have_fallbacks | 1 |
mcp88_replay_commands_complete | 1 |
mcp88_artifact_paths_and_hashes_recorded | 1 |
mcp88_artifacts_secret_free | 1 |
mcp92_contract_refs_exist | 1 |
mcp92_release_readiness_matrix_complete | 1 |
mcp92_docs_requirements_matrix_complete | 1 |
mcp92_sandbox_plan_complete | 1 |
mcp92_owner_links_complete | 1 |
mcp92_v1_scope_boundaries | 1 |
mcp92_artifacts_secret_free | 1 |
mcp96_contract_refs_exist | 1 |
mcp96_release_candidate_manifest_valid | 1 |
mcp96_m8_evidence_compatible | 1 |
mcp96_required_evidence_linked | 1 |
mcp96_negative_cases_stable | 1 |
mcp96_harness_command_documented | 1 |
mcp96_artifacts_secret_free | 1 |
mcp93_contract_refs_exist | 1 |
mcp93_reuses_m1_m8_evidence | 1 |
mcp93_smoke_coverage_complete | 1 |
mcp93_fail_closed_matrix_complete | 1 |
mcp93_machine_readable_report | 1 |
mcp93_artifacts_metadata_only_secret_free | 1 |
mcp95_contract_refs_exist | 1 |
mcp95_security_surfaces_complete | 1 |
mcp95_runtime_paths_no_payloads | 1 |
mcp95_release_artifacts_scanned | 1 |
mcp95_reviewer_checklist_complete | 1 |
mcp95_v1_scope_boundaries | 1 |
policy_core_cases_pass | 1 |
explicit_deny_precedence_matrix | 1 |
policy_versioned_decision_contract | 1 |
deny_reason_surface_contract | 1 |
m2_contract_complete | 1 |
oidc_jwt_validation_contract | 1 |
client_surface_restriction_enforced | 1 |
agent_authorization_contract | 1 |
policy_versioning_contract | 1 |
explicit_deny_diagnosis_contract | 1 |
m2_audit_contract | 1 |
revocation_event_contract | 1 |
m2_exit_path_observable | 1 |
agent_registry_contract | 1 |
agent_instance_identity_contract | 1 |
agent_registry_fail_closed | 1 |
policy_simulation_api_contract | 1 |
policy_simulation_api_cases_pass | 1 |
policy_simulation_api_secret_free | 1 |
policy_simulation_v1_scoped | 1 |
revocation_event_model_complete | 1 |
revocation_outbox_internal_queue | 1 |
revoked_new_calls_fail_closed | 1 |
revocation_active_sessions_handled | 1 |
revocation_audit_secret_free | 1 |
m4_contract_complete | 1 |
credential_resolution_success | 1 |
credential_resolution_missing_denied | 1 |
credential_audit_secret_free | 1 |
direct_private_route_decision | 1 |
connector_route_decision | 1 |
connector_health_contract | 1 |
connector_disable_revoke_fail_closed | 1 |
m4_sandbox_plan_scoped | 1 |
credential_binding_disable_revoke_fail_closed | 1 |
connector_disable_revoke_route_guard | 1 |
connectivity_audit_event_coverage | 1 |
connectivity_audit_secret_free | 1 |
revocation_policy_simulation_composition | 1 |
mcp205_client_surface_contract_complete | 1 |
mcp205_surface_scope_and_permissions | 1 |
mcp205_surface_lifecycle_revocation_safe | 1 |
mcp205_policy_agent_canonical_refs | 1 |
mcp205_no_secret_output | 1 |
mcp205_cli_json_parity | 1 |
managed_fleet_projection_v2_bytes_exact | 1 |
managed_fleet_projection_v3_fields_exact | 1 |
managed_fleet_projection_v3_canonical_bytes_exact | 1 |
managed_fleet_inventory_authority_exact | 1 |
managed_fleet_component_heartbeats_exact | 1 |
managed_fleet_preactivation_fetch_exact | 1 |
managed_fleet_pod_bound_auth_exact | 1 |
managed_fleet_database_roles_exact | 1 |
managed_fleet_settlement_sql_authority_exact | 1 |
managed_fleet_sources_exact | 1 |
managed_fleet_settlement_cutoff_exact | 1 |
managed_fleet_projection_cutoff_exact | 1 |
managed_fleet_freshness_boundary_exact | 1 |
managed_fleet_rate_boundary_exact | 1 |
managed_fleet_ordered_drain_exact | 1 |
managed_fleet_mixed_rollout_exact | 1 |
managed_fleet_isolation_boundaries_exact | 1 |
managed_fleet_http_routes_exact | 1 |
managed_fleet_http_security_exact | 1 |
managed_fleet_http_session_exact | 1 |
managed_fleet_http_projection_exact | 1 |
managed_fleet_http_heartbeat_exact | 1 |
managed_fleet_http_readiness_exact | 1 |
managed_fleet_http_errors_isolation_exact | 1 |
managed_valkey_scope_keys_exact | 1 |
managed_valkey_canonical_hashes_exact | 1 |
managed_valkey_function_abi_exact | 1 |
managed_valkey_fixed_point_exact | 1 |
managed_valkey_rolling_restore_exact | 1 |
managed_valkey_reconciliation_exact | 1 |
managed_valkey_crash_readiness_exact | 1 |
managed_valkey_serving_acl_exact | 1 |
managed_valkey_version_lanes_exact | 1 |
managed_valkey_isolation_proof_exact | 1 |
Fixture inventory
Section titled “Fixture inventory”| Fixture | Description | Expect keys | Metadata-only |
|---|---|---|---|
admin-identity-permissions | Admin identity is normalized, role bindings are scoped/default-deny, MCP server read responses include allowed actions, and registry mutations enforce auth-derived actors server-side without exposing raw token claims. | 7 | no |
agent-registry-m1-backend | Agent Registry M1 backend persists governed agent records, exposes admin list/detail/mutation/revoke APIs, projects approved records into runtime actor resolution, and fails closed before upstream for unapproved agent identities. | 6 | no |
api-backed-tool-call-audit | Approved generated API-backed tool can route through adapter and emit API adapter audit. | 2 | no |
api-contract-import-auth-coverage | API-to-MCP import preserves provider auth requirements, supports per-operation credential bindings, and resolves credentials before broker-aware upstream injection. | 1 | yes |
api-contract-import-gateway-hosted-runtime | Generic API contract import turns selected approved OpenAPI or Swagger operations into gateway-hosted MCP tools through the API adapter, not hosted backend MCP servers. | 5 | yes |
api-contract-import-postman-collection | Postman Collection v2.1 import creates selected approved gateway-hosted API-backed MCP tools without exposing collection values or creating a backend MCP server. | 6 | yes |
api-host-allowlist | API adapter blocks arbitrary upstream hosts and SSRF-like routes. | 2 | no |
api-operation-selection-approval | OpenAPI import does not publish unselected or unapproved operations. | 2 | no |
api-schema-validation | Invalid generated tool input schema is rejected before upstream API call. | 2 | no |
api-source-mcp-streamable-http-client-compat | Gateway API-source MCP endpoints behave as stateless Streamable HTTP MCP servers for broad SDK clients using standard bearer auth. | 1 | yes |
api-source-registry-api | API source registry API accepts owner-submitted selected OpenAPI operations and rejects unsafe or broad exposure before catalog publication. | 5 | no |
audit-siem-export | Audit export can target customer SIEM without requiring hosted vendor runtime. | 1 | no |
consumer-oauth-phase0-contracts | Phase 0 freezes consumer OAuth trust, endpoint-resource, discovery, lifecycle, diagnostics, revocation, and upstream administration contracts without changing production request-path authentication. | 11 | no |
consumer-oauth-phase1-auth-enforcement | Phase 1 makes endpoint authentication one shared, fail-closed gate for MCP-server and API-source traffic, with a single bearer parse, mandatory full-dimension revocation proof, typed version capabilities, activation-time validators, and required client-surface validation. | 9 | no |
credential-binding-redaction | Secrets never appear in audit event payloads. | 1 | no |
credential-bindings-control-plane-v1 | Credential bindings are first-class governed control-plane resources with list/detail/update/rotate/disable/revoke APIs, CLI parity, metadata-only audit, and runtime lifecycle enforcement. | 4 | no |
data-plane-zoo-runtime-live-smoke | Focused live proof for a governed mcp-zoo registered MCP server smoke report. | 7 | no |
data-plane-zoo-runtime-smoke | Data-plane runtime contract for a governed mcp-zoo registered MCP server smoke. | 6 | no |
discovery-filtering | Unauthorized tools are hidden during MCP discovery, not merely denied at execution. | 2 | no |
explicit-deny-wins | Explicit deny overrides allow. | 2 | no |
m1-discovery-endpoint | M1 discovery endpoint returns only policy-authorized tools from registered visible servers and omits unauthorized tools from the response. | 4 | no |
m1-first-governed-call | M1 first governed MCP call has runtime request/response, manifest, and audit contracts without secret material. | 4 | no |
m1-governed-call | M1 governed MCP call fixture covers unregistered backend denial, policy-filtered discovery, allowed tool execution, and audit evidence. | 27 | no |
m1-policy-core | M1 Cedar policy core default-denies missing policy and missing allow, filters discovery, and evaluates allowed and denied tool execution with reason codes and policy version. | 13 | no |
m2-agent-registry | M2 agent registry validates approved agent definitions, typed runtime instance identity, and fail-closed disabled or unapproved agents. | 3 | no |
m2-identity-policy-revocation | M2 contract baseline covers OIDC/JWT validation, actor chain attribution, agent/client-surface authorization, policy versioning, simulation, explicit deny diagnosis, audit, and revocation. | 9 | no |
m3-registry-catalog-approval-health-audit | M3 registry and catalog-lite contract baseline proves owner submit, platform approve, developer discovery, unauthorized capability hiding, health, change history, and admin audit. | 10 | no |
m4-credential-broker-private-connectivity | M4 contract baseline for credential brokering, private routing, connector health, redaction, disable/revoke, and sandbox verification scope. | 10 | no |
m4-credential-redaction-wrappers | M4 credential redaction wrappers keep credential material out of logs, traces, metrics, audit events, errors, fixtures, snapshots, prompts, CLI responses, and admin responses. | 5 | no |
m5-session-lifecycle-affinity-revocation-audit | M5 session lifecycle contracts cover create, reattach, reconnect, drain, terminate, revoke, affinity, durable metadata, and secret-free audit evidence. | 8 | no |
m7-admin-workflow-cli-parity | M7 admin workflow and CLI parity completion contract for platform and security users completing V1 work through the admin API and machine-readable CLI without database access or custom scripts. | 8 | no |
managed-cached-policy-outage | Managed data plane enforces cached policy during a bounded control-plane outage, then fails closed. | 1 | no |
managed-deployment-model | V1 exposes managed SaaS and fully self-hosted deployment modes with explicit managed egress and bounded cached-policy continuity. | 5 | no |
managed-runtime-admission-api | Managed-only backend plan, usage, invocation-status, assignment, and support-credit API contract with server-owned scope and commercial authority. | 7 | no |
managed-runtime-admission-fleet-compatibility | Executable Phase 5 contract for byte-stable v2/v3 projections, Kubernetes-authoritative fleet membership and routing withdrawal, projected workload authentication and replay-safe heartbeats, least-privilege database ingress, independent projection/settlement cutoffs, current-state readiness, rollout safety, and self-hosted isolation. | 17 | no |
managed-runtime-admission-fleet-http-abi | Executable contract for the four production-composed managed-only Fleet HTTP operations, their committed SQL authority, exact schemas and headers, replay boundary, closed errors, generated names, compatibility, and bounded isolation claim. | 7 | no |
managed-runtime-admission-supervision | Executable production supervision contract for managed runtime admission deadlines, reservations, cancellation, recovery checkpoints, settlement ambiguity, abort quarantine, bounded queues, and fenced lease authority. | 15 | no |
managed-runtime-admission-valkey | Executable Phase 6 contract for managed Valkey scope and keys, canonical hashes, complete RESP2 function ABI, exact rate math, reconciliation, DB0-confined data-plane and global executor ACLs, compatibility lanes, and proof boundaries. | 11 | no |
managed-runtime-admission | Executable contract for production-composed managed runtime admission ordering, counting, modes, invocation identity, and error envelopes. Environment-scale evidence remains an operational gate. | 8 | no |
managed-runtime-projection-wiring | Managed data planes fetch a scoped operational projection from the control plane, including a valid deny-all bootstrap projection before any server or policy exists, retain the last good projection on reload failure, and export metadata-only runtime audit while self-hosted installs remain file-backed. | 7 | no |
mcp-155-onboarding-approval-workflows | UI-3 onboarding and approval workflow contract for Demo A private MCP server governance and Demo B OpenAPI-to-MCP approval, including agent registration, audit events, catalog visibility, disable touchpoints, role denial, and browser evidence expectations. | 6 | no |
mcp-156-policy-editor-simulator-deny-diagnostics | UI-4 policy editor, policy simulator, version traceability, publish/rollback impact preview, deny diagnostics, CLI parity, and metadata-only evidence contract. | 6 | no |
mcp-157-credentials-connectors-sessions-audit-emergency-response | UI-5 operational security surface for credentials, connectors, active sessions, audit investigation, emergency disable, Demo C, and Demo A disable/session continuity. | 6 | no |
mcp-158-enterprise-readiness-topology-rbac-accessibility-hardening | UI-6 enterprise readiness surface for topology clarity, SIEM/export, telemetry, backup/restore, upgrade/drain, license/artifact status, RBAC, accessibility, production states, documentation links, CLI parity, and evidence. | 6 | no |
mcp-164-stateful-onboarding-workflows | Stateful SaaS onboarding, import, operation selection, reviewer approval, batch decision, timeline, and metadata-only UI contract for MCP server registration, API source import, and agent registration. | 5 | no |
mcp-18-actor-chain-client-surface | MCP-18 resolves approved client-surface identity, typed actor chains, and human delegator context before policy and audit. | 4 | no |
mcp-20-policy-version-explicit-deny | M2 policy decisions carry durable policy versions and explicit deny overrides allow across actor, surface, environment, API source, and tool input dimensions while default deny remains distinct. | 15 | no |
mcp-204-connector-registry-read-model | Connector registry list/detail read model exposes scoped, paginated, metadata-only UI and CLI contracts without client-side joins or private backend leakage. | 7 | no |
mcp-205-client-surfaces-management | Canonical client-surface management backend contract for list/detail, policy simulator and agent refs, safe disable/revoke, session impact, CLI parity, and metadata-only output. | 6 | no |
mcp-206-data-planes-topology-health | Backend data-plane topology and health read model for list/detail API, shared deployment-status aggregation, CLI parity, trust boundary, and metadata-only diagnostics. | 6 | no |
mcp-207-deny-diagnostics-bundle | Backend deny diagnostics bundle endpoint and CLI parity for metadata-only remediation-ready denied request explanations. | 6 | no |
mcp-208-admin-activity-read-model | MCP-208 Admin Activity read model exposes curated metadata-only admin actions for UI list/detail, permission gates, OpenAPI, and gatewayctl JSON parity. | 7 | no |
mcp-209-environments-settings-read-model | Read-only tenant environment settings list/detail contract for UI Settings > Environments. | 5 | no |
mcp-21-policy-simulation-api | Policy simulation API previews V1 gateway policy decisions without mutating runtime state or exposing secret material. | 4 | no |
mcp-210-telemetry-siem-webhooks-admin-contract | Telemetry and SIEM/webhook settings expose customer-controlled, metadata-only admin configuration with safe lifecycle actions and audit evidence. | 1 | no |
mcp-211-license-entitlement-read-model | Backend-owned License settings read model with safe entitlement metadata, deployment summary, CLI parity, permission gates, and no raw license material. | 5 | no |
mcp-214-hosted-email-user-management | Hosted-only transactional email and hosted user-management contract for verification challenges, invites, lifecycle notifications, Cloudflare Email Sending delivery-plan shape, and self-hosted separation. | 16 | no |
mcp-215-hosted-runtime-allowlisted-endpoints | Hosted-only runtime endpoint contract for customer-owned HTTPS API and HTTPS MCP endpoints, DAC egress allowlisting, reachability fail-closed behavior, metadata-only runtime decisions, and self-hosted separation. | 12 | no |
mcp-216-hosted-audit-payload-capture-operator-controls | Hosted-only audit, payload-capture, TTL purge, hosted incident/operator controls, abuse-control posture, metadata-only runtime audit, and self-hosted separation. | 13 | no |
mcp-218-hosted-saas-qa-closeout | Hosted SaaS QA1 closeout contract proving docs, evals, E2E smoke, child PR/check audit, no-secret evidence, and self-hosted regression coverage across R1-R44, F1-F7, AE1-AE10, and success criteria. | 7 | no |
mcp-22-policy-simulation-cli | gatewayctl policy simulation accepts M2 actor, surface, environment, server/API source, and tool context and emits stable deny diagnosis without secret material. | 4 | no |
mcp-221-hosted-org-trial-lifecycle | Hosted-only org trial lifecycle contract for verified signup, trial expiry hard-lock, upgrade request, DAC operator actions, and self-hosted separation. | 6 | no |
mcp-222-hosted-domain-membership | Hosted-only same-domain membership contract for provisional domain association, join requests, owner/security-admin review, public-domain no-grouping, DAC separate-workspace exceptions, deferred DNS/SSO claims, and self-hosted separation. | 8 | no |
mcp-223-hosted-be1-integration-closeout | BE1 closeout contract proving hosted plan entitlements, trial hard-lock, domain membership, same-domain join requests, public-domain no-grouping, DAC exceptions, deferred DNS/SSO claim, downstream boundaries, and self-hosted separation work together. | 6 | no |
mcp-227-hosted-saas-secret-handling | Hosted SaaS OPS2 contract for Hetzner pilot secret source selection, reference-only Helm wiring, self-hosted guardrails, and no-secret artifacts. | 6 | no |
mcp-229-hosted-saas-ops4-deploy-stack | Hosted SaaS OPS4 deploy contract for Hetzner k3s substrate, Helm-hosted stack resources, first-admin bootstrap, rollback, upgrade, and self-hosted separation. | 6 | no |
mcp-230-hosted-saas-ops5-networking-egress | Hosted SaaS OPS5 contract for public ingress, DNS, TLS, DAC egress identity, customer allowlisting, private runtime isolation, connector deferral, and sanitized live evidence. | 8 | no |
mcp-231-hosted-saas-ops6-observability-backups | Hosted SaaS OPS6 contract for hosted observability, alerts, backups, restore drill, exposure checks, retention posture, incident runbooks, and re-runnable launch checklist. | 9 | no |
mcp-234-hosted-saas-ops8-live-profile-smoke-access | Hosted SaaS OPS8 parent contract for live Helm profile digest pinning, guarded smoke access, sanitized live baseline, and no-go evidence until DNS/TLS/LB/egress/secret projection are ready. | 6 | no |
mcp-235-hosted-release-images-profile | Hosted SaaS OPS8A release image and deployable hosted profile contract for GHCR images, SBOM/provenance/signing, API-adapter runtime wrapper, Kubernetes bind envs, and self-hosted separation. | 9 | no |
mcp-236-hosted-public-smoke-api | Hosted-only public HTTP smoke API surface for MCP-233 live hosted signup, verification, trial, same-domain join, runtime readiness, metadata audit posture, expiry hard-lock, and self-hosted disabled responses. | 10 | no |
mcp-237-hosted-saas-ops8c-edge-readiness | Hosted SaaS OPS8C contract for DNS/TLS/secret projection, Hetzner LB target health, DAC egress identity publication, smoke access, rollback, destroy readiness, and sanitized blocker evidence. | 7 | no |
mcp-241-hosted-account-entry-baseline | ACCT0 hosted account-entry baseline contract map for signup, login, email confirmation, social sign-in, one workspace, same-domain join, invites, role selection, logout, first-run owner setup, no-secret evidence, and self-hosted separation. | 1 | yes |
mcp-243-hosted-auth-session-lifecycle | Hosted SaaS ACCT2 email-password signup, confirmation, restricted pre-activation sessions, active-session identity, logout, password reset, password change, and self-hosted separation. | 4 | no |
mcp-245-hosted-account-routing-admin | Hosted SaaS ACCT4 work-domain owner routing, public-domain owner blocking, same-domain join admin decisions, recipient-bound contractor invites, invite-to-join convergence, neutral conflicts, notifications, and self-hosted separation. | 4 | no |
mcp-247-hosted-account-settings-security-ui | Hosted SaaS ACCT6 workspace settings, access review, invite management, combined Profile & security, role selection, and self-hosted separation coverage. | 4 | no |
mcp-249-hosted-social-auth | Hosted-only social signup/login with real Google and GitHub redirect/callback, reference-only provider config, state/nonce protection, verified-email linking, callback UI states, app-only logout, and self-hosted separation. | 8 | no |
mcp-250-hosted-account-entry-closeout | Hosted SaaS ACCT9 closeout for account-entry OpenAPI contracts, eval coverage, docs, browser evidence, no-secret evidence, and self-hosted non-regression. | 8 | no |
mcp-253-hosted-governance-profile-resolver | Hosted SaaS GOV1 governance-profile resolver contract for typed profile presets, distinct-review metadata, hard safety gates, and self-hosted strict default semantics. | 5 | no |
mcp-254-hosted-activation-policy-settings | Hosted SaaS GOV2 activation policy settings persistence, hosted simple defaults, self-hosted strict separation, admin mutation acknowledgement, and audit-safe metadata. | 5 | no |
mcp-255-hosted-governance-transition-wiring | Hosted SaaS GOV3 resource transition wiring uses governance profile resolver decisions for MCP server, credential binding, API source, and approval queue activation paths while preserving hard gates and self-hosted distinct review. | 4 | no |
mcp-256-hosted-governance-contracts-client-dtos | Hosted SaaS GOV4 contract, eval, and generated client DTO coverage for activation-policy decision metadata on resource read surfaces. | 6 | no |
mcp-258-hosted-enterprise-identity-mapping | Hosted SaaS GOV6 keeps enterprise OIDC/SAML group-claim mapping as an optional extension point while preserving normal hosted local/social account entry and governance-profile separation. | 6 | no |
mcp-259-hosted-governance-closeout | Hosted SaaS GOV7 terminal closeout for governance-profile docs, browser evidence, no-secret review, self-hosted non-regression, and hosted deploy gate honesty. | 5 | no |
mcp-260-hosted-canonical-domain-reset | Hosted SaaS canonical-domain, automated TLS, and destructive reset contract for app.amelfi.ai. | 6 | no |
mcp-27-registry-lifecycle | MCP-27 registry lifecycle foundation uses typed submitted, under_review, approved, rejected, disabled, and archived states with owner/reviewer metadata, fail-closed invalid transitions, durable change history, and shared catalog/audit projections. | 5 | no |
mcp-28-server-registry-api | MCP-28 server registry API accepts owner-submitted MCP server manifests, rejects unsafe submissions with V1-safe reasons, returns registry metadata, and gates catalog visibility on platform approval. | 5 | no |
mcp-30-registry-admin-audit-events | MCP-30 registry admin audit evidence covers M3 submit, approval, rejection, disable, archive, ownership, validation, health, and catalog visibility lifecycle actions without payload material. | 4 | no |
mcp-31-server-health-checks | MCP-31 proves registered MCP server health checks use typed safe status metadata and can warn or hide catalog-lite capabilities without leaking secrets. | 1 | no |
mcp-32-catalog-lite-query-api | Catalog-lite query API returns only approved capabilities visible to the actor context and omits unauthorized, unapproved, disabled, unhealthy-blocked, or unselected capabilities. | 3 | no |
mcp-38-credential-broker-mock | MCP-38 deterministic credential broker trait and mock broker outcomes. | 4 | no |
mcp-41-vault-compatible-credential-broker-adapter | Vault-compatible M4 credential broker adapter maps approved bindings to Vault KV paths, returns safe failures, and never emits raw credential material. | 5 | no |
mcp-42-direct-private-endpoint-routing | MCP-42 direct private endpoint routing for governed tool calls, including fail-closed denied targets, safe route audit, and SSRF/host guardrails. | 6 | no |
mcp-43-outbound-connector-model-health | MCP-43 outbound connector records use typed governance metadata, safe health failures, scoped catalog visibility, and route-mode evidence without becoming a connector marketplace. | 5 | no |
mcp-44-connector-disable-revoke-audit | MCP-44 connector and credential binding disable/revoke decisions fail closed before upstream calls and emit safe audit evidence that composes with revocation and policy simulation. | 5 | no |
mcp-50-session-state-machine-durable-metadata | MCP-50 models typed client/backend session IDs, explicit state transitions, PostgreSQL durable metadata, expiry/idle handling, and fail-closed invalid transitions. | 5 | no |
mcp-51-valkey-active-session-index-affinity-routing | MCP-51 proves the Valkey hot active-session index supports safe lookup, affinity route hits, stale-state fail-closed behavior, drain/revoke fanout lookup, and Valkey-unavailable denial. | 6 | no |
mcp-52-reconnect-drain-terminate | MCP-52 proves reconnect is gated by eligible state and backend resume support, drain stops new stateful sessions and handles timeout/complete paths, terminate updates durable and hot state, and bounded buffers plus lifecycle hooks are secret-free. | 7 | no |
mcp-53-active-session-revocation-fanout | MCP-53 proves M2 revocation outbox events fan out through the V1 PostgreSQL internal queue into M5 active-session termination, blocked new calls, duplicate replay safety, and secret-free observable failures. | 6 | no |
mcp-54-session-lifecycle-audit-evidence | MCP-54 proves session lifecycle audit evidence records create, reattach, reconnect, drain, terminate, revoke, expiry, and denied lifecycle operations with safe context, request/correlation IDs, machine-readable reasons, and no secret material. | 5 | no |
mcp-59-api-to-mcp-contract-baselines | M6 OpenAPI import and API-to-MCP contracts require selected approved operations, guarded adapter validation, credential binding, size limits, and secret-free audit. | 5 | no |
mcp-60-openapi-parser-runtime | M6 runtime implementation parses approved OpenAPI specs, publishes only selected approved operations, and denies unsafe adapter calls before upstream. | 1 | no |
mcp-61-api-to-mcp-mapping-generator | M6 API-to-MCP mapping generator emits deterministic MCP tool mappings only for selected approved OpenAPI operations, with generated schema boundaries, response metadata, discovery compatibility, and secret-free artifacts. | 5 | no |
mcp-62-rest-runtime-adapter | M6 REST runtime adapter executes approved generated API tools through governed preflight, schema validation, size and timeout limits, stable fail-closed reasons, and secret-free audit evidence. | 1 | no |
mcp-63-api-credential-binding-integration | M6 API-backed tools resolve approved credential bindings through the credential broker before adapter execution and keep credential audit evidence secret-free. | 4 | no |
mcp-64-openapi-import-cli-diagnostics | M6 gatewayctl import-openapi emits deterministic selected/rejected operation diagnostics, stable fail-closed reason codes, and mapping evidence links for approved OpenAPI fixtures. | 4 | no |
mcp-66-api-adapter-conformance-sandbox-evidence | M6 API adapter conformance covers selected OpenAPI import, generated MCP tool execution, denial of unselected operations, host allowlist, schema validation, credentials, size limits, audit redaction, and Tier 4 Sprite evidence. | 4 | no |
mcp-82-self-hosted-readiness-sandbox-matrix | M8 baseline contract for design-partner self-hosted and managed readiness, including the sandbox evidence matrix for install, governed call, integrations, no-outbound behavior, backup/restore, upgrade/drain, Cosign, SBOM, and release evidence bundles. | 6 | no |
mcp-83-helm-values-install-diagnostics | M8 local Helm hardening contract for values schema validation, required V1 install diagnostics, rendered resources, and sandbox smoke deferral. | 8 | no |
mcp-84-self-hosted-hybrid-config | M8 follow-on contract for self-hosted dependency configuration and managed runtime-continuity surfaces, including PostgreSQL, Valkey, OIDC/SAML, secret manager, OTel, SIEM/export, stable diagnostics, and non-V1 dependency guardrails. | 5 | no |
mcp-85-runtime-controls | M8 runtime-control contract for deterministic network policy, resource requests/limits, HPA/PDB, readiness, and no-outbound runtime checks. | 6 | no |
mcp-86-backup-restore-upgrade-drain-workflows | M8 backup/restore and upgrade/drain workflow evidence contract for self-hosted state, including deterministic local checks, Tier 4 sandbox scenarios, failure modes, rollback boundaries, and metadata-only artifact rules. | 7 | no |
mcp-87-release-evidence-bundle-gates | M8 release evidence gate for deterministic Cosign, SBOM, provenance, self-hosted readiness linkage, and secret-free release evidence bundle validation. | 6 | no |
mcp-88-self-hosted-hybrid-sandbox-conformance | Historical M8 self-hosted sandbox conformance evidence from the former hybrid model; retained for release traceability and no-secret proof. | 8 | no |
mcp-92-release-readiness-matrix | M9 V1 release-readiness matrix and sandbox plan for PRD sections 22 and 23, including M1-M8 evidence, downstream owner issues, validation commands, Sprite reuse-before-new, V1 boundaries, and no-secret artifact rules. | 7 | no |
mcp-93-release-candidate-smoke | M9 release-candidate smoke harness that replays M1-M8 evidence for governed MCP calls, generated API tools, policy simulation, credential and connector fail-closed behavior, session lifecycle, audit/SIEM export, and admin/CLI readiness while emitting a metadata-only JSON report. | 6 | no |
mcp-95-release-security-no-secret-review-gates | M9 release security gate for deterministic no-secret scans, runtime surface payload review, release evidence artifacts, sandbox/PR evidence, reviewer checklist, and V1 boundaries. | 6 | no |
mcp-96-release-candidate-evidence | M9 release-candidate package evidence gate for versioned RC bundle metadata, M8 compatibility, Helm digest, Cosign, SBOM, provenance, backup/restore, upgrade/drain, rollback, release notes, checksum manifest, and no-secret artifact rules. | 7 | no |
mcp-protocol-compatibility-engine | Enterprise MCP compatibility engine contract for dual-era protocol mediation, configurable bindings and routes, safe cleartext profiles, extension policy, route-local probing, and fail-closed runtime resolution. | 11 | no |
mcp-server-live-reprobe | An authorized registry owner can re-probe only an approved registered immutable live MCP snapshot, with strict secret-safe endpoint validation, public-network transport guards, cross-replica compare-and-swap health persistence, metadata-only audit, exact hosted scope, and conditional rollback on audit failure. | 6 | no |
mcp-server-registry-immutable-snapshots | MCP server registry submissions are immutable, approval is submission/hash scoped, runtime reads the live approved snapshot, Versions lists approved snapshots only, and compare uses backend snapshot IDs. | 11 | no |
mcp-server-registry-list-backend | MCP server registry list and detail backend returns UI-ready immutable snapshot/submission metadata, permissions, activity, compare, and action endpoints without pushing approvals queue into this slice. | 8 | no |
mcp-server-registry-read-api | MCP server registry read APIs expose UI-ready list/detail/activity/compare contracts without local manifest diffing. | 9 | no |
openapi-import-cli | CLI can import OpenAPI 3.x specs and create candidate operations for explicit selection. | 1 | no |
policy-intent-authoring | Intent-driven policy authoring, impact review, publish readiness, runtime projection, governed testing, and audit correlation. | 5 | no |
policy-management-m1-backend | Policy Management M1 backend contract for list, detail, draft update, validation, publish, archive, permissions, audit, and metadata-only policy bodies. | 10 | no |
policy-ref-resolver-backend | Policy refs in MCP server tools resolve to same-tenant, same-environment policy_version records and are runtime-valid only when published. | 6 | no |
policy-simulation | Policy simulation previews allow outcomes before publishing. | 2 | no |
production-auth-boundary-be | Production admin/runtime auth boundary supports trusted proxy and direct OIDC JWT modes, exposes safe provider diagnostics, and manages gateway role bindings without raw token, SAML assertion, secret, or full claim payload leakage. | 6 | no |
production-auth-local-identity-be | Production local identity mode supports bootstrap admin, local users and groups, browser sessions, service-account tokens, and role-binding evaluation without trusting gateway headers or exposing password/token hashes. | 2 | no |
registered-backend-routing | Gateway denies unregistered MCP backend routing. | 5 | no |
revocation-event-model | M2 revocation event model uses typed subjects, PostgreSQL outbox semantics, fail-closed call denial, session revoke handling, and secret-free audit. | 5 | no |
self-hosted-no-outbound-runtime | Self-hosted mode has no required external vendor runtime dependency. | 1 | no |
session-affinity-externalized | Stateful sessions have affinity and metadata outside worker memory. | 2 | no |
session-drain-backpressure | Drain is explicit and streaming buffers are bounded. | 2 | no |
session-revocation | Agent revocation blocks new calls and terminates affected active sessions. | 3 | no |
stdio-sse-runtime-projection-contracts | Foundation contracts for transport-aware registered MCP servers, CP-published runtime projections, schema passthrough, lifecycle reload state, and metadata-only runtime audit export. | 14 | no |
telemetry-customer-controlled | Telemetry is disabled or exported only to customer-controlled systems by default. | 1 | no |
tenant-environment-isolation | Dev, staging, and production policy/routing boundaries are isolated. | 2 | no |
What can go wrong?
Section titled “What can go wrong?”- A fixture uses an unsupported expectation key and fails as
unsupported_expect_key. - A metadata-only fixture omits the metadata reason.
- Docs cite an eval ID that no longer exists.
- Fixture examples carry payloads, tokens, prompts, credentials, or customer data instead of safe metadata.
Source truth
Section titled “Source truth”- docs/evals/scenario-contract.md
- fixtures/evals/
- fixtures/evals/schema.json
- apps/docs/scripts/generate-reference-pages.mjs
- Read Quickstart for the first governed-call proof.
- Read Source Map for docs coverage status.
- Read Security Review for no-secret evidence rules.
Type set in Geist, Source Serif 4, and Departure Mono.