Evals
Reference
Generated source hash: sha256:f678020de84eab7b6a541a1ea03b918bc309f4d0c9c6cf673028cbe8acf414ad.
Use evaluation fixtures to test a specific gateway behavior. This inventory maps supported expectation keys to their fixtures so changes can be checked against the existing behavior contract.
A passing fixture is evidence for the tested scenario. It does not prove that a deployed gateway or external integration works.
Evaluation inventory
Section titled “Evaluation inventory”234supported expectation keys are read from the scenario contract.164fixture files are scanned.- Metadata-only fixtures must declare
metadata_only: trueand a non-empty reason, per the scenario contract.
Supported expectation keys
Section titled “Supported expectation keys”| Expectation key | Fixture count |
|---|---|
decision | 8 |
reason | 4 |
visible_tools | 4 |
hidden_tools | 4 |
route_allowed | 1 |
route_reason | 1 |
unregistered_backend_route_allowed | 2 |
unregistered_backend_route_reason | 2 |
unregistered_backend_upstream_attempted | 2 |
api_call_allowed | 0 |
api_call_reason | 0 |
allowed_tool_call_success | 1 |
allowed_tool_call_policy_decision | 1 |
allowed_tool_call_policy_reason | 1 |
allowed_tool_call_response_status | 1 |
denied_tool_call_denied | 1 |
denied_tool_call_policy_decision | 1 |
denied_tool_call_policy_reason | 1 |
denied_tool_call_response_status | 1 |
denied_tool_call_error_code | 1 |
denied_tool_call_upstream_attempted | 1 |
audit_emitted | 1 |
audit_event_type | 1 |
audit_policy_decision | 1 |
audit_credential_mode | 1 |
audit_policy_version | 1 |
denied_audit_emitted | 1 |
denied_audit_event_type | 1 |
denied_audit_policy_decision | 1 |
denied_audit_credential_mode | 1 |
denied_audit_policy_version | 1 |
denied_audit_error_code | 1 |
runtime_contract_complete | 1 |
discovery_response_contract_complete | 1 |
unauthorized_tool_hidden | 1 |
manifest_contract_complete | 1 |
audit_contract_complete | 1 |
audit_secret_free | 3 |
session_terminated | 1 |
backend_sessions_count_max | 1 |
bounded_buffers | 1 |
session_metadata_externalized | 1 |
drain_supported | 1 |
m5_session_lifecycle_contract_complete | 1 |
session_create_contract | 1 |
session_reattach_reconnect_contract | 1 |
session_drain_terminate_revoke_contract | 1 |
session_affinity_index_contract | 1 |
session_lifecycle_audit_secret_free | 1 |
m5_session_scope_boundaries | 1 |
m5_acceptance_matrix_complete | 1 |
mcp50_session_state_machine_durable_metadata | 1 |
mcp50_session_id_boundary_validation | 1 |
mcp50_invalid_transition_denied | 1 |
mcp50_expiry_idle_contract | 1 |
mcp50_session_metadata_secret_free | 1 |
mcp51_active_index_contract | 1 |
mcp51_affinity_hit | 1 |
mcp51_stale_index_fail_closed | 1 |
mcp51_drain_revoke_lookup | 1 |
mcp51_valkey_unavailable_fail_closed | 1 |
mcp51_index_secret_free | 1 |
mcp52_reconnect_gate | 1 |
mcp52_reattach_gate | 1 |
mcp52_drain_no_new_sessions | 1 |
mcp52_drain_complete_timeout | 1 |
mcp52_terminate_fail_closed | 1 |
mcp52_backpressure_contract | 1 |
mcp52_lifecycle_hooks_secret_free | 1 |
mcp53_outbox_fanout_contract | 1 |
mcp53_revocation_scope_coverage | 1 |
mcp53_active_session_termination | 1 |
mcp53_blocked_new_calls | 1 |
mcp53_duplicate_replay_idempotent | 1 |
mcp53_failure_modes_observable_secret_free | 1 |
mcp54_audit_event_sequence | 1 |
mcp54_safe_context_and_correlation | 1 |
mcp54_denied_lifecycle_auditable | 1 |
mcp54_session_audit_secret_free | 1 |
mcp54_contract_alignment | 1 |
outbound_runtime_required | 2 |
telemetry_customer_controlled | 1 |
cached_policy_enforced | 1 |
siem_truthfulness_enforced | 1 |
m7_critical_workflow_coverage | 1 |
m7_admin_api_surface_complete | 1 |
m7_cli_parity_contract | 1 |
m7_cli_diagnostics_executable | 1 |
m7_safe_deny_no_secret_surfaces | 1 |
m7_no_db_or_custom_script_dependency | 1 |
mcp82_contract_refs_exist | 1 |
mcp82_acceptance_matrix_complete | 1 |
mcp82_sandbox_matrix_complete | 1 |
mcp82_sprite_reuse_checked | 1 |
mcp82_release_evidence_bundle_complete | 1 |
mcp82_artifacts_secret_free | 1 |
mcp83_contract_refs_exist | 1 |
mcp83_schema_cases_complete | 1 |
mcp83_rendered_manifest_requirements_complete | 1 |
mcp83_install_diagnostics_machine_readable | 1 |
mcp83_local_validation_recorded | 1 |
mcp83_v1_scoped_defaults | 1 |
mcp83_artifacts_secret_free | 1 |
mcp84_config_surfaces_complete | 1 |
mcp84_hybrid_modes_customer_controlled | 0 |
mcp84_failure_diagnostics_stable | 1 |
mcp84_no_non_v1_dependencies | 1 |
mcp84_artifacts_secret_free | 1 |
mcp85_contract_refs_exist | 1 |
mcp85_network_controls_complete | 1 |
mcp85_resource_controls_complete | 1 |
mcp85_no_outbound_runtime_observable | 1 |
mcp85_sandbox_proof_listed | 1 |
mcp85_artifacts_secret_free | 1 |
mcp87_contract_refs_exist | 1 |
mcp87_release_bundle_valid | 1 |
mcp87_self_hosted_readiness_linked | 1 |
mcp87_provenance_negative_cases_stable | 1 |
mcp87_scripts_gate_release_fields | 1 |
mcp87_artifacts_secret_free | 1 |
mcp88_contract_refs_exist | 1 |
mcp88_blocker_chain_complete | 1 |
mcp88_sprite_reuse_and_checkpoint_recorded | 1 |
mcp88_sandbox_tracks_complete | 1 |
mcp88_unavailable_capabilities_have_fallbacks | 1 |
mcp88_replay_commands_complete | 1 |
mcp88_artifact_paths_and_hashes_recorded | 1 |
mcp88_artifacts_secret_free | 1 |
mcp92_contract_refs_exist | 1 |
mcp92_release_readiness_matrix_complete | 1 |
mcp92_docs_requirements_matrix_complete | 1 |
mcp92_sandbox_plan_complete | 1 |
mcp92_owner_links_complete | 1 |
mcp92_v1_scope_boundaries | 1 |
mcp92_artifacts_secret_free | 1 |
mcp96_contract_refs_exist | 1 |
mcp96_release_candidate_manifest_valid | 1 |
mcp96_m8_evidence_compatible | 1 |
mcp96_required_evidence_linked | 1 |
mcp96_negative_cases_stable | 1 |
mcp96_harness_command_documented | 1 |
mcp96_artifacts_secret_free | 1 |
mcp93_contract_refs_exist | 1 |
mcp93_reuses_m1_m8_evidence | 1 |
mcp93_smoke_coverage_complete | 1 |
mcp93_fail_closed_matrix_complete | 1 |
mcp93_machine_readable_report | 1 |
mcp93_artifacts_metadata_only_secret_free | 1 |
mcp95_contract_refs_exist | 1 |
mcp95_security_surfaces_complete | 1 |
mcp95_runtime_paths_no_payloads | 1 |
mcp95_release_artifacts_scanned | 1 |
mcp95_reviewer_checklist_complete | 1 |
mcp95_v1_scope_boundaries | 1 |
policy_core_cases_pass | 1 |
explicit_deny_precedence_matrix | 1 |
policy_versioned_decision_contract | 1 |
deny_reason_surface_contract | 1 |
m2_contract_complete | 1 |
oidc_jwt_validation_contract | 1 |
client_surface_restriction_enforced | 1 |
agent_authorization_contract | 1 |
policy_versioning_contract | 1 |
explicit_deny_diagnosis_contract | 1 |
m2_audit_contract | 1 |
revocation_event_contract | 1 |
m2_exit_path_observable | 1 |
agent_registry_contract | 1 |
agent_instance_identity_contract | 1 |
agent_registry_fail_closed | 1 |
policy_simulation_api_contract | 1 |
policy_simulation_api_cases_pass | 1 |
policy_simulation_api_secret_free | 1 |
policy_simulation_v1_scoped | 1 |
revocation_event_model_complete | 1 |
revocation_outbox_internal_queue | 1 |
revoked_new_calls_fail_closed | 1 |
revocation_active_sessions_handled | 1 |
revocation_audit_secret_free | 1 |
m4_contract_complete | 1 |
credential_resolution_success | 1 |
credential_resolution_missing_denied | 1 |
credential_audit_secret_free | 1 |
direct_private_route_decision | 1 |
connector_route_decision | 1 |
connector_health_contract | 1 |
connector_disable_revoke_fail_closed | 1 |
m4_sandbox_plan_scoped | 1 |
credential_binding_disable_revoke_fail_closed | 1 |
credential_binding_activation_refs_validated | 1 |
connector_disable_revoke_route_guard | 1 |
connectivity_audit_event_coverage | 1 |
connectivity_audit_secret_free | 1 |
revocation_policy_simulation_composition | 1 |
mcp205_client_surface_contract_complete | 1 |
mcp205_surface_scope_and_permissions | 1 |
mcp205_surface_lifecycle_revocation_safe | 1 |
mcp205_policy_agent_canonical_refs | 1 |
mcp205_no_secret_output | 1 |
mcp205_cli_json_parity | 1 |
managed_fleet_projection_v2_bytes_exact | 1 |
managed_fleet_projection_v3_fields_exact | 1 |
managed_fleet_projection_v3_canonical_bytes_exact | 1 |
managed_fleet_inventory_authority_exact | 1 |
managed_fleet_component_heartbeats_exact | 1 |
managed_fleet_preactivation_fetch_exact | 1 |
managed_fleet_pod_bound_auth_exact | 1 |
managed_fleet_database_roles_exact | 1 |
managed_fleet_settlement_sql_authority_exact | 1 |
managed_fleet_sources_exact | 1 |
managed_fleet_settlement_cutoff_exact | 1 |
managed_fleet_projection_cutoff_exact | 1 |
managed_fleet_freshness_boundary_exact | 1 |
managed_fleet_rate_boundary_exact | 1 |
managed_fleet_ordered_drain_exact | 1 |
managed_fleet_mixed_rollout_exact | 1 |
managed_fleet_isolation_boundaries_exact | 1 |
managed_fleet_http_routes_exact | 1 |
managed_fleet_http_security_exact | 1 |
managed_fleet_http_session_exact | 1 |
managed_fleet_http_projection_exact | 1 |
managed_fleet_http_heartbeat_exact | 1 |
managed_fleet_http_readiness_exact | 1 |
managed_fleet_http_errors_isolation_exact | 1 |
managed_valkey_scope_keys_exact | 1 |
managed_valkey_canonical_hashes_exact | 1 |
managed_valkey_function_abi_exact | 1 |
managed_valkey_fixed_point_exact | 1 |
managed_valkey_rolling_restore_exact | 1 |
managed_valkey_reconciliation_exact | 1 |
managed_valkey_crash_readiness_exact | 1 |
managed_valkey_serving_acl_exact | 1 |
managed_valkey_version_lanes_exact | 1 |
managed_valkey_isolation_proof_exact | 1 |
Fixture inventory
Section titled “Fixture inventory”| Fixture | Description | Expect keys | Metadata-only |
|---|---|---|---|
activation-credential-reconnect | Reconnect stages replacement OAuth credentials while the current binding remains active; atomic publication retires only the exclusive prior-snapshot binding. | 0 | yes |
activation-runtime-verification | Internal product service verification traverses the deployed data plane without changing public endpoint authentication. | 0 | yes |
admin-identity-permissions | Admin identity is normalized, role bindings are scoped/default-deny, MCP server read responses include allowed actions, and registry mutations enforce auth-derived actors server-side without exposing raw token claims. | 7 | no |
agent-registry-m1-backend | Agent Registry M1 backend persists governed agent records, exposes admin list/detail/mutation/revoke APIs, projects approved records into runtime actor resolution, and fails closed before upstream for unapproved agent identities. | 6 | no |
api-contract-import-auth-coverage | API-to-MCP import preserves provider auth requirements, supports per-operation credential bindings, and resolves credentials before broker-aware upstream injection. | 1 | yes |
audit-siem-export | SIEM delivery becomes active only from a current live worker observation and reports health from durable receiver evidence. | 1 | no |
consumer-oauth-phase0-contracts | Phase 0 freezes consumer OAuth trust, endpoint-resource, discovery, lifecycle, diagnostics, revocation, and upstream administration contracts without changing production request-path authentication. | 11 | no |
consumer-oauth-phase1-auth-enforcement | Phase 1 makes endpoint authentication one shared, fail-closed gate for MCP-server traffic, with a single bearer parse, mandatory full-dimension revocation proof, typed version capabilities, activation-time validators, and required client-surface validation. | 10 | no |
credential-binding-projection-routes | Managed v3/v4 runtime projections preserve one credential binding across approved target routes while rejecting duplicate routes, cross-route references and inconsistent shared binding state. Rust regressions execute parser behavior; this harness check anchors the contract and regression cases. | 1 | no |
credential-binding-redaction | Secrets never appear in audit event payloads. | 1 | no |
credential-bindings-control-plane-v1 | Credential bindings are first-class governed control-plane resources with list/detail/update/rotate/disable/revoke APIs, CLI parity, metadata-only audit, and runtime lifecycle enforcement. | 4 | no |
critical-tool-explicit-selection | Critical tools default off, accept explicit owner selection, and remain subject to the existing governance profile before publication. | 1 | no |
data-plane-zoo-runtime-live-smoke | Focused live proof for a governed mcp-zoo registered MCP server smoke report. | 7 | no |
data-plane-zoo-runtime-smoke | Data-plane runtime contract for a governed mcp-zoo registered MCP server smoke. | 6 | no |
diagnostic-recordings | Server-scoped diagnostic recordings retain encrypted MCP and gateway-observed OAuth events and immutable actor/client/upstream identity snapshots for a bounded investigation window. Bodies are redacted by default; an explicit operator configuration enables unredacted testing with truthful provenance. | 0 | yes |
discovery-filtering | Unauthorized tools are hidden during MCP discovery, not merely denied at execution. | 2 | no |
explicit-deny-wins | Explicit deny overrides allow. | 2 | no |
external-consumer-settings | External consumers require explicit server-bound OAuth access and exact tool selection. | 0 | yes |
gmail-native-mcp | Gmail uses ordinary remote MCP registration with audience-bound authorization and all Gmail v1 methods. | 0 | yes |
google-workspace-native-mcp | All six Google Workspace services share normal MCP registration with per-service authorization, complete pinned catalogs and local official-provider icons. | 0 | yes |
hosted-deploy | Hosted staging has one concurrent-recreate deploy path, while normal control-plane shutdown unregisters projection-reload membership and crash leases remain the recovery boundary. | 2 | no |
hosted-domain-signup-routing | Verified signup applies exact-domain routing policy, explicit invitation precedence, installation restrictions, and durable shared trial behavior. | 0 | yes |
hosted-workspace-membership-scope | Multiple active memberships use explicit workspace context for browser authority while preserving session-local selection and management-token scope. | 0 | yes |
m1-discovery-endpoint | M1 discovery endpoint returns only policy-authorized tools from registered visible servers and omits unauthorized tools from the response. | 4 | no |
m1-first-governed-call | M1 first governed MCP call has runtime request/response, manifest, and audit contracts without secret material. | 4 | no |
m1-governed-call | M1 governed MCP call fixture covers unregistered backend denial, policy-filtered discovery, allowed tool execution, and audit evidence. | 27 | no |
m1-policy-core | M1 Cedar policy core default-denies missing policy and missing allow, filters discovery, and evaluates allowed and denied tool execution with reason codes and policy version. | 13 | no |
m2-agent-registry | M2 agent registry validates approved agent definitions, typed runtime instance identity, and fail-closed disabled or unapproved agents. | 3 | no |
m2-identity-policy-revocation | M2 contract baseline covers OIDC/JWT validation, actor chain attribution, agent/client-surface authorization, policy versioning, simulation, explicit deny diagnosis, audit, and revocation. | 9 | no |
m3-registry-catalog-approval-health-audit | M3 registry and catalog-lite contract baseline proves owner submit, platform approve, developer discovery, unauthorized capability hiding, health, change history, and admin audit. | 9 | no |
m4-credential-broker-private-connectivity | M4 contract baseline for credential brokering, private routing, connector health, redaction, disable/revoke, and sandbox verification scope. | 11 | no |
m4-credential-redaction-wrappers | M4 credential redaction wrappers keep credential material out of logs, traces, metrics, audit events, errors, fixtures, snapshots, prompts, CLI responses, and admin responses. | 5 | no |
m5-session-lifecycle-affinity-revocation-audit | M5 session lifecycle contracts cover create, reattach, reconnect, drain, terminate, revoke, affinity, durable metadata, and secret-free audit evidence. | 8 | no |
m7-admin-workflow-cli-parity | M7 admin workflow and CLI parity completion contract for platform and security users completing V1 work through the admin API and machine-readable CLI without database access or custom scripts. | 8 | no |
managed-cached-policy-outage | Managed data plane enforces cached policy during a bounded control-plane outage, then fails closed. | 1 | no |
managed-deployment-model | V1 exposes managed SaaS and fully self-hosted deployment modes with explicit managed egress and bounded cached-policy continuity. | 5 | no |
managed-runtime-admission-api | Managed-only backend plan, usage, invocation-status, assignment, and support-credit API contract with server-owned scope and commercial authority. | 8 | no |
managed-runtime-admission-fleet-compatibility | Executable Phase 5 contract for byte-stable v2/v3 projections, Kubernetes-authoritative fleet membership and routing withdrawal, projected workload authentication and replay-safe heartbeats, least-privilege database ingress, independent projection/settlement cutoffs, current-state readiness, rollout safety, and self-hosted isolation. | 17 | no |
managed-runtime-admission-fleet-http-abi | Executable contract for the four production-composed managed-only Fleet HTTP operations, their committed SQL authority, exact schemas and headers, replay boundary, closed errors, generated names, compatibility, and bounded isolation claim. | 7 | no |
managed-runtime-admission-supervision | Executable production supervision contract for managed runtime admission deadlines, reservations, cancellation, recovery checkpoints, settlement ambiguity, abort quarantine, bounded queues, and fenced lease authority. | 15 | no |
managed-runtime-admission-valkey | Executable Phase 6 contract for managed Valkey scope and keys, canonical hashes, complete RESP2 function ABI, exact rate math, reconciliation, DB0-confined data-plane and global executor ACLs, compatibility lanes, and proof boundaries. | 11 | no |
managed-runtime-admission | Executable contract for production-composed managed runtime admission ordering, counting, modes, invocation identity, and error envelopes. Environment-scale evidence remains an operational gate. | 9 | no |
managed-runtime-projection-wiring | Managed data planes fetch scoped operational projections. Self-hosted installs remain file-backed by default and may explicitly enable authenticated CP publication with durable scoped audit authority. Historical audit delivery never grants current readiness. | 9 | no |
management-mcp-parity | Every Admin API operation receives explicit management exposure review; MCP, operation API and live CLI share the generated operation catalog and preserve the human/operator authorization boundary. | 3 | no |
mcp-155-onboarding-approval-workflows | UI-3 onboarding and approval workflow contract for Demo A private MCP server governance and Demo B OpenAPI-to-MCP approval, including agent registration, audit events, catalog visibility, disable touchpoints, role denial, and browser evidence expectations. | 6 | no |
mcp-156-policy-editor-simulator-deny-diagnostics | UI-4 policy editor, policy simulator, version traceability, publish/rollback impact preview, deny diagnostics, CLI parity, and metadata-only evidence contract. | 6 | no |
mcp-157-credentials-connectors-sessions-audit-emergency-response | UI-5 operational security surface for credentials, connectors, active sessions, audit investigation, emergency disable, Demo C, and Demo A disable/session continuity. | 6 | no |
mcp-158-enterprise-readiness-topology-rbac-accessibility-hardening | UI-6 enterprise readiness surface for topology clarity, SIEM/export, telemetry, backup/restore, upgrade/drain, license/artifact status, RBAC, accessibility, production states, documentation links, CLI parity, and evidence. | 6 | no |
mcp-164-stateful-onboarding-workflows | Stateful SaaS onboarding, import, operation selection, reviewer approval, batch decision, timeline, and metadata-only UI contract for MCP server registration, API source import, and agent registration. | 5 | no |
mcp-18-actor-chain-client-surface | MCP-18 resolves approved client-surface identity, typed actor chains, and human delegator context before policy and audit. | 4 | no |
mcp-20-policy-version-explicit-deny | M2 policy decisions carry durable policy versions and explicit deny overrides allow across actor, surface, environment, API source, and tool input dimensions while default deny remains distinct. | 15 | no |
mcp-204-connector-registry-read-model | Connector registry list/detail read model exposes scoped, paginated, metadata-only UI and CLI contracts without client-side joins or private backend leakage. | 7 | no |
mcp-205-client-surfaces-management | Canonical client-surface management backend contract for list/detail, policy simulator and agent refs, safe disable/revoke, session impact, CLI parity, and metadata-only output. | 6 | no |
mcp-206-data-planes-topology-health | Backend data-plane topology and health read model for list/detail API, shared deployment-status aggregation, authenticated customer-safe aggregate health, CLI parity, trust boundary, and metadata-only diagnostics. | 6 | no |
mcp-207-deny-diagnostics-bundle | Backend deny diagnostics bundle endpoint and CLI parity for metadata-only remediation-ready denied request explanations. | 6 | no |
mcp-208-admin-activity-read-model | MCP-208 Admin Activity read model exposes curated metadata-only admin actions for UI list/detail, permission gates, OpenAPI, and gatewayctl JSON parity. | 7 | no |
mcp-209-environments-settings-read-model | Read-only tenant environment settings list and detail API contract with fail-closed CLI access. | 5 | no |
mcp-21-policy-simulation-api | Policy simulation API previews V1 gateway policy decisions without mutating runtime state or exposing secret material. | 4 | no |
mcp-210-telemetry-siem-webhooks-admin-contract | Telemetry remains fail-closed while SIEM exposes metadata-only configuration, observed worker delivery, and bounded recovery controls. | 1 | no |
mcp-211-license-entitlement-read-model | Backend-owned License settings read model with safe entitlement metadata, deployment summary, CLI parity, permission gates, and no raw license material. | 5 | no |
mcp-214-hosted-email-user-management | Hosted-only transactional email and hosted user-management contract for verification challenges, invites, lifecycle notifications, Cloudflare Email Sending delivery-plan shape, and self-hosted separation. | 16 | no |
mcp-215-hosted-runtime-allowlisted-endpoints | Hosted-only runtime endpoint contract for customer-owned HTTPS API and HTTPS MCP endpoints, DAC egress allowlisting, reachability fail-closed behavior, metadata-only runtime decisions, and self-hosted separation. | 12 | no |
mcp-216-hosted-audit-payload-capture-operator-controls | Hosted-only audit, payload-capture, TTL purge, hosted incident/operator controls, abuse-control posture, metadata-only runtime audit, and self-hosted separation. | 13 | no |
mcp-218-hosted-saas-qa-closeout | Hosted SaaS QA1 closeout contract proving docs, evals, E2E smoke, child PR/check audit, no-secret evidence, and self-hosted regression coverage across R1-R44, F1-F7, AE1-AE10, and success criteria. | 7 | no |
mcp-22-policy-simulation-cli | gatewayctl policy simulation accepts M2 actor, surface, environment, server, and tool context and emits stable deny diagnosis without secret material. | 4 | no |
mcp-221-hosted-org-trial-lifecycle | Hosted-only org trial lifecycle contract for verified signup, trial expiry hard-lock, upgrade request, DAC operator actions, and self-hosted separation. | 6 | no |
mcp-222-hosted-domain-membership | Hosted-only same-domain membership contract for provisional domain association, join requests, owner/security-admin review, public-domain no-grouping, DAC separate-workspace exceptions, deferred DNS/SSO claims, and self-hosted separation. | 8 | no |
mcp-223-hosted-be1-integration-closeout | BE1 closeout contract proving hosted plan entitlements, trial hard-lock, domain membership, same-domain join requests, public-domain no-grouping, DAC exceptions, deferred DNS/SSO claim, downstream boundaries, and self-hosted separation work together. | 6 | no |
mcp-227-hosted-saas-secret-handling | Hosted SaaS OPS2 contract for Hetzner pilot secret source selection, reference-only Helm wiring, self-hosted guardrails, and no-secret artifacts. | 6 | no |
mcp-229-hosted-saas-ops4-deploy-stack | Hosted SaaS OPS4 deploy contract for Hetzner k3s substrate, Helm-hosted stack resources, first-admin bootstrap, rollback, upgrade, and self-hosted separation. | 6 | no |
mcp-230-hosted-saas-ops5-networking-egress | Hosted SaaS OPS5 contract for public ingress, DNS, TLS, DAC egress identity, customer allowlisting, private runtime isolation, connector deferral, and sanitized live evidence. | 8 | no |
mcp-231-hosted-saas-ops6-observability-backups | Hosted SaaS OPS6 contract for hosted observability, alerts, backups, restore drill, exposure checks, retention posture, incident runbooks, and re-runnable launch checklist. | 9 | no |
mcp-234-hosted-saas-ops8-live-profile-smoke-access | Hosted SaaS OPS8 parent contract for live Helm profile digest pinning, guarded smoke access, sanitized live baseline, and no-go evidence until DNS/TLS/LB/egress/secret projection are ready. | 6 | no |
mcp-235-hosted-release-images-profile | Hosted SaaS OPS8A release image and deployable hosted profile contract for GHCR images, SBOM/provenance/signing, Kubernetes bind envs, and self-hosted separation. | 8 | no |
mcp-236-hosted-public-smoke-api | Hosted-only public HTTP smoke API surface for MCP-233 live hosted signup, verification, trial, same-domain join, runtime readiness, metadata audit posture, expiry hard-lock, and self-hosted disabled responses. | 10 | no |
mcp-237-hosted-saas-ops8c-edge-readiness | Hosted SaaS OPS8C contract for DNS/TLS/secret projection, Hetzner LB target health, DAC egress identity publication, smoke access, rollback, destroy readiness, and sanitized blocker evidence. | 7 | no |
mcp-241-hosted-account-entry-baseline | ACCT0 hosted account-entry baseline contract map for signup, login, email confirmation, social sign-in, one workspace, same-domain join, invites, role selection, logout, first-run owner setup, no-secret evidence, and self-hosted separation. | 1 | yes |
mcp-243-hosted-auth-session-lifecycle | Hosted SaaS ACCT2 email-password signup, confirmation, restricted pre-activation sessions, active-session identity, logout, password reset, password change, and self-hosted separation. | 4 | no |
mcp-245-hosted-account-routing-admin | Hosted SaaS normal work-domain signup creates an independent owner workspace regardless of domain routing policy; explicit join admin decisions and recipient-bound owner invitations remain available. Public-domain owner blocking, neutral conflicts, notifications, and self-hosted separation remain enforced. | 4 | no |
mcp-247-hosted-account-settings-security-ui | Hosted SaaS ACCT6 workspace settings, access review, invite management, combined Profile & security, role selection, and self-hosted separation coverage. | 4 | no |
mcp-249-hosted-social-auth | Hosted-only social signup/login with real Google and GitHub redirect/callback, reference-only provider config, state/nonce protection, verified-email linking, callback UI states, app-only logout, and self-hosted separation. | 8 | no |
mcp-250-hosted-account-entry-closeout | Hosted SaaS ACCT9 closeout for account-entry OpenAPI contracts, eval coverage, docs, browser evidence, no-secret evidence, and self-hosted non-regression. | 8 | no |
mcp-253-hosted-governance-profile-resolver | Hosted SaaS GOV1 governance-profile resolver contract for typed profile presets, Simple verified-owner MCP and credential activation, higher-profile distinct-review metadata, hard safety gates, and self-hosted strict default semantics. | 5 | no |
mcp-254-hosted-activation-policy-settings | Hosted SaaS GOV2 activation policy settings persistence, hosted Simple owner-activation semantics, self-hosted strict separation, admin mutation acknowledgement, and audit-safe metadata. | 5 | no |
mcp-255-hosted-governance-transition-wiring | Hosted SaaS GOV3 resource transition wiring uses current governance decisions for Simple owner MCP and credential activation with scoped ownership, audited compare-and-swap, hard gates, and higher-profile distinct review. | 4 | no |
mcp-256-hosted-governance-contracts-client-dtos | Hosted SaaS GOV4 contract, eval, and generated client DTO coverage for activation-policy decision metadata on resource read surfaces. | 6 | no |
mcp-258-hosted-enterprise-identity-mapping | Hosted SaaS GOV6 keeps enterprise OIDC/SAML group-claim mapping as an optional extension point while preserving normal hosted local/social account entry and governance-profile separation. | 6 | no |
mcp-259-hosted-governance-closeout | Hosted SaaS GOV7 terminal closeout for governance-profile docs, browser evidence, no-secret review, self-hosted non-regression, and hosted deploy gate honesty. | 5 | no |
mcp-260-hosted-canonical-domain-reset | Hosted SaaS canonical-domain, automated TLS, and destructive reset contract for app.amelfi.ai. | 6 | no |
mcp-27-registry-lifecycle | MCP-27 registry lifecycle foundation uses typed submitted, under_review, approved, rejected, disabled, and archived states with owner/reviewer metadata, fail-closed invalid transitions, durable change history, and shared catalog/audit projections. | 5 | no |
mcp-28-server-registry-api | MCP-28 server registry API accepts owner-submitted MCP server manifests, rejects unsafe submissions with V1-safe reasons, returns registry metadata, and gates catalog visibility on platform approval. | 5 | no |
mcp-30-registry-admin-audit-events | MCP-30 registry admin audit evidence covers M3 submit, approval, rejection, disable, archive, ownership, validation, health, and catalog visibility lifecycle actions without payload material. | 4 | no |
mcp-31-server-health-checks | MCP-31 proves registered MCP server health checks use typed safe status metadata and can warn or hide catalog-lite capabilities without leaking secrets. | 1 | no |
mcp-32-catalog-lite-query-api | Catalog-lite query API returns only approved capabilities visible to the actor context and omits unauthorized, unapproved, disabled, unhealthy-blocked, or unselected capabilities. | 3 | no |
mcp-38-credential-broker-mock | MCP-38 deterministic credential broker trait and mock broker outcomes. | 4 | no |
mcp-41-vault-compatible-credential-broker-adapter | Vault-compatible M4 credential broker adapter maps approved bindings to Vault KV paths, returns safe failures, and never emits raw credential material. | 5 | no |
mcp-42-direct-private-endpoint-routing | MCP-42 direct private endpoint routing for governed tool calls, including fail-closed denied targets, safe route audit, and SSRF/host guardrails. | 6 | no |
mcp-43-outbound-connector-model-health | MCP-43 outbound connector records use typed governance metadata, safe health failures, scoped catalog visibility, and route-mode evidence without becoming a connector marketplace. | 5 | no |
mcp-44-connector-disable-revoke-audit | MCP-44 connector and credential binding disable/revoke decisions fail closed before upstream calls and emit safe audit evidence that composes with revocation and policy simulation. | 5 | no |
mcp-50-session-state-machine-durable-metadata | MCP-50 models typed client/backend session IDs, explicit state transitions, PostgreSQL durable metadata, expiry/idle handling, and fail-closed invalid transitions. | 5 | no |
mcp-51-valkey-active-session-index-affinity-routing | MCP-51 proves the Valkey hot active-session index supports safe lookup, affinity route hits, stale-state fail-closed behavior, drain/revoke fanout lookup, and Valkey-unavailable denial. | 6 | no |
mcp-52-reconnect-drain-terminate | MCP-52 proves reconnect is gated by eligible state and backend resume support, drain stops new stateful sessions and handles timeout/complete paths, terminate updates durable and hot state, and bounded buffers plus lifecycle hooks are secret-free. | 7 | no |
mcp-53-active-session-revocation-fanout | MCP-53 proves M2 revocation outbox events fan out through the V1 PostgreSQL internal queue into M5 active-session termination, blocked new calls, duplicate replay safety, and secret-free observable failures. | 6 | no |
mcp-54-session-lifecycle-audit-evidence | MCP-54 proves session lifecycle audit evidence records create, reattach, reconnect, drain, terminate, revoke, expiry, and denied lifecycle operations with safe context, request/correlation IDs, machine-readable reasons, and no secret material. | 5 | no |
mcp-60-openapi-parser-runtime | M6 runtime implementation parses approved OpenAPI specs, publishes only selected approved operations, and denies unsafe adapter calls before upstream. | 1 | no |
mcp-82-self-hosted-readiness-sandbox-matrix | M8 baseline contract for design-partner self-hosted and managed readiness, including the sandbox evidence matrix for install, governed call, integrations, no-outbound behavior, backup/restore, upgrade/drain, Cosign, SBOM, and release evidence bundles. | 6 | no |
mcp-83-helm-values-install-diagnostics | M8 local Helm hardening contract for values schema validation, required V1 install diagnostics, rendered resources, and sandbox smoke deferral. | 8 | no |
mcp-84-self-hosted-hybrid-config | M8 follow-on contract for self-hosted dependency configuration and managed runtime-continuity surfaces, including PostgreSQL, Valkey, OIDC/SAML, secret manager, OTel, SIEM/export, stable diagnostics, and non-V1 dependency guardrails. | 5 | no |
mcp-85-runtime-controls | M8 runtime-control contract for deterministic network policy, resource requests/limits, HPA/PDB, readiness, and no-outbound runtime checks. | 6 | no |
mcp-86-backup-restore-upgrade-drain-workflows | M8 backup/restore and upgrade/drain workflow evidence contract for self-hosted state, including deterministic local checks, Tier 4 sandbox scenarios, failure modes, rollback boundaries, and metadata-only artifact rules. | 7 | no |
mcp-87-release-evidence-bundle-gates | M8 release evidence gate for deterministic Cosign, SBOM, provenance, self-hosted readiness linkage, and secret-free release evidence bundle validation. | 6 | no |
mcp-88-self-hosted-hybrid-sandbox-conformance | Historical M8 self-hosted sandbox conformance evidence from the former hybrid model; retained for release traceability and no-secret proof. | 8 | no |
mcp-92-release-readiness-matrix | M9 V1 release-readiness matrix and sandbox plan for PRD sections 22 and 23, including M1-M8 evidence, downstream owner issues, validation commands, Sprite reuse-before-new, V1 boundaries, and no-secret artifact rules. | 7 | no |
mcp-93-release-candidate-smoke | M9 release-candidate smoke harness that replays M1-M8 evidence for governed MCP calls, generated API tools, policy simulation, credential and connector fail-closed behavior, session lifecycle, audit/SIEM export, and admin/CLI readiness while emitting a metadata-only JSON report. | 6 | no |
mcp-95-release-security-no-secret-review-gates | M9 release security gate for deterministic no-secret scans, runtime surface payload review, release evidence artifacts, sandbox/PR evidence, reviewer checklist, and V1 boundaries. | 6 | no |
mcp-96-release-candidate-evidence | M9 release-candidate package evidence gate for versioned RC bundle metadata, M8 compatibility, Helm digest, Cosign, SBOM, provenance, backup/restore, upgrade/drain, rollback, release notes, checksum manifest, and no-secret artifact rules. | 7 | no |
mcp-oauth-discovery-connection | Registration discovers MCP tools with a short-lived person-bound OAuth connection; runtime approval remains independent. | 0 | yes |
mcp-protocol-compatibility-engine | Enterprise MCP compatibility engine contract for dual-era protocol mediation, configurable bindings and routes, safe cleartext profiles, extension policy, route-local probing, and fail-closed runtime resolution. | 11 | no |
mcp-server-live-reprobe | An authorized registry owner can re-probe only the original approved registered immutable live MCP snapshot, with strict secret-safe endpoint validation, public-network transport guards with connection fallback across the complete validated DNS address set, cross-replica compare-and-swap health persistence that rejects changed configuration, metadata-only audit, exact hosted scope, shared bearer normalization and forbidden transport-header rejection, authorized cross-owner credential revisions that preserve owner metadata and the authenticated submitter, reviewer probes limited to the scoped shared publication candidate while arbitrary discovery remains creator-scoped, and conditional rollback on audit failure. | 6 | no |
mcp-server-registry-immutable-snapshots | MCP server registry submissions are immutable, approval is submission/hash scoped, runtime reads the live approved snapshot, Versions lists approved snapshots only, and compare uses backend snapshot IDs. | 11 | no |
mcp-server-registry-list-backend | MCP server registry list and detail backend returns UI-ready immutable snapshot/submission metadata, permissions, activity, compare, and action endpoints without pushing approvals queue into this slice. | 8 | no |
mcp-server-registry-read-api | MCP server registry read APIs expose UI-ready list/detail/activity/compare contracts without local manifest diffing. | 9 | no |
native-signing-key-rollover | Native issuer public keys overlap during deployment-controlled signing-key rotation without extending token lifetime or admitting an unknown key. | 0 | yes |
nila-support-authority | Support trusts the received mailbox supplied by the exact OAuth workload client, resolves existing account/member authority, and enforces workspace opt-in, immutable message/case binding, permissions and scope without customer ownership confirmation. | 2 | no |
oauth-resource-onboarding | Authoritative prepublication OAuth resource and explicit endpoint audience onboarding across activation, registration and detail. | 0 | yes |
openapi-server-registration | OpenAPI registration pins selected mappings; publication checks HTTPS reachability, static-header and OAuth application bindings use the API origin, and v4 projects adapters while preserving MCP bytes. OAuth import regressions cover provider bindings, credential safety and generic client-credentials prefill from applicable unambiguous security metadata. | 1 | no |
outbound-delivery-stage1-contract-freeze | Stage 1 freezes the shared outbound delivery and SIEM architecture before shared storage or product implementation begins. | 1 | no |
outbound-delivery-stage2-security-foundation | Stage 2 provides shared endpoint and workload-scoped target security without a product event store or delivery queue. | 1 | no |
outbound-delivery-stage3-worker-kernel | Stage 3 proves shared delivery mechanics through separate test-only SIEM and webhook repositories, a live PostgreSQL database, and a strict TLS receiver. | 1 | no |
outbound-delivery-stage4-siem-hmac | Stage 4 proves the first production SIEM workload from a committed audit row through atomic PostgreSQL fan-out, the dedicated SIEM worker, and strict receiver evidence. | 1 | no |
outbound-delivery-stage5-generic-webhooks | Current-main integration preserves canonical client-surface receipts, atomic policy audits, and hosted credential authority while Stage 5 proves a narrow catalog of transactionally published control plane events through activation-position fan-out, the dedicated generic worker, CloudEvents HMAC, and strict receiver evidence. | 1 | no |
outbound-delivery-stage6-splunk-hec | Stage 6 proves SIEM-only Splunk HEC delivery with explicit accepted and indexed evidence, durable acknowledgment takeover, safe history, and bounded operator re-drive. | 1 | no |
outbound-delivery-stage7-replay-lifecycle | Stage 7 proves bounded SIEM replay, live workload priority, webhook lifecycle operations, independent retention, storage admission, and PostgreSQL recovery without adding a broker or generic replay. | 1 | no |
per-request-body-capture | Request-body selection cannot enable global capture and never suppresses eligible response capture. | 0 | yes |
policy-intent-authoring | Intent-driven policy authoring, impact review, publish readiness, runtime projection, governed testing, and audit correlation. | 5 | no |
policy-management-m1-backend | Policy Management M1 backend contract for list, detail, draft update, validation, publish, archive, permissions, audit, and metadata-only policy bodies. | 10 | no |
policy-ref-resolver-backend | Policy refs in MCP server tools resolve to same-tenant, same-environment policy_version records and are runtime-valid only when published. | 6 | no |
policy-simulation | Policy simulation previews allow outcomes before publishing. | 2 | no |
production-auth-boundary-be | Production admin/runtime auth boundary supports trusted proxy and direct OIDC JWT modes, exposes safe provider diagnostics, and manages gateway role bindings without raw token, SAML assertion, secret, or full claim payload leakage. | 6 | no |
production-auth-local-identity-be | Production local identity mode supports bootstrap admin, local users and groups, browser sessions, service-account tokens, and role-binding evaluation without trusting gateway headers or exposing password/token hashes. | 2 | no |
public-endpoint-access | Explicit published public MCP bindings authorize anonymous endpoint access while private bindings remain protected. | 0 | yes |
registered-backend-routing | Gateway denies unregistered MCP backend routing. | 5 | no |
revocation-event-model | M2 revocation event model uses typed subjects, PostgreSQL outbox semantics, fail-closed call denial, session revoke handling, and secret-free audit. | 5 | no |
runtime-archived-server-projection | Archived routes leave runtime policy and credentials while pending registrations preserve existing live routes. | 0 | yes |
self-hosted-no-outbound-runtime | Self-hosted mode has no required external vendor runtime dependency. | 1 | no |
session-affinity-externalized | Stateful sessions have affinity and metadata outside worker memory. | 2 | no |
session-drain-backpressure | Drain is explicit and streaming buffers are bounded. | 2 | no |
session-revocation | Agent revocation blocks new calls and terminates affected active sessions. | 3 | no |
staging-runtime-oauth-recovery | Managed runtime startup tolerates the documented Fleet convergence window, projection reload reconciliation preserves monotonic row timestamps, and hosted UI failures avoid local-development advice. | 1 | no |
stdio-sse-runtime-projection-contracts | Foundation contracts for transport-aware registered MCP servers, CP-published runtime projections, schema passthrough, lifecycle reload state, and metadata-only runtime audit export. | 14 | no |
telemetry-customer-controlled | Telemetry is disabled or exported only to customer-controlled systems by default. | 1 | no |
tenant-environment-isolation | Dev, staging, and production policy/routing boundaries are isolated. | 2 | no |
tool-classification-removal | Tool execution uses explicit policy and per-invocation duplicate protection without gateway-owned side-effect classification. | 0 | yes |
tool-test-payload-capture | Console tool responses use governed capture while calls retain metadata diagnostics when capture is off. | 0 | yes |
upstream-connection-client-use | One stable user-owned upstream account with explicit independent per-client and tool use consent, atomic replacement, refresh and revocation. | 0 | yes |
upstream-oauth-browser-return | A verified upstream OAuth callback returns a console tab to its originating server or registration flow without exposing OAuth material or replaying a tool. | 0 | yes |
upstream-oauth-dcr-optional-management | DCR accepts core public-client credentials without interpreting unusable optional management metadata when no management token exists. | 0 | yes |
upstream-oauth-existing-application | Pre-registered authorization-code applications without DCR use encrypted client custody; denied callbacks preserve browser binding and terminate once. | 0 | yes |
upstream-oauth-grant-lifecycle | Reconnect atomically replaces same-authority grants; refresh retries only known pre-send failures and preserves fail-closed revocation. | 0 | yes |
upstream-oauth-initialized-streamable | Resolved user grants authenticate each initialized Streamable HTTP phase without sharing upstream sessions or exposing reflected credential material, and managed Streamable HTTP accepts JSON-RPC responses framed as JSON or SSE. | 0 | yes |
upstream-workspace-browser-consent | Browser identity proof binds explicit workspace ownership to independently revocable client consent. | 0 | yes |
verified-consumer-policy-audit | Trusted upstream issuer and subject follow native authorization into explicit policy and durable audit without workspace membership. | 0 | yes |
workspace-external-identity | Exact issuer and subject links require active workspace ownership and explicit epoch transitions. Relinked identity use rejects missing or pre-activation JWT issuance times before cache/material access. Client consent expires per call even while upstream token and cache remain valid. | 0 | yes |
workspace-runtime-provisioning | Signup creates durable runtime intent; scoped provisioning survives worker replacement and never claims serving readiness without Fleet evidence. | 0 | yes |
Usage notes
Section titled “Usage notes”Unsupported expectation keys and missing metadata reasons fail validation. Choose an existing supported key or extend the harness with a meaningful behavioral check before adding a new assertion.
Source truth
Section titled “Source truth”- docs/evals/scenario-contract.md
- fixtures/evals/
- fixtures/evals/schema.json
- apps/docs/scripts/generate-reference-pages.mjs
- Read Quickstart for the first governed-call proof.
- Read Source Map for docs coverage status.
- Read Security Review for no-secret evidence rules.
Amelfi documentation · Space Grotesk / Geist Mono