Skip to content

Secret-Manager Integration: Design Record

This note records the design that shipped for resolving credential references from a secret manager (issue #2413). The first release ships the reference scheme, a pluggable resolver trait, and one provider, HashiCorp Vault. AWS Secrets Manager, AWS KMS, and GCP Secret Manager resolvers remain future work.

Today

Secrets reach the router these ways:

  • Environment substitution: resolve_secret_value in src/infrastructure/config/secrets.rs expands ${VAR}, $VAR, and ${VAR:-default} inside config values.
  • Local files: a backend auth.key_file (for example a service-account JSON), and api_keys.api_keys_file for client keys.
  • Hub-managed backend credentials: on a Hub-enrolled router with backend sync enabled, Hub-authored backend snapshots carry a credential reference the Hub resolves and delivers (src/control_plane/backend_sync.rs, #1261). This path stays separate from the design below.
  • Secret-manager references: a credential field may hold secret://<path>#<field>, which the router resolves from the configured provider at config load and on hot reload. The provider is contacted only on those two events, never per request.

Shipped design

Configuration

secrets:
  provider: vault
  vault:
    address: https://vault.example.com
    token: ${VAULT_TOKEN}
    mount: secret
    kv_version: 2
    namespace: null
    timeout_seconds: 10
    ca_cert_file: null
  • The section is optional. Without it, and without any reference, the router makes no provider call and behaves exactly as before.
  • provider defaults to vault; that is the only provider implemented.
  • address must be an https URL. Plain http is accepted only for a loopback host (localhost, 127.0.0.1, ::1).
  • token may itself use ${VAR} substitution. It is masked in every Admin report and in Debug output.
  • mount defaults to secret and may not start or end with /, contain .. or whitespace, or contain \ or %. The last two follow the same URL-normalization rule as the reference path.
  • kv_version is 1 or 2. timeout_seconds is 1 to 120, defaulting to 10. namespace is sent as X-Vault-Namespace. ca_cert_file adds a PEM root for a private CA.
  • A malformed section is a load error, not a warning.

Reference grammar

  • The scheme marker is exactly secret://. A value is a reference only when the entire string starts with it, so Bearer secret://... stays literal.
  • The grammar is secret://<path>[#<field>]. <path> is relative to mount: non-empty, no whitespace or control characters, no leading /, no .. segment, and none of ?, \, or %. The last two are refused because URL normalization would turn a backslash into a path separator and a percent-encoded dot segment into .., moving the request outside the mount.
  • <field> selects a top-level key of the secret's data object and is required for Vault. The router does not guess the only field.
  • A resolved value must be a JSON string. A missing field, a non-string field, an empty path, or a field on a non-object is a hard error.
  • Vault KV v2 reads GET <address>/v1/<mount>/data/<path> and takes the field from data.data. KV v1 reads GET <address>/v1/<mount>/<path> and takes it from data. Both carry X-Vault-Token and, when configured, X-Vault-Namespace.

Which fields accept a reference

  • Only credential-bearing positions, determined by the same rules that mask configuration: api_key, secret, token, password, key_file, AWS and OAuth credential fields, redis.url, header maps, prompt_logging endpoints and patterns, alerting.webhook.url, and rate_limiting.scoped_bypass_keys[].key.
  • The api_keys section is special-cased: entry fields key, api_key, value, and secret resolve, while id, name, and other metadata do not.
  • A secret:// value in any other position is a load error naming the field path. It is never resolved and never left literal.
  • A reference inside the secrets section itself is a load error, because it would be a bootstrap cycle.

When resolution runs

  • During configuration load, after ${VAR} interpolation and before typed deserialization. Startup, hot reload, Admin history restore, and the CLI effective paths (config validate, config show --resolved, config diff) all resolve references.
  • Default config show, the MCP paths, and the masked configuration-history view keep references literal and do not contact the provider.
  • A final validation guard rejects a typed Config that still contains a string beginning with secret://. This catches Admin API, WebUI, and control-plane mutations that bypass the file loader.

Failure policy

  • Every failure is a load error naming the dotted field path and the reference path and field. The message never contains the resolved value. An example is backends[0].api_key: secret reference secret://openai#api_key could not be resolved: ....
  • An unresolvable reference never falls back to the literal string, so the router fails closed.
  • A failed reload keeps the previous configuration, so traffic keeps the last good credentials.
  • A reference with no secrets section is a load error.
  • The runtime logs a provider name and a resolution count, never values.

Masking

  • The masking rules already cover every position where a reference may appear. That is why a reference in a non-credential position is an error rather than a value that would pass through reports unmasked.
  • secrets.vault.token is masked by the existing token rule, while provider, address, and the other non-secret settings stay readable.
  • A resolved value never appears in logs, Debug output, Admin config GET or export, config reports, or config diff.

Rotation

  • Each load reads the secret again. There is no cache across loads and no TTL cache.
  • A rotated secret is visible on the next reload, without a restart.
  • The resolved value lives only in the live Config and is never written back to disk.

Build feature

  • secret-manager is a Cargo feature, included in full and in official release binaries, and it adds no crate dependency. The resolver uses the existing reqwest.
  • The secrets config type is always compiled, so a configuration file validates identically under any feature set.
  • A binary without the feature warns that the runtime is inert, and a secret:// value is still a hard load error.
  • The feature is absent from embed, whose dependency graph is unchanged.

Admin and MCP

  • The secrets section is startup-owned. PUT and PATCH /admin/config/secrets answer 409 STARTUP_OWNED_SECTION, and so do apply and rollback candidates that change the section.
  • Config export omits the section, and import keeps the live section when the candidate omits it.
  • The MCP hot-reload surface classifies secrets as requires_restart.
  • Admin API and WebUI mutations cannot set a reference, because the runtime API cannot resolve one. Edit the source configuration and reload it.

Remaining providers

  • AWS Secrets Manager, AWS KMS, and GCP Secret Manager resolvers are not implemented.
  • Adding one means implementing the SecretResolver trait and extending the provider enum and its factory. The reference scheme, the tree walker, the failure policy, and the masking rules stay unchanged.