inferplane

Documentation (English) · 한국어 문서 · Quickstart · Deployment profiles · Production readiness

inferplane — set the policy; let the data plane enforce it.

A governance plane for coding agents, with policy administration separated from inference traffic. Configure model access, sensitive-data handling, budget limits and approved routing choices. mayu applies them on requests and provider attempts; inferplaned distributes policy and budget authority without carrying prompts or response streams.

Enabled policies turn operating rules into request-time decisions: route detected PII to approved internal models, mask and reinspect content, select a model using configured context signals, or switch to a lower-cost target at a budget threshold. Hard caps and privacy restrictions still constrain the result and its fallbacks. Operators supply and validate the rules, model metadata and processing boundaries; PII detection has finite coverage. Why inferplane · 한국어.

Control-plane HTTP is not called for per-request admission. Shared admission still depends on Postgres; finite authority, readiness/staleness gates and credential renewal have their own failure contracts. See deployment profiles.

mayu is a component name, not a project name — it holds the same position in inferplane that ztunnel/waypoint hold in Istio. It runs on localhost or on each Kubernetes node, speaks your coding agent's native protocol (Anthropic Messages, OpenAI Chat Completions/Responses, Bedrock InvokeModel), and enforces the rules the control plane (inferplaned) hands it: per-user attribution, budget cutoffs, and model routing.

Goals

1. Give Claude Code, OpenCode, and Codex¹ users a single entry point to Anthropic, Amazon Bedrock, and OpenAI-compatible (vLLM/Ollama/etc.) providers.
2. Let each user choose which model they talk to.
3. Support cost-driven model substitution — swap to a cheaper model (e.g. Sonnet → GLM) when cost, not just capability, decides.
4. Set spend limits per team and per individual, block on breach, and always show how much has been spent.
5. Keep control-plane HTTP off the inference path, with explicit outage limits: node-local authority is finite, shared admission requires Postgres, and readiness/staleness gates and hard-authority expiry still fail closed.

These are goals, not blanket completion or no-SPOF guarantees. See Deployment profiles and Current limits.

Target users

Enterprise platform/SRE teams governing coding-assistant LLM traffic across many developers and teams. The on-ramp stays bottom-up — a single team can run mayu standalone in minutes with no control plane (see Quick start) — but the intended growth path is a platform team adopting inferplaned to govern that traffic fleet-wide once more than one team is on it.

Non-goals

• Not an MCP gateway. Routing MCP traffic is already well served by Envoy AI Gateway / Higress; that's not where inferplane differentiates.
• Not competing on data-plane inference performance. The core is governance — attribution, budget, audit — not inference optimization.
• No embeddings, image, audio, or rerank support in v1. Chat/completions traffic only until that lane is proven (see docs/roadmap.md).

Policy-aware routing

Basic JSON parsing is required. Additional inspection is conditional on the ingress and effective rules; Responses performs compatibility inspection even without policy; native Bedrock Converse also always checks structure. Structure-only paths skip finite PII detectors; applicable privacy and verified masking retain full detection, including beside context Shadow. Learned classification and legacy masking are separate opt-ins. See inspection conditions.

GovernancePolicy.sensitiveData selects approved internal destinations, complete masking, or refusal before egress. Independent routing.context rules classify weak/simple, optional normal, and strong/complex tasks. Context starts in Shadow; explicit stability and Enforce support compatible multi-turn/tool workflows and retain the actual successful model/provider between related requests. Privacy always enforces, including during Shadow (ADR-043/044).

Strict budget tiers (enforceTargets: true) constrain every later selection and retry after a switching threshold. A separate total hard cap remains binding. The soft switching threshold does not become a smaller blocking admission cap. Legacy optional tiers retain their behavior. Legacy ADR-043 two-class context selection remains limited to eligible, completely inspectable single-user-turn requests without history/tools/media/reasoning/structured output. ADR-044's extended context/stability rules support compatible tool/history sessions; pins are bounded and local to one gateway, not fleet-wide session state.

Footnotes

1. Responses ingress and native/stateless adapters have local protocol tests and installed Codex CLI tool-round-trip tests, including Bedrock Converse. The model-aware launcher supplies model selection and context limits. Backend tool availability and native-only features remain explicit; see Codex setup. ↩

This is the beginning of the README. Read the rest at the original. Read the full README ↗