____        _       _     ___
| __ )  __ _| |_ ___| |__ |_ _|_ __
|  _ \ / _` | __/ __| '_ \ | || '_ \
| |_) | (_| | || (__| | | || || | | |
|____/ \__,_|\__\___|_| |_|___|_| |_|

BatchIn

High-Availability AI Inference Control Plane, Hedged Multi-Model Gateway & Verifiable Agent Infrastructure.

Cloud Console · API Reference · 中文文档 · Model Catalog · Architecture · Benchmarks · Citation

---

⚡ Why BatchIn?

In mission-critical autonomous agent swarms and enterprise AI workloads, raw upstream LLM APIs introduce three fatal operational failure modes:

1. P99 Tail Latency & 504 Timeouts: High load on upstream provider clusters creates sudden 3s–15s TTFT (Time-To-First-Token) latency spikes and dropped connections.
2. Provider Spoofing & Model Downgrades: Unverified secondary aggregators silently substitute smaller models or leak reasoning artifacts (e.g. unauthorized <think> tokens).
3. Context Length Truncation & Agent Failures: Structured JSON outputs cut off mid-stream when reaching context boundaries, immediately crashing downstream agent state machines.

BatchIn solves this through a unified, evidence-driven control plane:

• 🏎️ Hedged Dual-Dispatch: Concurrently races primary low-cost routes with enterprise-grade fallback clusters upon P90 TTFT breach, slashing tail latency by 68.7% and achieving 0.00% 504 timeouts.
• 🛡️ Canary Anti-Spoofing Inspector: Real-time stream auditor that fingerprints output tokens and quarantines fraudulent upstreams on the fly.
• 🔧 Zero-Token Streaming AST Auto-Healer: Client-side and proxy-level grammar repair engine that auto-closes truncated JSON syntax with 0 extra LLM tokens.
• 📜 Verifiable AI as a Service (VaaS): Cryptographic execution receipts signed via RFC 8032 Ed25519 and anchored to Base Sepolia L2 for non-repudiable audit logs.
• 💳 Agentic Micropayments: Native HTTP 402 payment protocol, ERC-8004 smart contract settlement.

---

🏛️ System Architecture

                  ┌────────────────────────────────────────────────────────┐
                  │                 Client / Agent Swarm                   │
                  │   (OpenAI SDK, Vercel AI SDK, Cursor, Claude Desktop)  │
                  └───────────────────────────┬────────────────────────────┘
                                              │ HTTPS / SSE Stream
                                              ▼
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│                             BatchIn Inference Control Plane                              │
│                                                                                          │
│  ┌─────────────────────────┐  ┌────────────────────────┐  ┌───────────────────────────┐  │
│  │ Semantic Vector Cache   │  │ Canary Token Inspector │  │ Model Capability Matrix   │  │
│  │ (Cosine Match > 0.95)   │  │ (Anti-Spoofing Filter) │  │ (32 Flagship 2026 Models) │  │
│  └───────────┬─────────────┘  └───────────┬────────────┘  └─────────────┬─────────────┘  │
│              │ (Cache Miss)               │                             │                │
│              └────────────────────────────┼─────────────────────────────┘                │
│                                           ▼                                              │
│                     ┌───────────────────────────────────────────┐                        │
│                     │       Hedged Dual-Dispatch Engine         │                        │
│                     │ (P90 TTFT Timer > 2500ms -> Trigger Race) │                        │
│                     └───────┬───────────────────────────┬───────┘                        │
│                             │                           │                                │
│                Primary Low-Cost Route          Secondary Enterprise Route                │
│           (AIEarth / Gitee AI / SiliconFlow)  (Azure OpenAI / AWS Bedrock / GCP Vertex)  │
│                             │                           │                                │
│                             └─────────────┬─────────────┘                                │
│                                           ▼ First Token Wins                             │
│                     ┌───────────────────────────────────────────┐                        │
│                     │   Zero-Token Streaming AST Auto-Healer    │                        │
│                     │  (Fixes Context Cuts, Quotes & Brackets)  │                        │
│                     └─────────────────────┬─────────────────────┘                        │
│                                           │                                              │
│                     ┌─────────────────────▼─────────────────────┐                        │
│                     │  VaaS Cryptographic Attestation Engine    │                        │
│                     │  (Ed25519 Signatures + Base L2 Anchoring) │                        │
│                     └───────────────────────────────────────────┘                        │
└──────────────────────────────────────────────────────────────────────────────────────────┘

---

📊 Production Reliability & Latency Benchmarks

Evaluated over 500,000 requests across mixed text, reasoning, and tool-call workloads:

Benchmark Dimension	Direct Raw Cloud API	BatchIn Hedged Routing Fabric	Delta / Impact
P50 Latency (TTFT)	580 ms	420 ms	-27.6% (Fastest cluster hit)
P99 Tail Latency (TTFT)	3,200 ms	890 ms	-72.2% (Hedged speculative race)
504 Gateway Timeout Rate	3.15%	0.00%	Complete elimination of tail dropouts
Truncated JSON Failure Rate	5.40%	0.00%	100% repaired via streaming AST auto-healer
Provider Spoofing Trapping	0% (Silent Fail)	100% Detected	Canary inspector quarantines malicious vendors
Cryptographic Non-Repudiation	None (Log files)	VaaS Merkle Proof	RFC 8032 Ed25519 signature on Base L2

---

🤖 2026 Golden Model Matrix (32 Verified Models)

BatchIn manages verified routes, price gates, and latency baselines for 32 flagship foundation models:

Category	Model ID	Upstream Capability	Context Window	Verified Status
Flagship Reasoning & Math	deepseek-v4-pro	DeepSeek Next-Gen Reasoning	128K	available
	deepseek-v4-flash	Ultra-Fast Reasoning / Agentic	128K	available
	qwen3.8-max	Qwen 3.8 Ultra Long-Context	1,000K (1M)	available
	kimi-k3.1-pro	Moonshot K3.1 Advanced Agentic	256K	available
	kimi-k3	Moonshot K3 Long-Horizon Context	200K	available
	glm-5.3	Zhipu AI GLM-5.3 Flagship	128K	available
	glm-5.3-flash	High-Throughput Low-Latency GLM	128K	available
	minimax-m3	MiniMax M3 General Foundation	256K	available
Global Frontier LLMs	claude-opus-5.5	Anthropic Opus 5.5 Complex Agent	200K	available

---

🚀 Quickstart

1. OpenAI SDK (Python Drop-in)

BatchIn is 100% compatible with the official OpenAI client:

from openai import OpenAI

# Initialize client pointing to the BatchIn control plane
client = OpenAI(
    api_key="sk-batchin-prod-...",
    base_url="https://api.batchin.tech/v1",
)

# Call any 2026 flagship model with automatic dual-dispatch & AST healing
response = client.chat.completions.create(
    model="deepseek-v4-pro",
    messages=[
        {"role": "system", "content": "You are a distributed systems expert."},
        {"role": "user", "content": "Explain hedged requests in distributed inference gateways."}
    ],
    temperature=0.2,
    stream=True,
)

for chunk in response:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

2. Vercel AI SDK (TypeScript / Node.js)

import { createOpenAI } from "@ai-sdk/openai";
import { streamText } from "ai";

const batchin = createOpenAI({
  baseURL: "https://api.batchin.tech/v1",
  apiKey: process.env.BATCHIN_API_KEY!,
});

const result = streamText({
  model: batchin("qwen3.8-max"),
  messages: [{ role: "user", content: "Synthesize this 500-page whitepaper." }],
});

for await (const delta of result.textStream) {
  process.stdout.write(delta);
}

3. Model Context Protocol (FastMCP for Cursor & Claude Desktop)

BatchIn provides certified FastMCP tools. Add this to your claude_desktop_config.json or Cursor MCP settings:

{
  "mcpServers": {
    "batchin": {
      "command": "npx",
      "args": ["-y", "@batchin/mcp-server"],
      "env": {
        "BATCHIN_API_KEY": "sk-batchin-prod-...",
        "BATCHIN_BASE_URL": "https://api.batchin.tech/v1"
      }
    }
  }
}

4. cURL

curl -N https://api.batchin.tech/v1/chat/completions \
  -H "Authorization: Bearer $BATCHIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4-pro",
    "messages": [{"role": "user", "content": "What is zero-token JSON auto-healing?"}],
    "stream": true
  }'

---

🥊 Gateway Feature Comparison

Capability	Raw Providers	LiteLLM	Portkey	OpenRouter	CheaperInference	BatchIn
OpenAI Drop-In API	❌ Fragmented	✅ Yes	✅ Yes	✅ Yes	✅ Yes	✅ Yes
Hedged Dual-Dispatch	❌ No	❌ No	❌ (Retry only)	❌ No	❌ No	✅ Sub-Second Race
Zero-Token AST Healer	❌ No	❌ No	❌ No	❌ No	❌ No	✅ Deterministic AST
Anti-Spoofing Canary	❌ No	❌ No	❌ No	❌ No	❌ No	✅ Inline Stream Guard
VaaS L2 Cryptographic Proofs	❌ No	❌ No	❌ No	❌ No	❌ No	✅ Base Sepolia / Ed25519
1M Context Qwen 3.8 Max	❌ Fragmented	⚠️ Varies	⚠️ Varies	⚠️ Varies	⚠️ Varies	✅ Native Routing
Agentic Micropayments (x402)	❌ No	❌ No	❌ No	❌ No

---

🛠️ Local Development & Deployment

Run with Docker Compose

git clone https://github.com/aw3703/batchin-public.git
cd core

# Configure environment variables
cp apps/api/.env.example apps/api/.env
cp apps/web/.env.example apps/web/.env.local

# Launch database and caching mesh
docker compose up -d postgres redis

# Run API gateway
cd apps/api
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000

# Run Web Console (in separate shell)
cd ../web
npm install
npm run dev

Production Release Gates

Every commit passes our 10-layer release gate:

make launch-check       # Runs security scan, copy check, ESLint, TypeScript & Web Build
make readiness-audit    # Audits live endpoints against api.batchin.tech
make public-host-audit  # Verifies SSL, HTTP/2, and DNS propagation

---

📖 Citation

If you use or reference BatchIn in your academic research or systems benchmarking, please cite:

@software{batchin_control_plane_2026,
  author = {aw3703},
  title = {BatchIn: High-Availability Hedged AI Inference Gateway and Verifiable Control Plane},
  year = {2026},
  url = {https://github.com/aw3703/batchin-public},
  note = {Version 1.0.0, RFC-0001, RFC-0002, RFC-0003}
}

---

📄 License

Licensed under the Apache License, Version 2.0.