Bluebelly — Market Research Backend
Bluebelly is the sole intended public and private presentation product. This repository retains the existing canonical research backend, including source ingestion, transcript studies, evidence validation, chart requirements, thesis updates and append-only research memory. Its legacy repository name, internal IDs, machine APIs and historical research contracts remain unchanged during migration.
See Research Architecture and Rollout.
See the Editorial-Brain Design Pack for the Original release ledger, story command desk, macro operating system, History Cabinet and judge-panel recommendation.
Scheduled research engine
The Live Desk has two canonical Asia/Kuala_Lumpur research cycles:
- 09:30 full desk update (
morning, 01:30 UTC)
- 21:30 evening delta update (
evening, 13:30 UTC)
The audited GitHub Actions workflow Run Live Research Now owns those full-desk cycles and invokes the existing Vercel production handlers through short-lived GitHub OIDC. Each canonical slot also has a 15-minute safety trigger (09:45 / 21:45 MYT) using the same daily scheduled identity, so it no-ops after a successful primary run instead of duplicating research.
Legacy Vercel research cron routes remain paused. vercel.json schedules only creator/video intake jobs; full Live research stays GitHub-owned until scheduler ownership is changed in a reviewed migration.
OpenAI market-intelligence runtime
When OPENAI_API_KEY is configured, the canonical research publisher hands validated evidence to the server-only OpenAI intelligence runtime. Caller-supplied Story recalibrations are no longer authoritative while this runtime is active.
The runtime follows the persisted intelligence contract:
validated research intake
→ canonical evidence + source ancestry
→ market belief
→ material divergence
→ competing hypotheses
→ Challenger audit
→ asset scenarios
→ original Story synthesis
→ semantic deduplication
→ Story lifecycle
→ canonical Live Story
→ Hybrid feed
The model never writes directly to stories. Every stage uses strict structured output and is recorded in intelligence_stage_runs with model, provider request ID and token usage. Research completeness, missing requirements, source depth, confidence and Challenger verdict describe a Story as SUPPORTED, DEVELOPING, CONTESTED or EARLY; all four states may publish. A candidate is suppressed only when it lacks usable traceable evidence or semantic deduplication finds no material new state.
A traceable material update may publish with limited corroboration while its research state remains EARLY or DEVELOPING. Creator commentary remains a research lead rather than independent proof. When the event, thesis and mechanism substantially match an existing Story, semantic deduplication prefers updating that Story rather than creating another one. Canonical Stories do not require an external article URL.
The authenticated /api/intelligence-run route can run the engine directly or expose sanitized operational status. It never returns stage payloads, transcript text or credentials.
Default model routing is cost-aware:
- GPT-5.6 Terra for hypotheses, Challenger, scenarios and Story synthesis.
- GPT-5.6 Luna for market-belief extraction, divergence detection, semantic deduplication and lifecycle classification.
- Medium reasoning for complex stages and low reasoning for fast stages.
All model choices and reasoning effort can be overridden with the server-only environment variables documented in .env.example.
TranscriptAPI pipeline
Transcript intake is database-first and uses TranscriptAPI v2 as the only transcript provider:
- Read
research_intake_items for a completed transcript cache entry.
- Call the free
/youtube/info endpoint to validate the video and discover languages.
- Call
/youtube/transcript with the selected language priority and timestamps enabled.
- Persist the flattened text, canonical timestamped segments, language, duration, provider metadata and retrieval time.
- On a required-video failure, preserve the error taxonomy on the intake row and upsert one open
research_debt record.
Successful rows are protected from later failure writes. Network failures, timeouts, rate limits and provider 5xx responses use at most three attempts with exponential backoff and Retry-After support.
A targeted retry for an existing intake row is available to authenticated operators:
curl -H "Authorization: Bearer $RESEARCH_UPDATE_TOKEN" \
"https://<deployment>/api/video-intake?videoId=yNiWeHGBl98"
The response contains operational metadata, never transcript text or credentials.
Environment
Copy .env.example and configure these server-side Vercel values:
SUPABASE_SERVICE_ROLE_KEYRESEARCH_UPDATE_TOKENYOUTUBE_DATA_API_KEYTRANSCRIPT_API_KEYGEMINI_API_KEY (optional public-YouTube summary rescue)OPENAI_API_KEYCRON_SECRET
OPENAI_INTELLIGENCE_ENABLED=false is the kill switch for the model reasoning layer. If no OpenAI key is configured, the existing legacy recalibration path remains available rather than breaking research ingestion.
When GEMINI_API_KEY is present, an eligible transcript-provider failure can trigger public-YouTube analysis with GEMINI_VIDEO_MODEL (default gemini-3.8-flash). The result is stored only as GEMINI_SUMMARY_ONLY: it is not a recovered transcript, canonical evidence, or a Market Motion input. The original transcript failure and retry schedule remain authoritative.
CRON_SECRET authorizes only Vercel Cron requests. VERCEL_AUTOMATION_BYPASS_SECRET is preview-only and exists solely for protected deployment verification.
Authorised maintainers can run the canonical production research pipeline from the
Run Live Research Now GitHub Actions workflow. The workflow uses a short-lived
GitHub OIDC identity scoped to the exact workflow on main; the production route
validates that identity, keeps CRON_SECRET inside Vercel, and calls the existing
scheduled acquisition and intelligence handlers with one audited retry identity.
No research credential is copied into the workflow or returned to the operator.
The publisher contract and runbook live in Research Update Engine.
Verification
npm test runs the TranscriptAPI error-taxonomy and cache/debt tests. npm run build runs those tests before the Next.js production build. The SQL contract at supabase/tests/transcriptapi_persistence_contract.sql validates the production persistence fields, provider constraint, cache index, schedule slots and view security.
