Lumina Note

An AI voice notetaker with Google Sign-In, multilingual transcription and a 3D interactive interface.

---

Overview

Lumina Note records your voice — in French, English, Spanish or a dozen other languages — transcribes it server-side, and uses Google's Gemini models to structure it into clean notes. It works out whether you recorded a meeting, a lecture, a brainstorm or an interview, and produces the sections that fit, in the language you actually spoke.

Every account gets its own private space. Notes, transcripts and tags are scoped to the signed-in user and are never visible to anyone else.

---

Features

• Google Sign-In — one tap, no passwords. Notes are private per account.
• Records in any supported language — audio is captured with MediaRecorder, so it works in Chrome, Firefox and Safari, not only in browsers that ship the Web Speech API.
• Automatic language detection — leave the picker on Auto and the transcriber identifies the language itself; pick one explicitly for short or noisy recordings.
• Notes in your language — the summary, the bullets and even the section headings come back in the language of the recording. Section keys stay stable underneath so search and export keep working.
• Long recordings — up to three hours. Audio is captured in segments, converted to 16 kHz mono WAV and split at quiet moments into chunks, each transcribed with the tail of the previous one for consistent spelling. Nothing is held whole, so a three hour recording costs no more memory than a five minute one.
• Fail-safes throughout — model fallback chain, retries with backoff, a local extraction that still produces usable notes when the AI is down, retryable transcription, a downloadable copy of the audio, and a draft transcript that survives a page reload.
• Full transcript kept — stored alongside every note, searchable and exportable.
• Search, tag filtering, pagination
• Obsidian export — one note, or the whole library as a zip: properties, real #tags, the cited slide pages as attachments, the transcript in a folded callout, and the revision deck as cards for the Spaced Repetition plugin. Still plain Markdown, so any editor reads it.
• Claude can read your notes — a small MCP server searches and reads them with a personal token that can only read. See Connect Claude.
• 3D interactive scene — a floating notebook, a cursor-tracking pen and a paper airplane on note generation. Purely decorative: it is behind an error boundary and the app works without it.

---

Tech Stack

Frontend — React 19 (CRA + CRACO), Tailwind CSS, Framer Motion, Three.js, Axios, Google Identity Services.

Backend — FastAPI, Motor (async MongoDB), httpx, Pydantic v2, PyJWT, google-auth.

External — Google Gemini (transcription + structuring), MongoDB Atlas.

---

Architecture

┌──────────────────────────────────────────────┐
│                React Frontend                │
│  MediaRecorder ──► WAV 16 kHz mono ──► chunks│
│  Google Identity Services ──► ID token       │
│  Web Speech API ──► live captions (optional) │
└───────────────────┬──────────────────────────┘
                    │ HTTPS + Bearer token
┌───────────────────▼──────────────────────────┐
│                FastAPI Backend               │
│  POST /api/auth/google  verify ──► app JWT   │
│  POST /api/transcribe   audio ──► text+lang  │
│  POST /api/notes/summarize ──► sections      │
│       └─ model fallback ──► local extraction │
│  /api/notes  CRUD, scoped to the user        │
└───────────────────┬──────────────────────────┘
                    │
┌───────────────────▼──────────────────────────┐
│                MongoDB Atlas                 │
│  users : id, google_sub, email, name         │
│  notes : id, user_id, title, type, language, │
│          sections, labels, raw_transcript,   │
│          tags, source, duration, timestamps  │
└──────────────────────────────────────────────┘

---

Getting Started

Prerequisites

• Node.js 18+, Python 3.11+
• A Google AI Studio API key
• A MongoDB Atlas cluster
• A Google OAuth Web application client ID

1. Create the Google OAuth client

This is required — Google Sign-In cannot work without it, and the app has no other way in. It is free and takes a few minutes.

1. Go to Google Cloud Console and create a project (or pick an existing one).
2. APIs & Services → OAuth consent screen. Choose External, fill in the app name, your email as support and developer contact, and save. You do not need to submit it for verification: while it is in Testing, add your own Google account under Test users, or click Publish app to let anyone sign in. Nothing here needs a paid account.
3. APIs & Services → Credentials → Create credentials → OAuth client ID → Web application.
4. Under Authorised JavaScript origins, add every origin the frontend is served from — scheme and port included, no trailing slash:
   • http://localhost:3000 for local development
   • https://your-app.vercel.app (or your own domain) for production
5. Leave Authorised redirect URIs empty. Sign In With Google hands the credential back in the browser, so there is nothing to redirect to.
6. Copy the client ID — it looks like 1234567890-abc123.apps.googleusercontent.com — into GOOGLE_CLIENT_ID in backend/.env. It is not a secret; the frontend reads it from /api/config, so it only needs setting in one place.

If the sign-in button does not appear, the origin almost always does not match exactly. Origin changes can take a few minutes to propagate.

2. Backend

cd backend
pip install -r requirements.txt
cp .env.example .env     # then fill it in
python -m uvicorn server:app --reload

Required in backend/.env:

Variable	Purpose
MONGO_URL	MongoDB connection string
DB_NAME	Database name
GEMINI_API_KEY	Google AI Studio key
GOOGLE_CLIENT_ID	OAuth Web client ID
JWT_SECRET	Signs session tokens — set this in production
CORS_ORIGINS	Comma-separated frontend origins

.env.example documents the optional settings: model fallback order, rate limits, audio and transcript ceilings, and an email allow-list.

Two providers

Gemini is primary; Groq is tried for any window every Gemini model has failed on. That matters because a Gemini quota error takes out all Gemini models at once — the fallback chain within one provider is no help — so the honest next step is somewhere else entirely. It is decided window by window: when Gemini runs out halfway through a lecture, Groq writes the rest and the parts join up in order. A provider that reports a rate limit is rested for the rest of the pass, and windows every provider refused get one more try after WINDOW_RETRY_PAUSE seconds (60 by default), since most limits are per minute. Only a window that fails again is left out, and the notes say a stretch is missing. Generating them again redoes only that stretch: every window written is kept (window_notes, cleared once a run is complete, and after WINDOW_CACHE_TTL otherwise), so the quota is not spent twice on the rest. Local extraction stays as the last resort, but it produces visibly worse notes and says so in the UI.

GROQ_API_KEY_BACKUP is used when GROQ_API_KEY is rate limited. Groq counts limits per account, so it only helps if it comes from a different Groq account.

Groq covers both jobs: chat models for the notes, and Whisper for speech, which also gives the transcription path a way out when Gemini's audio quota runs dry. Set GROQ_API_KEY (free from console.groq.com) and it switches itself on.

Detail that does not thin out

Long transcripts are summarized window by window and merged, whichever provider answers.

Groq's free tier forces it — it refuses any single request over a few thousand tokens, and an hour of speech is well past that. Gemini does not force it: its context easily swallows a whole lecture. It is windowed anyway, because a model asked to summarize an hour in one pass covers the start and the end and thins out in the middle. On the same 78-minute lecture:

	one pass	windowed
Gemini	56 bullets, 6 sections	104 bullets, 8 sections
Groq	refused (413)	103 bullets, 8 sections

Two things keep it honest at any length. A window refused for its size is halved and requeued, so *_WINDOW_CHARS is a starting guess rather than a hard assumption — a language that packs more tokens into the same characters settles on a smaller window by itself, down to MIN_WINDOW_CHARS. A window that fails any other way is counted and skipped: losing one stretch beats discarding the rest, and the note is flagged partial so nobody mistakes it for complete. Rate limits are waited out rather than skipped, using the delay the service advertises, which is why summarizeTranscript has its own ten-minute client timeout.

Check the model chain occasionally. Google retires models without much notice, and a retired one answers 404 — which turns the fallback into a second failure instead of a rescue, exactly when the primary is rate-limited. List what your key can actually reach with curl "https://generativelanguage.googleapis.com/v1beta/models?key=$GEMINI_API_KEY", and keep the entries distinct: quota is counted per model, so a second entry only helps if it has its own budget.

3. Frontend

cd frontend
npm install --legacy-peer-deps
cp .env.example .env     # REACT_APP_BACKEND_URL
npm start

Open http://localhost:3000.

Microphone access requires a secure context. localhost counts; a plain-HTTP LAN address does not.

4. Tests

cd backend
pip install -r requirements-dev.txt
python -m pytest

cd ../frontend && CI=true npm test
cd ../worker && npm test
cd ../mcp-server && python -m pytest   # needs httpx and pytest-asyncio, from the backend's dev requirements

5. Transcription Worker (Cloudflare)

Audio is over 99% of the bytes the app moves. Sent through the Render backend, every chunk counted against Render's 5 GB of monthly outbound bandwidth, and running out suspended the whole backend. worker/ is a Cloudflare Worker that transcribes chunks instead. Its traffic is not metered, and the free plan's 100 MB body and 100,000 requests a day are far above what the app needs. The backend's /api/transcribe stays as the fallback. The app retries a chunk there only when the Worker itself fails (unreachable, crashing, misconfigured), not when every service is out of quota, since the backend holds the same keys.

Each chunk tries, in order: Gemini 3.5 Flash → 2.5 Flash → 3.1 Flash-Lite → Groq whisper-large-v3 → whisper-large-v3-turbo. Each has its own free quota; the free Gemini 3.5 Flash alone allows only 20 requests a day. A model that runs out is skipped for the rest of the lecture instead of being asked again for every chunk.

One-time setup (free Cloudflare account, no card):

cd worker
npm install
npx wrangler login                    # opens the browser once
npx wrangler secret put JWT_SECRET    # exactly the backend's JWT_SECRET
npx wrangler secret put GEMINI_API_KEY
npx wrangler secret put GROQ_API_KEY
# edit wrangler.toml: put your Vercel URL first in ALLOWED_ORIGINS
npx wrangler deploy                   # prints https://lumina-transcribe.<you>.workers.dev

Then in Vercel, set REACT_APP_TRANSCRIBE_URL to that URL and redeploy the frontend. It is read at build time. Without it the app uses the backend alone, as before. To check it's live: curl https://lumina-transcribe.<you>.workers.dev/health. To watch requests: npx wrangler tail.

For local work, put the same three secrets in worker/.dev.vars (git-ignored) and run npm run dev.

6. Connect Claude (MCP)

mcp-server/ lets Claude Desktop or Claude Code search and read your notes. It runs on your computer and talks to the backend with a personal API token. Tokens can only read: the backend refuses them on every route that creates, changes or deletes anything, and on the token routes themselves, so a token cannot make more tokens.

1. In the app, open the account menu → API tokens → Create. Copy the token (it is shown once) and the API URL under it.

2. Install the server's two dependencies in their own environment:

   python -m venv mcp-server/.venv
   mcp-server/.venv/Scripts/python -m pip install -r mcp-server/requirements.txt   # macOS/Linux: .venv/bin/python
   

   With uv you can skip this: uv run mcp-server/lumina_mcp.py installs them from the script's header.

3. Add it to Claude, using absolute paths.

   Claude Code

   claude mcp add --transport stdio --scope user lumina \
     --env LUMINA_API_URL=https://your-backend.onrender.com \
     --env LUMINA_TOKEN=lum_... \
     -- /full/path/to/ainote/mcp-server/.venv/Scripts/python /full/path/to/ainote/mcp-server/lumina_mcp.py
   

   Claude Desktop: add this to claude_desktop_config.json (Settings → Developer → Edit Config), then restart Claude Desktop.

   {
     "mcpServers": {
       "lumina": {
         "command": "C:\\full\\path\\to\\ainote\\mcp-server\\.venv\\Scripts\\python.exe",
         "args": ["C:\\full\\path\\to\\ainote\\mcp-server\\lumina_mcp.py"],
         "env": {
           "LUMINA_API_URL": "https://your-backend.onrender.com",
           "LUMINA_TOKEN": "lum_..."
         }
       }
     }
   }
   

Then ask things like "find my thermodynamics lectures from this month" or "quiz me on the last lecture". The tools:

Tool	What it returns
search_notes(query, tag, note_type, limit)	matching notes, newest first, with ids and a one-line preview
get_note(note_id, include_transcript)	the whole note as Markdown; the transcript only when asked, since it can run to 100k characters
list_tags()	every tag in use
get_flashcards(note_id)	the question and answer cards the app's revision deck uses

On the free Render plan the backend sleeps when idle, so the first call after a quiet spell can take up to a minute. The server waits 90 seconds, then says the backend is waking up. Revoke a token from the same menu if it leaks; it stops working at once.

---

Deployment notes

• Set JWT_SECRET. Without it the server generates a random secret at boot, so every restart signs everyone out.
• Python version on Render: runtime.txt is Heroku's convention and Render ignores it — it will build on whatever its current default is. To pin a version, set a PYTHON_VERSION environment variable in the Render dashboard, or add a .python-version file. The test suite is green on 3.11 through 3.13.
• Verify the dependency list in a clean environment before deploying. google-auth declares requests only as an extra, so import google.auth.transport.requests succeeded locally (where requests came in via something else) and failed on Render. python -m venv into a throwaway directory, pip install -r requirements.txt, then import server.
• CORS_ORIGINS must list your real frontend origin — no trailing slash.
• Health checks: point the platform at /api/health, which returns 200 whenever the process is alive. /api/health/ready returns 503 until the database and both credentials are usable — useful for a dashboard, but not as a liveness probe.
• Free-tier cold starts: the first request after an idle period can take 20–30 seconds. The frontend retries idempotent reads and shows a wake-up screen rather than an error.
• Stop the Atlas cluster pausing: a free M0 cluster is suspended after roughly 60 days with no connections, and only a human can resume it. .github/workflows/keepalive.yml pings /api/health twice a day, which opens a connection and resets that timer. Set a repository variable BACKEND_URL (Settings → Secrets and variables → Actions → Variables) to your API base URL, then run it once from the Actions tab to confirm. Two caveats: GitHub disables scheduled workflows on a repository with no activity for 60 days, so push something occasionally or re-enable it from the Actions tab; and if the cluster is already paused, this cannot wake it — resume it once in Atlas first. Changing the cron to */14 * * * * would also keep a Render free instance warm, at the cost of nearly all the monthly free instance hours.
• A paused or unreachable database returns 503 with a "may be waking up" message and a Try again button, rather than a generic failure.
• Gemini quotas: when the daily quota runs out, summarization falls back to a local extraction and the note is flagged in the UI. The transcript is never lost.

---

Project structure

backend/
├── server.py          # app wiring, routes, middleware
├── config.py          # environment settings
├── db.py              # Mongo client, indexes, legacy migration
├── auth.py            # Google token verification, app JWTs, read-only API tokens
├── gemini.py          # retries, model fallback, JSON recovery
├── transcription.py   # audio validation, chunking, speech-to-text
├── summarizer.py      # prompting, normalization, offline fallback
├── languages.py       # supported languages + offline detector
├── ratelimit.py       # per-user sliding window
├── models.py          # request/response schemas
├── retranscribe.py    # CLI: transcribe a file from disk, locally or via the API
└── test_*.py          # unit and API tests

frontend/src/
├── contexts/AuthContext.jsx     # Google Sign-In, session, boot config
├── hooks/
│   ├── useAudioRecorder.js      # MediaRecorder + level meter
│   └── useSpeechRecognition.js  # live captions (best-effort)
├── lib/
│   ├── audio.js                 # decode, resample, WAV, chunking
│   ├── notes.js                 # section shapes, labels, Markdown
│   ├── obsidian.js              # notes as Obsidian Markdown, file names, tags
│   ├── obsidianExport.js        # one note or the library, with slide pages
│   └── zip.js                   # a stored (uncompressed) zip writer
├── services/api.js              # axios instance, auth, retries
└── components/                  # Recorder, NoteOutput, NoteHistory, SignIn…

mcp-server/
├── lumina_mcp.py      # the MCP server: tools over stdio
├── lumina_api.py      # API client and notes as text, testable without the MCP SDK
└── test_lumina_api.py

---

Key implementation details

Why transcription moved to the server

The Web Speech API is Chrome-only in practice, needs a language fixed up front, and hands back text with no audio to check it against. Recording with MediaRecorder and transcribing server-side works everywhere, detects the language on its own, and keeps the audio available for a retry. The Web Speech API is still used, but only for optional live captions while you speak — if it fails, nothing is lost.

Audio conversion

Browsers record WebM/Opus (Chrome) or MP4/AAC (Safari). Rather than depend on container support, the client decodes locally and re-encodes to 16 kHz mono PCM, then splits long recordings at the quietest point near each boundary so a cut does not land mid-word.

Two details make the length limit three hours rather than about forty minutes:

• Decoding asks for 16 kHz up front. new AudioContext() adopts the output device's rate, usually 48 kHz, and decodeAudioData then materialises the audio at that rate before anything downsamples it — three times the memory, for samples about to be thrown away. Requesting the target rate on the decoding context skips that buffer entirely.
• Nothing is held whole. The recorder rotates MediaRecorder every five minutes, so capture produces a list of separately decodable segments rather than one blob. streamWavChunks decodes one segment at a time and yields each WAV chunk as the previous upload finishes, so peak memory follows the segment length instead of the recording length. Audio left over at the end of a segment is carried into the next, so a rotation does not show up as a short chunk.

Between them these replace a pipeline whose peak was roughly 1.2 GB for a one-hour recording — enough that an hour-long meeting failed outright, reported as a browser that could not decode the audio.

Transcribing a file you already have

The app only transcribes what it records, so backend/retranscribe.py is the way in for a file on disk — a recording saved after a failure, a voice memo, anything exported elsewhere. It writes plain text, ready to paste into the Text tab.

cd backend
python retranscribe.py lecture.webm --language fr

It has two engines. --engine local runs Whisper on your own machine: no API quota, no rate limit, and the audio never leaves the laptop, which matters for a long recording — the API path costs about one request per four minutes of audio, so three hours is ~45 requests and a free-tier key will not survive it. --engine gemini uses the same chunking and prompts as the app. With no flag it picks local when it is installed.

pip install faster-whisper
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12   # NVIDIA GPUs only

The default model is large-v3-turbo; --model large-v3 is more accurate on hard audio and several times slower, and --model small suits a machine with no usable GPU. Note this is a local tool, not something the deployed backend can do — a free Render instance has no GPU.

Lecture slides

Attach the lecture's PDF (the Slides button next to Voice/Text) and the notes are made with it. pdf.js reads the deck in the browser, so the file never crosses the metered backend: only each page's text goes along with the transcript, and each summary window is shown the few pages that share its vocabulary (rare words weigh more). The slides supply exact terms, notation and the formulas that were on the board, which the audio never contained; a bullet that draws on one ends with [p. 12], invented page numbers are dropped, and the chip opens the page.

When the note is saved, pictures of the cited pages only are kept with it (a separate slide_pages collection, never sent with a listing), so the reader can show them on any device. Pages are served with an immutable cache header and the deck's hash in the URL.

Security headers

frontend/vercel.json sets a Content-Security-Policy, X-Frame-Options, nosniff, a referrer policy and caching for the 3D model. The CSP allows the backend and Worker by their hosting domains (*.onrender.com, *.workers.dev); moving either to a custom domain means adding it to connect-src, or the app cannot reach it.

The 3D scene: one rig, baked in Blender

The notebook, the pen and the paper airplane are a single rigged model, frontend/public/models/desk.glb, built from scratch by a Blender script. The pen is a child of the book and its nib follows the very path the ink is laid along, so the browser never steers anything: it only picks which clip plays.

Clip	What happens
idle	the pen hovers, capped, over the page; the page breathes
to_book	the pen uncaps, posts the cap on its end and lands on line one
write	seven lines of handwriting, then the page turns (loops seamlessly)
to_rest	to_book backwards
fly	the written page peels off, folds into a dart and flies away

While you record, your voice sets how fast the pen writes; while the transcript and notes are made it writes on; when the notes are ready the page becomes the plane. The ink is revealed in the shader: every ink vertex carries the time the nib passes it, and the material hides what the clip has not reached yet.

The scene holds a still composition under prefers-reduced-motion, stops rendering when the tab is hidden or you are on the Notes tab, and every rate is per second rather than per frame. The logic is in Canvas3D/deskScene.js (plain JS); Scene3D.jsx only wraps it for React.

Rebuilding the 3D model

"C:/Program Files/Blender Foundation/Blender 5.2/blender.exe" -b --python frontend/models-src/build_desk.py

It writes models-src/desk.raw.glb (~1.6 MB, git-ignored) and packs it with gltfpack (meshopt) into public/models/desk.glb (~0.35 MB). The pen is the one outside asset, credited in models-src/CREDITS.md (CC BY 4.0).

Stable keys, localized labels

The model returns English snake_case section keys plus a labels map holding the same headings in the transcript's language. The UI shows the labels; search, export and older notes keep working off the keys.

---

3D Models

Sourced from Sketchfab under Creative Commons licences — notebook, pen and paper airplane, all CC Attribution. See each model's Sketchfab page for full attribution.

---

License

MIT