🎓 LectureNote Agent
Live lecture simultaneous interpretation + bilingual structured notes — Eazo Global Youth AI Agent Hackathon (Singapore)
Frontend: https://lecture-note-tau.vercel.app
Backend: wss://lecture-note-2we1.onrender.com
MCP Server: https://mcp-server-9mz7.onrender.com/mcp
📸 Screenshots
Start Screen
Dark themed landing page with PPT upload and history access.
Recording View (Dual Panel)
| Left: Real-time Bilingual Subtitles | Right: AI-Generated Outline |
|---|
| Live English speech recognition → Chinese translation | Auto-generated structured lecture notes |
| Pause / Resume / Stop controls | Editable rich-text with highlight toolbar |
| Yellow banner when paused | Practice questions after class |
Replay View
Tabbed view: outline, subtitles, practice questions — with Word/PDF export.
✨ Features
| Feature | Description |
|---|
| 🎙️ Real-time ASR | Azure Speech Services, en-IN model optimized for Indian English |
| 🌐 Simultaneous Translation | DeepSeek API: real-time English → Chinese, partial-text streaming for low-latency interpretation |
| 🧹 Filler Word Filtering | Automatically filters hesitation words (yeah, OK, um…) from the subtitle feed |
| 📝 AI Outline | Auto-generated bilingual structured outline every 30 seconds, merges with manual edits |
| 📎 Courseware Upload | Upload .pptx / .pdf files to provide lecture context; image-based PDFs auto-OCR |
| ✏️ 15 Exam Questions | End-of-class: 15 core quiz questions prioritized by exam hints (MCQ / short answer / true-false) |
| ⏯️ Pause / Resume | Adapt to lecture breaks and discussions |
| 🎨 Rich Text Editing | 4-color highlighting (key point / understanding / exam point / formula), H1/H2/H3 headings, Bold/Italic/Underline |
| 🔍 Smart Auto-Scroll | Auto-follows new subtitles; pauses when user scrolls up to review past content |
| 📥 Export | Word document (.doc), PDF print, copy to clipboard |
| 📚 Session History | Auto-save lectures to localStorage (max 20), browse and replay past sessions |
| 🇬🇧 English-Only Filter | Automatically skips Chinese speech / pinyin to avoid noise |
| 🛡️ Auto Wake-Up | Pings Render backend on page load to wake from free-tier sleep |
| 🔌 MCP Server | Self-hosted Model Context Protocol server with Streamable HTTP — 4 AI tools (translate, outline, questions, health) |
| 🌍 Bilingual UI | English/Chinese interface toggle (EN default), persisted in localStorage |
🏗️ Architecture
Browser (React + Vite + TailwindCSS)
│ WebSocket (audio binary + JSON messages)
│ HTTP POST (courseware upload)
│ MCP Streamable HTTP (AI tools)
▼
Node.js Backend (Render)
├── Azure Speech Services (ASR, en-IN, 16kHz PCM)
├── DeepSeek API (translation + outline + questions)
├── adm-zip (PPTX text extraction)
└── pdfjs-dist + Tesseract.js (PDF text + OCR)
MCP Server (Render, standalone)
├── @modelcontextprotocol/server v2
├── DeepSeek API (translate + outline + questions)
└── Streamable HTTP transport
Frontend Deployed on: Vercel
Backend Deployed on: Render
MCP Server Deployed on: Render
🚀 Quick Start (Local Development)
Prerequisites
- Node.js 18+
- Azure Speech Services API key (japaneast region)
- DeepSeek API key
Setup
# Clone
git clone https://github.com/Stevenchen213/lecture-note.git
cd lecture-note
# Frontend
npm install
npm run dev # → http://localhost:3000
# Backend (in another terminal)
cd server
npm install
cp ../.env.example .env # then edit .env with your API keys
node index.js # → ws://localhost:8080
Environment Variables
Create server/.env:
AZURE_SPEECH_KEY=your_azure_speech_key
AZURE_SPEECH_REGION=japaneast
DEEPSEEK_API_KEY=your_deepseek_api_key
Optional — switch the LLM to any OpenAI-compatible provider (e.g. Kimi/Moonshot). These override DeepSeek when set:
LLM_BASE_URL=https://api.moonshot.ai/v1
LLM_MODEL=kimi-k2.6
LLM_API_KEY=your_kimi_api_key
Note: when a custom provider is set, temperature is NOT sent (Kimi's current models lock it and reject explicit values); set LLM_TEMPERATURE to force one.
For production frontend:
VITE_WS_URL=wss://your-render-backend.onrender.com
📦 Tech Stack
| Layer | Technology |
|---|
| Frontend | React 18, Vite 5, TailwindCSS 3 |
| Backend | Node.js 20, WebSocket (ws) |
| ASR | Azure Speech Services (en-IN, continuous recognition) |
| LLM | DeepSeek API (default), or any OpenAI-compatible provider (e.g. Kimi/Moonshot) via LLM_BASE_URL / LLM_MODEL / LLM_API_KEY |
| PPT Parsing | adm-zip (PPTX = ZIP of XML) |
| PDF Parsing | pdfjs-dist + canvas (text extraction + render) |
| OCR | tesseract.js (image-based PDF fallback) |
| Frontend Hosting | Vercel (free) |
| Backend Hosting | Render (free) |
| MCP Server | @modelcontextprotocol/server v2, @modelcontextprotocol/express, Zod v4, Streamable HTTP |
📁 Project Structure
lecture-note/
├── index.html # Entry HTML
├── vite.config.js # Vite config
├── tailwind.config.js # TailwindCSS config
├── vercel.json # Vercel deployment config
├── src/
│ ├── main.jsx # React entry
│ ├── App.jsx # Root component + routing
│ ├── index.css # Global styles + animations
│ ├── components/
│ │ ├── StartScreen.jsx # Landing page + courseware upload
│ │ ├── SubtitlePanel.jsx # Real-time bilingual subtitles
│ │ ├── OutlinePanel.jsx # Editable outline + questions
│ │ ├── RichTextEditor.jsx # contentEditable + floating toolbar
│ │ ├── HistoryPanel.jsx # Past sessions list
│ │ └── ReplayView.jsx # View past session
│ ├── hooks/
│ │ ├── useAudioRecorder.js # Browser mic → Int16 PCM
│ │ └── useWebSocket.js # WebSocket connection manager
│ └── utils/
│ ├── outlineMerge.js # AI + user edit merge logic
│ ├── export.js # Word / PDF / clipboard export
│ └── sessionStore.js # localStorage session persistence
├── server/
│ ├── index.js # WebSocket + HTTP + MCP server
│ ├── azure-speech.js # Azure ASR integration
│ ├── deepseek.js # DeepSeek API (translate + outline + questions)
│ ├── pdf-utils.js # PDF text extraction + OCR fallback
│ └── package.json # Server dependencies
├── mcp-server/
│ ├── index.js # Standalone MCP server (Streamable HTTP)
│ ├── package.json # MCP server dependencies
│ └── .gitignore # Node modules + .env
├── docs/ # PE6201: report, cost analysis, recording guide
├── data/ # PE6201: evaluation dataset (segments + key points)
├── evals/ # PE6201: evaluation harness (run_evals.mjs)
└── .env.example # Environment variables template
🎓 PE6201 Course Deliverables (NTU)
End-of-course project materials for PE6201 Emerging AI Technologies:
To reproduce the evaluation:
node evals/run_evals.mjs # requires DEEPSEEK_API_KEY in server/.env
🏆 Hackathon
- Event: Eazo Global Youth AI Agent Hackathon
- Track: Personal Agent Challenge
- Location: Singapore
- Deadline: August 10, 2026, 22:00 (UTC+8)
👤 Author
Steven Chen — GitHub
