EchoCue

A dark-themed web app with two tabs — Speech and Video — that measure behavioral patterns and explain the results with AI. Plain HTML, CSS and JavaScript on the front end, with small serverless functions on Vercel. No framework and no TypeScript.

Live app: autism-speech-pattern-analyzer.vercel.app

For educational and research use only. It is not a diagnostic tool.

What it does

• Transcribes audio with Groq Whisper Large V3, with word-level timestamps
• Measures initial response time, speech rate (wpm), average and maximum word duration, and word count
• Flags delayed responses, reduced speech rate, extended words and repeated phrases (possible echolalia)
• Shows a speech timeline with pauses highlighted, and a transcript that follows the audio as it plays
• Runs a live webcam session and measures gaze, head position, and repetitive arm/hand movement entirely on-device
• Writes an educational interpretation of the measured numbers through OpenRouter — never of raw audio/video

Architecture

flowchart LR
    subgraph Browser["Browser (public/)"]
        UI["index.html\ntab UI"]
        Speech["app.js\nSpeech tab"]
        Video["video.js\nVideo tab"]
        MP["MediaPipe\nFaceLandmarker + HandLandmarker\n(WASM, on-device)"]
    end

    subgraph Vercel["Vercel serverless functions (api/)"]
        T["transcribe.js"]
        A["analyze.js"]
        AV["analyze-video.js"]
    end

    Groq["Groq Whisper\nLarge V3"]
    OR["OpenRouter\nLLM"]

    UI --> Speech
    UI --> Video
    Speech -- "trimmed 16kHz mono WAV" --> T
    T --> Groq
    Groq -- "transcript + word timings" --> T
    T --> Speech
    Speech -- "speech metrics only" --> A
    A --> OR
    OR -- "interpretation text" --> A
    A --> Speech

    Video --> MP
    MP -- "landmarks, in-browser only" --> Video
    Video -- "gaze/head/movement metrics only\n(never video)" --> AV
    AV --> OR
    OR -- "interpretation text" --> AV
    AV --> Video

API keys (GROQ_API_KEY, OPENROUTER_API_KEY) are read from environment variables on the server side only — the browser never sees them.

Files

Piece	Role
public/index.html	Single page with two tabs (Speech, Video); tab switching is a tiny inline script
public/app.js	Speech tab's logic
public/video.js	Video tab's logic (its DOM ids are all prefixed v- so the two tabs' scripts never collide)
api/transcribe.js	Sends the audio to Groq Whisper and returns word timings
api/analyze.js	Builds the prompt from the measured speech metrics and calls OpenRouter
api/analyze-video.js	Builds the prompt from the measured video metrics and calls OpenRouter

Speech tab

The browser decodes the file, trims it to the first 120 seconds, and resamples it to 16 kHz mono WAV before upload. That keeps the request under Vercel's 4.5 MB body limit and replaces the librosa step from an earlier Python version.

sequenceDiagram
    participant U as User
    participant B as Browser (app.js)
    participant T as api/transcribe.js
    participant G as Groq Whisper
    participant A as api/analyze.js
    participant O as OpenRouter

    U->>B: Upload/record audio
    B->>B: Trim to 120s, resample to 16kHz mono WAV
    B->>T: POST audio
    T->>G: Transcribe (word timestamps)
    G-->>T: Transcript + word timings
    T-->>B: Word timings
    B->>B: Compute response time, wpm,\nword durations, repeated phrases
    B->>A: POST metrics only
    A->>O: Prompt built from metrics
    O-->>A: Educational interpretation
    A-->>B: Interpretation text
    B->>U: Timeline + transcript + interpretation

Video tab

Runs a live webcam session and measures four things in real time, entirely on-device:

• Starts the camera with getUserMedia, records up to 60 seconds with MediaRecorder (for playback afterward), and samples the live feed at roughly 6–7 frames/second while it runs
• Runs MediaPipe FaceLandmarker (gaze-direction blendshapes + the 468-point face mesh) and HandLandmarker (21-point hand skeleton) in the browser via WebAssembly, both on CPU delegate — nothing is ever uploaded, detection and recording both happen on-device
• Live feedback while recording: a vector ring is drawn around each eye (green/amber with gaze), the five head-pose points light up amber when turned/tilted, the hand skeleton flashes red on a repetitive pattern, and a running counter shows gaze/head/movement/squeeze counts updating live
• Gaze: % of time looking toward the camera vs away, longest unbroken streak, number of distinct away-glances — from FaceLandmarker's eye-direction blendshapes
• Head position: calibrates a "facing forward" baseline from the first ~1.2s of the session (distance ratios between nose/forehead/chin/cheeks — the same vector-ratio technique as an Eye Aspect Ratio blink check), then flags sustained turning or tilting away from that baseline
• Arm/hand movement (flapping, rocking): sustained oscillation in wrist position (direction reversals/sec over a sliding window)
• Hand-squeezing: sustained oscillation in hand openness (average distance from wrist to fingertips, normalized by hand size)
• Sends only the resulting numbers (never the video) to api/analyze-video.js, which reports on exactly these four things in a short, direct, non-diagnostic explanation

flowchart TD
    Cam["getUserMedia webcam feed"] --> Sample["Sample ~6-7 fps"]
    Sample --> FL["FaceLandmarker\n(gaze blendshapes, 468-pt mesh)"]
    Sample --> HL["HandLandmarker\n(21-pt hand skeleton)"]
    FL --> Gaze["Gaze metrics"]
    FL --> Head["Head position metrics"]
    HL --> Move["Arm/hand movement metrics"]
    HL --> Squeeze["Hand-squeezing metrics"]
    Gaze --> Metrics["Metrics only"]
    Head --> Metrics
    Move --> Metrics
    Squeeze --> Metrics
    Metrics --> AV["api/analyze-video.js"]
    AV --> OR["OpenRouter"]
    OR --> Result["Non-diagnostic explanation"]

Deploy to Vercel

1. Import this repository at vercel.com/new. No build settings are needed.

2. Add these environment variables under Project Settings, then redeploy:

   Name	Where to get it
   GROQ_API_KEY	https://console.groq.com/keys
   OPENROUTER_API_KEY	https://openrouter.ai/keys
   OPENROUTER_MODEL (optional)	Tried first if set; otherwise a built-in list of free models is tried in order

Run locally

npm i -g vercel
cp .env.example .env   # then fill in your keys
vercel dev