VISION — CNN Facial Emotion Recognition

VISION is a web app that classifies the facial expression in a photo with a Convolutional Neural Network trained from scratch. It recognises seven emotions: angry, disgust, fear, happy, neutral, sad and surprise. You can take a photo with your phone camera, pick one from your gallery, or upload a file. The app shows the predicted emotion, the score for every class, and unknown when the model is less than 60% confident.

Live demo: https://vision-cnn.onrender.com (free Render instance — the first visit after it has been idle can take about a minute to wake up)

• Stage 1 – training: train_model.py trains the CNN (Keras) and saves it to model/.
• Stage 2 – web app: app/ is a FastAPI backend plus an HTML/CSS/JS frontend that loads the saved model once at startup.

Project structure

facial_expression_cnn/
├── app/
│   ├── main.py             FastAPI app: GET /, GET /health, POST /predict, security headers
│   ├── config.py           all settings (threshold, upload limits, paths), overridable by env vars
│   ├── preprocessing.py    upload validation + exactly the training preprocessing
│   ├── model_service.py    loads the model once, interprets the softmax output, applies the threshold
│   └── static/             index.html, styles.css, js/app.js, js/logic.js, fonts/, favicon.svg
├── model/
│   ├── emotion_cnn_model.keras   trained model (27 MB)
│   ├── class_names.json          output index → emotion
│   ├── results.json              test metrics, confusion matrix, training history
│   ├── confusion_matrix.png
│   └── training_log.txt
├── tests/                  pytest (API, preprocessing, threshold, real model), Node tests, browser tests
├── train_model.py          CNN training script (stage 1)
├── plot_confusion_matrix.py
├── Dockerfile              web app image (local + Render)
├── Dockerfile.train        GPU training image
├── render.yaml             Render Blueprint
├── requirements.txt        web app runtime
└── requirements-dev.txt    + test tools

dataset/ is not committed. It is merged_emotion_dataset.zip extracted into dataset/merged_emotion_dataset/.

The model

File	model/emotion_cnn_model.keras (Keras 3.15, TensorFlow 2.21)
Input	(1, 64, 64, 1) float32, grayscale, pixel values 0–1
Output	7 softmax probabilities, in the order of class_names.json: angry, disgust, fear, happy, neutral, sad, surprise
Architecture	4 blocks of [Conv3×3 → BN → Conv3×3 → BN → MaxPool → Dropout] (32→64→128→256 filters), Dense 256, Dense 7 softmax · 2.2 M parameters
Training data	merged_emotion_dataset: 63,419 train images (10% held out for validation), 15,854 test images
Test result	71.8% accuracy, macro F1 0.70 (see model/results.json, model/confusion_matrix.png)

The test images come from the same merged dataset as the training images. The dataset includes pre-augmented copies, and its duplicate check only catches near-identical images, so the test score is probably somewhat optimistic. Real phone photos usually score lower.

Preprocessing: training = inference

Training used ImageDataGenerator(rescale=1/255).flow_from_directory(target_size=(64, 64), color_mode="grayscale"). That means Keras load_img defaults: PIL convert to L, a nearest-neighbour resize to 64×64, then ×1/255. app/preprocessing.py:to_model_input does exactly the same.

This was checked against all 15,854 test images:

• The pixel values differ from Keras' own loader by at most 6×10⁻⁸ (float rounding).
• Accuracy through the app pipeline is 71.83%, against 71.81% recorded at training time.

Before that step, a phone photo goes through two extra steps that match it to the training data (tight face crops of at most 96×96):

1. EXIF orientation fix. Phone photos are often stored rotated.
2. Face crop (FACE_CROP=true). OpenCV's Haar cascade finds the largest face, crops it with a 9% margin, and downscales it to 96×96. The margin is calibrated on the training images: the detector box covers about 85% of a training crop. If no face is found, the whole image is used and the UI says so. Images of 96×96 or smaller are treated as face crops already.

The model's last layer is already a softmax, so its output is used as-is. There is no second softmax.

The 60% "unknown" threshold

UNKNOWN_THRESHOLD in app/config.py (env var UNKNOWN_THRESHOLD, default 0.60): if the top score is below the threshold, the API returns "label": "unknown". It still returns the tentative top_class and all the scores, and the UI shows both.

Softmax scores are not calibrated probabilities. A CNN can be confidently wrong, especially on photos unlike its training data. The threshold filters out clearly uncertain cases; it does not guarantee correctness.

API

Endpoint	Description
GET /	The web app
GET /health	status: ok means the service is up; model_ready says whether the CNN can predict. Returns 503 if the model is not loaded. Also returns the classes, threshold and upload limit.
POST /predict	multipart field file (JPG/PNG/WEBP, ≤ 10 MB)
GET /api/docs	Interactive API documentation

Example /predict response:

{
  "status": "ok", "label": "surprise", "top_class": "surprise", "confidence": 0.97808,
  "is_confident": true, "threshold": 0.6,
  "probabilities": [{"label": "angry", "probability": 0.0001}, "... 7 entries in model order ..."],
  "inference_ms": 19.0, "face": "detected", "face_detected": true
}

face is detected, not_found, or skipped (the image is already training-sized). Errors return {"status": "error", "detail": "<message safe to show>"}:

• 400 for a corrupt or unsupported file
• 413 for a file that is too large
• 422 when the file field is missing
• 503 when the model is not loaded
• 500 for an inference failure (no internal details)

Images are processed in memory and never stored.

Run locally (Windows, VS Code)

Why Docker: Windows Smart App Control on this PC blocks TensorFlow's unsigned DLLs, so import tensorflow fails in the Windows Python. Docker runs the app in Linux, the same environment Render uses. Docker Desktop must be running.

Open the folder in VS Code (File → Open Folder → C:\facial_expression_cnn), open a PowerShell terminal, and run:

docker build -t vision-app .
docker run --rm -p 8000:8000 --name vision vision-app

Open http://localhost:8000. Stop the app with Ctrl+C, or docker stop vision.

Without Docker works only where TensorFlow can load (another PC, or with Smart App Control off). It needs Python 3.11:

py -3.11 -m venv .venv
.venv\Scripts\Activate.ps1          # if blocked: Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 8000

Troubleshooting

Problem	Fix
DLL load failed … Application Control policy	That's Smart App Control; use the Docker commands above.
/health returns 503 / model_ready: false	The model files are missing. Check model/emotion_cnn_model.keras and model/class_names.json, or set MODEL_PATH / CLASS_NAMES_PATH.
port is already allocated / address in use	Use another port: docker run --rm -p 8080:8000 vision-app, then open http://localhost:8080.
ModuleNotFoundError without Docker	Activate the venv and run pip install -r requirements.txt again.
docker: error during connect	Start Docker Desktop and wait until it says "running".

Open it on your phone (same Wi-Fi)

1. Start the app with the Docker command above. It listens on all interfaces.
2. Find your PC's IP address. Run ipconfig and use the IPv4 Address of the Wi-Fi adapter, for example 192.168.1.23.
3. Allow the port through Windows Firewall once. In an Administrator PowerShell: New-NetFirewallRule -DisplayName "VISION 8000" -Direction Inbound -Protocol TCP -LocalPort 8000 -Action Allow -Profile Private
4. On the phone, open http://192.168.1.23:8000.

Camera and Gallery use the phone's native picker (<input type="file" capture>), which works over plain HTTP. The live webcam preview on a desktop needs a secure context: localhost, or HTTPS such as the Render URL. Use the HTTPS Render deployment to share the app outside your home network. Don't forward port 8000 on your router.

Tests

# Python: API, preprocessing parity, threshold, error handling, real model inference (40 tests)
docker run --rm -u root -v "${PWD}:/work" -w /work vision-app sh -c "pip install -q -r requirements-dev.txt && python -m pytest -q tests"

# Frontend logic (6 tests)
node --test tests/frontend/logic.test.mjs

# Browser end-to-end at desktop, tablet and phone sizes (44 checks, screenshots in tests/e2e/screenshots)
docker network create vision-net
docker run -d --rm --name vision-test --network vision-net vision-app
docker run --rm --network vision-net -v "${PWD}\tests:/tests" mcr.microsoft.com/playwright/python:v1.55.0-noble sh -c "pip install -q playwright==1.55.0 && python /tests/e2e/e2e_browser.py http://vision-test:8000 /tests/e2e/screenshots"
docker stop vision-test

Deploy to Render

The model is 27 MB, small enough for the GitHub repo. Memory measured in the container with the model loaded is about 250 MB, so it fits Render's free instance (512 MB). One CPU inference takes about 20 ms. Free instances sleep after about 15 minutes without traffic, so the first request after that takes roughly a minute while the container starts.

1. Push this project to a GitHub repository. .gitignore already excludes dataset/.
2. On https://dashboard.render.com, choose New → Blueprint, connect the repository, and confirm. Render reads render.yaml and builds the Dockerfile. Alternative: New → Web Service → Docker, health check path /health. Don't set a start command, because the Dockerfile has one.
3. Wait for the build (about 5–10 minutes the first time), then open https://<service-name>.onrender.com/health. You should see "model_ready": true.

The container binds to 0.0.0.0:$PORT, so Render's PORT is picked up automatically.

Environment variable	Default	Meaning
UNKNOWN_THRESHOLD	0.60	below this top score the result is unknown
MAX_UPLOAD_MB	10	upload size limit
FACE_CROP	true	crop the largest detected face before classifying
ALLOWED_ORIGINS	(empty)	comma-separated origins allowed to call the API from another site (CORS); not needed for a plain link
MODEL_PATH, CLASS_NAMES_PATH, RESULTS_PATH	model/...	model file locations

Training (stage 1)

docker build -f Dockerfile.train -t emotion-cnn .
docker run --rm --gpus all -v "${PWD}:/work" -w /work emotion-cnn python train_model.py
python plot_confusion_matrix.py        # works in Windows Python (no TensorFlow needed)