Design notes
Why the first version missed subtle emotions, and what changed
Version 1 only recognised sad and angry when they were exaggerated, and often missed disgust and fear. The model itself was the main problem. The way its output was used made it worse.
1 · Diagnosis
- The model was too small and trained on the wrong data. face-api's FaceExpressionNet is a ~330 KB CNN. It works on low-resolution crops and was trained on a small set of mostly posed photos, so it learned the extreme versions of each emotion. Real sadness is often just raised inner eyebrows and slightly lowered lip corners, and that pattern looks almost nothing like its training examples.
- Neutral and happy dominate the data. Most facial-expression data is neutral or happy. In AffectNet, the largest in-the-wild set, happy plus neutral make up 73% of images, while disgust, fear and contempt are about 1–2% each. A model trained this way learns that "neutral" is almost always the safe answer, so anything subtle collapses to it.
- Everyone's resting face is different. A naturally low brow or downturned mouth looks like mild anger or sadness to a generic model. A very relaxed face hides small changes. Without knowing what your neutral looks like, the model can't detect small departures from it.
- The detector missed faces. On the fearful test photo, the old Tiny Face Detector found no face at all, so no emotion model could have labelled it.
- The decision rule was crude. The app took the argmax of one frame's probabilities. A 30%-sad, 60%-neutral face always read as neutral.
2 · Options considered
| Technique | What it fixes | Cost / risk | Decision |
|---|---|---|---|
| Better pretrained model (HSEmotion EfficientNet-B0, trained on AffectNet) | Much better at subtle, real-world expressions. Adds contempt, valence and arousal. | 16 MB download (cached after the first visit). ~25–60 ms per face. | Adopted |
| Better face pipeline (MediaPipe Face Landmarker) | Finds more faces, gives 478 landmarks for alignment, plus 52 blendshapes. | 4 MB model + 14 MB runtime | Adopted |
| Facial action units (FACS) from blendshapes | Directly measures the small muscle movements behind sad (AU1+4+15), anger (AU4+7+24), fear (AU1+2+4+5+20) and disgust (AU9+10). | Blendshapes are noisy for some movements (nose wrinkle is weak), so they nudge the result rather than decide it. | Adopted, as supporting evidence |
| Personal neutral calibration | Re-centres the model on your resting face, so small departures count. Also stops a "resting angry face" from being mislabelled. | Needs 2.5 s of your cooperation | Adopted |
| Class-prior (logit) adjustment | Counters the bias toward neutral and happy learned from imbalanced data. | Too much makes neutral faces read as surprise. Kept gentle and adjustable. | Adopted (Sensitivity slider) |
| Few-shot personal training | Learns how you show an emotion, however subtly. | A small nearest-class-mean classifier trained in the browser on a few seconds of examples per emotion. | Adopted (optional) |
| Retrain / fine-tune a model on AffectNet, RAF-DB or ExpW with class-balanced or focal loss | The most principled long-term fix, especially for disgust and fear. | Needs GPUs, days of training and datasets that are licensed for research only. It doesn't fit a static website. Better, ready-trained models already exist, and that's what HSEmotion is. | Deferred. Use the pretrained model instead. |
| Larger models (EfficientNet-B2, vision transformers) | A few more points of accuracy | 2–5× slower and larger. Not real-time on typical laptops in the browser. | Rejected for now |
| Temporal smoothing | Stops flicker | Too much makes it slow to react | Kept (adjustable) |
3 · What the app does now
- Detect and map the face. MediaPipe Face Landmarker finds up to 4 faces. For each one it gives 478 3-D landmarks and 52 blendshape coefficients.
- Align. The face is rotated so the eyes are level, then cropped square to 224×224.
- Classify. HSEmotion EfficientNet-B0 (ONNX, run with ONNX Runtime Web on WebAssembly, multi-threaded) gives 8 emotion logits plus valence and arousal.
- Measure action units. 17 FACS-style action units are measured above this face's own baseline. The baseline comes from your calibration, or else it is learned automatically: it drops quickly and rises slowly, so held expressions aren't absorbed into it. These are scored against EMFACS emotion prototypes.
- Fuse.
final logits = CNN − calibration offsets − s·τ·log prior − s·neutral bias + s·AU evidence + personal classifier. Heresis the Sensitivity slider. Then softmax, then per-face smoothing over time. - Track. IoU matching keeps a stable ID per face. Boxes are interpolated between analyses.
4 · Results on test images
The same 7 labelled photos, run through both versions in this browser (no calibration, default settings):
| True label | v1 (face-api) | v2 (this app) |
|---|---|---|
| Angry | Angry 100% | Angry 84% |
| Disgusted | Angry 62% | Disgusted 97% |
| Fearful | No face found | Fearful 98% |
| Happy | Happy 100% | Happy 100% |
| Neutral | Neutral 98% | Neutral 42% (Surprised 37% next) |
| Sad | Sad 100% | Sad 99% |
| Surprised | Surprised 100% | Surprised 98% |
v2 gets 7 of 7, against v1's 5 of 7. The neutral photo is a close call, and that is deliberate. Her eyebrows are slightly raised, and v2 is tuned to notice small departures from neutral. That's the trade-off: being more sensitive to subtle sadness and anger also makes it more sensitive to small brow movements. Calibrating on your own neutral face settles this for you, or you can lower Sensitivity.
These are posed, exaggerated photos, so they show the pipeline works. They don't prove it handles subtle expressions. Subtle expressions are where calibration and teaching matter, and they can only be judged live on a real face.
5 · How to get the best accuracy
- Use even, front-facing light and keep your face within about 1 m of the camera.
- Calibrate Neutral first, then teach the emotions that still get missed. Two or three teaching rounds for sad, angry, disgust and fear make the biggest difference.
- If neutral moments get labelled with an emotion, lower Sensitivity. If subtle emotions still show as neutral, raise it.
6 · Limitations
Facial expression is not the same as felt emotion. People can feel sad with a neutral face, and expression norms vary between people and cultures. AffectNet-trained models are among the best available, but even state-of-the-art accuracy on 8-class AffectNet is only about 60–65%. Disgust, fear and contempt remain the hardest classes. Personal calibration and teaching are what close the gap for any one person. HSEmotion models are Apache-2.0 licensed; AffectNet itself is licensed for research only.