Emotion Tracker

Loading…

Start your camera to begin

Everything runs on your device. No video leaves this browser.

Personalize to your face

Auto baseline

Step 1: click Neutral and relax your face for 2.5 s. The app then measures expressions against your resting face, which matters most for subtle sadness and anger. Step 2 (optional): teach any emotion it gets wrong. Click it, then hold that expression. Even a subtle version is fine; that is what it will learn.

Saved only in this browser.

Design notes

Why the first version missed subtle emotions, and what changed

Version 1 only recognised sad and angry when they were exaggerated, and often missed disgust and fear. The model itself was the main problem. The way its output was used made it worse.

1 · Diagnosis

2 · Options considered

TechniqueWhat it fixesCost / riskDecision
Better pretrained model (HSEmotion EfficientNet-B0, trained on AffectNet)Much better at subtle, real-world expressions. Adds contempt, valence and arousal.16 MB download (cached after the first visit). ~25–60 ms per face.Adopted
Better face pipeline (MediaPipe Face Landmarker)Finds more faces, gives 478 landmarks for alignment, plus 52 blendshapes.4 MB model + 14 MB runtimeAdopted
Facial action units (FACS) from blendshapesDirectly measures the small muscle movements behind sad (AU1+4+15), anger (AU4+7+24), fear (AU1+2+4+5+20) and disgust (AU9+10).Blendshapes are noisy for some movements (nose wrinkle is weak), so they nudge the result rather than decide it.Adopted, as supporting evidence
Personal neutral calibrationRe-centres the model on your resting face, so small departures count. Also stops a "resting angry face" from being mislabelled.Needs 2.5 s of your cooperationAdopted
Class-prior (logit) adjustmentCounters the bias toward neutral and happy learned from imbalanced data.Too much makes neutral faces read as surprise. Kept gentle and adjustable.Adopted (Sensitivity slider)
Few-shot personal trainingLearns how you show an emotion, however subtly.A small nearest-class-mean classifier trained in the browser on a few seconds of examples per emotion.Adopted (optional)
Retrain / fine-tune a model on AffectNet, RAF-DB or ExpW with class-balanced or focal lossThe most principled long-term fix, especially for disgust and fear.Needs GPUs, days of training and datasets that are licensed for research only. It doesn't fit a static website. Better, ready-trained models already exist, and that's what HSEmotion is.Deferred. Use the pretrained model instead.
Larger models (EfficientNet-B2, vision transformers)A few more points of accuracy2–5× slower and larger. Not real-time on typical laptops in the browser.Rejected for now
Temporal smoothingStops flickerToo much makes it slow to reactKept (adjustable)

3 · What the app does now

  1. Detect and map the face. MediaPipe Face Landmarker finds up to 4 faces. For each one it gives 478 3-D landmarks and 52 blendshape coefficients.
  2. Align. The face is rotated so the eyes are level, then cropped square to 224×224.
  3. Classify. HSEmotion EfficientNet-B0 (ONNX, run with ONNX Runtime Web on WebAssembly, multi-threaded) gives 8 emotion logits plus valence and arousal.
  4. Measure action units. 17 FACS-style action units are measured above this face's own baseline. The baseline comes from your calibration, or else it is learned automatically: it drops quickly and rises slowly, so held expressions aren't absorbed into it. These are scored against EMFACS emotion prototypes.
  5. Fuse. final logits = CNN − calibration offsets − s·τ·log prior − s·neutral bias + s·AU evidence + personal classifier. Here s is the Sensitivity slider. Then softmax, then per-face smoothing over time.
  6. Track. IoU matching keeps a stable ID per face. Boxes are interpolated between analyses.

4 · Results on test images

The same 7 labelled photos, run through both versions in this browser (no calibration, default settings):

True labelv1 (face-api)v2 (this app)
AngryAngry 100%Angry 84%
DisgustedAngry 62%Disgusted 97%
FearfulNo face foundFearful 98%
HappyHappy 100%Happy 100%
NeutralNeutral 98%Neutral 42% (Surprised 37% next)
SadSad 100%Sad 99%
SurprisedSurprised 100%Surprised 98%

v2 gets 7 of 7, against v1's 5 of 7. The neutral photo is a close call, and that is deliberate. Her eyebrows are slightly raised, and v2 is tuned to notice small departures from neutral. That's the trade-off: being more sensitive to subtle sadness and anger also makes it more sensitive to small brow movements. Calibrating on your own neutral face settles this for you, or you can lower Sensitivity.

These are posed, exaggerated photos, so they show the pipeline works. They don't prove it handles subtle expressions. Subtle expressions are where calibration and teaching matter, and they can only be judged live on a real face.

5 · How to get the best accuracy

6 · Limitations

Facial expression is not the same as felt emotion. People can feel sad with a neutral face, and expression norms vary between people and cultures. AffectNet-trained models are among the best available, but even state-of-the-art accuracy on 8-class AffectNet is only about 60–65%. Disgust, fear and contempt remain the hardest classes. Personal calibration and teaching are what close the gap for any one person. HSEmotion models are Apache-2.0 licensed; AffectNet itself is licensed for research only.