Silent Speech

A constrained vocabulary and a short calibration can beat a much larger general model running cold on a stranger's webcam. If that holds, personalization is worth more than model size for this problem. Measuring the first classifier against ground truth has already settled one part of it: rules I hand tuned on a single face scored 11.6% on strangers, so anything that does not generalise across faces is not worth building.

1. 01

   Running now: MediaPipe Face Landmarker tracks 478 3D points and 52 blendshape coefficients at 30fps, entirely on-device.

2. 02

   Running now: lip geometry (gap, width, rounding, lip press, lower-lip roll) is measured against eye span so it survives distance, then scored into nine mouth shapes. Every score is a readable expression over measurable features rather than a black box, so the page can show the mechanism instead of asserting a result.

3. 03

   Running now: one of those nine classes is explicitly ambiguous. Sounds formed behind the teeth are invisible to a camera, so the classifier reports that rather than guessing between them.

4. 04

   Running now: a small network replaces the rules I wrote by hand. Per frame viseme labels were derived from the corpus word alignments, then a 17,000 parameter network was trained on 27 speakers and measured on six it never saw. Split by speaker, never by clip, because splitting by clip lets the same face appear on both sides and measures memory rather than generalisation.

5. 05

   The model ships as a 266KB JSON file and runs as two matrix multiplies in the browser, so no inference runtime is loaded and the page costs nothing extra. You can switch between it and my original hand written rules in the demo and watch the difference.

6. 06

   What moved the number was time and honesty about timing, not detail. Feeding the model the full 40 point lip contour instead of nine derived measurements was worth 0.1 points. Widening the window from a quarter of a second to just over half a second was worth seven. A mouth shape is not visible in a frame, it is visible in the movement into and out of it.

7. 07

   Corpus speakers read quietly in a studio, and their widest vowel is a lip gap of 0.078. Anyone deliberately testing this reaches four times that, which is input the model had never seen, so it fell back to reporting silence. Training now includes the same clips rescaled between 0.7 and 5 times around a resting mouth. Only the features that genuinely grow are scaled: opening wider does not widen the corners of your mouth or press your lips harder, and scaling those together made the model collapse.

8. 08

   The largest single gain came from fixing the labels rather than the model. Phonemes inside each word were originally given equal time, which is wrong: vowels carry most of the duration of speech and stops are brief. Equal spacing labelled three quarters of a word like green as unreadable when a listener hears mostly the vowel, so the model was taught that vowels barely exist. Weighting by realistic duration moved recall on the EE shape from 21% to 45%.

9. 09

   Running now: twenty seconds of calibration fits every threshold to your own face. Measured across corpus speakers, resting mouth width varies about 7 percent and maximum lip rounding by over 100 percent, which is wider than the range the classifier uses to make its decisions. An average of other people's mouths is a poor fit for any individual one.

10. 10

    Running now: whole words. Lip motion is segmented into utterances, each complete word is resampled to a fixed length so speaking speed cancels out, and a second model reads it against the 51 word corpus vocabulary. Measured on six unseen speakers: 38.7 percent with no constraint at all, where chance is 2 percent, and 62.2 percent when the grammar says which slot a word belongs to.

11. 11

    Word boundaries are the bottleneck, and the cost is measured. With perfect boundaries the grammar constrained figure is 68.7 percent. Out by one frame it is 66.3, by two 61.6, by three 56.4. A frame is 40 milliseconds, so every frame the detector misses costs about two and a half points.

12. 12

    Next: optional calibration fits a personalization layer to your face in about 60 seconds and shows you the accuracy difference it made.

Landmarks tracked

478

plus 52 blendshape coefficients, every frame

Kept, stored, or sent

Nothing

no recording, no upload, no account, no server to receive it

Reads a mouth shape

38.4%

on six speakers the model never saw, while they move four times as much as the corpus does, which is what someone testing a demo actually does. The previous model scored 34.4% under the same condition. Overall frame accuracy is 62.2%, but silence and unreadable consonants are most frames, so that number flatters it

Reads a whole word

62.2%

across 51 words on six speakers it never saw, with the grammar narrowing each position to its candidates. Chance is 2 percent and always guessing the commonest word is 4.3 percent. Without the grammar it reads 38.7 percent

Calibration

20 sec

fits the thresholds to your face, held in memory only and destroyed when you leave the page or close the tab

• Raise mouth shape accuracy above 32%, which is better than the rules and still not good
• Replace the even phoneme spacing in the labels, which is the likeliest ceiling on all of this
• Train the word recognizer on GRID and publish its real accuracy, not a target
• Report the accuracy difference calibration makes, measured rather than asserted
• Ship the confusion matrix, including the homophenes no model can separate
• Test degradation under low light, facial hair, and faster speech