Silent Speech
Maps 478 points on your face in the browser and reads the shape your mouth is making, 30 times a second, with the sound off.
Turn on your camera and this maps 478 points on your face, 30 times a second, then reads the shape your mouth is making. Nothing is recorded and nothing is sent.
Everything above happens inside your browser. No frame, landmark, or measurement is recorded, uploaded, or kept. There is no account, no storage, and no server that could receive it. Stopping the camera or leaving this page discards all of it.
Hypothesis
A constrained vocabulary and 60 seconds of calibration can beat a much larger general model running cold on a stranger's webcam. If that holds, personalization is worth more than model size for this problem.
How it works
- 01
Running now: MediaPipe Face Landmarker tracks 478 3D points and 52 blendshape coefficients at 30fps, entirely on-device.
- 02
Running now: lip geometry (gap, width, rounding, lip press, lower-lip roll) is measured against eye span so it survives distance, then scored into nine mouth shapes. Every score is a readable expression over measurable features rather than a black box, so the page can show the mechanism instead of asserting a result.
- 03
Running now: one of those nine classes is explicitly ambiguous. Sounds formed behind the teeth are invisible to a camera, so the classifier reports that rather than guessing between them.
- 04
Next: a sequence model, quantized under 3MB, decodes shape sequences into words against the GRID corpus grammar of command, colour, preposition, letter, digit, adverb.
- 05
Next: optional calibration fits a personalization layer to your face in about 60 seconds and shows you the accuracy difference it made.
Measured so far
- Landmarks tracked
- 478
- Kept, stored, or sent
- Nothing
- Mouth shapes read
- 8 of 9
plus 52 blendshape coefficients, every frame
no recording, no upload, no account, no server to receive it
the ninth class is everything formed behind the teeth, which no camera can separate
What's next
- Train the word recognizer on GRID and publish its real accuracy, not a target
- Add the 60 second calibration flow, with the before and after number shown live
- Ship the confusion matrix, including the homophenes no model can separate
- Test degradation under low light, facial hair, and faster speech