NEO 2: AI Dubbing and Lip Sync

Abstract

Dubbing video into other languages has historically required either re-recording with a native speaker or accepting the uncanny mismatch between audio and lip movements that comes with simple voice-over. Neither approach scales. The first destroys the original performance. The second looks wrong and audiences notice immediately.

NEO 2 takes a different approach. Given a video of a person speaking and a target language audio track, the system maps phonemes from the target language onto the speaker's existing facial movements. It modifies jaw position, lip shape, and tongue visibility frame by frame while preserving the speaker's identity, skin texture, and surrounding scene. The output is a photorealistic video where the person appears to natively speak the target language.

The entire pipeline runs in a single forward pass. There is no iterative refinement, no manual correction, no post-processing. The model handles the phoneme-to-viseme mapping, facial deformation, and temporal smoothing in one step.

How it works

The pipeline has four stages. Each runs sequentially but the full chain executes as a single inference call.

The challenge of cross-lingual lip-sync

Languages differ not just in sound but in how they use the face. Mandarin relies on tonal variation with relatively constrained lip movement. Arabic uses pharyngeal consonants that engage the throat and back of the mouth in ways English never does. Spanish vowels require more pronounced lip rounding than their English equivalents.

Previous approaches to AI dubbing fell into two categories. The first modified the audio but left the face untouched, producing a visible mismatch between what you hear and what you see. The second warped the entire face to match new audio, often destroying the speaker's identity or producing artifacts around the jawline.

NEO 2 takes a third path. It operates only on the articulatory region (lips, jaw, tongue, cheeks) while leaving everything else pixel-identical to the original. The key insight is that viseme targets can be predicted directly from phoneme sequences without needing an intermediate 3D face model.

Architecture

The system is built around a conditional diffusion model trained on paired multilingual speech data. Given a source frame and a target phoneme, the model generates a modified mouth region that blends seamlessly with the untouched portions of the face.

Three components work together. A phoneme encoder converts the target language audio into a dense sequence of articulatory features. A spatial attention module identifies which pixels in the source frame correspond to the articulatory region. And a diffusion decoder generates the modified pixels conditioned on both the articulatory features and the surrounding facial context.

The temporal consistency module sits on top. It takes a window of generated frames and enforces smooth transitions, removing the flickering artifacts that plague frame-by-frame generation approaches. This runs as a lightweight post-process, adding minimal latency.

Additional technical details will be added here from the research team's brief. This section will expand to cover training data, loss functions, ablation studies, and comparison with prior work.