ByteDance releases SeedRealTime, a large audio and video full-duplex model

The Zhitong Finance App learned that on August 5, ByteDance officially launched SeedRealTime, a full-duplex native audio and video model. Currently, SeedRealTime has launched all Doubao apps, and is the first in the industry to achieve large-scale implementation of audio and video full-duplex technology.

As a key step towards full modal interaction, SeedRealTime uses a unified architecture to natively blend audio, video, and text to interact in real time on a continuous multi-modal stream of information, bringing a new experience of “watching, listening, and speaking”. It achieved the following three core breakthroughs:

Joint audio and video understanding: Native support for deep integration of sound, image, and timing information. The model can not only combine scenes to resolve homonym ambiguities, but also accurately understand visual timing references, so that what you see, hear, and say are integrated.

Active interaction ability: Ability to continuously perceive the environment and actively express oneself. The model can detect changes in the state of the screen (such as when a key target appears) and can call tools to incorporate expressions, so that interaction can be upgraded from passive response to active collaboration.

Smooth interaction rhythm: It can sense the user's conversation status and communication rhythm in real time, and naturally pick up, pause, and respond at the right time; at the same time, it has strong anti-interference ability, can distinguish between other people's chatter and background noise, and maintain smooth and consistent communication.

ByteDance said that the end-to-end manual evaluation results showed that compared to the cascaded model, SeedRealTime's audio and video conversation rhythm problems were reduced by half — the model grasped the timing of speech more naturally, and stuck phenomena such as “being stolen without speaking, slow response after speaking, or mistriggered by background noise and small talk” were significantly reduced; at the same time, the probability of a single conversation being able to fully and smoothly communicate was also significantly improved.