WebCodecs - Synchronization

[kixelated]

Audio and video are rendered out of sync. This isn't a big deal with zero latency, but matters more at higher end targets.

[chrisprobst]

Hi @kixelated, very cool projects. Quick question and that is related to this issue's topic: How do you deal with clock drift?

Meaning, an audio packet with 128 samples (as an example) has a clear duration given by the rate and number of samples. However, 20 ms on my machine are not 20 ms on your machine. That means, there is ALWAYS a difference. WebRTC solved this massive problem (it's really hard to solve) using NetEQ (a library used by Chrome). It is able to shrink and stretch audio in a non-hearable fashion to compensate for that. Otherwise, buffers will (even in a perfect world) always run under or over over time.

Now, de-synchronization is one effect of this issue. I guess your current sync issue has nothing to do with clock drift. But it will arise later, even after you implement sync. Just wanted to share my experience with this topic so you can keep this on your radar. Otherwise, great projects and nice work!!