My experience with multimedia standards, such as MPEG-4 BIFS, made me used to very good synchronization between video streams and graphics. So I wanted to check that the same was possible with web technologies, in particular with the plugin-less HTML5. To be more precise, I would like to display graphics on top of a video in a synchronized manner, with frame accuracy. The synchronization should be preserved when: playing, at different speed; paused; seeked; when the video stalls, …. I investigated several options and present some of the results. My first option was to use a graphics track within the HTML 5 video element, in a similar way to a subtitle track. Unfortunately, the track element seems limited to text tracks. So you have to embed your graphics (i.e. SVG) within a WebVTT text track as suggested. I tried it and it does not seem to work in Chrome Canary (see test here), even if some things happen. This is probably normal as the syntax for WebVTT does not allow embedding SVG in a cue. Anyway, to my mind, it’s using a hammer to kill a fly. SVG already specifies timing information, and allows specifying complex relationships (repetitions for instance) that you can’t do with WebVTT. In WebVTT, you have to specify/repeat the graphics at every key frame (see file). This might be acceptable in some cases but not always. Additionally, storing a text track in a text file is not very efficient for random access. In the end, I don’t think this option is satisfactory.

A second option was to use the video timeupdate events and to draw the graphics on a <canvas> element on top. Here is the test. It almost works in Firefox Nightly, Chrome Canary, Opera Next. Almost because timeupdate events don’t seem to be fired at every frame and the playback is not smooth. It seems that 3-4 events are generated per second. Is it a problem of overhead? I don’t know. Almost because the graphics seem to lag behind the video during normal playback. But, at least during the frame-by-frame playback, the synchronization is good. On the overall during playback, the rendering is not satisfactory, not smooth enough. It’s hard to know if it comes from the JavaScript layer or the canvas layer.

The final option was to use an SVG document overlayed on top of the video and to start/pause/resume the animations in the SVG when the video is started/paused/resumed; and for frame-by-frame playback to seek in the SVG using the setCurrentTime method with the currentTime of the video. The test is here. This solution is almost good: Playback is smoother than with the other solution, but still not perfect, there are some rendering artefacts. Frame-by-frame playback is correct. The only problem is that there is no way to control the speed of the SVG playback.

The remaining problems are:

• Precision of the timestamps. Even if my video is 25 fps exactly, the timestamps are not exact multiple of 40 ms.
• Implementation problems in different browsers: Chrome & namely of B-frames in MP4, Safari & rendering problems of transparent graphics on top of the video.

So to me, the best solution would be to let the browser control the timing of the playback of the SVG, i.e. control its timeline, if indicated by the author. One way would be to allow SVG content to be referenced from a <track> element. Another way would be to allow the mediagroup attribute on the <svg> element. Another way would be to allow SVG content to be referenced from a <video> element, after all animated SVG is ‘ostensibly video data’.

Update: some of the existing web pages from which I got inspired:

• Two videos : a really bad example of synchronization, at least on my machine.
• Usage of timeupdate event
• W3C test page for media events
• HTML5 video and frame accuracy: the author seems to be happy about the result, I’m not that convinced: frame-by-frame it’s ok, but live playback is not synchronized.
• Explanation of the mediagroup attribute
• A tutorial on the track element.