In a previous post, I described a bit the support in GPAC for the HTML 5 video and audio elements in SVG documents. My initial idea was to be able to play, with the Media Source Extensions API, adaptive streaming content (such as DASH) in SVG. So I started implementing MSE and it is now possible to play DASH content within an SVG presentation using GPAC. This post details how it was implemented.
The Media Source Extensions API is a specification developed by the W3C HTML Working Group. It defines JavaScript interfaces to enable the playback in the browser of media data provided by the JavaScript layer. It is currently supported in Google Chrome Canary (including Android), in Internet Explorer 11, and in the nightly build of Firefox. It basically works as depicted in this figure: the HTML document and JavaScript code are loaded, the JavaScript fetches video content and passes it to the media engine in the browser for decoding. The rendering and control of the decoding is handled through the HTML <video> element.
MSE relies on the support of the following tasks:
- Parsing of container formats such as MP4 (or MPEG-2 or Ogg);
- Media decoding and rendering;
- Executing of JavaScript code;
- Processing of HTML Media Elements.
GPAC supported already the first 3 parts. The fourth one is more complex and was partly added. The HTML Media Elements are the following HTML 5 elements: <video>, <audio>, <source>, <track>. The processing of the HTML Media Elements can be decomposed in the parsing, rendering and execution of JS related interfaces.
GPAC does not support the parsing and rendering of HTML documents. But recently, the SVG Working Group decided to support the HTML Media Elements. Since GPAC already supports parsing and rendering of SVG documents, the support for the HTML Media elements was added to that SVG support (see this post).
GPAC is now (almost) MSE-compatible, as depicted in the figure below, an SVG renderer is used instead of the HTML renderer but ideally the same MSE and media-related JavaScript code should be reusable between a browser and GPAC.
The whole MSE spec is not yet supported but it can already be interestingly tested. Namely, you can create a MediaSource object, attach it to a video (or audio) element, get notified when the source is open, create MP4 source buffers, append MP4 segments (or entire files) in order, out-of-order (using the abort(“continuation”) method), of different quality. See the example code below. I’ve also put some working examples here.
function start() { var ms = new MediaSource(); var bloburl = URL.createObjectURL(ms); var v = document.getElementById('MyVideo'); /* GPAC currently dispatches the SourceOpen event on the video element and not on the MediaSource object */ v.ms = ms; v.addEventListener("sourceopen", onSourceOpen); v.src = bloburl; v.play(); } function onSourceOpen = function (event) { var ms = event.target.ms; var sb = ms.addSourceBuffer("video/mp4"); getNextSegment(sb); }
The JavaScript code to use GPAC’s MSE support is not yet fully compliant with the W3C specification. However, with the current support, one of the test I made was playing DASH content using MSE on GPAC on iPad. That made me happy, as I believe this is a first.
It is not yet possible to reuse the DASH JS or DASH IF players but I expect to have better support soon. Some parts of MSE are not fully implemented yet, it is missing:
- event dispatch (except sourceopen),
- frame removal and dependent frame removal,
- audio splicing,
- MPEG-2 TS segments,
- text tracks …
This work was reported during the MPEG DASH meeting in the following input contribution and demonstrated during the MPEG 105th meeting:
[textimport http://biblio.telecom-paristech.fr/cgi-bin/ws/biblio.cgi?type=standardisation&etat=submitted&id=13810]
I assume you have to prepare the MP4 files specifically for MSE. Do you have some advice on that?
Hi Silvia,
MSE unfortunately requires MP4 files to be fragmented. You can fragment a file offline with MP4Box (see this documentation) or online in JS with MP4Box.js. Fragmentation is a little different from DASH segmentation but DASH segmentation implies fragmentation, so DASHed MP4 files should work with MSE. HTH.