Category Archives: Multimedia

Late news

Just for the record, in case you haven’t noticed. Our team organized two standardisation meetings this September:

We are working on both topics at the moment. We should have some demos soon.

XML and whitespaces

I thought handling of white spaces in XML documents was simple. Now, I know better. Let me share my understanding. There are 2 main levels of processing for white spaces:

  • at the parser level;
  • at the application level either for manipulating text in the DOM or for rendering text.

Parsing

White spaces may appear almost anywhere but for use in an application, only those in attribute values and in text content matter. XML 1.0 (or 1.1, for that respect it’s the same) indicates what should be done with white spaces:

An XML processor MUST always pass all characters in a document that are not markup through to the application. A validating XML processor MUST also inform the application which of these characters constitute white space appearing in element content.

XML defines what is markup and what is not.

Markup takes the form of start-tags, end-tags, empty-element tags, entity references, character references, comments, CDATA section delimiters, document type declarations, processing instructions, XML declarations, text declarations, and any white space that is at the top level of the document entity (that is, outside the document element and not inside any other markup).

So attribute values are markup as part of the start-tag construct and are not affected by the above quote and white space in attributes should be dealt differently:

Before the value of an attribute is passed to the application or checked for validity, the XML processor MUST normalize the attribute value by applying the algorithm below, or by using some other method such that the value passed to the application is the same as that produced by the algorithm.

Additionally, XML defines the xml:space attribute but at the parsing level it’s useless.

DOM Manipulations

The characters in the DOM are received by the parser for text and cdata section nodes, in the form of a DOMString. In DOM3 it is possible to retrieve/set the text content of a DOM subtree without scanning the whole tree using the textContent atribute. This attribute of a DOM node returns the text content with all whitespaces if no validation is performed or without the whitespaces not specifically allowed in the associated grammar (DTD, Schema, RNG,…) :

On getting, no serialization is performed, the returned string does not contain any markup. No whitespace normalization is performed and the returned string does not contain the white spaces in element content (see the attribute Text.isElementContentWhitespace).

Rendering

The rendering algorithm should handle the white spaces. For example, SVG overrides the semantics of the XML-defined xml:space attribute to apply normalization (convert carriage return, line feed, tabs into space, remove duplicated, trailing and leading spaces) when the value of xml:space is “default” and render the result without modifying the DOM tree value; and when the value is “preserve”, only the conversion of CR, LF and TAB is done, all spaces are displayed. In HTML context, CSS defines the white-space property and associated processing model.

Thanks to Robin for his help in understanding all of this mess.

Video support in Android

Today (Oct. 8, 2009), I started playing around with a new phone, an HTC hero, which runs Android (v. 1.5). I tested the video support and here are the preliminary results.

Streaming

Summary

Generic support for

  • RTSP (DESCRIBE, SETUP, PLAY) and reception of SDP, including with MPEG-4 IOD

Stream types supported

  • video/MP4V-ES
  • video/H263-1998
  • audio/AMR
  • audio/mpeg4-generic

Stream types not supported

  • video/H264
  • video/mpeg4-generic
  • video/X-SV3V-ES
  • audio/X-QDM

Darwin Streaming Server Details

sample_100kbit.mov: Media type not supported.

v=0
o=StreamingServer 3463999147 1214234604000 IN IP4 137.194.232.94
s=/sample_100kbit.mov
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:94
t=0 0
a=control:*
a=range:npt=0-  70.00000
m=video 0 RTP/AVP 96
b=AS:79
a=rtpmap:96 X-SV3V-ES/90000
a=control:trackID=3
m=audio 0 RTP/AVP 97
b=AS:14
a=rtpmap:97 X-QDM/22050/2
a=control:trackID=4
a=x-bufferdelay:4.97

sample_300kbit.mov: Media type not supported.

v=0
o=StreamingServer 3463999278 1214234604000 IN IP4 137.194.232.94
s=/sample_300kbit.mov
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:253
t=0 0
a=control:*
a=range:npt=0-  70.00000
m=video 0 RTP/AVP 96
b=AS:209
a=rtpmap:96 X-SV3V-ES/90000
a=control:trackID=3
m=audio 0 RTP/AVP 97
b=AS:44
a=rtpmap:97 X-QDM/32000/2
a=control:trackID=4
a=x-bufferdelay:3.66

sample_50kbit.3gp: Success audio+video

v=0
o=StreamingServer 3463999304 1214234604000 IN IP4 137.194.232.94
s=/sample_50kbit.3gp
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:50
t=0 0
a=control:*
a=range:npt=0-  70.00000
m=video 0 RTP/AVP 96
b=AS:41
a=rtpmap:96 MP4V-ES/90000
a=control:trackID=3
a=cliprect:0,0,144,176
a=framesize:96 176-144
a=fmtp:96 profile-level-id=1;config=000001B008000001B50EE040C0CF0000010000000120
008440FA282C2090A21F
m=audio 0 RTP/AVP 97
b=AS:8
a=rtpmap:97 AMR/8000/1
a=control:trackID=4
a=fmtp:97 octet-align=1

sample.mp3: Fail

sample_100kbit.mp4: Success audio+video

v=0
o=StreamingServer 3463999476 1214234604000 IN IP4 137.194.232.94
s=/sample_100kbit.mp4
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:96
t=0 0
a=control:*
a=mpeg4-iod:"data:application/mpeg4-iod;base64,AoJrAE///w/z/wOBdgABQNhkYXRhOmFwc
GxpY2F0aW9uL21wZWc0LW9kLWF1O2Jhc2U2NCxBWUVDQVV3Rkh3TklBTWtnQUdVRUx5QVJBRzNkQUFFa
ytBQUJKUGdGSUFBQUFiRHpBQUFCdFE3Z1FNRFBBQUFCQUFBQUFTQUFoRUQ2S0RBZzhxSWZCaEFBUkFBQ
UFsZ0FBQUFBSUFBQUFBQURBVElDbndNdUFHVUFCSUNBZ0JSQUZRQVlBQUFBVGlBQUFFNGdCWUNBZ0FJV
mtBWVFBRVFBQUI5QUFBQWZRQ0FnQUFBQUF3PT0EDQEFAADIAAAAAAAAAAAGCQEAAAAAAAAAAANpAAJAR
mRhdGE6YXBwbGljYXRpb24vbXBlZzQtYmlmcy1hdTtiYXNlNjQsd0JBU2daTUNvRmNtRUVIOEFBQUIvQ
UFBQkVLQ0tDbjQEEgINAABkAAAAAAAAAAAFAwAAYAYJAQAAAAAAAAAA"
a=isma-compliance:1,1.0,1
a=range:npt=0-  70.00000
m=video 0 RTP/AVP 96
b=AS:76
a=rtpmap:96 MP4V-ES/90000
a=control:trackID=3
a=cliprect:0,0,242,192
a=framesize:96 192-242
a=fmtp:96 profile-level-id=1;config=000001B0F3000001B50EE040C0CF0000010000000120
008440FA283020F2A21F
a=mpeg4-esid:201
m=audio 0 RTP/AVP 97
b=AS:20
a=rtpmap:97 mpeg4-generic/8000/2
a=control:trackID=4
a=fmtp:97 profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdelt
alength=3;config=1590
a=mpeg4-esid:101

sample_300kbit.mp4: Success audio+video

v=0
o=StreamingServer 3464001266 1214234604000 IN IP4 137.194.232.94
s=/sample_300kbit.mp4
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:258
t=0 0
a=control:*
a=mpeg4-iod:"data:application/mpeg4-iod;base64,AoJrAE///w/z/wOBdgABQNhkYXRhOmFwc
GxpY2F0aW9uL21wZWc0LW9kLWF1O2Jhc2U2NCxBWUVDQVV3Rkh3TklBTWtnQUdVRUx5QVJBVUpFQUFOY
llBQURXMkFGSUFBQUFiRHpBQUFCdFE3Z1FNRFBBQUFCQUFBQUFTQUFoRUQ2S0Y4aDRLSWZCaEFBUkFBQ
UFCNEFBQUFBSUFBQUFBQURBVElDbndNdUFHVUFCSUNBZ0JSQUZRQVlBQUFBdTRBQUFMdUFCWUNBZ0FJV
GtBWVFBRVFBQUZZaUFBQldJaUFnQUFBQUF3PT0EDQEFAADIAAAAAAAAAAAGCQEAAAAAAAAAAANpAAJAR
mRhdGE6YXBwbGljYXRpb24vbXBlZzQtYmlmcy1hdTtiYXNlNjQsd0JBU2daTUNvRmNtRUVIOEFBQUIvQ
UFBQkVLQ0tDbjQEEgINAABkAAAAAAAAAAAFAwAAYAYJAQAAAAAAAAAA"
a=isma-compliance:1,1.0,1
a=range:npt=0-  70.00000
m=video 0 RTP/AVP 96
b=AS:209
a=rtpmap:96 MP4V-ES/90000
a=control:trackID=3
a=cliprect:0,0,480,380
a=framesize:96 380-480
a=fmtp:96 profile-level-id=1;config=000001B0F3000001B50EE040C0CF0000010000000120
008440FA285F21E0A21F
a=mpeg4-esid:201
m=audio 0 RTP/AVP 97
b=AS:48
a=rtpmap:97 mpeg4-generic/22050/2
a=control:trackID=4
a=fmtp:97 profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdelt
alength=3;config=1390
a=mpeg4-esid:101

sample_h264_100kbit.mp4: Impossible to read this video

v=0
o=StreamingServer 3463999589 1214234604000 IN IP4 137.194.232.94
s=/sample_h264_100kbit.mp4
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:2097172
t=0 0
a=control:*
a=isma-compliance:2,2.0,2
a=range:npt=0-  70.00000
m=video 0 RTP/AVP 96
b=AS:2097151
a=rtpmap:96 H264/90000
a=control:trackID=3
a=cliprect:0,0,242,192
a=framesize:96 192-242
a=fmtp:96 packetization-mode=1;profile-level-id=4D400C;sprop-parameter-sets=J01A
DKkYYELxCA==,KM4JiA==
a=mpeg4-esid:201
m=audio 0 RTP/AVP 97
b=AS:20
a=rtpmap:97 mpeg4-generic/8000/2
a=control:trackID=4
a=fmtp:97 profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdelt
alength=3;config=1590
a=mpeg4-esid:101

sample_h264_300kbit.mp4: Impossible to read this video

v=0
o=StreamingServer 3463999675 1214234605000 IN IP4 137.194.232.94
s=/sample_h264_300kbit.mp4
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:2097200
t=0 0
a=control:*
a=isma-compliance:2,2.0,2
a=range:npt=0-  70.00000
m=video 0 RTP/AVP 96
b=AS:2097151
a=rtpmap:96 H264/90000
a=control:trackID=3
a=cliprect:0,0,480,380
a=framesize:96 380-480
a=fmtp:96 packetization-mode=1;profile-level-id=4D401E;sprop-parameter-sets=J01A
HqkYMB73oA==,KM4NiA==
a=mpeg4-esid:201
m=audio 0 RTP/AVP 97
b=AS:48
a=rtpmap:97 mpeg4-generic/22050/2
a=control:trackID=4
a=fmtp:97 profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdelt
alength=3;config=1390
a=mpeg4-esid:101

sample_h264_1mbit.mp4: Impossible to read this video

v=0
o=StreamingServer 3463999710 1214234605000 IN IP4 137.194.232.94
s=/sample_h264_1mbit.mp4
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:2097279
t=0 0
a=control:*
a=isma-compliance:2,2.0,2
a=range:npt=0-  70.00000
m=video 0 RTP/AVP 96
b=AS:2097151
a=rtpmap:96 H264/90000
a=control:trackID=3
a=cliprect:0,0,480,380
a=framesize:96 380-480
a=fmtp:96 packetization-mode=1;profile-level-id=4D401E;sprop-parameter-sets=J01A
HqkYMB73oA==,KM4C+IA=
a=mpeg4-esid:201
m=audio 0 RTP/AVP 97
b=AS:127
a=rtpmap:97 mpeg4-generic/48000/2
a=control:trackID=4
a=fmtp:97 profile-level-id=15;mode=AAC-hbr;sizelength=13;indexlength=3;indexdelt
alength=3;config=1190
a=mpeg4-esid:101

movie_av_hint.mp4: Media file type not supported

v=0
o=StreamingServer 3463999989 1214235184000 IN IP4 137.194.232.94
s=/movie_av_hint.mp4
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:401
t=0 0
a=control:*
a=x-copyright: MP4/3GP File hinted with GPAC 0.4.5-DEV (build 24) (C)2000-2005 -
 http://gpac.sourceforge.net
a=range:npt=0- 195.02333
m=video 0 RTP/AVP 96
b=AS:339
a=rtpmap:96 mpeg4-generic/90000
a=control:trackID=65536
a=fmtp:96 profile-level-id=245; config=000001b0f5000001b509000001000000012000868
7fffb0aad8482098a31000001b244697658393939623030306e000001b25876694430303239; str
eamType=4; mode=generic; objectType=32; DTSDeltaLength=14
a=framesize:96 288-152
m=audio 0 RTP/AVP 97
b=AS:62
a=rtpmap:97 mpeg4-generic/44100/2
a=control:trackID=65537
a=fmtp:97 profile-level-id=41; config=1210; streamType=5; mode=AAC-hbr; objectTy
pe=103; sizeLength=13; indexLength=3; indexDeltaLength=3

counter.mp4: Media file type not supported

v=0
o=StreamingServer 3464000048 1214234604000 IN IP4 137.194.232.94
s=/counter.mp4
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:117
t=0 0
a=control:*
a=x-copyright: MP4/3GP File hinted with GPAC 0.4.3-DEV (C)2000-2005 - http://gpa
c.sourceforge.net
a=range:npt=0-  10.00000
m=video 0 RTP/AVP 96
b=AS:117
a=rtpmap:96 mpeg4-generic/90000
a=control:trackID=65536
a=fmtp:96 profile-level-id=245; config=000001b0f5000001b509000001000000012000868
400670c0f1030518f000001b244697658393939623030306e000001b25876694430303239; strea
mType=4; mode=generic; objectType=32; DTSDeltaLength=14
a=framesize:96 120-96

cvid.mp4: Media file type not supported

v=0
o=StreamingServer 3464000094 1214327349000 IN IP4 137.194.232.94
s=/cvid.mp4
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:117
t=0 0
a=control:*
a=x-copyright: MP4/3GP File hinted with GPAC 0.4.5-DEV (build 24) (C)2000-2005 -
 http://gpac.sourceforge.net
a=isma-compliance=1,1.0,1
a=mpeg4-iod:"data:application/mpeg4-iod;base64,AoJJAE8BAf/1AQOBVwACQLBkYXRhOmFwc
GxpY2F0aW9uL21wZWc0LW9kLWF1O2Jhc2U2NCxBV2NCWlFVZkEyRUF5U0FBQWdSSUlCRUFEOGNBQWlaW
UFBSEtPQVU1QUFBQnNQVUFBQUcxQ1FBQUFRQUFBQUVnQUlhRUFHY01EeEF3VVk4QUFBR3lSR2wyV0RrN
U9XSXdNREJ1QUFBQnNsaDJhVVF3TURJNUJoQUFKQUFCWDVBQUFBQUFJQUFBQUFRRAQPAQUAAGkAAAAAA
AAAAAUABhAA1AAAAlgAAAAAIAAAAAADA2YAAWA2ZGF0YTphcHBsaWNhdGlvbi9tcGVnNC1iaWZzLWF1O
2Jhc2U2NCx3QkFTWUVLQ0tDblFUd0E9AAIEFgINAAALAAAAAAAAAAAFBwAAcAeABgAGEADUAAACWAAAA
AAgAAAAAAM="
a=range:npt=0-  10.00000
m=video 0 RTP/AVP 96
b=AS:117
a=rtpmap:96 mpeg4-generic/90000
a=mpeg4-esid:201
a=control:trackID=65736
a=fmtp:96 profile-level-id=245; config=000001b0f5000001b509000001000000012000868
400670c0f1030518f000001b244697658393939623030306e000001b25876694430303239; strea
mType=4; mode=generic; objectType=32; DTSDeltaLength=14
a=framesize:96 120-96

test.3gp: Success – Video only

v=0
o=StreamingServer 3464000121 1214234605000 IN IP4 137.194.232.94
s=/test.3gp
u=http:///
e=admin@
c=IN IP4 0.0.0.0
b=AS:91
t=0 0
a=control:*
a=x-copyright: MP4/3GP File hinted with GPAC 0.4.3-DEV (C)2000-2005 - http://gpa
c.sourceforge.net
a=range:npt=0- 158.60000
m=video 0 RTP/AVP 96
b=AS:91
a=rtpmap:96 H263-1998/90000
a=control:trackID=65536
a=cliprect:0,0,96,128
a=framesize:96 128-96

GPAC MP4Streamer Details

SDP Multicast session

  • local: not reckognised
  • remote, served as text/plain: not supported
  • remote, served as application/sdp: recognized but cannot connect ? Multicast problem ?

SDP Multicast session (simplified):

HTTP or local files Details

Video 1

  • local: success
  • remote, served as text/plain: not supported
  • remote, served as video/mp4: cannot open this file ??