NinSuna: Metadata-driven Media Adaptation and Delivery

Our Model-driven media delivery platform is a fully integrated platform for multimedia content adaptation and delivery. Its basic design is inspired by the principles of XML-driven content adaptation techniques, while its final design and the implementation thereof are based on Semantic Web technologies such as the Resource Description Framework (RDF), Web Ontology Language (OWL), and SPARQL Protocol And RDF Query Language (SPARQL).

Because the platform is fully based on Semantic Web technologies (see above), advanced queries can be constructed by a client application and efficiently processed by the platform. This way, fully customized search applications can be developed to interact with the platform while searching for media content. Because media resources can be annotated down to the scene level, it is possible to only return the relevant media fragments that correspond to the search query (instead of the full media resource).

Features

The most important features of our media delivery platform are:

• format-independence: the platform software is independent of media formats (i.e., both coding and container formats);
• extensibility: new media formats can be added by means of plugins;
• scalability: platform modules can be distributed over different machines;
• media delivery: the platform supports streaming media delivery through RTSP, RTMP, HTTP progressive download (MP4, Ogg, 3GPP, MPEG-2 TS), and HTTP Live streaming;
• RDF-based metadata: a fully RDF-based storage backend for descriptive metadata which is accessible through a SPARQL endpoint;
• multimedia ontology: a central multimedia ontology couples the media resources and any available metadata (originating from any RDF-based metadata model);
• metadata models: a number of metadata parsers import and convert XML-based metadata to an equivalent RDF representation (e.g., NewsML) while parsers for other metadata schemes can be added as plugins;
• adaptation: The platform supports various forms of content adaptation: scenes or shots can be extracted from any media resource to facilitate fine-grained search queries, requested media fragments can be adapted depending on the scalability provisions in the media stream, and multiple track selection is supported as well (e.g., video stream selection based on bit rate);
• W3C Media Fragments: the platform has a server-side reference implementation of the W3C Media Fragments URI 1.0 specification. More information can be found here.

Motivation

Current multimedia landscape is characterized by

• growing amount of multimedia content (e.g., user-generated content)
• growing diversity in usage environments (e.g., mobile devices)
• growing interest in personalized multimedia content (e.g, intelligent video summarization)
• growing amount of multimedia metadata formats (e.g., MPEG-7, Dublin Core, …)
• growing amount of multimedia coding formats (e.g., H.264/AVC, VC-1, …)
• growing amount of multimedia delivery formats (e.g., MP4, Ogg, …)

In order to deal with the current multimedia landscape characteristics, efficient techniques for multimedia content adaptation and delivery are desired:

• coding-format independent multimedia adaptation and delivery engines
• seamless integration with multimedia metadata formats
• fully integrated, flexible, and responsive multimedia delivery platform

Architecture

A tight coupling exists between the design of the media delivery platform and a model for describing structural, content, and scalability information of multimedia bitstreams. This model, implemented by making use of OWL, provides support for a seamless integration of the adaptation operations and semantic metadata. Therefore, it enables the definition of adaptation operations on a higher level (i.e., based on the model). Furthermore, when existing coding formats are mapped to this model, they can be adapted in a format-independent way.

Within our platform, support for the authoring and retrieval of multimedia is provided, together with the ability to adapt and deliver the multimedia content in a format-independent way. Clients can request media content through RTSP, RTMP, and HTTP. Before media content becomes available through the platform, this media needs to be ingested. Information regarding multimedia content can be obtained via a SPARQL endpoint. Requested content is selected, adapted, and packaged using format-independent adaptation and packaging engines and delivered to the client by making use of the RTP/RTSP streaming service, the RTMP streaming service, or the HTTP download service. A high-level overview of the architecture of the NinSuna platform is also depicted in Figure 1.

Technical Details

• The adaptation and packaging engines are implemented in Java.
• The RDF repository is an Openlink Virtuoso database engine.
• The RTP/RTSP streaming service is a Darwin Streaming Server with a customized module.
• The RTMP streaming service is a Wowza streaming server with a customized module

Publications

• Davy Van Deursen, Wim Van Lancker, and Rik Van de Walle. On Media Delivery Protocols in the Web. Accepted for publication in the International Conference on Multimedia & Expo (ICME), 2010.
• Davy Van Deursen, Wim Van Lancker, Sarah De Bruyne, Wesley De Neve, Erik Mannens, and Rik Van de Walle. Format-independent and metadata-driven media resource adaptation using semantic web technologies. Multimedia Systems, Vol. 16, Nr. 2, March 2010. Available on http://www.springerlink.com/content/20434747039865n5/.
• Davy Van Deursen, Wim Van Lancker, Wesley De Neve, Tom Paridaens, Erik Mannens, and Rik Van de Walle. NinSuna: a fully integrated platform for format-independent multimedia content adaptation and delivery using semantic web technologies. Multimedia Tools and applications - Special Issue on Data Semantics for Multimedia Systems, Vol. 46, Nr. 2-3, January 2010. Available on http://www.springerlink.com/content/461380502m756877/.
• Davy Van Deursen. Format-Independent Media Resource Adaptation and Delivery. PhD Thesis, Ghent University, Ghent, Belgium, June 18, 2009 (pdf).
• Davy Van Deursen, Wesley De Neve, Wim Van Lancker and Rik Van de Walle. Semantic adaptation of synchronized multimedia streams in a format-independent way. In proceedings of the 27th Picture Coding Symposium. pp. 1-4 on CD-ROM, May 2009, Chicago, USA.
• Davy Van Deursen, Wim Van Lancker, Tom Paridaens, Wesley De Neve, Erik Mannens and Rik Van de Walle. NinSuna: a Format-independent Multimedia Content Adaptation Platform based on Semantic Web Technologies. In Proceedings of the 10th International Symposium on Multimedia, pp. 491-492, December 2008, Berkeley, USA.
• Davy Van Deursen, Sarah De Bruyne, Wim Van Lancker, Wesley De Neve, Davy De Schrijver, Hermann Hellwagner and Rik Van de Walle. MuMiVA: a Multimedia Delivery Platform using Format-agnostic, XML-driven Content Adaptation. In proceedings of the 9th International Symposium on Multimedia, pp. 131 -138, December 2007, Taichung, Taiwan.

Name Origin

Our media delivery platform is called ‘NinSuna’, which stands for “The NinSuna INtelligent Search framework for UNiversal media Access”. Note the origin of the term ‘NinSuna’: A minor Sumerian goddess noted for her wisdom, and whose house was in Erech (Uruk). She is the wife of the deified king Lugulbanda and mother of the great hero Gilgamesh. In the Gilgamesh epic she appears as a counselor of her son and interprets dreams. Her name means “queen of the wild cow” (Encyclopedia Mythica).