image_annotation.html 112 KB

Raw Blame History Permalink

<?xml version='1.0' encoding='utf-8' standalone="yes"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xml:lang="en-US" xmlns="http://www.w3.org/1999/xhtml" lang="en-US">
  <head>
    <title>Image Annotation on the Semantic Web</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
    <style type="text/css">

.new { color: #FF0000 }

.example {font-family: monospace; }

.figure {
         font-weight: bold;
         text-align: center; }

div.example {
    padding: 1em;
    margin: 0.1em 3.5em 0.1em 0.1em;
    background-color: #efeff5;
    border: 1px solid #cfcfcf; }

div.exampleOuter {
   margin: 0em;
   padding: 0em;
}

div.exampleInner {
   color: black;
   background-color: #efeff5;
   border-top-style: double;
   border-top-color: #d3d3d3;
   border-bottom-width: 1px;
   border-bottom-style: double;
   border-bottom-color: #d3d3d3;
   padding: 4px;
   margin: 0em;
}

div.exampleInner pre {
  margin-left: 0em;
  margin-top: 0em;
  margin-bottom: 0em;
  font-family: monospace;
}

div.c1 {text-align:center}

  </style>
  <link
   href="http://www.w3.org/StyleSheets/TR/base.css"
   type="text/css" rel="stylesheet"/>
 </head>

 <body>
  <div class="head">
   <p>
    <a href="http://www.w3.org/">
     <img alt="W3C" src="http://www.w3.org/Icons/w3c_home"
      height="48" width="72"/>
    </a>
   </p>

   <h1>Image Annotation on the Semantic Web</h1>
   <h2>Editors' Draft $Date: 2007/02/15 16:03:24 $ $Revision: 1.226 $</h2>

   <dl>
    <dt>This version: </dt>
    <dd>
     <a href="http://www.w3.org/2001/sw/BestPractices/MM/image_annotation.html">
      http://www.w3.org/2001/sw/BestPractices/MM/image_annotation.html</a></dd>
    <dt>Latest version: </dt>
    <dd>
     <a href="http://www.w3.org/2005/Incubator/mmsem/XGR-image-annotation/">
      http://www.w3.org/2005/Incubator/mmsem/XGR-image-annotation/</a></dd>
    <dt>Previous version:</dt>
    <dd>
     <a href="http://www.w3.org/TR/2006/WD-swbp-image-annotation-20060322/">
      http://www.w3.org/TR/2006/WD-swbp-image-annotation-20060322/
     </a>
    </dd>


    <dt>Editors: </dt>
    <dd><a href="http://www.cwi.nl/~jrvosse/">Jacco van Ossenbruggen</a>, Center for Mathematics and Computer Science (CWI Amsterdam)</dd>
    <dd><a href="http://www.cwi.nl/~troncy/">Rapha&#235;l Troncy</a>, Center for Mathematics and Computer Science (CWI Amsterdam)</dd>
    <dd><a href="http://www.image.ntua.gr/~gstam/">Giorgos Stamou</a>, IVML, National Technical University of Athens</dd>
    <dd><a href="http://www.csd.abdn.ac.uk/~jpan/">Jeff Z. Pan</a>, University of Aberdeen (Formerly University of Manchester)</dd>

    <dt>Contributors: </dt>
    <dd><a href="http://www.mindswap.org/~chris/">Christian Halaschek-Wiener</a>, University of Maryland</dd>
    <dd><a href="mailto:nsimou@image.ece.ntua.gr">Nikolaos Simou</a>, IVML, National Technical University of Athens</dd>
    <dd><a href="mailto:tzouvaras@image.ntua.gr">Vassilis Tzouvaras</a>, IVML, National Technical University of Athens</dd>

    <dt>&#160; </dt>
    <dd>Also see <a href="#acknowledgments">Acknowledgements</a>.</dd>
      </dl>

      <p>
    <a href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">
      Copyright
    </a>
    &#169; 2006
    <a href="http://www.w3.org/">
      <acronym title="World Wide Web Consortium">W3C</acronym>
    </a><sup>&#174;</sup>

    (<a href="http://www.csail.mit.edu/"><acronym
    title="Massachusetts Institute of
    Technology">MIT</acronym></a>, <a
    href="http://www.ercim.org/"><acronym title="European Research
    Consortium for Informatics and
    Mathematics">ERCIM</acronym></a>, <a
    href="http://www.keio.ac.jp/">Keio</a>), All Rights
    Reserved. W3C <a
    href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>,
    <a
    href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a>
    and <a
    href="http://www.w3.org/Consortium/Legal/copyright-documents">document
    use </a> rules apply.
      </p>
      <hr/>
    </div>

    <h2>
      <a id="abstract" name="abstract">
    Abstract
      </a>
    </h2>

    <p>
      Many applications that process multimedia content make use of
      some form of metadata that describe the multimedia content.  The
      goals of this document are to explain the advantages of using
      Semantic Web languages and technologies for the creation,
      storage, manipulation, interchange and processing of image
      metadata.  In addition, it provides guidelines for Semantic
      Web-based image annotation, illustrated by use cases.  Relevant
      RDF and OWL vocabularies are discussed, along with a short
      overview of publicly available tools.
    </p>

    <h2>
      <a id="roadmap" name="roadmap">
    Document Roadmap
      </a>
    </h2>
    <p>
      After reading this document, readers may
      turn to separate documents discussing individual image
      annotation <a
      href="http://www.w3.org/2001/sw/BestPractices/MM/resources/Vocabularies.html">vocabularies</a>,
      <a
      href="http://www.w3.org/2001/sw/BestPractices/MM/resources/Tools.html">tools</a>,
      and other <a
      href="http://www.w3.org/2001/sw/BestPractices/MM/resources/Resources.html">relevant
      resources</a>.
      Most of the current approaches to image annotation are not based on Semantic
      Web languages. Interoperability between these technologies and
      RDF and OWL-based approaches is <em>not</em> the topic of this document.
    </p>

    <h2>
      <a id="targetaudience" name="targetaudience">
    Target Audience
      </a>
    </h2>

    <p>
      This document is target at everybody with an interest in image
      annotation, ranging from non-professional end-users that are
      annotating their personal digital photos to professionals
      working with digital pictures in image and video banks,
      audiovisual archives, museums, libraries, media production and
      broadcast industry, etc.
    </p>

    <h2>
      <a id="objectives" name="objectives">
    Objectives
      </a>
    </h2>

    <ul>
      <li>
    To illustrate the benefits of using semantic technologies in image annotations.</li>

      <li>
    To provide guidelines for applying semantic technologies in this area.</li>

      <li>
    To collect currently used vocabularies for Semantic Web-based
    image annotation.</li>

      <li>
    To provide use cases with examples of Semantic Web-based
    image annotation.</li>
    </ul>

    <h2>
      <a id="status" name="status">Status of this document</a>
    </h2>

    <p>
      This is an editors' draft intended to become a
      <a href="http://www.w3.org/2003/06/Process-20030618/tr.html#q71">
    Working Group Note</a> produced by the <a
    href="http://www.w3.org/2001/sw/BestPractices/MM/">Multimedia
    Annotation on the Semantic Web Task Force</a> of the <a
    href="http://www.w3.org/2001/sw/BestPractices/">W3C Semantic
    Web Best Practices &amp; Deployment Working Group</a>, which
    is part of the <a href="http://www.w3.org/2001/sw/">W3C
    Semantic Web</a> activity.
    </p>

    <p>Discussion of this document
    is invited on the public mailing list <a
    href="mailto:public-swbp-wg@w3.org">public-swbp-wg@w3.org</a>
    (<a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/">public
    archives</a>). Public comments should include "comments: [MM]"
    at the start of the Subject header. </p> <p>Publication as a
    Working Group Note does not imply endorsement by the W3C
    Membership. This is a draft document and may be updated,
    replaced or obsoleted by other documents at any time. It is
    inappropriate to cite this document as other than work in
    progress.  Other documents may supersede this document.
    </p>


  <div class="toc">
   <h2 class="notoc">
    <a id="contents" name="contents">Table of Contents</a>
   </h2>

   <ul id="toc" class="toc">
    <li class="tocline"><a href="#use_cases"><b>1. Use Cases</b></a>
     <ul class="toc">
      <li class="tocline"><a href="#photo_collection">1.1. Use Case: Management of Personal Digital Photo Collections</a></li>
      <li class="tocline"><a href="#cultural_heritage">1.2. Use Case: Cultural Heritage</a></li>
      <li class="tocline"><a href="#television_archive">1.3. Use Case: Television Archive</a></li>
      <li class="tocline"><a href="#large_collection">1.4. Use Case: Large-scale Image Collections at NASA</a></li>
      <li class="tocline"><a href="#biomed">1.5. Use Case: Biomedical Images</a></li>
     </ul>
    </li>
    <li class="tocline"><a href="#introduction"><b>2. Introduction to Image Annotation and to the Semantic Web</b></a>
     <ul class="toc">
      <li class="tocline"><a href="#annot_intro">2.1. Image Annotation Issues</a></li>
      <li class="tocline"><a href="#semweb_intro">2.2. Semantic Web Basics</a></li>
     </ul>
    </li>
    <li class="tocline"><a href="#vocabularies"><b>3. Vocabularies for Image Annotation</b></a></li>
    <li class="tocline"><a href="#tools"><b>4. Available Tools for Semantic Image Annotation</b></a></li>
    <li class="tocline"><a href="#examples"><b>5. Example Solutions to the Use Cases</b></a>
     <ul class="toc">
      <li class="tocline"><a href="#solution_personal">5.1. Use Case: Management of Personal Digital Photo Collections</a></li>
      <li class="tocline"><a href="#solution_culture">5.2. Use Case: Cultural Heritage</a></li>
      <li class="tocline"><a href="#solution_TVarchive">5.3. Use Case: Television Archive</a></li>
      <li class="tocline"><a href="#solution_NASA">5.4. Use Case: Large-scale Image Collections at NASA</a></li>
      <li class="tocline"><a href="#solution_biomed">5.5. Use Case: Biomedical Images</a></li>
     </ul>
    </li>
    <li class="tocline"><a href="#conclusions"><b>6. Conclusions</b></a></li>
    <li class="tocline"><a href="#references"><b>References</b></a></li>
    <li class="tocline"><a href="#acknowledgments"><b>Acknowledgments</b></a></li>
    <li class="tocline"><a href="#changelog"><b>Change log</b></a></li>
   </ul>
  </div>

  <h2>
   <a name="use_cases">
    1. Use Cases
   </a>
  </h2>

  <p>
   The need for annotating digital image data is recognized in a wide
   variety of different applications, covering both professional and
   personal usage of image data. At the time of writing, most work
   done in this area does not yet use semantic-based technologies.
   This document explains the advantages of using Semantic Web
   languages and technologies for image annotations and provides
   guidelines for doing so. It is organized around a number of
   representative use cases, and a description of Semantic Web
   vocabularies and tools that could be used to help accomplish the
   task mentioned in the uses cases.
  </p>

  <p>
   The use cases are intended as a representative set of examples,
   providing both personal and industrial, and open and closed domain
   examples.  They will be used later to discuss the vocabularies and
   tools that are relevant for image annotation on the Semantic
   Web. Example scenarios are given in <a href="#examples">Section
   5</a>.  The use cases differ in the topics depicted by the images
   or their usage community.  These criteria often determine the tools
   and vocabularies used in the annotation process.
  </p>

  <h3>
   <a name="photo_collection">
    1.1. Use Case: Management of Personal Digital Photo Collections
   </a>
  </h3>

  <p>
   Many personal users have thousands of digital photos from vacations,
   parties, traveling, friends and family, everyday life, etc. Typically, the
   photos are stored on personal computer hard drives in a simple
   directory structure without any metadata. The user wants generally
   to easily access this content, view it, use it in his homepage,
   create presentations, make part of it accessible for other people
   or even sell part of it to image banks. Too often, however, the
   only way for this content to be accessed is by browsing the
   directories, their name providing usually the date and the
   description with one or two words of the original event captured by
   the specific photos. Obviously, this access becomes more and more
   difficult as the number of photos increases and the content becomes
   quickly unused in practice.  More sophisticated users leverage
   simple photo organizing tools allowing them to provide keyword
   metadata, possibly along with a simple taxonomy of categories. This
   is a first step towards a semantically-enabled solution. <a
   href="#solution_personal">Section 5.1</a> provides an example
   scenario for this use case using Semantic Web technologies.
  </p>

  <h3>
   <a name="cultural_heritage">
    1.2. Use Case: Cultural Heritage
   </a>
  </h3>
  <p>
   Let us imagine that a museum in fine arts has asked a specialized
   company to produce high resolution digital scans of the most
   important art works of their collections. The museum's quality
   assurance requires the possibility to track when, where and by whom
   every scan was made, with what equipment, etc. The museum's
   internal IT department, maintaining the underlying image database,
   needs the size, resolution and format of every resulting image. It
   also needs to know the repository ID of the original work of
   art. The company developing the museum's website additionally
   requires copyright information (that varies for every scan,
   depending on the age of the original work of art and the collection
   it originates from). It also want to give the users of the website
   access to the collection, not only based on the titles of the
   paintings and names of their painters, but also based on the topics
   depicted ('sun sets'), genre ('self portraits'), style
   ('post-impressionism'), period ('fin de si&#232;cle'), region
   ('west European'). <a href="#solution_culture">Section 5.2</a>
   shows how all these requirements can be fulfilled using Semantic
   Web technologies.
  </p>


  <h3>
   <a name="television_archive">
    1.3. Use Case: Television Archive
   </a>
  </h3>

  <p>
   Audiovisual archive centers are used to manage very large
   multimedia databases. For instance, INA, the French Audiovisual
   National Institute, has been archiving TV documents since the fifties,
   the radio documents since the forties, and stores more than 1 million
   hours of broadcast programs. They have recently put online and in free access
   more than 10000 hours of French TV programs. The images and sound archives kept at
   INA are primarily intended for professional use (journalists, film
   directors, producers, audiovisual and multimedia programmers and
   publishers, in France and worldwide) or communicated for research
   purposes (for a public of students, research workers, teachers and
   writers), but they are, now, more and more available for the general public.
   In order to allow an efficient access to the data stored,
   most of the parts of these video documents are described and
   indexed by their content. The global multimedia information system
   should then be fine-grain enough detailed to support some very
   complex and precise queries. For example, a journalist or a film
   director client might ask for an excerpt of a previously
   broadcasted program showing the first goal of a given football
   player in its national team, scored with its head. The query could
   additionally contain some more technical requirements such that the
   goal action should be available according to both the front camera
   view and the reverse angle camera view. Finally, the client might
   or might not remember some general information about this football
   game, such that the date, the place and the final score. <a
   href="#solution_TVarchive">Section 5.3</a> gives a possible
   solution for this use case using Semantic Web technologies.
  </p>

  <h3>
   <a name="large_collection">
    1.4. Use Case: Large-scale Image Collections at NASA
   </a>
  </h3>

  <p>
   Many organizations maintain extremely large-scale image
   collections. The National Aeronautics and Space Administration
   (NASA), for example, has hundreds of thousands of images, stored in
   different formats, levels of availability and resolution, and with
   associated descriptive information at various levels of detail and
   formality. Such an organization also generates thousands of images
   on an ongoing basis that are collected and cataloged. Thus, a
   mechanism is needed to catalog all the different types of image
   content across various domains. Information about both the image
   itself (e.g., its creation date, dpi, source) and about the
   specific content of the image is required. Additionally, the
   associated metadata must be maintainable and extensible so that
   associated relationships between images and data can evolve
   cumulatively. Lastly, management functionality should provide
   mechanisms flexible enough to enforce restriction based on content
   type, ownership, authorization, etc.  <a
   href="#solution_NASA">Section 5.4</a> gives an example solution for
   this use case.
  </p>

  <h3>
   <a name="biomed">
    1.5. Use Case: Biomedical Images
   </a>
  </h3>

  <p>
      An health-care provider might be interested in annotating medical
      images for clinical applications, education or other purposes.
      Diagnostic images involved in patient care need to be properly
      annotated from different points of view: the image modality
      (radiograph, MRI, etc) and its acquisition parameters (including
      time), the patient and anatomical body part depicted by the
      image, the associated report data such as diagnosis reports and
      electronic patient records. When such images are used for
      medical education, anonymization is needed in order
      to not reveal personal data to unauthorized users. Educational
      images need a clear description of modality, anatomical parts,
      and morphological and diagnostic comments. Such metadata may
      refer to the complete image or, often, to just one or more
      subregions, which need to be identified in some way. Educational
      metadata could be directly shown to users or maintained hidden
      to develop self-learning skills.
  <a
   href="#solution_biomed">Section 5.5</a> gives an example solution addressing
   these requirements.
  </p>

  <h2>
   <a name="introduction" id="introduction">
    2. Introduction to Image Annotation and to the Semantic Web
   </a>
  </h2>
  <p>
   Before discussing example solutions to the use cases, this section
   provides a short overview of image annotation in general, focussing
   on the issues that often make image annotation much more complex
   that initially expected.  In addition, it briefly discusses the
   basic Semantic Web concepts that are required to understand the
   solutions provided later.
  </p>

  <h3>
   <a name="annot_intro">
    2.1 Image Annotation Issues
   </a>
  </h3>

  <p>
   Annotating images on a small scale for personal usage, as described
   in the first use case above, can be relatively simple, as
   examplified by popular keyword-based tagging systems such as <a
   href="#Flickr">Flickr</a>.

   Unfortunately, for more ambitious annotation tasks, the situation
   quickly becomes less simple.  Larger scale industrial strength
   image annotation is notoriously complex. Trade offs along several
   dimensions make the professional multimedia annotations difficult:
  </p>

  <ol>
   <li>
    <p>
     <em>Production versus post-production annotation</em>
    </p>
    <p>
     A general rule is that it is much easier to annotate earlier
     rather than later.  Typically, most of the information that is
     needed for making the annotations is available during production
     time. Examples include time and date, lens settings and other
     EXIF metadata added to JPEG images by most digital cameras at the
     time a picture is taken, experimental data in scientific and
     medical images, information from scripts, story boards and edit
     decision lists in creative industry, etc.  Indeed, maybe the
     single most best practice in image annotation is that in general,
     adding metadata during the production process is much cheaper and
     yields higher quality annotations than adding metadata in a later
     stage (such as by automatic analysis of the digital artifact or
     by manual post-production data).
    </p>
   </li>

   <li>
    <p>
     <em>
      Generic vs task-specific annotation
     </em>
    </p>
    <p>
     Annotating images without having a specific context, goal or task
     in mind is often not cost effective: after the target application
     has been developed, it may turn out that images have been
     annotated with information that insufficiently covers the new
     requirements.  Redoing the annotations is then an unavoidable,
     but costly solution.  On the other hand, annotating with
     <em>only</em> the target application in mind may also not be cost
     effective.  The annotations may work well with that one
     application, but if the same metadata is to be reused in the
     context of other applications, it may turn out to be too
     specific, and unsuited for reuse in a different context.  In most
     situations the range of applications in which the metadata will
     be used in the future is unknown at the time of annotation.  When
     lacking a crystal ball, the best the annotator can do in practice
     is use an approach that is sufficiently specific for the
     application under development, while avoiding unnecessary
     application-specific assumptions as much as possible.
    </p>
   </li>

   <li>
    <p>
     <em>
      Manual versus automatic annotation and the "Semantic Gap"
     </em>
    </p>
    <p>
     In general, manual annotation can provide image descriptions at
     the right level of abstraction. It is, however, time consuming
     and thus expensive. In addition, it proves to be highly
     subjective: different human annotators tend to "see" different
     things in the same image. On the other hand, annotation based on
     automatic feature extraction is relatively fast and cheap, and
     can be more systematic. It tends to result, however, in image
     descriptions that are too low level for many applications. The
     difference between the low level feature descriptions provided by
     image analysis tools and the high level content descriptions
     required by the applications is often referred to, in the
     literature, as the <em>Semantic Gap</em>. In the remainder, we
     will discuss use cases solutions, vocabularies and tools for both
     manual and automatic image annotation.
    </p>
   </li>

   <li>
    <p>
     <em>
      Different types of metadata
     </em>
    </p>
    <p>
     While various classifications of metadata have been described in
     the literature, every annotator should at least be aware of the
     difference between annotations describing properties of the image
     itself, and those describing the subject matter of the image,
     that is, the properties of the objects, persons or concepts
     depicted by the image.  In the first category, typical
     annotations provide information about title, creator, resolution,
     image format, image size, copyright, year of publication, etc.
     Many applications use a common, predefined and relatively small
     vocabulary defining such properties.  Examples include the <a
     href="#DublinCore">Dublin Core</a> and <a href="#VraCore">VRA
     Core</a> vocabularies. The second category describes what is
     depicted by the image, which can vary wildly with the type of
     image at hand.  In many applications, it is also useful to
     distinguish between objective observations (<em>'the person in the
     white shirt moves his arm from left to right'</em>) versus subjective
     interpretations (<em>'the person seems to perform a martial arts
     exercise'</em>).  As a result, one sees a large variation in
     vocabularies used for this purpose. Typical examples vary from
     domain-specific vocabularies (for example, with terms that are
     very specific for astronomy images or sport images) to
     domain-independent ones (for example, a vocabulary with terms
     sufficiently generic to describe any news photo). In addition,
     vocabularies tend to differ in size, granularity, formality etc.
     In the remainder, we discuss the above metadata categories.  Note
     that in the first type it is not uncommon that a vocabulary only
     defines the properties and defers the definitions of the values
     of those properties to another vocabulary.  This is true, for
     example, for both Dublin Core and VRA Core.  This means that,
     typically, in order to annotate a single image one needs terms
     from multiple vocabularies.
    </p>
   </li>

   <li>
    <p>
     <em>
      Lack of Syntactic and Semantic Interoperability
     </em>
    </p>
    <p>
     Many different file formats and tools for image annotations are
     currently in use.  Reusing metadata created by another tool is
     often hindered by a lack of interoperability. First, a tool may
     use a different syntax for its file formats. As a consequence,
     other tools are not able to read in the annotations produced by
     this tool.  Second, a particular tool may assign a different
     meaning (semantics) to the same annotation.  In this situation,
     the tool may be able to read annotations from other tools, but
     will fail to process them in the way originally intended.
    </p>
    <p>
     Both problems can be solved by using Semantic Web technology.
     First, the Semantic Web provides developers with means to
     explicitly specify the syntax and semantics of their metadata.
     Second, it allows tool developers to make explicit how their
     terminology relates to that of other tools.  The document will
     provide examples of both cases.
    </p>
   </li>
  </ol>

  <h3>
   <a name="semweb_intro">
    2.2 Semantic Web Basics
   </a>
  </h3>

  <p>
   This section briefly describe the role of Semantic Web technologies
   in image annotations. The aim of the Semantic Web is to augment the
   existing Web so that resources (Web pages, images etc.) are more
   easily interpreted by programs (or "intelligent agents").  The idea
   is to associate Web resources with semantic categories which
   describe the contents and/or functionalities of Web resources.
  </p>

  <p>
   Annotations alone do not establish the semantics of what is being
   marked-up. One way generally followed to introduce semantics to
   annotations is to get a community to agree on what a set of
   concepts mean, and what terms have to be used for them.
  </p>

  <h4>Informal versus formal semantics</h4>
  <p>
   Consider the following example of an image anntotated using the <a
   href="#DublinCore">Dublin Core</a> Metadata Element Set.  The
   Dublin Core provides 15 "core" information properties, such as
   "Title", "Creator" or "Date".  The following RDF/XML example code
   represents the statements: 'There is an image <tt>Ganesh.jpg</tt>
   created by <tt>Jeff Z. Pan</tt> whose title is <tt>An image of
   the Elephant Ganesh</tt>'.  The first four lines define the <a
   href="#XML-NS">XML namespaces</a> used in this description. A good
   starting point for more information on RDF is the <a
   href="#RDF-Primer">RDF Primer</a>.
  </p>

  <div class="exampleInner" style="clear: both">
   <pre>
&lt;rdf:RDF xml:base=&quot;http://example.org/&quot;
         xmlns=&quot;http://example.org/&quot;
         xmlns:dc=&quot;http://purl.org/dc/elements/1.1/&quot;
         xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot;&gt;

  &lt;rdf:Description rdf:about=&quot;Ganesh.jpg&quot;&gt;
    &lt;dc:title&gt;An image of the Elephant Ganesh&lt;/dc:title&gt;
    &lt;dc:creator&gt;Jeff Z. Pan&lt;/dc:creator&gt;
  &lt;/rdf:Description&gt;
&lt;/rdf:RDF&gt;</pre>
  </div>

  <p>
   The disadvantage of the example above is that the meaning of all
   the metadata values are strings, which cannot easily be interpreted
   by automated procedures.  Applications that need to process the
   image based on this meaning, need to have this processing hardwired
   into their code.  For example, it would be hard for an application
   to find out that this image could match queries such as "show me
   all images with animals" or "show me all images of Indian deities".
   An approach to solve this is to take the informal agreement about
   the meaning of the terms used in the domain as a starting point and
   to convert this into a formal, machine processable version.  This
   version is than typically referred to as an <em>ontology</em>.  It
   provides, for a specific domain, a shared and common
   <em>vocabulary</em>, including important concepts, their
   properties, mutual relations and constraints.  This formal
   agreement can be communicated between people and heterogeneous,
   distributed application systems.  The ontology-based approach might
   be more difficult to develop. It is, however, more powerful than
   the approach that is based only on informal agreement.  For example,
   machines can use the formal meaning for reasoning about, completing
   and validating the annotations.  In addition, human users can more
   thoroughly define and check the vocabulary using axioms expressed
   in a logic language.  Ideally, the concepts and properties of an
   ontology should have both formal definitions and natural language
   descriptions to be unambiguously used by humans and software
   applications.
  </p>

  <p>
   For example, suppose the community that is interested in this
   picture has agreed to use concepts from the <a
   href="#WordNet">WordNet</a> vocabulary to annotate its images.  One
   could simple add two lines of RDF under the <tt>dc:creator</tt> line
   of the example above:
  </p>

  <div class="exampleInner" style="clear: both">
   <pre>
&lt;dc:subject rdf:resource="http://www.w3.org/2006/03/wn/wn20/instances/synset-Indian_elephant-noun-1"/&gt;
&lt;dc:subject rdf:resource="http://www.w3.org/2006/03/wn/wn20/instances/synset-Ganesh-noun-1"/&gt;
   </pre>
  </div>

  <p>
   Now, by refering to the formally defined concepts of "Indian
   Elephant" and "Ganesh" in WordNet, even applications that have
   never encountered these concepts before could request more
   information about the WordNet definitions of the concepts and find
   out, for example, that "Ganesh" is a hyponym of the concept "Indian
   Deity", and that "Indian Elephant" is a member of the "Elephas"
   genus of the "Elephantidae" family in the animal kingdom.
  </p>

  <h2>
   <a name="vocabularies">
    3. Vocabularies for Image Annotation
   </a>
  </h2>

  <p>
      Choosing which vocabularies to use for annotating image is a key
      decision in an annotation project.  Typically, one needs more
      than a single vocabulary to cover the different relevant aspects
      of the images.  A separate document named
      <a href="http://www.w3.org/2001/sw/BestPractices/MM/resources/Vocabularies.html">Vocabularies
      Overview</a> discusses a number of individual vocabularies that
      are relevant for images annotation.  The remainder of this
      section discusses more general issues.
    </p>

    <p>
      Many of the relevant vocabularies have been developed prior to
      the Semantic Web, and <a
      href="http://www.w3.org/2001/sw/BestPractices/MM/resources/Vocabularies.html">Vocabularies
      Overview</a> lists
      many translations of such vocabularies to RDF or OWL.  Most
      notably, the key International Standard in this area, the <a
      href="#MPEG-7">Multimedia Content Description</a> standard,
      widely known as MPEG-7, is defined using XML Schema.  At the
      time of writing, there is no commonly accepted mapping from the
      XML Schema definitions in the standard to RDF or OWL.  Several
      alternative mappings, however, have been developed so far and
      are discussed in the overview.
    </p>

    <p>
      Another relevant vocabulary is the <a href="#VraCore">VRA
      Core</a>.  Where the <a href="#DublinCore">Dublin Core</a> (DC)
      specifies a small and commonly used vocabulary for on-line
      resources in general, VRA Core defines a similar set targeted
      especially at visual resources, specializing the DC elements.
      Dublin Core and VRA Core both refer to terms in their
      vocabularies as <em>elements</em>, and both use
      <em>qualifiers</em> to refine elements in similar way. All the
      elements of VRA Core have either direct mappings to comparable
      fields in Dublin Core or are defined as specializations of one
      or more DC elements.  Furthermore, both vocabularies are defined
      in a way that abstracts from implementation issues and
      underlying serialization languages. A key difference, however,
      is that for Dublin Core, there exists a commonly accepted
      mapping to RDF, along with the associated schema. At the time of
      writing, this is not the case for VRA Core, and the overview
      discusses the pros and cons of the alternative mappings.
    </p>

    <p>
      Many annotations on the Semantic Web are about an entire
      resource. For example, a <tt>&lt;dc:title&gt;</tt> property
      applies to the entire document. For images and other multimedia
      documents, one often needs to annotate a specific part of a
      resource (for example, a region in an image). Sharing the
      metadata dealing with the localization of some specific part of
      multimedia content is important since it allows to have multiple
      annotations (potentially from multiple users) referring to the
      same content.
    </p>

    <p>
      There are at least two possibilities to annotate a spatially-localized
      region of an image:
    </p>

    <ol>
      <li>
    In the ideal situation, the target image already specifies this specific
    part, using a (anchor) name that is addressable in the URI fragment
    identifier. This can be done, for example, in <a href="#SVG">SVG</a>.
      </li>
      <li>
    Otherwise, the region needs to be described in the metadata itself, as
    it is done in <a href="#MPEG-7">MPEG-7</a>. The drawback of this strategy is that
    various different annotations of a single region have then to redefine each time
    the borders of this region. The region definition could be defined once for all in
    an independant and identified file, but then, the annotation will be <em>about</em>
    this XML snippet definition and not <em>about</em> the image region content.
      </li>
    </ol>

    <p>
      The next section discusses general characteristics of the existing annotation
      tools for images.
   </p>

    <h2>
      <a name="tools">
    4. Available Tools for Semantic Image Annotation
      </a>
    </h2>

    <p>
    Among the numerous tools used for image archiving and description, some of them
    may be used for semantic annotation. The aim of this section is to
    identify some key characteristics of semantic image annotation tools, such as the type of
    content they can handle or if they allow fine-grained annotations, so as to provide
    some guidelines for their proper use. Using these characteristics as criteria, users of these
    tools could choose the most appropriate for a specific application.
    </p>

    <p>
    <strong>Type of Content.</strong> A tool can annotate different type of content.
    Usually, the raw content is an image, whose format can be jpg, png, tif, etc. but there
    are tools that can annotate videos as well.
    </p>

    <p>
    <strong>Type of Metadata.</strong> An annotation can be targeted for different use.
    Following the categorization
    provided by <a href="http://sunsite.berkeley.edu/moa2/wp-v2.html">The Making of
    America II project</a>, the metadata can be <em>descriptive</em> (for description
    and identification of information), <em>structural</em>
    (for navigation and presentation), or <em>administrative</em> (for management and
    processing). Most of the tools can be used in order to provide
    descriptive metadata and for some of them, the user can also provide structural and
    administrative information.
    </p>

    <p>
    <strong>Format of Metadata.</strong> An annotation can be expressed in different format.
    This format is important since it should ensure
    interoperability with other (semantic web) applications. MPEG-7 is often used as
    the metadata format for exchanging automatic analysis results whereas OWL and RDF are
    better appropriate in the Semantic Web world.
    </p>

    <p>
    <strong>Annotation level.</strong> Some tools give to the user the
    opportunity to annotate an image using vocabularies while others allow free text
    annotation only. When ontologies are used (in RDF or OWL format),
    the annotation level is considered to be controlled since the semantics is generally
    provided in a more formal way, whereas if they are not, the annotation level is
    considered to be free.
    </p>

    <p>
    <strong>Client-side Requirement.</strong> This characteristic refers to
    whether users can use a Web browser to access the service(s) or need to install
    a stand-alone application.
    </p>

    <p>
    <strong>License Conditions.</strong> Some of the tools are open source while some
    others are not. It is important for the user and for potential researchers and
    developers in the area of multimedia annotation to know this issue
    before choosing a particular tool.
    </p>

    <p>
    <strong>Collaborative or individual.</strong> This characteristic refers
    to the possible usage of the tool as an annotation framework for web-shared image
    databases or as an individual user multimedia content annotation tool.
    </p>

    <p>
    <strong>Granularity.</strong> Granularity specifies whether annotation is
    segment based or file based. This is an important characteristic since in some applications,
    it could be crucial to provide the structure of the image. For example, it is
    useful to provide annotations for different regions of the image, describing several cues of
    information (like a textual part or sub-images) or defining and describing different objects
    visualized in the image (e.g. people).
    </p>

    <p>
    <strong>Threaded or unthreaded.</strong> This characteristic refers to the
    ability of the tool to respond or add to a previous annotation and to stagger/structure
    the presentation of annotations to reflect this.
    </p>

    <p>
    <strong>Access control.</strong> This refers to the
    access provided for different users to the metadata. For example, it is important to
    distinguish between users that have simple access (just view) and users that have full
    access (view or change).
    </p>

    <p>
    Concluding, the appropriateness of a tool depends on the nature of annotation that the user
    requires and cannot be predetermined. A separate web page is maintained with
    <a href="http://www.w3.org/2001/sw/BestPractices/MM/resources/Tools.html">Semantic Web
    Image Annotation Tools</a>, and categorizes most of the annotation tools found in the Internet,
    according to the characteristics described above. Any comments, suggestions or new tools
    annoucements will be added to this separate document. The tools can be used for different
    types of annotations, depending on the use cases, as shown in the following section.
    </p>

    <!-- EXAMPLES -->
  <h2>
   <a name="examples">
    5. Example Solutions to the Use Cases
   </a>
  </h2>
  <p>
   This section describes possible scenarios for how
   Semantic Web technology could be used for supporting the
   use cases presented in <a href="#use_cases">Section 1</a>.
   These scenarios are provided purely as illustrative examples and do not imply
   endorsement by the W3C membership or the Semantic Web Best
   Practices and Deployment Working Group.
  </p>

  <h3>
   <a name="solution_personal">
    5.1 Use Case: Management of Personal Digital Photo Collections
   </a>
  </h3>

  <h4 id="personal_solution">Possible Semantic Web-based solution</h4>

  <div style="float: right; width: 30%; border: 1px solid gray; padding: 1%; margin: 1%">
   <a href="images/examples/Personal.jpg">
    <img style="width: 100%;;"
     src="images/examples/Personal.jpg"
     alt="A photo from a personal collection"/></a>
   <br/>
   A photo from a personal collection
  </div>

  <p>
  The proposed scenario for the use case of the management of Personal Digital
  Photo Collections, described  in <a href="#photo_collection">Section 1.1</a>,
  is based on the annotation of the photos using multiple vocabularies to
  describe their properties and their content. The properties of the image itself,
  e.g. the creator, the resolution, the date of capture, are described with terms
  from the DC and VRA vocabularies. For the description of the content of the images
  a variety of existing vocabularies are used and few domain specific ontologies
  are created, because the potential subject of a photo from a personal digital
  collection is very wide, and may include notions such as vacations, special events
  of the personal history, landscapes, locations, people, objects, etc.
  </p>
  <p>
  As a sample from a personal digital photo collection, a photo that depicts the vacations
  of a person named <tt>Katerina Tzouvara</tt> in Thailand is chosen. The FOAF vocabulary is
  then used to describe the person depicted in the image. Three domain specific
  ontologies: the location, the landscape and the event ontology are used to describe the
  location, the landscape and the event depicted on the photo. The RDF format is chosen
  as the most appropriate for all the vocabularies, the ontologies and the annotation file.
  Any existing editor can be used for editing the ontologies and the annotation file.
  The corresponding RDF annotation file is available <a
   href="http://www.w3.org/2001/sw/BestPractices/MM/images/examples/usecase_personal_photo_management.rdf">here</a>.
  </p>

  <h4>The Annotation process</h4>
  <p>
    Firstly, a decision about the format of the annotation file has to be made.
    The RDF/XML syntax of RDF is chosen in order to gain interoperability.
    The RDF/XML syntax of RDF is well-formed XML with an overlay of constraints
    and is parsable with XML technology.
  </p>
  <p>
    The first line of the annotation file is the XML declaration line. The next
    element is the RDF element that encloses the RDF based content. Within the
    RDF element, the various namespaces associated with
    each element in the file are declared as attributes. These namespaces reference existing
    RDF schemas and ontologies from which particular elements have been used to
    annotate the target image. Specifically, the following schemas are used:
  </p>
  <ul>
    <li>The Simple DC RDF Schema, version 2003-03-24, defines the terms for Simple Dublin
    Core (the original 15 elements) and is located on
<a href="http://purl.org/dc/elements/1.1/">http://purl.org/dc/elements/1.1/</a></li>
    <li>The RDF/OWL schema of VRA Core located on
<a href="http://www.vraweb.org/vracore3.htm">http://www.vraweb.org/vracore3.htm</a></li>
    <li>The RDFS/OWL FOAF ontology located on
<a href="http://xmlns.com/foaf/0.1/">http://xmlns.com/foaf/0.1/</a></li>
    <li>And three domain-specific ontologies:
      <a href="http://www.w3.org/2001/sw/BestPractices/MM/images/examples/event.rdf">personal history event</a>,
      <a href="http://www.w3.org/2001/sw/BestPractices/MM/images/examples/location.rdf">location</a> and
      <a href="http://www.w3.org/2001/sw/BestPractices/MM/images/examples/landscape.rdf">landscape</a>.</li>
   </ul>

    <div class="exampleInner" style="clear: both">
       <pre>
&lt;!-- Declaration of the namespaces of the schemas and ontologies being used --&gt;
&lt;?xml version="1.0" encoding="UTF-8"?&gt;

&lt;rdf:RDF
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:owl="http://www.w3.org/2002/07/owl#"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:foaf="http://xmlns.com/foaf/0.1/"
  xmlns:event="http://www.altova.com/ontologies/personal_history_event#"
  xmlns:locat="http://www.altova.com/ontologies/location#"
  xmlns:lsc="http://www.altova.com/ontologies/landscape#"
  xmlns:perph="http://www.altova.com/ontologies/usecase_personal_photo_management#"
  xmlns:vra="http://www.vraweb.org/vracore/vracore3#" &gt;
       </pre>
    </div>

    <p>
    The inner element that follows (<tt>rdf:Description</tt>), wraps the subject-predicate-object
    triples. The subject, declared by the <tt>rdf:about</tt> attribute is the image
    (identified by its URI) that will be annotated, the predicates are properties of the image
    and the objects are the values of these properties.
    </p>

    <div class="exampleInner" style="clear: both">
         <pre>
  &lt;rdf:Description rdf:about="http://www.w3.org/2001/sw/BestPractices/MM/images/examples/Personal.jpg"&gt;
       </pre>
   </div>

    <p>
    The following block of triples provides general information about the entire photo.
    The Dublin Core terms are mainly used to provide this kind of information,
    since they are general enough and widely accepted by the metadata communities.
    The first predicate-object pair states that the object of the annotation file is an image,
    using the DC term-property type (<tt>dc:type</tt>) as the predicate, and the DCMI Type
    term-concept <tt>Image</tt> as the object.<br/>
    The DC term-property description (<tt>dc:description</tt>) in the next line
    provides a short linguistic human-understandable description of the subject of the image.<br/>
    The DC term-property creator (<tt>dc:creator</tt>) is then used for specifying the
    creator of the image. In order to define that the creator is a person
    (and e.g. not an organization or a company), the FOAF term-class Person (<tt>foaf:Person</tt>)
    is used. The name of the person is defined using the FOAF term-properties familyname
    (<tt>foaf:familyname</tt>) and firstname (<tt>foaf:firstname</tt>). The values of these
    properties are RDF Literal. Note that these two properties are preferred to the general
    <tt>foaf:name</tt> property since the absence of any further rules or standards for the
    construction of individual names could mislead the reasoning engines
    that would not be able to discriminate the first name from the family name.<br/>
    The DC property date (<tt>dc:date</tt>) completes the description describing the capture day
    of the photo. The RDF Literal value of this property
    is conformed to the ISO 8601 specifications for representing dates and times.
  </p>

      <div class="exampleInner" style="clear: both">
           <pre>
  &lt;!-- Description and general information about the entire photo, e.g. description, creator, date --&gt;
  &lt;!-- using Dublin Core and FOAF --&gt;
    &lt;dc:type rdf:resource="http://purl.org/dc/dcmitype/Image"/&gt;
    &lt;dc:description&gt;Photo of Katerina Tzouvara during Vacations in Thailand&lt;/dc:description&gt;
    &lt;dc:creator&gt;
      &lt;foaf:Person&gt;
        &lt;foaf:familyname&gt;Stabenaou&lt;/foaf:familyname&gt;
        &lt;foaf:firstname&gt;Arne&lt;/foaf:firstname&gt;
      &lt;/foaf:Person&gt;
    &lt;/dc:creator&gt;
    &lt;dc:date&gt;2002-12-40&lt;/dc:date&gt;
   </pre>
     </div>

    <p>
    Some more technical information about the format and the resolution of the photo
    are included in the annotation file using the DC property format (<tt>dc:format</tt>) and
    the term JPEG of the MIME vocabulary, and the VRA property measurements.resolution
    (<tt>vra:measurements.resolution</tt>).  Though they may not seem really important for this
    specific use case, these properties should exist for the purpose of interchanging
    personal digital photos among users with different hardware and software capabilities.
    </p>

      <div class="exampleInner" style="clear: both">
           <pre>
  &lt;!-- Technical information, e.g. format, resolution, using DC, VRA, MIME --&gt;
    &lt;dc:format>JPEG&lt;/dc:format&gt;
    &lt;vra:measurements.resolution&gt;300 x 225px&lt;/vra:measurements.resolution&gt;
           </pre>
     </div>

    <p>
    The rest of the annotation file focuses on the description of the content of the
    image. In order to provide a complete description of the content, it is often
    useful to try to answer the questions "when", "where" and "why" the photo has been taken,
    and "who" and "what" is depicted on the photo, because these are the most probable questions
    that the end-user would like to be queried during the retrieval process.
    </p>

    <p>
    Firstly, the location and the landscape depicted on the photo are described.
    To do so, two domain ontologies are built as simple as needed for the
    purposes of the annotation of this image: the location ontology, whose prefix is
    <tt>locat</tt>, and the landscape ontology, whose prefix is <tt>lsc</tt>. The location ontology
    provides concepts such as continent (<tt>locat:Continent</tt>), country (<tt>locat:Country</tt>),
    city (<tt>locat:City</tt>), and properties for the representation
    of the fact that an image is taken in a specific continent (<tt>locat:located_in_Continent</tt>),
    country (<tt>locat:located_in_Country</tt>) and city (<tt>locat:located_in_City</tt>).
    The landscape ontology classes represent concepts such as mountain (<tt>lsc:Mountain</tt>),
    beach (<tt>lsc:Beach</tt>), sand (<tt>lsc:Sand</tt>) and tree (<tt>lsc:Tree</tt>).<br/>
    The FOAF property depicts (<tt>foaf:depicts</tt>) stands for the relation
    that an image depicts something or someone. The FOAF property name (<tt>foaf:name</tt>)
    informs that any kind of location or landscape has a specific name. The
    <a href="http://www.getty.edu/research/conducting_research/vocabularies/tgn/">TGN vocabulary</a>
    is used for the names of the locations and landscapes, e.g. <tt>Thailand</tt>, and the
    <a href="http://www.fao.org/aims/ag_intro.htm">AGROVOC thesaurus</a>
    is used for the names related to agriculture, e.g. <tt>Phoenix Dactyliphera</tt>.
    For the representation of the relation that a city belongs to a specific country, the property
    <tt>locat:belongs_to_Country</tt> has been used.
    </p>

      <div class="exampleInner" style="clear: both">
        <pre>
  &lt;!-- Information about the location and the objects depicted using a landscape ontology --&gt;
  &lt;!-- and the TGN and AGROVOC thesaurus --&gt;
    &lt;locat:located_in_Continent&gt;
      &lt;locat:Continent&gt;Asia&lt;/locat:Continent&gt;
    &lt;/locat:located_in_Continent&gt;
    &lt;locat:located_in_Country&gt;
      &lt;locat:Country&gt;Thailand&lt;/locat:Country&gt;
    &lt;/locat:located_in_Country&gt;
    &lt;locat:located_in_City&gt;
      &lt;locat:City&gt;
        &lt;foaf:name>Phi Phi&lt;/foaf:name&gt;
        &lt;locat:belongs_to_Country&gt;
          &lt;locat:Country&gt;Thailand&lt;/locat:Country&gt;
        &lt;/locat:belongs_to_Country&gt;
      &lt;/locat:City&gt;
    &lt;/locat:located_in_City&gt;
    &lt;foaf:depicts&gt;
      &lt;lsc:Beach/&gt;
    &lt;/foaf:depicts&gt;
    &lt;foaf:depicts&gt;
      &lt;lsc:Palm_Tree&gt;Phoenix Dactyliphera&lt;/lsc:Palm_Tree&gt;
    &lt;/foaf:depicts&gt;
    &lt;foaf:depicts rdf:resource="http://www.altova.com/ontologies/landscape#Sand"/&gt;
        </pre>
      </div>

    <p>
    The advantages of using Semantic Web technologies in this use case can be already shown
    at this stage. Suppose that the annotation file did not include the lines that
    fully declare explicitly in which country the picture has been taken, but that it included
    only the lines that state that the photo was taken in the city Phi Phi which belongs to the
    country Thailand. When the end-user will query for retrieving photos depicting Thailand,
    the application for the management of the digital photo collection using reasoning techniques
    should return as a result all the photos that are explicitly declared to show Thailand,
    but also all the photos that are implicitly declared to portray Thailand, e.g. by being explicitly
    declared to be captured in a city that belongs to Thailand.
    </p>

    <p>
    To describe the events captured on the image, a simple ontology representing the important
    events for a person is created. This ontology conceptualizes occasions such as business
    traveling (<tt>event:Business_Traveling</tt> or <tt>event:Conference</tt>),
    vacations (<tt>event:Vacations</tt>), sports activities
    (<tt>event:Sports_Activity</tt>), celebrations (<tt>event:Celebration</tt> or
    <tt>event:Birthday_Party</tt>), etc.
    </p>

      <div class="exampleInner" style="clear: both">
        <pre>
  &lt;!-- Information about the kind of event depicted by the image --&gt;
    &lt;foaf:depicts rdf:resource="http://www.altova.com/ontologies/event#Vacations"/&gt;
        </pre>
      </div>

    <p>
    Finally, the <tt>foaf:depicts</tt> property and the <tt>foaf:Person</tt> concept are used
    for identifying the person shown on the image.
    </p>

       <div class="exampleInner" style="clear: both">
         <pre>
  &lt;!-- Information about the persons depicted by the image --&gt;
    &lt;foaf:depicts&gt;
      &lt;foaf:Person&gt;
        &lt;foaf:familyname&gt;Tzouvara&lt;/foaf:familyname&gt;
        &lt;foaf:firstname&gt;Katerina&lt;/foaf:firstname&gt;
      &lt;/foaf:Person&gt;
    &lt;/foaf:depicts&gt;
  &lt;/rdf:Description&gt;
&lt;/rdf:RDF&gt;
        </pre>
      </div>

    <h4>Conclusion</h4>
    <p>
    It is obvious, from this analysis of the annotation process, that there
    are many important issues under discussion in order to improve the efficiency
    of Semantic Web technologies. Firstly, the various vocabularies, such as FOAF,
    VRA, TGN, do not have yet a standardized representation in a standardized
    ontology language, such as OWL or RDF. Secondly, there do not exist many domain
    specific vocabularies about the description of the content of visual document,
    e.g. there are not any landscape, personal history event vocabularies, nor
    vocabularies about the possible objects depicted by an image. This deficiency
    of annotation standards provokes interoperability problems between the different
    types of annotated content during the management and retrieval process.
    Finally, another important concern is to which depth the annotation process
    should proceed. In the example above, one might find it important to annotate
    the clothing of the person too, but someone else may believe that it is more
    important to annotate the objects on the beach, etc. This is a matter of the
    annotation approach which should be defined according to the end-user needs.
    </p>

    <p>
    In conclusion, there is still much work to be done for improving the use of
    Semantic Web technologies for describing multimedia material.
    However, as shown in this use case analysis, deploying
    ontology-based annotations on personal computers applications will
    enable a much more focused and integrated content management
    and ease the exchange of personal photos.
  </p>

  <h3>
   <a name="solution_culture">
    5.2 Use Case: Cultural Heritage
   </a>
  </h3>

  <h4 id="ec_solution">Possible Semantic Web-based solution</h4>

  <div style="float: right; width: 30%; border: 1px solid gray; padding: 1%; margin: 1%">
   <img style="width: 100%;;"
    src="http://www.artchive.com/artchive/m/monet/adresse.jpg"
    alt="Image of Monet's painting 'Garden at Sainte-Adresse'"/>
   Claude Monet, Garden at Sainte-Adresse.<br/>
   Image courtesy of <a href="http://www.artchive.com">Mark
    Harden</a>, used with permission.
  </div>

  <p>
   Many of the requirements of the use case described in <a href="#cultural_heritage">Section 1.2</a>
   can be met by using the vocabulary developed by the <a href="#VraCore">VRA</a> in
   combination with domain-specific vocabularies such as Getty's AAT
   and ULAN.

   In this section, we provide as an example a set of RDF annotations
   of a painting by Claude Monet, which is in English known as "Garden
   at Sainte-Adresse".  It is part of the collection of the
   <a href="http://www.metmuseum.org/">Metropolitan Museum of Art</a> in New York.
   The corresponding RDF file
   is <a href="images/examples/eculture-use-case.rdf">available as a
   separate document</a>.  No special annotation tools where used to
   create the annotations.  We assume that cultural heritage
   organizations that need to publish similar metadata will do so by
   exporting existing information from their collection database to
   RDF.  Below, we discuss the different annotations used in this
   file.
  </p>


  <h4 id="housekeeping">House keeping</h4>
    <p>
   The file starts as a typical RDF/XML file, by defining the XML
   version and encoding and defining entities for the RDF and VRA
   namespaces that will be used later.  Note that we use the <a
    href="#VraRDF">RDF/OWL schema of VRA Core</a> developed by Mark
   van Assem.
  </p>

  <div class="exampleInner" style="clear: both">
<pre>
&lt;?xml version='1.0' encoding='ISO-8859-1'?&gt;
&lt;!DOCTYPE rdf:RDF [
    &lt;!ENTITY rdf        "http://www.w3.org/1999/02/22-rdf-syntax-ns#"&gt;
    &lt;!ENTITY vra        "http://www.vraweb.org/vracore/vracore3#"&gt;
      </pre>
    </div>

  <h5 id="work_or_image">Work versus Image</h5>
  <p>
   The example includes annotations about two different images of the
   same painting.  An important distinction made by VRA vocabulary is
   the distinction between annotations describing a work of art itself
   and annotations describing (digital) images of that work. This
   example also uses this distinction.  In RDF, to say something about
   a resource, that resource needs to have a URI. We will thus not
   only need the URIs of the two images, but also a URI for the
   painting itself:
  </p>

  <div class="exampleInner">
<pre>
    &lt;!ENTITY image1    "http://www.metmuseum.org/Works_Of_Art/images/ep/images/ep67.241.L.jpg"&gt;
    &lt;!ENTITY image2    "http://www.artchive.com/artchive/m/monet/adresse.jpg"&gt;
    &lt;!ENTITY painting  "http://thing-described-by.org/?http://www.metmuseum.org/Works_Of_Art/images/ep/images/ep67.241.L.jpg"&gt;
]&gt;
   </pre>
  </div>

  <h5 id="uri_conventions">URI and ID conventions</h5>
  <p>
   VRA Core does not specify how works, images or annotation records
   should be identified.  For the two images, we have chosen for the
   most straightforward solution and use the URI of the image as the
   identifying URI.  We did not have, however, a similar URI that
   identifies the painting itself. We could not reuse the URI of one
   of the images.  This is not only conceptually wrong, but would also
   lead to technical errors: it would make the existing instance of
   <tt>vra:Image</tt> also an instance of the <tt>vra:Work</tt> class,
   while this is not allowed by the schema.
  </p>
  <p>
   In the example, we have decided to 'mint' the URI of the painting
   by arbitrary selecting the URI of one of the images, and prefixing
   it by <tt><a href="http://thing-described-by.org/">
   http://thing-described-by.org/?</a></tt>.  This creates a new URI
   that is distinct from the image itself, but when a the browser
   resolves it, it will be redirected to the image URI by the
   <tt>thing-described-by.org</tt> web server (one could argue if the
   use of an http-based URI is actually appropriate here.  See <a
   href="#HTTP-URI">What do HTTP URIs Identify?</a> and <a
   href="#httpRange-14">[httpRange-14]</a> for more details on this
   discussion).
  </p>
  <p>
   Warning: The annotations described below also contain a
   <tt>vra:idNumber.currentRepository</tt> element, that defines the
   identifier used <em>locally</em> in the museum's repositories.
   These local identifiers should not be confused with the globally
   unique identifier that is provided by the URI.
  </p>

  <h5 id="housekeeping2">More housekeeping: starting the RDF block</h5>
  <p>
   The next line opens the RDF block, declares the namespaces using
   the XML entities defined above. Out of courtesy, it uses
   <tt>rdf:seeAlso</tt> to help agents find the VRA schema that is
   used.
  </p>

  <div class="exampleInner">
<pre>
&lt;rdf:RDF  xmlns:rdf="&amp;rdf;" xmlns:vra="&amp;vra;"
  rdf:seeAlso="http://www.w3.org/2001/sw/BestPractices/MM/vracore3.rdfs"
&gt;
   </pre>
  </div>

  <h4 id="work">Description of the work (painting)</h4>
  <p>
   The following lines describe properties of the painting itself: we
   will deal with the properties of the two images later.  First, we
   provide general information about the painting such as the title,
   its creator and the date of creation. For these properties, the VRA
   closely follows the Dublin Core conventions:
  </p>
  <div class="exampleInner">
   <pre>
  &lt;!-- Description of the painting --&gt;
  &lt;vra:Work rdf:about="&amp;painting1;"&gt;
    &lt;!-- General information --&gt;
    &lt;vra.title&gt;Jardin &#224;; Sainte-Adresse&lt;/vra.title&gt;
    &lt;vra:title.translation&gt;Garden at Sainte-Adresse&lt;/vra:title.translation&gt;
    &lt;vra:creator&gt;Monet, Claude&lt;/vra:creator&gt;                            &lt;!-- ULAN ID:500019484 --&gt;
    &lt;vra:creator.role&gt;artist&lt;/vra:creator.role&gt;                         &lt;!-- ULAN ID:31100     --&gt;
    &lt;vra:date.creation&gt;1867&lt;/vra:date.creation&gt;
   </pre>
  </div>

  <h5 id="text_or_controlled">Text fields and controlled vocabularies</h5>
  <p>
   Many values are filled with RDF Literals, of which the value is not
   further constraint by the schema.  But many of these values are
   actually terms from other controlled vocabularies, such as the
   Getty <a href="#refAAT">AAT</a>, <a href="#refULAN">ULAN</a> or a image
   type defined by <a href="#refMIME-2">MIME</a>.  Using controlled
   vocabularies solves many problems associated with free text
   annotations.  For example, ULAN recommends a spelling when an
   artist's name is used for indexing, so for the <tt>vra:creator</tt>
   field we have exactly used this spelling ("Monet, Claude"). The
   ULAN identifiers of the records describing Claude Monet and the
   "artist" class are given in XML comments above.  The use of
   controlled vocabulary can avoid confusion and the need for
   "smushing" different spellings for the same name later.
  </p>

  <p>
   However, using controlled vocabularies does not solve the problem
   of ambiguous terms.  The annotations below use three different meanings
   for "oil paint", "oil paintings" and "oil painting (technique)".
   The first refers to the type of paint used on the canvas, the
   second to the type of work (e.g. the work is an oil painting, and
   not an etching) and the last to the painting technique used by
   artist.  All three terms refer to different concepts that are part
   of different branches of the AAT term hierarchy (the AAT
   identifiers of these concepts are mentioned in XML comments).
   However, the use of terms that are so similar for different
   concepts is bound to lead to confusion.  Instead, one could switch
   from using <tt>owl:datatypeProperties</tt> to using
   owl:objectProperties, and replace the literal text by a reference
   to the URI of the concept used.  For example, one could change:

   <br/><tt>&lt;vra:material.medium&gt;oil paint&lt;/vra:material.medium&gt;</tt>
   <br/>to
   <br/><tt>&lt;vra:material.medium rdf:resource="http://www.getty.edu/aat#300015050"/&gt;</tt>
   </p>

  <p>
   This approach requires, however, that an unambiguous URI-based
   naming scheme is defined for all terms in the target vocabulary
   (and in this case, such a URI-based naming scheme does not yet
   exist for AAT terms).  Additional Semantic Web-based processing is
   also only possible once these vocabularies become available in RDF
   or OWL.
  </p>

  <div class="exampleInner">
   <pre>
    &lt;!-- Technical information --&gt;
    &lt;vra:measurements.dimensions&gt;98.1 x 129.9 cm&lt;/vra:measurements.dimensions&gt;
    &lt;vra:material.support&gt;unprimed canvas&lt;/vra:material.support&gt;        &lt;!-- AAT ID:300238097 --&gt;
    &lt;vra:material.medium&gt;oil paint&lt;/vra:material.medium&gt;                &lt;!-- AAT ID:300015050 --&gt;
    &lt;vra:type&gt;oil paintings&lt;/vra:type&gt;                                  &lt;!-- AAT ID:300033799 --&gt;
    &lt;vra.technique&gt;oil painting (technique)&lt;/vra.technique&gt;             &lt;!-- AAT ID:300178684 --&gt;

    &lt;!-- Associated style, etc. --&gt;
    &lt;vra:stylePeriod&gt;Impressionist&lt;/vra:stylePeriod&gt;                    &lt;!-- AAT ID:300021503 --&gt;
    &lt;vra:culture&gt;French&lt;/vra:culture&gt;                                   &lt;!-- AAT ID:300111188 --&gt;
   </pre>
  </div>

  <h5 id="subject_matter">Annotating subject matter</h5>
  <p>
   For many applications, it is useful to know what is actually
   depicted by the painting.  One could add annotations of this style
   to an arbitrary level of detail.  To keep the example simple, we
   have chosen to record only the names of the people that are
   depicted on the painting, using the <tt>vra:subject</tt> field.
   Also for simplicity, we have chosen not to annotate specific parts
   or regions of the painting.  This might have been appropriate, for
   example, to identify the associated regions that depict the various
   people in the painting:
  </p>

  <div class="exampleInner">
   <pre>
    &lt;!-- Subject matter: (who/what is depicted by this work --&gt;
    &lt;vra:subject&gt;Jeanne-Marguerite Lecadre (artist's cousin)&lt;/vra:subject&gt;
    &lt;vra:subject&gt;Madame Lecadre (artist's aunt)&lt;/vra:subject&gt;
    &lt;vra:subject&gt;Adolphe Monet (artist's father)&lt;/vra:subject&gt;
   </pre>
  </div>

  <h5 id="provenance">Provenance: annotating the past</h5>
  <p>
   Many of the fields below do not contain information about the
   current situation of the painting, but information about places and
   collections the painting has been in the past. This provides
   provenance information that is important in this domain.
  </p>

  <div class="exampleInner">
   <pre>
    &lt;!-- Provenance --&gt;
    &lt;vra:location.currentSite&gt;Metropolitan Museum of Art, New York&lt;/vra:location.currentSite&gt;
    &lt;vra:location.formerSite&gt;Montpellier&lt;/vra:location.formerSite&gt;
    &lt;vra:location.formerSite&gt;Paris&lt;/vra:location.formerSite&gt;
    &lt;vra:location.formerSite&gt;New York&lt;/vra:location.formerSite&gt;
    &lt;vra:location.formerSite&gt;Bryn Athyn, Pa.&lt;/vra:location.formerSite&gt;
    &lt;vra:location.formerSite&gt;London&lt;/vra:location.formerSite&gt;
    &lt;vra:location.formerRepository&gt;
      Victor Frat, Montpellier (probably before 1870 at least 1879;
      bought from the artist); his widow, Mme Frat, Montpellier (until 1913)
    &lt;/vra:location.formerRepository&gt;
    &lt;vra:location.formerRepository&gt;Durand-Ruel, Paris, 1913&lt;/vra:location.formerRepository&gt;
    &lt;vra:location.formerRepository&gt;Durand-Ruel, New York, 1913&lt;/vra:location.formerRepository&gt;
    &lt;vra:location.formerRepository&gt;
      Reverend Theodore Pitcairn and the Beneficia Foundation, Bryn Athyn, Pa. (1926-1967),
      sale, Christie's, London, December 1, 1967, no. 26 to MMA
    &lt;/vra:location.formerRepository&gt;
    &lt;vra:idNumber.currentRepository&gt;67.241&lt;/vra:idNumber.currentRepository&gt; &lt;!-- MMA ID number --&gt;
   </pre>
  </div>

  <h5 id="copyright">Copyright and origin of metadata</h5>
  <p>
   The remaining properties describe the origin the sources used for
   creating the metadata and a rights management statement. We have
   used the <tt>vra:description</tt> element to provide a link to a
   web page with additional descriptive information:
  </p>
  <div class="exampleInner">
   <pre>
    &lt;!-- extra information, source of this information and copyright issues --&gt;
    &lt;vra:description&gt;For more information, see
  http://www.metmuseum.org/Works_Of_Art/viewOne.asp?dep=11&amp;viewmode=1&amp;item=67%2E241&amp;section=description#a&lt;/vra:description&gt;
    &lt;vra:source&gt;Metropolitan Museum of Art, New York&lt;/vra:source&gt;
    &lt;vra:rights&gt;Metropolitan Museum of Art, New York&lt;/vra:rights&gt;
      </pre>
    </div>

  <h4 id="images">Image properties</h4>
  <p>
   Finally, we define the properties that are specific to the two
   images of the painting, which differ in resolution, copyright, etc.
   The first set of annotations describe a 500x300 pixel image that is
   located at the website of the Metropolitan itself, while the second
   set describes the properties of a larger resolution (1075 x 778px)
   image at Mark Harden's <a
   href="http://www.artchive.com/">Artchive</a> website.
   Note that VRA Core does not specify how Works and their associated Images
   should be related.  In the example we follow <a href="#VraRDF">Van
   Assem's suggestion</a> and use <tt>vra.relation.depicts</tt> to
   explicitly link the Image to the Work it depicts.
  </p>

  <div class="exampleInner">
   <pre>
  &lt;!-- Description of the first online image of the painting --&gt;
  &lt;vra:Image rdf:about="&amp;image1a;"&gt;
    &lt;vra:type&gt;digital images&lt;/vra:type&gt;                                &lt;!-- AAT ID: 300215302 --&gt;
    &lt;vra:relation.depicts rdf:resource="&amp;painting1;"/&gt;
    &lt;vra.measurements.format&gt;image/jpeg&lt;/vra.measurements.format&gt;                   &lt;!-- MIME --&gt;
    &lt;vra.measurements.resolution&gt;500 x 380px&lt;/vra.measurements.resolution&gt;
    &lt;vra.technique&gt;Scanning&lt;/vra.technique&gt;
    &lt;vra:creator&gt;Anonymous employee of the museum&lt;/vra:creator&gt;
    &lt;vra:idNumber.currentRepository&gt;ep67.241.L.jpg&lt;/vra:idNumber.currentRepository&gt;
    &lt;vra:rights&gt;Metropolitan Museum of Art, New York&lt;/vra:rights&gt;
  &lt;/vra:Image&gt;
   </pre>
  </div>
  <div class="exampleInner">
   <pre>
  &lt;!-- Description of the second online image of the painting --&gt;
  &lt;vra:Image rdf:about="&amp;image1b;"&gt;
    &lt;vra:type&gt;digital images&lt;/vra:type&gt;                                &lt;!-- AAT ID: 300215302 --&gt;
    &lt;vra:relation.depicts rdf:resource="&amp;painting1;"/&gt;
    &lt;vra:creator&gt;Mark Harden&lt;/vra:creator&gt;
    &lt;vra.technique&gt;Scanning&lt;/vra.technique&gt;
    &lt;vra.measurements.format&gt;image/jpeg&lt;/vra.measurements.format&gt;                   &lt;!-- MIME --&gt;
    &lt;vra.measurements.resolution&gt;1075 x 778px&lt;/vra.measurements.resolution&gt;
    &lt;vra:idNumber.currentRepository&gt;adresse.jpg&lt;/vra:idNumber.currentRepository&gt;
    &lt;vra:rights&gt;Mark Harden, The Artchive, http://www.artchive.com/&lt;/vra:rights&gt;
  &lt;/vra:Image&gt;
&lt;/rdf:RDF&gt;
   </pre>
  </div>

  <h4 id="ch_conclusion">Conclusion and discussion</h4>
  <p>
   The example above reveals several technical issues that are still
   open.  For example, the way the URI for the painting was minted is
   rather arbitrary.  Preferably, there would have been a commonly
   accepted URI scheme for paintings (c.f. the <a href="#lsid">LSID</a>
   scheme used to identify concepts from the life sciences).  At the
   time of writing, the VRA, AAT and ULAN vocabulary used have
   currently no commonly agreed upon RDF or OWL representation, which
   reduces the interoperability of the chosen approach.  Tool support
   is another issue.  While some major database vendors already start
   to support RDF, generating the type of RDF as shown here from
   existing collection databases will in many cases require non
   trivial custom conversion software.
  </p>

  <p>
   From a modeling point of view, subject matter annotations are
   always non-trivial.  As stated above, it is hard to give general
   guidelines about what should be annotated and to what depth, as
   this can be very application dependent.  Note that in this example,
   we annotated the persons that appear in the painting, and we
   modeled this information as properties of the painting URI, not of
   the two image URIs. But if we slightly modify our use case and
   assume one normal image and one X-ray image that reveals an older
   painting under this one, it might make more sense to model more
   specific subject matter annotations as properties of the specific
   images.
  </p>

  <p>
   Nevertheless, the example shows that a large part of issues
   described by the use case can be solved using current Semantic Web
   technology.  It shows how RDF can be used together with existing
   vocabularies to annotate various aspects of paintings and the
   images that depict them.
  </p>

  <h3>
   <a name="solution_TVarchive">
    5.3 Use Case: Television News Archive
   </a>
  </h3>

  <h4>Possible Semantic Web-based solution</h4>

  <div style="float: right; width: 30%; border: 1px solid gray; padding: 1%; margin: 1%">
   <img style="width: 100%;;"
    src="images/examples/soccer.png"
    alt="Drawing representing an action in a football game"/>

   Drawing representing an offside position of a football player while his partner is
   scoring with his head during a game.<br/>
   Personal artwork, used with permission.
  </div>

  <p>
   The use case described in <a href="#television_archive">Section 1.3</a>
   is typically one that requires the use of multiple
   vocabularies. Let us imagine that the image to be described is about
   a refused goal of a given soccer player (e.g.
   <a href="http://www.jeanalainboumsong.fr/">J.A Boumsong</a>) for
   an active offside position during a particular game (e.g. Auxerre-Metz played on 2002/03/16).
   First, the image can be extracted from a weekly sports
   magazine broadcasted on a TV channel. This program may be fully
   described using the vocabulary developed by the <a href="#TVA">
   [TV Anytime forum]</a>. Second, this image shows the player
   Jean-Alain Boumsong scoring with his head during the game
   Auxerre-Metz.  The context of this football game could be described
   using the <a href="#MPEG-7">[MPEG-7]</a> vocabulary while the
   action itself might be described by a soccer ontology such as the
   one developed by <a href="#Tsinaraki">[Tsinaraki]</a>. Finally, a
   soccer fan may notice that this goal was actually refused for an
   active offside position of another player. On the image, a circle
   could highlight this player badly positioned. Again, the description
   could merge MPEG-7 vocabulary for delimiting the relevant image
   region and a domain specific ontology for describing the action
   itself.

   In the following, we provide as an example a set of RDF annotations
   illustrating these three levels of description as well as the
   vocabularies involved.
  </p>

  <h4>The image context</h4>
  <p>
   Let us consider that the image comes from a weekly sports magazine named <a
   href="http://sport.france2.fr/stade2/">Stade 2</a> broadcasted on
   March, 17th 2002 on the French public channel <a
   href="http://www.france2.fr/">France 2</a>. This context can be
   represented using the TV Anytime vocabulary which allows for a TV
   (or radio) broadcaster to publish its program listings on the web
   or in an electronic program guide. Therefore, this vocabulary
   provides the necessary concepts and relations for cataloging the
   programs, giving their intended audience, format and genre, or some
   parental guidance. The vocabulary contains also the vocabulary for
   describing afterwards the real audience and the peak viewing times
   which are of crucial importance for the broadcasters in order to
   adapt their advertisement rates.
  </p>

  <div style="clear: both;" class="exampleOuter">
   <div class="c1">
    <a id="exampleTV" name="exampleTV">
RDF description of the program from which the image comes from</a>
   </div>
   <div class="exampleInner">
    <pre>
&lt;?xml version='1.0' encoding='ISO-8859-1'?&gt;
&lt;!DOCTYPE rdf:RDF [
    &lt;!ENTITY rdf        "http://www.w3.org/1999/02/22-rdf-syntax-ns#"&gt;
    &lt;!ENTITY xsd        "http://www.w3.org/2001/XMLSchema#"&gt;
]&gt;
&lt;rdf:RDF
  xmlns:rdf="&amp;rdf;"
  xmlns:xsd="&amp;xsd;"
  xmlns:tva="urn:tva:metadata:2002"&gt;

  &lt;tva:Program rdf:about="program1"&gt;
    &lt;tva:hasTitle&gt;Stade 2&lt;/tva:hasTitle&gt;
    &lt;tva:hasSynopsis&gt;Weekly Sports Magazine broadcasted every Sunday&lt;/tva:hasSynopsis&gt;
    &lt;tva:Genre rdf:resource="urn:tva:metadata:cs:IntentionCS:2002:Entertainment"/&gt;
    &lt;tva:Genre rdf:resource="urn:tva:metadata:cs:FormatCS:2002:Magazine"/&gt;
    &lt;tva:Genre rdf:resource="urn:tva:metadata:cs:ContentCS:2002:Sports"/&gt;
    &lt;tva:ReleaseInformation&gt;
      &lt;rdf:Description&gt;
        &lt;tva:ReleaseDate xsd:date="2002-03-17"/&gt;
        &lt;tva:ReleaseLocation&gt;fr&lt;/tva:ReleaseLocation&gt;
      &lt;/rdf:Description&gt;
    &lt;/tva:ReleaseInformation&gt;
  &lt;/tva:Program&gt;
&lt;/rdf:RDF&gt;
    </pre>
   </div>
  </div>

<!--

  <h4>The description of the action</h4>
  <p>
   To be done.
  </p>

  <h4>The description of particular region</h4>
  <p>
   Discuss the pros and cons of having either 2 separate files (one
   expressing the localization of the region and one representing the
   content annotation) or 1 RDF file having both description.
  </p>

-->

  <h3>
   <a name="solution_NASA">
    5.4 Use Case: Large-scale Image Collections at NASA
   </a>
  </h3>

  <div style="float: right; width: 300px; border: 1px solid gray; padding: 1%; margin: 1%">
   <a href="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg">
    <img style="width: 300px"
     src="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg"
     alt="Apollo 7 Saturn rocket launch"/></a>
   <br/>
   Apollo 7 Saturn rocket launch -
   October, 10th 1968. Image courtesy of NASA, available at <a href="http://grin.hq.nasa.gov/">GRIN</a>,
   used with permission.
  </div>

  <h4 id="large-collection-solution">Possible Semantic Web-based solution</h4>
  <p>
    One possible solution for the requirements expressed in the use case description
    in <a href="#large_collection">Section 1.4</a>
    is an annotation environment that enables users to annotate information
    about images and/or their regions using concepts in ontologies
    (OWL and/or RDFS). More specifically, subject matter experts will
    be able to assert metadata elements about images and their
    specific content. Multimedia related ontologies can be used to
    localize and represent regions within particular images. These
    regions can then be related to the image via a
    depiction (or annotation) property. This functionality
    (to represent images, image regions, etc.) can be provided,
    for example, by the <a
    href="http://www.mindswap.org/2005/owl/digital-media">MINDSWAP
    digital-media ontology</a>, in conjunction with <a
    href="http://xmlns.com/foaf/0.1/">FOAF</a> (to assert image
    depictions). Additionally, in order to represent the low level
    image features of regions, the <a
    href="http://www.acemedia.org/aceMedia/reference/resource/index.html">aceMedia
    Visual Descriptor Ontology</a> can be used.
    </p>

  <h4>Domain Specific Ontologies</h4>

  <p>
   In order to describe the content of such images, a mechanism to
   represent the domain specific content depicted within them is
   needed. For this use case, domain ontologies that define space
   specific concepts and relations can be used. Such ontologies are
   freely available and include, but are not limited to the following:
  </p>
  <ul>
   <li>Shuttle related
(<a href="http://semspace.mindswap.org/2004/ontologies/ShuttleMission-ont.owl">OWL</a> /
<a href="http://semspace.mindswap.org/2004/ontologies/ShuttleMission-ont.rdfs">RDFS</a>)</li>
   <li>Space vehicle system related
(<a href="http://semspace.mindswap.org/2004/ontologies/System-ont.owl">OWL</a> /
<a href="http://semspace.mindswap.org/2004/ontologies/System-ont.rdfs">RDFS</a>)</li>
  </ul>

  <h4>Visual Ontologies</h4>
  <p>
   As discussed above, this scenario requires the ability to state
   that images (and possibly their regions) depict certain things. For
   example, consider a picture of the <a
   href="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg">Apollo
   7 Saturn rocket launch</a>. One would want to make assertions that
   include that the image <i>depicts</i> the Apollo 7 launch, the
   Apollo 7 Saturn IB space vehicle is depicted in a rectangular
   <i>region</i> around the rocket, the image <i>creator</i> is NASA,
   etc. One possible way to accomplish this is to use a combination of
   various multimedia related ontologies, including <a
   href="http://xmlns.com/foaf/0.1/">FOAF</a> and the <a
   href="http://www.mindswap.org/2005/owl/digital-media">MINDSWAP
   digital-media ontology</a>. More specifically, image depictions can
   be asserted via a <i>depiction</i> property (a sub-property of
   <tt>foaf:depiction</tt>) defined in the MINDSWAP Digital Media
   ontology. Thus, images can be semantically linked to instances
   defined on the Web. Image regions can defined via an
   <i>ImagePart</i> concept (also defined in the MINDSWAP Digital
   Media ontology). Additionally, regions can be given a bounding box
   by using a property named <i>svgOutline</i>, allowing localizing of
   image parts. Essentially SVG outlines (SVG XML literals) of the
   regions can be specified using this property. Using the <a
   href="http://dublincore.org/schemas/rdfs/">Dublin Core</a>
   and the <a href="http://www.w3.org/2003/12/exif/">EXIF
   Schema</a> more general annotations about the image can be asserted
   including its creator, size, etc. A subset of these sample
   annotations are shown in the RDF graph below in <a
   href="#figure2">Figure 2</a>.
  </p>

  <div style="text-align: center">
   <a href="images/examples/nasaRDFDiagram.png">
   <img alt="RDF Graph Describing the Apollo 7 Launch Image"
    style="width: 90%;"
    src="images/examples/nasaRDFDiagram.png"/></a>
   <br/>
   <a id="figure2" name="figure2">Figure 2: An RDF Graph
    Describing the Apollo 7 Launch Image</a>
  </div>

  <p>
   <a href="#figure2">Figure 2</a> illustrates how the approach links metadata to the image:
  </p>
  <ul>
   <li>image content, e.g., Apollo 7 Launch, is identified by
    <code>http://www.mindswap.org/2005/owl/digital-media#depicts</code></li>
   <li>image subparts are identified by the property
    <code>http://www.mindswap.org/2005/owl/digital-media#hasRegion</code></li>
   <li>image regions are localized using
    <code>http://www.mindswap.org/2005/owl/digital-media#svgOutline</code> and an SVG snippet</li>
  </ul>

  <p>
   Additionally, the entire annotations of the Apollo 7 launch are shown below in RDF/XML.
  </p>

  <div class="c1">
   <a id="exampleApollo7" name="exampleApollo7">RDF/XML annotations of Apollo 7 launch </a>
  </div>

  <div class="exampleInner">
   <pre>
&lt;rdf:RDF
  xmlns:j.0="http://www.w3.org/2003/12/exif/ns#"
  xmlns:j.1="http://www.mindswap.org/2005/owl/digital-media#"
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:j.2="http://semspace.mindswap.org/2004/ontologies/System-ont.owl#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:owl="http://www.w3.org/2002/07/owl#"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:j.3="http://semspace.mindswap.org/2004/ontologies/ShuttleMission-ont.owl#"
  xml:base="http://example.org/NASA-Use-Case" &gt;

  &lt;rdf:Description rdf:about="A0"&gt;
    &lt;j.1:depicts rdf:resource="#Saturn_1B"/&gt;
    &lt;rdf:type rdf:resource="http://www.mindswap.org/~glapizco/technical.owl#ImagePart"/&gt;
    &lt;rdfs:label&gt;region2407&lt;/rdfs:label&gt;
    &lt;j.1:regionOf rdf:resource="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg"/&gt;
    &lt;j.1:svgOutline&gt;
      &lt;svg xml:space="preserve" width="451" heigth="640" viewBox="0 0 451 640"&gt;
        &lt;image xlink:href="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg" x="0" y="0"  width="451" height="640" /&gt;
        &lt;rect x="242.0" y="79.0" width="46.0" height="236.0" style="fill:none; stroke:yellow; stroke-width:1pt;"/&gt;
      &lt;/svg&gt;
    &lt;/j.1:svgOutline&gt;
  &lt;/rdf:Description&gt;

  &lt;rdf:Description rdf:about="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg"&gt;
    &lt;j.0:imageLength&gt;640&lt;/j.0:imageLength&gt;
    &lt;dc:date&gt;10/11/1968&lt;/dc:date&gt;
    &lt;dc:description&gt;Taken at Kennedy Space Center in Florida&lt;/dc:description&gt;
    &lt;j.1:depicts rdf:resource="#Apollo_7_Launch"/&gt;
    &lt;j.1:hasRegion rdf:nodeID="A0"/&gt;
    &lt;dc:creator&gt;NASA&lt;/dc:creator&gt;
    &lt;rdf:type rdf:resource="http://www.mindswap.org/~glapizco/technical.owl#Image"/&gt;
    &lt;j.0:imageWidth&gt;451&lt;/j.0:imageWidth&gt;
  &lt;/rdf:Description&gt;

  &lt;rdf:Description rdf:about="#Apollo_7_Launch"&gt;
    &lt;j.3:launchDate&gt;10/11/1968&lt;/j.3:launchDate&gt;
    &lt;j.3:codeName&gt;Apollo 7 Launch&lt;/j.3:codeName&gt;
    &lt;j.3:has_shuttle rdf:resource="#Saturn_1B"/&gt;

    &lt;rdfs:label&gt;Apollo 7 Launch&lt;/rdfs:label&gt;
    &lt;j.1:depiction rdf:resource="http://grin.hq.nasa.gov/IMAGES/SMALL/GPN-2000-001171.jpg"/&gt;
    &lt;rdf:type rdf:resource="http://semspace.mindswap.org/2004/ontologies/ShuttleMission-ont.owl#Launch"/&gt;
  &lt;/rdf:Description&gt;

  &lt;rdf:Description rdf:about="#Saturn_1B"&gt;
    &lt;rdfs:label&gt;Saturn_1B&lt;/rdfs:label&gt;
    &lt;j.1:depiction rdf:nodeID="A1"/&gt;
    &lt;rdfs:label&gt;Saturn 1B&lt;/rdfs:label&gt;
    &lt;rdf:type rdf:resource="http://semspace.mindswap.org/2004/ontologies/System-ont.owl#ShuttleName"/&gt;
    &lt;j.1:depiction rdf:nodeID="A0"/&gt;
  &lt;/rdf:Description&gt;

&lt;/rdf:RDF&gt;
   </pre>
  </div>

  <p>
   In order to represent the low level features of images, the <a
   href="http://www.acemedia.org/aceMedia/reference/resource/index.html">aceMedia
   Visual Descriptor Ontology</a> can be used. This ontology contains
   representations of MPEG-7 visual descriptors and models Concepts
   and Properties that describe visual characteristics of objects. For
   example, the dominant color descriptor can be used to describe the
   number and value of dominant colors that are present in a region of
   interest and the percentage of pixels that each associated color
   value has.
  </p>

  <h4>Available Annotation Tools</h4>
  <p>
   Existing toolkits, such as <a href="#PS">PhotoStuff</a> and <a
   href="#mOnto">M-OntoMat-Annotizer</a>, currently provide
   graphical environments to accomplish the annotation tasks mentioned
   above. Using such tools, users can load images, create regions
   around parts of the image, automatically extract low-level features
   of selected regions (via M-OntoMat-Annotizer), assert statements
   about the selected regions, etc. Additionally, the resulting
   annotations can be exported as RDF/XML (as shown above), thus
   allowing them be shared, indexed, and used by advanced
   annotation-based browsing (and searchable) environments.
   </p>

  <h3>
   <a name="solution_biomed">
    5.5 Use Case: Biomedical Images
   </a>
  </h3>

  <h4>Possible Semantic Web-based solution</h4>

  <div style="float: right; width: 30%; border: 1px solid gray; padding: 1%; margin: 1%">
   <img style="width: 100%;;"
     src="images/examples/biomed.jpg"
     alt="Example showing a microscopic image of lymph tissue"/>
   Microscopic image of lymph tissue.<br/>
   Image courtesy of prof. C.A. Beltrami, University of Udine,
   used with permission.
  </div>
  <p>
   A solution for the requirements expressed in the use case
   described in <a href="#biomed">Section 1.5</a> includes support
   for annotating information about images and their regions.

   Given the numerous controlled vocabularies (including thesauri and
   ontologies such as <a href="http://icd9cm.chrisendres.com/">ICD9</a>,
   <a href="http://www.nlm.nih.gov/mesh/meshhome.html">MeSH</a>,
   <a href="http://www.snomed.org/">SNOMED</a>) that are already in use in
   the biomedical domain, the annotations should be
   able to make use of the concepts of these controlled vocabularies, and
   not be free text or keywords only.  And given the many data stores that are already
   available (including electronic patient records), annotation should
   not be a fully manual process. Instead, metadata should be
   (semi)automatically linked to, or extracted from, these existing
   sources.

   For some medical imaging, image annotation is already done through
   image formats able to carry metadata, like
   <a href="http://medical.nema.org/">DICOM</a> for
   radiology. Preferably, Semantic Web-solutions should be made
   interoperable with such embedded metadata image formats.
  </p>

  <h4>Domain Specific Ontologies</h4>

  <p>
   In order to describe the content of such images, a mechanism to
   represent the domain specific content depicted within them is
   needed. For biomedical concepts, many vocabularies are available,
   including, but not limited to:
  </p>
  <ul>
    <li>
    The ICD9-CM classification includes pathology and procedures
    concepts.
   </li>
   <li>
    SNOMED aims at the complete description of a clinical
    record. Eleven modules include terms for anatomy, diagnosis,
    morphological aspects, organisms, procedures and techniques, and more.
   </li>
   <li>
    MeSH is a terminology developed for describing scientific literature
    items.
   </li>
   <li>
    NCI Thesaurus, more recently developed, is a complete ontology
    devoted to concepts related to oncology.
   </li>
  </ul>

  <p>
   The National Library of Medicine also developed
   <a href="http://umlsinfo.nlm.nih.gov/">UMLS</a> (Unified Medical Language System),
   including a meta-thesaurus where terms coming from over 60
   biomedical terminologies in many languages are connected in a
   semantic network.  The orginal formats of all these ontologies are
   not in RDF or OWL. However, they can be (or already have been)
   converted into these languages. All metadata described above is
   covered by one or more of these vocabularies. Furthermore, thanks
   to UMLS, it is also possible to move from one vocabulary to another
   using the meta-thesaurus as a bridge.
  </p>

  <p>
   As a sample, the gastric mucosa can be brought from MeSH, where it
   has unique ID <tt>D005753</tt>, and is placed in the digestive system
   subtree as a part of the stomach as well as in the tissue subtree
   among the membranes.
  </p>

  <h4>Using semantic annotations</h4>
  <p>
   Such kind of metadata may help in retrieving images with greater
   recall than usual. In fact, a student might search for images of
   the stomach. If annotated as suggested, the student will be able to
   discover this image of the gastric mucosa, since it is a part of the
   stomach according to the MeSH ontology. On the other side, when looking for
   samples of membranes, she will again find also this image, since it
   is also depicting a specific kind of membrane. The same could be
   said for queries involving epithelium, which is not the main
   subject of the image, but it is still a part of it.
  </p>

  <h4>A possible scenario</h4>
  <p>
   As a scenario, let's consider an image from the medical field of
   histology, a field that studies the microscopic structure of
   tissue. The image is taken using a microscope, and stored for
   educational aims. Due to this specific purpose, there is no need
   to annotate the image with sensible patient' data. However, some
   personal data are needed to interpret the image meaning: for example, age
   and gender meaning are sometimes useful variables, as also patient personal and familiar
   history are. These could become annotations of the image and
   eventually also of other images of the same patient, acquired with
   the same tool or even some other modality. This may correspond to
   the following RDF code, where we also consider that to the same
   patient, many images could be associated (in this case, image1.jpg
   and image2.jpg):
  </p>
  <div class="exampleInner">
   <pre>
&lt;?xml version="1.0"?&gt;
&lt;rdf:RDF
     xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
     xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
     xmlns:dc="http://purl.org/dc/elements/1.1/"
     xmlns:dcterms="http://purl.org/dc/terms/"&gt;
  &lt;rdf:Description rdf:about="http://www.telemed.uniud.it/cases/case1/"&gt;
    &lt;dc:creator&gt;Beltrami, Carlo Alberto&lt;/dc:creator&gt;
    &lt;dc:contributor&gt;Della Mea, Vincenzo&lt;/dc:contributor&gt;
    &lt;dc:title&gt;Sample annotated image&lt;/dc:title&gt;
    &lt;dc:subject&gt;
      &lt;dcterms:MESH&gt;
        &lt;rdf:value&gt;A10.549.400&lt;/rdf:value&gt;
        &lt;rdfs:label&gt;lymph nodes&lt;/rdfs:label&gt;
      &lt;/dcterms:MESH&gt;
    &lt;/dc:subject&gt;
    &lt;dc:subject&gt;
      &lt;dcterms:MESH&gt;
        &lt;rdf:value&gt;C04.697.650.560&lt;/rdf:value&gt;
        &lt;rdfs:label&gt;lymphatic metastasis&lt;/rdfs:label&gt;
      &lt;/dcterms:MESH&gt;
    &lt;/dc:subject&gt;
    &lt;dc:subject&gt;
      &lt;dcterms:MESH&gt;
        &lt;rdf:value&gt;C04.557.470.200.025.200&lt;/rdf:value&gt;
        &lt;rdfs:label&gt;carcinoid tumor&lt;/rdfs:label&gt;
      &lt;/dcterms:MESH&gt;
    &lt;/dc:subject&gt;
    &lt;dc:date&gt;2006-05-30&lt;/dc:date&gt;
    &lt;dc:language&gt;EN&lt;/dc:language&gt;
    &lt;dc:description&gt;Female, 86 yrs. Some clinical history. This field might be further structured.&lt;/dc:description&gt;
    &lt;dc:identifier&gt;http://www.telemed.uniud.it/cases/case1/&lt;/dc:identifier&gt;
    &lt;dc:relation.hasPart rdf:resource="http://www.telemed.uniud.it/cases/case1/image1.jpg"/&gt;
    &lt;dc:relation.hasPart rdf:resource="http://www.telemed.uniud.it/cases/case1/image2.jpg"/&gt;
  &lt;/rdf:Description&gt;
&lt;/rdf:RDF&gt;
   </pre>
  </div>

  <p>
   The subject has been described using terms from MeSH.  Another
   level of description regards the biological sample and its
   preparation for microscopy. The biological sample comes from a body
   organ. It has been sectioned and stained using some specific
   chemical product. Associated to the sample, there are one or more
   diagnostic hypotheses, which derive from both patient data and
   image (or images) content. Let's consider to have a chromogranin
   stained section of a lymph node (chromogranin is an
   immunohistochemical antibody), showing signs of metastatis from a
   carcinoid tumor.  A final level of description regards the
   acquisition device, constituted by a microscope with specific
   optical properties (the most important of which are objective
   magnification and numerical aperture, which influence the optical
   resolution and thus the visible detail), and the camera, with its
   pixel resolution and color capabilities. In this scenario, the
   image might have been acquired with a 10x objective, 0.20 NA, at a
   resolution of 1600x1200 pixel.
  </p>

  <div class="exampleInner">
    <pre>
&lt;rdf:Description rdf:about="http://www.telemed.uniud.it/cases/case1/image1.jpg"&gt;
  &lt;dp:image dp:staining="chromogranin" dp:magnification="10" dp:na="0.20"
     dp:xsize="1600" dp:ysize="1200" dp:depth="24" dp:xres="1.2" dp:yres="1.2"/&gt;
  &lt;dc:Description&gt;chromogranin-stained section of lymph node&lt;/dc:Description&gt;
  &lt;dc:subject&gt;
    &lt;dcterms:MESH&gt;
      &lt;rdf:value&gt;C030075&lt;/rdf:value&gt;
      &lt;rdfs:label&gt;chromogranin A&lt;/rdfs:label&gt;
    &lt;/dcterms:MESH&gt;
  &lt;/dc:subject&gt;
  &lt;dc:subject&gt;
    &lt;dcterms:MESH&gt;
      &lt;rdf:value&gt;E05.200.750.551.512&lt;/rdf:value&gt;
      &lt;rdfs:label&gt;immunohistochemistry&lt;/rdfs:label&gt;
    &lt;/dcterms:MESH&gt;
  &lt;/dc:subject&gt;
  &lt;dc:subject&gt;
    &lt;dcterms:MESH&gt;
      &lt;rdf:value&gt;E05.595&lt;/rdf:value&gt;
      &lt;rdfs:label&gt;microscopy&lt;/rdfs:label&gt;
    &lt;/dcterms:MESH&gt;
  &lt;/dc:subject&gt;
&lt;/rdf:Description&gt;
   </pre>
  </div>

  <p>
   In addition to global metadata referring to the whole image,
   multimedia related ontologies can be used to localize and represent
   regions within particular images. Annotations can provide for a
   description of specific morphological features, including a
   specification of the depicted anatomical details, presence and kind
   of abnormal cells, and generally speaking, features supporting the
   above mentioned diagnostic hypotheses. In such an image, metastatic
   cells are evidentiated in brown by the staining, and can be
   semantically annotated. In this case, the MeSH terminology does not
   provide sufficient detail, so that we must chose another source of
   terms: the NCI Thesaurus is one possibility.
  </p>

  <div class="exampleInner">
   <pre>
&lt;rdf:Description id="area1"&gt;
  &lt;dp:rectangle&gt;some rectangle definition&lt;/dp:rectangle&gt;
&lt;/rdf:Description&gt;
   </pre>
  </div>
  <div class="exampleInner">
   <pre>
&lt;rdf:Description about="#area1"&gt;
  &lt;dc:Description&gt;metastatic cells&lt;/dc:Description&gt;
  &lt;dc:subject&gt;
    &lt;dcterms:NCI&gt;
      &lt;rdf:value&gt;C4904&lt;/rdf:value&gt;
      &lt;rdfs:label&gt;Metastatic_Neoplasm_to_Lymph_Nodes&lt;/rdfs:label&gt;
    &lt;/dcterms:NCI&gt;
  &lt;/dc:subject&gt;
  &lt;dc:subject&gt;
    &lt;dcterms:NCI&gt;
      &lt;rdf:value&gt;C12917&lt;/rdf:value&gt;
      &lt;rdfs:label&gt;Malignant_Cell&lt;/rdfs:label&gt;
    &lt;/dcterms:NCI&gt;
  &lt;/dc:subject&gt;
&lt;/rdf:Description&gt;
   </pre>
  </div>

  <!-- ===================================================================== -->
  <h2>
   <a name="conclusions" id="conclusions">
    6. Conclusions
   </a>
  </h2>
  <p>
   Current Semantic Web technologies are sufficiently generic to support
   annotation of a wide variety of Web resources, including image
   resources. This document provides examples of the use of Semantic
   Web languages and tools for image annotation, based on use cases
   for a wide variety of domains. It also briefly surveys some
   currently available vocabularies and tools that can be used to
   semantically annotate images so that machine can better process them.
   The use of Semantic Web technologies have significant advantages in applications areas in
   which the interoperability of heterogeneous metadata is important
   and in areas that require an explicitly defined and formal
   semantics of the metadata in order to perform reasoning tasks.
  </p>

  <p>
   Still, many things need to be improved. Commonly accepted, widely
   used vocabularies for image annotation are still missing.  Having
   such vocabularies would help in sharing metadata across
   applications and across multiple domains. Especially, a standard
   means to address subregions withing an image is still missing.  In
   addition, tool support needs to improve dramatically before
   Semantic Web-based image annotation can be applied on an industrial
   scale: support needs to be integrated in the entire production and
   distribution chain. Finally, many existing approaches for image
   metadata are not based on Semantic Web technologies, and work is
   required to make these approaches interoperable with the Semantic Web.
  </p>

  <!-- ===================================================================== -->
  <h2>
   <a name="references" id="references">
    References</a>
  </h2>

  <dl>

   <dt>
    <a id="refAAT" name="refAAT">[AAT]</a>
   </dt>
   <dd><span class="title">Art and Architecture Thesaurus</span>.
    The J.  Paul Getty Trust, 2004.
    (See <a href="http://www.getty.edu/research/conducting_research/vocabularies/aat/">
     http://www.getty.edu/research/conducting_research/vocabularies/aat/</a>)
   </dd>

   <dt>
    <a id="DublinCore" name="DublinCore">[Dublin Core]</a>
   </dt>
   <dd>
    The Dublin Core Metadata Initiative,
    <a href="http://dublincore.org/documents/dces/">Dublin Core Metadata Element Set, Version 1.1: Reference Description</a>
   </dd>

   <dt>
    <a id="Flickr" name="Flickr">[Flickr]</a>
    </dt>
   <dd>
    <a href="http://www.flickr.com/">Flickr</a> online photo management and sharing application
   </dd>

   <dt>
    <a id="httpRange-14" name="httpRange-14">
     [httpRange-14]</a></dt>
   <dd>
    TAG's issue list, issue 14, see
    <a href="http://www.w3.org/2001/tag/issues.html?type=1#httpRange-14">
     http://www.w3.org/2001/tag/issues.html?type=1#httpRange-14</a>
   </dd>

   <dt>
    <a id="HTTP-URI" name="HTTP-URI">
     [HTTP-URI]</a></dt>
   <dd>
    Tim Berners-Lee, What do HTTP URIs Identify?  Available at <a
    href="http://www.w3.org/DesignIssues/HTTP-URI">http://www.w3.org/DesignIssues/HTTP-URI</a>
   </dd>

   <dt>
    <a id="Hunter01" name="Hunter01">[Hunter, 2001]</a>
   </dt>
   <dd>
    J. Hunter.
    <!-- official link broken? a href="http://www.semanticweb.org/SWWS/program/full/paper59.pdf" -->
    <a href="http://www.itee.uq.edu.au/~jane/jane-hunter/swws.pdf">Adding
    Multimedia to the Semantic Web &#8212; Building an MPEG-7
    Ontology</a>. In <i><a
    href="http://www.semanticweb.org/SWWS/">International Semantic Web
    Working Symposium (SWWS 2001)</a></i>, Stanford University,
    California, USA, July 30 - August 1, 2001
      </dd>

   <dt>
    <a id="lsid" name="lsid">
     [LSID]
    </a>
   </dt>
   <dd>
    Life Sciences Identifier specification, <a
     href="http://www.omg.org/cgi-bin/doc?dtc/04-05-01">
     http://www.omg.org/cgi-bin/doc?dtc/04-05-01</a>
   </dd>

   <dt>
    <a name="refMIME-2" id="refMIME-2">[MIME-2]</a>
   </dt>
   <dd>
    <a href="ftp://ftp.isi.edu/in-notes/rfc2046.txt">
     RFC 2046: Multipurpose Internet Mail Extensions (MIME) Part Two:
     Media Types
    </a>.
    N. Freed, N. Borenstein, November 1996.  Available at
    <a href="ftp://ftp.isi.edu/in-notes/rfc2046.txt">ftp://ftp.isi.edu/in-notes/rfc2046.txt</a>
   </dd>

   <dt>
    <a id="mOnto" name="mOnto">[M-OntoMat-Annotizer]</a>
   </dt>
   <dd>
    M-OntoMat-Annotizer Project Homepage at <a
    href="http://www.acemedia.org/aceMedia/results/software/m-ontomat-annotizer.html">
    http://www.acemedia.org/aceMedia/results/software/m-ontomat-annotizer.html</a>
   </dd>

   <dt>
    <a id="MPEG-7" name="MPEG-7">[MPEG-7]</a>
   </dt>
   <dd>
    Information Technology - Multimedia Content Description Interface (MPEG-7).
    Standard No. ISO/IEC 15938:2001, International Organization for Standardization(ISO), 2001
   </dd>

   <dt>
    <a id="Ossenbruggen04" name="Ossenbruggen04">[Ossenbruggen, 2004]</a>
   </dt>
   <dd>
    J. van Ossenbruggen, F. Nack, and L. Hardman. That Obscure Object of Desire: Multimedia Metadata on the Web (Part I). In:
    IEEE Multimedia 11(4), pp. 38-48 October-December 2004
   </dd>

   <dt>
    <a id="Ossenbruggen05" name="Ossenbruggen05">[Ossenbruggen, 2005]</a>
   </dt>
   <dd>
    F. Nack, J. van Ossenbruggen, and L. Hardman. That Obscure Object of Desire: Multimedia Metadata on the Web (Part II). In:
    IEEE Multimedia 12(1), pp. 54-63 January-March 2005
   </dd>

   <dt>
    <a name="OWL-Guide" id="OWL-Guide">[OWL Guide]</a>
   </dt>
   <dd>
     <cite>
       <a href="http://www.w3.org/TR/2004/REC-owl-guide-20040210/">
        OWL Web Ontology Language Guide</a></cite>, Michael K.
        Smith, Chris Welty, and Deborah L. McGuinness, Editors, W3C
        Recommendation, 10 February 2004,
        http://www.w3.org/TR/2004/REC-owl-guide-20040210/ .
        <a href="http://www.w3.org/TR/owl-guide/">Latest
        version</a> available at http://www.w3.org/TR/owl-guide/
   </dd>

   <dt>
    <a name="OWL" id="OWL">[OWL Semantics and Abstract Syntax]</a></dt>
   <dd>
    <cite>
     <a href=
      "http://www.w3.org/TR/2004/REC-owl-semantics-20040210/">OWL Web
     Ontology Language Semantics and Abstract Syntax</a></cite>, Peter
     F. Patel-Schneider, Patrick Hayes, and Ian Horrocks, Editors, W3C
     Recommendation 10 February 2004,
     http://www.w3.org/TR/2004/REC-owl-semantics-20040210/ . <a href=
     "http://www.w3.org/TR/owl-semantics/">Latest version</a>
     available at http://www.w3.org/TR/owl-semantics/</dd>

   <dt>
    <a id="PS" name="PS">[PhotoStuff]</a>
   </dt>
   <dd>
    PhotoStuff Project Homepage at <a
    href="http://www.mindswap.org/2003/PhotoStuff/">http://www.mindswap.org/2003/PhotoStuff/</a>
   </dd>

   <dt><a id="RDF-Primer" name="RDF-Primer">[RDF Primer]</a></dt>
   <dd>
     <cite><a href="http://www.w3.org/TR/2004/REC-rdf-primer-20040210/">RDF
   Primer</a></cite>, F. Manola, E. Miller, Editors, W3C Recommendation, 10 February 2004.
   <a href="http://www.w3.org/TR/2004/REC-rdf-primer-20040210/">This
   version</a> is
   http://www.w3.org/TR/2004/REC-rdf-primer-20040210/. The <a href="http://www.w3.org/TR/rdf-primer/">latest version</a> is at
   http://www.w3.org/TR/rdf-primer/
   </dd>

   <dt><a id="RDF" name="RDF"></a>[RDF Syntax]</dt>
   <dd>
    <cite>
     <a href="http://www.w3.org/TR/2004/REC-rdf-syntax-grammar-20040210/">
      RDF/XML Syntax Specification (Revised)</a>
    </cite>, Dave Beckett,
      Editor, W3C Recommendation, 10 February 2004,
      http://www.w3.org/TR/2004/REC-rdf-syntax-grammar-20040210/ .  <a
      href="http://www.w3.org/TR/rdf-syntax-grammar/">Latest
      version</a> available at
      http://www.w3.org/TR/rdf-syntax-grammar/
   </dd>

   <dt>
    <a id="Stamou05" name="Stamou05">[Stamou, 2005]</a>
   </dt>
   <dd>
    G. Stamou and S. Kollias (eds). Multimedia Content and the
    Semantic Web: Methods, Standards and Tools.  John Wiley &amp; Sons
    Ltd, 2005
   </dd>

   <dt>
    <a id="Troncy03" name="Troncy03">[Troncy, 2003]</a>
   </dt>
   <dd>
    R. Troncy. <a
    href="http://springerlink.metapress.com/openurl.asp?genre=article&amp;issn=0302-9743&amp;volume=2870&amp;spage=566">
    Integrating Structure and Semantics into Audio-visual
    Documents</a>. In <i><a
    href="http://iswc2003.semanticweb.org/">Second International
    Semantic Web Conference (ISWC 2003)</a></i>, pages 566 &#8211;
    581, Sanibel Island, Florida, USA, October 20-23,
    2003. Springer-Verlag Heidelberg
   </dd>

   <dt>
    <a id="Tsinaraki" name="Tsinaraki">
     [Tsinaraki]</a></dt>
   <dd>
    Tsinaraki, C.: OWL soccer ontology available at
    <a href="http://elikonas.ced.tuc.gr/ontologies/soccer.zip">http://elikonas.ced.tuc.gr/ontologies/soccer.zip</a>
   </dd>

   <dt>
    <a id="TVA" name="TVA">[TV Anytime]</a>
   </dt>
   <dd>
    TV Anytime Forum,
    <a href="http://www.tv-anytime.org/">
     http://www.tv-anytime.org/
    </a>
   </dd>

   <dt>
    <a id="refULAN" name="refULAN">[ULAN]</a>
   </dt>
   <dd><span class="title">Union List of Artist Names</span>.
    The J.  Paul Getty Trust, 2004.
    (See <a href="http://www.getty.edu/research/conducting_research/vocabularies/ulan/">
     http://www.getty.edu/research/conducting_research/vocabularies/ulan/</a>)
   </dd>

   <dt>
    <a id="VDO" name="VDO">[VDO]</a>
   </dt>
   <dd>
    aceMedia Visual Descriptor Ontology, available from <a
    href="http://www.acemedia.org/aceMedia/reference/resource/index.html">
    http://www.acemedia.org/aceMedia/reference/resource/index.html</a>
   </dd>

   <dt>
    <a id="VraCore" name="VraCore">[VRA Core]</a>
   </dt>
   <dd>
    Visual Resources Association Data Standards Committee,
    <a href="http://www.vraweb.org/vracore3.htm">
     VRA Core Categories, Version 3.0</a>.  Available at:
    <a href="http://www.vraweb.org/vracore3.htm">
     http://www.vraweb.org/vracore3.htm</a>
   </dd>

   <dt>
    <a id="VraRDF" name="VraRDF">[VRA in RDF/OWL]</a>
   </dt>
   <dd>
    Mark van Assem. <a href="http://www.w3.org/2001/sw/BestPractices/MM/vra-conversion.html">
     http://www.w3.org/2001/sw/BestPractices/MM/vra-conversion.html</a>
    describes the RDFS schema of VRA Core 3.0 used in
    <a href="#solution_culture">section 5.2</a>
   </dd>

   <dt>
    <a id="SVG" name="SVG">[SVG]</a>
   </dt>
   <dd>
    <a href="http://www.w3.org/Graphics/SVG/">Scalable Vector Graphics (SVG)</a> -
XML Graphics for the Web. W3C Working Group
   </dd>

   <dt>
    <a id="WordNet" name="WordNet">[WordNet in RDF/OWL]</a>
   </dt>
   <dd>
    Mark van Assem, Aldo Gangemi and Guus Schreiber.
    <a href="http://www.w3.org/2001/sw/BestPractices/WNET/wn-conversion.html">
     RDF/OWL Representation of WordNet
    </a>.  W3C Editor's draft, work in progress
   </dd>

    <dt>
     <a id="XML-NS" name="XML-NS">[XML NS]</a>
    </dt>
    <dd>
     <cite>
     <a href="http://www.w3.org/TR/1999/REC-xml-names-19990114/">Namespaces
     in XML</a></cite>, Bray T., Hollander D., Layman A.
     (Editors), World Wide Web Consortium, 14 January 1999. <a href="http://www.w3.org/TR/1999/REC-xml-names-19990114/">This
     version</a> is http://www.w3.org/TR/1999/REC-xml-names-19990114/. The <a href="http://www.w3.org/TR/REC-xml-names/">latest version</a>
     is http://www.w3.org/TR/REC-xml-names/
    </dd>

  </dl>

  <!-- ======================================================================== -->

  <h2>
   <a id="acknowledgments" name="acknowledgments">Acknowledgments</a>
  </h2>

  <p>
   Thanks to Vincenzo Della Mea (University of Udine) for contributing the biomedical images use case.
   The editors would also like to thank
   Chris Catton (University of Oxford),
   Karl Dubost (W3C),
   John Smith (IBM T. J. Watson Research Center)
   and the following Working Group members
   for their feedback on earlier versions of this document:
   Mark van Assem,
   Jeremy Caroll,
   Jane Hunter,
   Libby Miller,
   Guus Schreiber and
   Michael Uschold.
  </p>


  <p>
    This document is a product of the <a
    href="http://www.w3.org/2001/sw/BestPractices/MM/">Multimedia
    Annotation on the Semantic Web Task Force</a> of the <a
    href="http://www.w3.org/2001/sw/BestPractices/">W3C Semantic
    Web Best Practices &amp; Deployment Working Group</a>, which
    is part of the <a href="http://www.w3.org/2001/sw/">W3C
    Semantic Web</a> activity.
  </p>

  <!-- ======================================================================== -->

  <h2>
   <a id="changelog" name="changelog">Change log</a>
  </h2>

  <p>
   The following list described the changes made to this document
   since the <a
   href="http://www.w3.org/TR/2006/WD-swbp-image-annotation-20060322/">previous
   public version (March 22, 2006)</a>.  It also describes all
   responses of the editors to comments on the previous version.
  </p>

  <ul>
   <li>Sept 4. Troncy revised the whole document for final publication.</li>

   <li>July 20. Troncy revised the various XML/RDF examples typos, see also <a
href="http://lists.w3.org/Archives/Public/public-xg-mmsem/2006Jul/0043.html">the announcement on the
list</a>.
   </li>

   <li>June 15. Van Ossenbruggen added and edited the biomedical use case provided by Vincenzo Della Mea of the University of Udine.
       Also added Vincenzo to the Acknowledgments section.
   </li>
   <li>
    May 4.  Van Ossenbruggen revised <a href="#semweb_intro">Semantic
    Web Basics</a> section, see also <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006May/0022.html">the
    announcement on the list</a>.
   </li>

   <li>
     May 4. Troncy substancially revised the <a href="#television_archive">Television Archive
     use case</a> and its solution (<a href="#solution_TVarchive">section 5.3</a>).
     See also the <a href="">announcement</a>.
   </li>

   <li>
    May 2. Response to <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Apr/0035.html">Annotation
    Context mail by Karl Dubost</a>: Van Ossenbruggen <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006May/0007.html">revised</a>
    the text in the <a href="#annot_intro">paragraph mentioned</a>.
    Also added Karl to Acknowledgments.
   </li>

   <li>
    April 27. Response to <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Apr/0037.html">Use
    cases mail by Karl Dubost</a>: Van Ossenbruggen <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Apr/0097.html">followed
    suggestions made by Karl</a>.</li>


   <li>
    April 27. Response to <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Apr/0033.html">Prose
    Improvement mail by Karl Dubost</a>: Van Ossenbruggen <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Apr/0093.html">changed
    prose of abstract</a>, <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Apr/0095.html">changed
    prose of interoperability section</a>.
   </li>

   <li>
    April 11. Stamou and Tzouvaras revised section
    <a href="#solution_personal">
     5.1 Use Case: Management of Personal Digital Photo Collections
     </a>. See also the <a
     href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Apr/0023.html">announcement
     on the list</a>. The broken links reported by
     <a href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Apr/0050.html">Mike Uschold</a>
     have been <a href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006May/0015.html">fixed</a>.
   </li>

   <li>
    March 28. Response to <a
    href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Mar/0110.html">Mathias
    Lux</a>: Van Ossenbruggen decided not change this document because tools mentioned are
    not Semantic Web based, instead we added relevant pointers to <a
    href="
    http://www.w3.org/2001/sw/BestPractices/MM/resources/Resources.html">Resources</a>
    overview.
   </li>

   <li>March 28. Response to <a
   href="http://lists.w3.org/Archives/Public/public-swbp-wg/2006Mar/0102.html">
   Vincenzo Della Mea </a>:  The editors have kindly invited Vincenzo
   to submit a use case before April 28.
   </li>


  </ul>
 </body>
</html>