index.html 22.3 KB

Raw Blame History Permalink

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-US" lang="en-US">
<head>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
  <title>Publishing Open Government Data</title>
  <link href="http://www.w3.org/StyleSheets/TR/W3C-WD" rel="stylesheet"
  type="text/css" />
  <style type="text/css">
	.leader {
		text-transform: uppercase;
	}
	img {
		padding-top: 2em;
		padding-bottom: 2em;
	}
	</style>
</head>

<body>

<div class="head">
<a href="http://www.w3.org/"><img height="48" width="72" alt="W3C"
src="http://www.w3.org/Icons/w3c_home" /></a>

<h1 id="title">Publishing Open Government Data</h1>

<h2 id="W3C-doctype">W3C Working Draft 8 September 2009</h2>
<dl class="meta control">
  <dt>This version:</dt>
    <dd><a href="http://www.w3.org/TR/2009/WD-gov-data-20090908/"
      id="this-version-url">http://www.w3.org/TR/2009/WD-gov-data-20090908/</a></dd>
  <dt>Latest version:</dt>
    <dd><a
      href="http://www.w3.org/TR/gov-data/">http://www.w3.org/TR/gov-data/</a></dd>
  <!--
                    			<dt>Previous version:</dt>
                    			<dd><a href="http://www.w3.org/TR/2009/foo/">http://www.w3.org/TR/2009/foo/</a> (<a href="http://www.w3.org/TR/2009/foo/diff-from-20090421">color-coded diff</a>)</dd>
                    			-->
</dl>
<dl>
  <dt>Authors:</dt>
    <dd><a href="http://citizencontact.myopenid.com/">Daniel Bennett</a> &lt;<a
      class="email"
      href="mailto:daniel@citizencontact.com">daniel@citizencontact.com</a>&gt;
      (Invited Expert)</dd>
    <dd>Adam Harvey (Invited Expert)</dd>
  <dt></dt>
</dl>

<p>Please refer to the <a href="/2007/eGov/IG/wiki/Glossary">glossary</a> for
this document.</p>
<hr />

<p class="copyright"><a
href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a> ©
2009 <a href="http://www.w3.org/"><acronym
title="World Wide Web Consortium">W3C</acronym></a><sup>®</sup> (<a
href="http://www.csail.mit.edu/"><acronym
title="Massachusetts Institute of Technology">MIT</acronym></a>, <a
href="http://www.ercim.org/"><acronym
title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>,
<a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a
href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>,
<a
href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a>
and <a href="http://www.w3.org/Consortium/Legal/copyright-documents">document
use</a> rules apply.</p>
</div>
<hr />

<div id="abstract">
<h2>Abstract</h2>

<p>Every day, governments and government agencies publish more data on the
Internet. Sharing this data enables greater transparency; delivers more
efficient public services; and encourages greater public and commercial use and
re-use of government information. Some governments have even created catalogs
or portals (such as <a href="http://www.data.gov">data.gov</a>) to make it easy
for the public to find and use this data.</p>

<p>Although the reasons may vary, the logistics and practicalities of opening
government data are the same. To help governments open and share their data,
the W3C eGov Interest Group has developed the following guidelines. These
straightforward steps emphasize standards and methodologies to encourage
publication of government data, allowing the public to use this data in new and
innovative ways.</p>
</div>

<div id="status">
<h2 class="no-toc no-num">Status of this Document</h2>

<p><em>This section describes the status of this document at the time of its
publication. Other documents may supersede this document. A list of current W3C
publications and the latest revision of this technical report can be found in
the <a href="http://www.w3.org/TR/">W3C technical reports index</a> at
http://www.w3.org/TR/.</em></p>

<p>The <a href="http://www.w3.org/2007/eGov/IG/">W3C eGovernment Interest
Group</a>is pleased to offer this First Public Working Draft which briefly
explains how to publish government data on the Web, based on the our gathered
experience. We encourage anyone with additional experience to join the group or
send written comment to help with the next version.</p>

<p>The <a href="http://www.w3.org/2007/eGov/IG/">W3C eGovernment Interest
Group</a> views this as an evolving document and seeks public feedback on this
Working Draft. Please send your comments to <a
href="mailto:public-egov-ig@w3.org">public-egov-ig@w3.org</a> (<a
href="http://lists.w3.org/Archives/Public/public-egov-ig/">archives</a>). If
possible, please offer specific changes to the text that would address your
concern. You may also wish to check the <a
href="http://www.w3.org/2007/eGov/IG/wiki/Data-gov-memo">Wiki Version</a> of
this document and see if the relevant text has already been updated.</p>

<p>Publication as a Working Draft does not imply endorsement by the W3C
Membership. This is a draft document and may be updated, replaced or obsoleted
by other documents at any time. It is inappropriate to cite this document as
other than work in progress.</p>

<p>This document was produced by a group operating under the <a
href="http://www.w3.org/Consortium/Patent-Policy-20040205/">5 February 2004 W3C
Patent Policy</a>. The group does not expect this document to become a W3C
Recommendation. W3C maintains a <a rel="disclosure"
href="http://www.w3.org/2004/01/pp-impl/42481/status">public list of any patent
disclosures</a> made in connection with the deliverables of the group; that
page also includes instructions for disclosing a patent. An individual who has
actual knowledge of a patent which the individual believes contains <a
href="http://www.w3.org/Consortium/Patent-Policy-20040205/#def-essential">Essential
Claim(s)</a> must disclose the information in accordance with <a
href="http://www.w3.org/Consortium/Patent-Policy-20040205/#sec-Disclosure">section
6 of the W3C Patent Policy</a>.</p>
</div>
<hr />

<div id="toc">
<h2>Table of Contents</h2>
<ul>
  <li><a href="#steps">Straightforward Steps to Publish Government Data</a></li>
  <li><a href="#concepts">Open Data Gives Real Power to the People</a>
    <ul>
      <li><a href="#concepts.identify">Identify</a></li>
      <li><a href="#concepts.document">Document</a></li>
      <li><a href="#concepts.link">Link</a></li>
      <li><a href="#concepts.preserve">Preserve</a></li>
      <li><a href="#concepts.expose">Expose Interfaces</a></li>
      <li><a href="#concepts.coverage">Create Standard Names/URIs for All
        Government Objects</a></li>
    </ul>
  </li>
  <li><a href="#what">Choosing What Data to Publish</a></li>
  <li><a href="#formats">Choosing the Right Format for the Data</a></li>
  <li><a href="#rights">Restrictions on the Use of the Data</a></li>
  <li><a href="#help">Learn More</a></li>
  <li><a href="#references">References</a></li>
</ul>
</div>

<div class="section" id="steps">
<h2>Straightforward Steps to Publish Government Data</h2>

<div class="step" id="step.1">
<p><strong class="leader">Step 1: </strong>The quickest and easiest way to make
data available on the Internet is to publish the data in its raw form (e.g., an
XML file of polling data from past elections). However, the data should be
well-structured. Structure allows others to successfully make automated use of
the data. Well-known formats or structures include XML, RDF and CSV. Formats
that only allow the data to be seen, rather than extracted (for example,
pictures of the data), are not useful and should be avoided.</p>
</div>

<div class="step" id="step.2">
<p><strong class="leader">Step 2: </strong>Create an online catalog of the raw
data (complete with documentation) so people can discover what has been
posted.</p>

<p>These raw datasets should be reliably structured and documented, otherwise
their usefulness is negligible.  Most governments already have mechanisms in
place to create and store data (e.g., Excel, Word, and other software-specific
file formats).</p>

<p>Posting raw data, with an online catalog, is a great starting point, and
reflects the next-step evolution of the Internet - "website as fileserver".</p>
</div>

<div class="step" id="step.3">
<p class="lead-in"><strong class="leader">Step 3: </strong>Make the data both
human- and machine-readable:</p>
<ul>
  <li>enrich your existing (X)HTML resources with semantics, metadata, and
    identifiers;</li>
  <li>encode the data using open and industry standards - especially XML - or
    create your own standards based on your vocabulary;</li>
  <li>make your data human-readable by either converting to (X)HTML, or by
    using real-time transformations through CSS or XSLT.  Remember to follow
    accessibility requirements;</li>
  <li>use permanent patterned and/or discoverable "<a
    href="http://www.w3.org/TR/cooluris/">Cool URIs</a>";</li>
  <li>allow for electronic citations in the form of standardized (anchor/id
    links or XLINKs/XPointers) hyperlinks.</li>
</ul>

<p>These steps will help the public to easily find, use, cite and understand
the data. The data catalog should explain any rules or regulations that must be
followed in the use of the dataset. Also, the data catalog itself is considered
"data" and should be published as structured data, so that third parties can
extract data about the datasets. Thoroughly document the parts of the web
page, using valid XHTML, and choose easily patterned and discoverable URLs for
the pages. Also syndicate the data for the catalog (using formats such as RSS)
to quickly and easily advertise new datasets upon publication.</p>

<p><img alt="How to Publich Government Data, both for people and machines"
src="HowToPublish.jpg"
style="display: block; text-align: center; margin-left: auto; margin-right: auto"
width="800" height="379" /></p>
</div>
</div>

<div class="section" id="concepts">
<h2>Open Data Gives Real Power to the People</h2>

<div id="concepts.identify">
<h3>Identify</h3>

<p>The ability to identify things by a URI/URL is the foundation of the
Internet. Best practices for "open government data" depend on applying the
architecture described in <a href="http://www.w3.org/TR/webarch/">Architecture
of the World Wide Web, Volume 1</a> [<cite><a
href="#WEBARCH">WEBARCH</a></cite>]. If you give permanent, patterned and/or
discoverable URI/URLs to your data, processes and people will be able to find
and use it more easily. URI/URLs can be used in databases and metadata as
universal, unique identifiers (e.g. by appending a serial number or other
internal naming system to a domain: <span
class="uri example">http://www.example.gov/objects/optional-hierarchy/serial12345678.html</span>).</p>

<p>Use internal IDs to identify specific data for machine re-use (e.g., <a
href="http://www.census.gov/main/www/popclock.html">http://www.census.gov/main/www/popclock.html</a>). By
using URIs, and incorporating internal links inside large files or documents,
people can authoritatively cite the information on their own websites.</p>
</div>

<div id="concepts.document">
<h3>Document</h3>

<p>Without documentation, data is not very useful. When possible, you should
use industry standards such as those based on XML/RDF, as they tend to be
self-documenting. Create a web page with a plain language description of the
dataset to help search engines find the data, so people can use it. The
description should explain how to find the pages and/or files, and describe the
contents of the files. At a minimum, document the title, description,
publication date and authoritative source for the data. Use clear language, and
link to related information to help put the data into context. When possible,
also fully document the description of each datum and the data standard and/or
data validation (e.g., if datum is a temperature, also explain the related
standard, such as Celsius, Fahrenheit or Kelvin). As standards emerge for
describing the datasets, take advantage of them. Document search tools and
ReSTful methods of obtaining the data when applicable.</p>
</div>

<div id="concepts.link">
<h3>Link</h3>

<p>The next step beyond raw data is fully linked data. In other words, raw data
can only be linked to, there are no links back out - but linked data contains
links out to other data and documentation. There is a spectrum from raw to
fully linked data, and standards such as RDF that can help you link your
data.</p>

<p class="lead-in">Here are four expectations for the linked data web:</p>
<ol>
  <li>URIs are names for things;</li>
  <li>HTTP URIs help people find those things;</li>
  <li>When someone looks up a URI, they should find useful information; and</li>
  <li>Links within your data to other URIs help people discover more, related
    things.</li>
</ol>

<p>Using URIs as described provides for a rich and easily connected web of data
that is easier for the public to search and use.  However, many databases do
not build URIs, but instead use internally derived unique identifiers. To link
this data into the larger web, we need to use systems that can anticipate or
build the URIs prior to publishing the data.</p>
</div>

<div id="concepts.preserve">
<h3>Preserve</h3>

<p>Preserving the data we publish is a top concern for governments. How can we
ensure that data is findable and can be referenced for as long as people need
it?  People may continue to cite older data at the original URI decades after
it was initially published.</p>

<p>To maintain the history of a dataset, think about how to integrate new URIs
for upgraded and new datasets, and structure your URIs accordingly.  Use
versioning of datasets so people can cite and link to present and past
versions. New and upgraded datasets can refer back to original datasets.  If
needed, provide tools to translate older data.  Carefully document the changes
between versions, and embed the version number/indicator within the data if
possible. Consider the file format carefully, and opt for open standards that
are not software specific.</p>

<p class="seealso">See the <a href="#references">References</a> section for
more information on standards and best practices for data preservation.</p>
</div>

<div id="concepts.expose">
<h3>Expose Interfaces</h3>

<p>If you really want to help people discover and explore the data you are
publishing, there are some useful W3C standards which can help.  It is
possible to make data human-readable by using XSLT for XML- and RDF-based
formats. Make web pages machine-readable by embedding semantic information in
the HTML, using RDFa and/or Microformats. (Note that search engines can take
advantage of embedded semantic information, helping more people to find your
data.) If it is too difficult to append XSL or CSS styles to old data, it is
still possible to document how to transform the data with a specific or generic
XSLT engines (for example, see <a
href="http://www.xmldatasets.net">http://www.xmldatasets.net</a>).</p>

<p>External parties can create new and exciting interfaces that may not be
obvious to the data publishers. For that reason, do not compromise the
integrity of the data to create flashy interfaces. If you must create an
interface, then publish the data separate from the interface, and ensure
external parties have direct access to the raw data, so they can build their
own interfaces if they wish.</p>

<p>Additionally, since all web documents using (X)HTML, XML and RDF can be used
as an object database or ReSTful API, the public can create software, Web
applications and mash-ups that use the datasets, and link back to the
authoritative version on the government web site. Two W3C standard languages
that make that possible are XQuery and SPARQL.  Once your data is published,
governments themselves can also create a user-friendly, transparent interface
that allows users to play with and refine the information in which they are
interested.</p>
</div>

<div id="concepts.coverage">
<h3>Create Standard Names/URIs for All Government Objects</h3>

<p>Having a unique identifier for every registered company or school (object)
is as important as having information about those companies or schools. In
addition to using open industry standards, an effort should be made to catalog
all reference-able government entities, officers and objects.  The URIs can
then be used within all of the data. This aids in discoverability, improves
metadata, and ensures authenticity.</p>
</div>
</div>

<div class="section" id="what">
<h2>Choosing What Data to Publish</h2>

<p>All data that can be shared with the public should be opened for public
dissemination. Data should be published in compliance with applicable laws and
regulations, and only after addressing issues of security and privacy.</p>

<p>First, publish all data that is already available in other formats (such as
print), or already being collected and shared with the public in other ways.
Publish both highly structured data and text documents. Publish laws,
regulations and codes, and public documents related to government agency
mission, vision, values and goals.</p>

<p>There are many standards for such documents (such as AIIM's emerging
Strategy Markup Language (StratML), XBRL and others). Use of standards enables
stakeholders to easily discover and provide feedback on items of interest to
them. Creating and publishing new documents in various standard formats is
easier than fixing after the fact.</p>

<p>As you create your documentation for the data, include and reference XML
Schemas, DTDs and taxonomies. Explain how to discover all of the
files/documents in a collection. Use syndication standards and tools to
announce additions or changes. Provide complete documentation to enable
automatically-generated data dictionaries and provide query services, which
will make it easier for the public to find your documents and datasets.</p>
</div>

<div class="section" id="formats">
<h2>Choosing the Right Format for the Data</h2>

<p>There are many different data formats, but which will work best with your
data?  The primary format for human-readable data is (X)HTML.</p>

<p>Raw data is more likely to be produced using formats customized to the
specific data, the tools used, or industry standards. The W3C has pioneered
XML and RDF, which allow for excellent manipulation and standardized tool sets.
RDF and XML files can be accessed like databases, using SPARQL, XQuery,
JavaScript and many other computer languages. When possible, use established
open standards, and tools that allow easy and efficient production and
publishing of the data. See the <a href="#references">References</a> section
for a list of current tools. Also keep in mind the power of linked data.</p>
</div>

<div class="section" id="rights">
<h2>Restrictions on the Use of the Data</h2>

<p>Once you've made your data available, be sure to document clearly any legal
or regulatory restrictions on the use of the data. In many cases, there are
defined standards to insert copyright and/or licensing information right into
the data (see metadata standards, such as Dublin Core). It is very important to
be clear regarding the expected uses, credits, limitations and responsibilities
that particular parties have in regard to working with and providing the
data.</p>
</div>

<div class="section" id="help">
<h2>Learn More</h2>

<p>We still have much to learn about the best ways for governments to open up
their data. The W3C eGov Interest Group plans to publish relevant use cases to
illustrate working examples of current thinking and best practices. The W3C
technologies and approaches described in this document can be implemented
quickly, and often with relatively little cost. If you want to learn more,
there are a number of communities that can help. If you are a government
agency, think about joining the W3C and getting involved in e-Government
activities. You can connect with people from other governments who are facing
similar issues and sharing their experiences. Even if you don't join the W3C,
please let us know how you are getting on. We can help.</p>
</div>

<div class="section" id="references">
<h2>References</h2>
<dl>
  <dt id="EGOV-IMPROVING">[EGOV-IMPROVING]</dt>
    <dd><a href="http://www.w3.org/TR/egov-improving/"><cite>Improving Access
      to Government through Better Use of the Web</cite></a>, S. Accar, J.
      Alonso, K. Novak, Editors, W3C Group Note, 12 May 2009.</dd>
  <dt id="EU-PSID">[EU-PSID]</dt>
    <dd><a type="application/pdf"
      href="http://ec.europa.eu/information_society/policy/psi/docs/pdfs/directive/psi_directive_en.pdf"><cite>Directive
      2003/98/EC on the Re-Use of Public Sector Information</cite></a>,
      European Parliament and Council, 17 November 2003.</dd>
  <dt id="OGD-CIVIC">[OGD-CIVIC]</dt>
    <dd><a
      href="http://razor.occams.info/pubdocs/opendataciviccapital.html"><cite>Open
      Data is Civic Capital: Best Practices for "Open Government
      Data"</cite></a>, J. Tauberer, 20 July 2009</dd>
  <dt id="TBL-GOV">[TBL-GOV]</dt>
    <dd><a href="http://www.w3.org/DesignIssues/GovData"><cite>Design Issues:
      Putting Government Data Online</cite></a>, T. Berners-Lee.</dd>
  <dt id="TBL-LD">[TBL-LD]</dt>
    <dd><a href="http://www.w3.org/DesignIssues/LinkedData"><cite>Design
      Issues: Linked Data</cite></a>, T. Berners-Lee.</dd>
  <dt id="TUT-LD">[TUT-LD]</dt>
    <dd><a
      href="http://sites.wiwiss.fu-berlin.de/suhl/bizer/pub/LinkedDataTutorial/"><cite>How
      to Publish Linked Data on the Web</cite></a>, C. Bizer, R. Cyganiak, T.
      Heath, 27 July 2007.</dd>
  <dt id="US-OBMEMO">[US-OBMEMO]</dt>
    <dd><a
      href="http://www.whitehouse.gov/the_press_office/TransparencyandOpenGovernment/"><cite>Memorandum
      for the Heads of Executive Departments and Agencies on Transparency and
      Open Government</cite></a>, B. Obama.</dd>
  <dt id="US-GSATRAN">[US-GSATRAN]</dt>
    <dd><a type="application/pdf"
      href="http://www.usaservices.gov/events_news/documents/Transparency_000.pdf"><cite>Intergovernmental
      Solutions Newsletter: Transparency and Open Government</cite></a>, GSA
      Office of Citizen Services and Communications, Spring 2009 Issue.</dd>
  <dt id="WEBARCH">[WEBARCH]</dt>
    <dd><a href="http://www.w3.org/TR/webarch/"><cite>Architecture of the World
      Wide Web, Volume One</cite></a>, I. Jacobs, N. Walsh, Editors, W3C
      Recommendation, 15 December 2005.</dd>
  <dt id="WEBSELF">[WEBSELF]</dt>
    <dd><a
      href="http://www.w3.org/2001/tag/doc/selfDescribingDocuments"><cite>The
      Self-Describing Web</cite></a>, N. Walsh, Editor, W3C TAG Finding, 7
      February 2009.</dd>
</dl>
</div>
</body>
</html>