certif 23.2 KB

Raw Blame History Permalink

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
      "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
  <title>W3C - QA - W3C Certification Activity Proposal DRAFT</title>
  <meta name="Keywords" content="qa, quality assurance, conformance">
  <meta name="Description" content="Briefing Package for the QA Activity.">
  <link rel="stylesheet" type="text/css"
  href="http://www.w3.org/StyleSheets/base.css" />
  <link rel="stylesheet" type="text/css"
  href="http://www.w3.org/2003/03/site-css/css/1.css" />
  <style type="text/css">
h1 { clear: left; background: white; color: black }</style>
</head>

<body>
<p class="banner"><a href="http://www.w3.org/"><img alt="W3C" width="72"
height="48" src="http://www.w3.org/Icons/w3c_home"></a> <a
href="http://www.w3.org/QA/"><img alt="QA" src="/QA/images/qa"></a></p>

<h1>Study of a W3C Certification Activity.</h1>

<p></p>

<p>Editor: <strong>Daniel Dardailler</strong>, W3C.</p>

<p>Date of original release: April 2003.</p>

<p><em>Note: as of June 2007: W3C hasn't started a Certification
activity.</em></p>

<h2>I. Executive Summary</h2>

<p>This document is a study of the whereabouts, pros and cons of a
<strong>W3C Certification activity</strong>. No decision has yet been made
with regard to starting such an activity. This document still provides a
basis for discussing options.</p>

<p><em>Note that even though the term activity is used in this document, it
doesn't really carry the same formalism attached to the word Activity in the
<a href="../06/Process-20030618/process.html#Activities">W3C process
document</a>.</em></p>

<p>This new activity would be closely integrated with the existing <a
href="/QA/">W3C QA activity</a> and aims to create and manage a certification
program for potentially: Web content, Web applications, and Web
developers.</p>

<p>The <strong>basic premises</strong> behind this program is that:</p>
<ul>
  <li>W3C <strong>Working Groups</strong> are responsible for releasing
    quality test materials and validator for their specs/formats</li>
  <li>passing a W3C test constitutes an <strong>objective assessment of a
    claim</strong> (i.e. reproduceable test results) but doesn't necessarily
    mean attaining a level of conformance to a spec, only that it passed the
    given test.</li>
  <li><strong>who runs the tests</strong>: the being-certified party (i.e.
    self-certification), a third-party (acting on W3C's behalf) or W3C
    itself, is an important parameter of the system which has to be decided
    early on.</li>
</ul>

<p>The benefit foreseen by certification is that this will create a market
need for <strong>better recognition of quality</strong> of Web resources,
quality sought by the user community, and that it will therefore improve the
quality of the Web and the demand of good products by users.</p>

<p>The risks on the other hand are that we spend precious resources on such
an initiative, for no good result, and worse, that we end up alienating part
of the industry or the Web community for wanting to play what could be seen
as a new <strong>police or a commercial role</strong> for the W3C.</p>

<h2>2. Context</h2>

<h3>Definitions</h3>

<p>A <strong>certification</strong> is an <em>acknowledgement</em> by a
certifying authority (the organism issuing a certificate or branding) that a
set of criteria has been met.</p>

<p>It is different from conformance, which defines an <em>ability</em> to
meet a set of requirements defined by a specification.</p>

<p>Certification in itself makes no claim about a given specification, and
its relationship to a specification is <strong>transitive</strong> in nature
(e.g. a product is certified to pass a set of SVG tests, which themselves
defines - or not - a level of conformance to the SVG specs).</p>

<p>It is an expression of <strong>trust</strong> and therefore a form of
liability, while a conformance claim is more of a technical assertion.</p>

<p>Whether or not the tests are run by the party seeking certification, by
W3C, or by a third party acting on its behalf, the <strong>novel</strong>
idea behind this program is that W3C would <strong>engage</strong> its name
and brand for things that it hasn't directly produced (e.g. software products
on the market, Web development agencies).</p>

<p>It is therefore very important that a good control over this new W3C
liability be recognized and properly organized within the W3C and the QA
activity, involving the membership and the Web community. Given that the cost
of the branding program itself, i.e. the MarComm around it, wil probably be
very resource intensive, it is also very important not to underestimate them
in any future planning.</p>

<p></p>

<h3>Current status</h3>

<p>The <a href="/QA">Quality Assurance (QA) Activity at W3C</a> has a dual
focus: to solidify and extend current quality practices, and to educate by
sharing our understanding of coordination, certification, funding, and
tracking of the quality of products and services related to W3C technologies.
The mission of the QA Team is to <strong>improve the quality of W3C
specification implementation in the field</strong>.</p>

<p>The QA activity is mainly focused toward the <em>inreach</em>, that is,
the quality of the W3C own working groups processes and deliverables (e.g.
specification).</p>

<p>Certification, on the other hand, is partly about <strong>judging</strong>
the quality of other people's work.</p>

<p>The closest to something resembling today that at W3C is the <a
href="/Consortium/Process-20010719/process.html#Submission">submission</a>
process whereby W3C express its opinion on the technical work of its members
doen outside W3C. This is a limited comparison as it concerns only
specifications (not products or services) and W3C, when accepting a
submission, only asserts that the work is <strong>good enough</strong> for
becoming a W3C note.</p>

<p>But this constitutes nevertheless some kind of approval stamp we're giving
to external work that we may want to draw experience from (in terms of
liability and public reaction, press coverage, etc).</p>

<p></p>

<p>The <a href="/QA/Activity">QA activity proposal</a> mentions certification
as a <strong>topic</strong> of the QA Interest Group but it is specifically
marked as <strong>currently</strong><strong>out-of-scope</strong> in the QA
<a href="/QA/IG/charter.html#Scope">IG charter</a> and as something that
needs more thinking and shared understanding.</p>

<p>So if we move forward with that, we do have to <strong>ask our
membership</strong>, since it's not in any existing activity scope.</p>

<p>However, in recent W3C meetings (TP March 2003) of the QA chairs, staff,
but also the AB, and some WGs, it appeared that a successful certification
program could have the following <strong>benefits</strong>:</p>
<ul>
  <li>create a market need for quality recognition</li>
  <li>improve the validity of software implementing our specs in a scalable
    way</li>
  <li>improve the awareness for the importance of Web standard</li>
  <li>make more widely known the W3C brand.</li>
  <li>potentially attract new W3C members</li>
</ul>

<p>It was also recognized that there are associated <strong>risks</strong>
and difficulties attached to such a new endeavor:</p>
<ul>
  <li>It will be effort demanding for several already overloaded W3C
    constituencies: Management, Comm, QA.</li>
  <li>if started too early with no good QA ts materials to back it up, if may
    be detrimental to the W3C brand instead of reinforcing it.</li>
  <li>If noone is interested in being W3C certified, it will be a complete
    loss of resources, if it is financially successful, it may change the W3C
    mission and role in the face of profit gains..</li>
</ul>

<p></p>

<p>But the potentially high payoff calls for exploring this further and
propose a plan to the W3C members at some point, once we have reached
internal consensus on such a plan.</p>

<h2>3. Scope</h2>

<h3>Taxonomy of certification</h3>

<p>We need to study both the different classes of <strong>objects</strong>
being certified and the different <strong>methods</strong> for certifying
them.</p>

<p><span style="text-decoration: underline">On the objects of
certification.</span></p>

<p>The QA activity has released a <a href="/QA/Taxonomy.html">taxonomy of
test materials</a> which presents various categories of test materials.</p>

<p>Using that list as a starting point, we see three kinds of potential
certification activities:</p>
<ul>
  <li>certification of Web content (e.g. site validity, accessibility)</li>
  <li>certification of Software programs (e.g. browsers, authoring tools)</li>
  <li>certification of Service providers (e.g. Web design shops)
    <p></p>
  </li>
</ul>

<p>Note: certification of documentation, like HTML books. could fall into
services, if we consider that the author/editor of the doc is being
certified. Or it could fall onto products, e.g. a book is a product. Or we
could have products splitted into programs (products above) and docs.</p>

<p><span style="text-decoration: underline">On the certification
methods.</span></p>

<p>At some point, someone <strong>has to run</strong> a series of tests over
something (the objects above).</p>

<p>At some other point, the <strong>W3C name</strong> is attached to this
something as a guarantee that the tests were successful.</p>

<p>Between those two moments, W3C has to make sure its brand is not misused
or abused.</p>

<p>In decreasing order of "assurance" and "cost for W3C", the tests could be
run:</p>
<ul>
  <li>by the W3C staff (high guarantee, high cost)</li>
  <li>by a third party specialized in running tests (good guarantee, high
    management)</li>
  <li>by the party seeking certification (self-certification)</li>
</ul>

<p></p>

<p>For <strong>each</strong> of these categories of objects and methods, we
need to examine what a particular certification program would mean (what are
the nature of the tests being saught and the exact objects of the
certification, e.g. SVG or WCAG) and could look like (difficulties, existing
test framework, etc).</p>

<p>See the <a href="#L780">decision matrix</a> below.</p>

<p>The relationship between certification and testing is of course very
strong, as the first cannot usually be obtained without the availability of
the second, so it is important to know the <strong>status
of</strong><strong>test development at W3C</strong> and availibility in each
category.</p>

<p></p>

<h3>Available tests</h3>

<p>One of the main goals of the QA activity is to have each W3C WG develops
and being <strong>responsible</strong> for test materials that support their
technology.</p>

<p>In order to achieve that goal, the W3C QA activity is developing a <a
href="/TR/qaframe-intro/">framework</a> of operational, specification and
tests development guidelines for promoting and facilitating these WG quality
practices.</p>

<p>The <a href="/QA/TheMatrix">QA Matrix</a> lists the current achievement of
WGs in terms of availibility of validation tools and test suites in their
area of work.</p>

<p>It is a fair statement to say that we're <strong>far from</strong>
achieving this goal today (of having each WG producing tests of quality), and
many W3C groups do not have any test for checking either the content or the
products implementing their specification.</p>

<p>This in itself is an indication that whatever we start now will
<strong>not be comprehensive</strong> across W3C activities but will have to
work incrementally over time.</p>

<p>This also means that we may want to wait another year to get better test
materials available from the WG to embark into certification.</p>

<p>On the other hand, the QA Matrix shows some promising result that makes it
<strong>possible</strong> to start such a program given the availibility of
good validation and test suites programs in some W3C activities.</p>

<p>Let's now examine each potential certification activities in turn.</p>

<h2>4. Certification of Web content</h2>

<p>The tests used in that category are usually called
<strong>validators</strong> or <strong>evaluation</strong> programs.</p>

<p>The objects of the certification are Web pages or Web sites.</p>

<p>We already have several test tools for various W3C specifications, like
the HTML validator or the XML Schema validator.</p>

<p>W3C even provides <strong>logos</strong> for HTML, CSS and WAI WCAG
compliance, but this is not a certification approach as it is clearly
indicated that content providers are solely responsible for the use of our
logos (it can't even be called self-certification, but more
self-evaluation).</p>

<p>The real <strong>difficulty</strong> in that category lies not so much in
the development or the running of the tests but in the
<strong>amount</strong> of potential things to be tested: the Web itself!</p>

<p>If we start a certification program for Web content, one may wonder how
this can scale, or more to the point, how does the return on investment
compare to other forms of certification (e.g. authoring tools)?</p>

<p>Maybe the self-evaluation approach we've taken so far is the
<strong>best</strong> one we can hope to achieve, even it it is sometimes
abused (Web pages carry our logo without being valid or accessible at the
level they claim, often for versioning reasons).</p>

<p>A related case of Web content certification program is the notion of <a
href="/WAI/EO/Drafts/galleryintro.html">gallery</a> of Accessible Web Content
that the WAI EO group has been working on. This effort already shows the
range of difficulty associated with Web content related claims per se: need
for a human liaison responsible for the content, need for tracking the site
changes (from valid to not valid or the opposite), need for a selection
process, etc.</p>

<h2>5. Certification of Web applications</h2>

<p>The tests used in that category are usually called <strong>test
suites</strong>, TS for short.</p>

<p>The objects of the certification are Web applications like <strong>user
agents</strong> (e.g. browsers), <strong>authoring tools</strong> (e.g. HTML
editors), or <strong>web servers</strong>.</p>

<p>There is existing market of such certification already for platform
integration (e.g. <a
href="http://www.microsoft.com/windowsserver2003/partners/isvs/cfw.mspx">Certified
for Windows</a>) and one can imagine that a <em>W3C Certified for the Open
Web</em> (or some such attractive name) program would appeal to the product
vendors as well.</p>

<p>The issues there lie in the developement of <strong>objective</strong> and
performant test suites for, in particuliar, authoring tools, which cover one
of the most important aspect of the certification chain. Indeed, if the tools
that are used to generate Web content improves, Web Content itself improves
widely.</p>

<p>Some of our most important specification (e.g. HTML) are also difficult to
test as their semantics are difficult to define in the specs and are as a
result sometimes ill-defined (and therefore we do not have good test for
them).</p>

<p>Each sub-category of tools mentioned above has its own set of specific
difficulties, both technical and logistics, that each deserve a plan
description.</p>

<h2>6. Certification of Web developers and services</h2>

<p>The tests used in that category are usually called <strong>exams</strong>,
or <strong>training</strong> sessions, and are both technical and process
oriented. They are used to recognize a human or a human structure
qualification, not a technology like the other two above.</p>

<p>The objects of the certification are people (Web developers), or
organizations (e.g. Web design agencies).</p>

<p>We do not have test support for that today at W3C, but we have a some
expertise in judging the production made by Web designers wrt to applying our
standards, and we have some good base materials (slides, tutorials).</p>

<p>There is a large existing market of Certified Engineers for various
products and systems, and we could create a <em>Certified W3C Designers</em>
program that follow the same model used today for proprietary systems.</p>

<p>We would have to create several sets of <strong>educational</strong> and
<strong>checking</strong> materials and a framework for passing the tests
(e.g. exams) or/and certifying the quality of the internal process used in
relationg to Web design.</p>

<p>Another approach would be the development of a <em>Good Web Design
Practices</em> document that people would agree to comply with (process wise)
and that we (or a third party on our behalf) could audit on-demand, for a
fee, much like the ISO9000 way of doing thing.</p>

<p>This is potentially as area that can bring a lot of visibility to W3C and
the importance of standard development work.</p>

<h2><a name="L780" id="L780">7. Decision matrix</a></h2>

<p>Here's a matrix that gives some insight on the pros (+) and cons (-) of
each classes of objects and methods and for each couple, what we should pay
attention to.</p>

<table border="1">
  <caption></caption>
  <tbody>
    <tr>
      <td>
        <table border="0">
          <caption></caption>
          <tbody>
            <tr>
              <td></td>
              <td><em>Object of the tests</em></td>
            </tr>
            <tr>
              <td><em>Who runs the tests</em></td>
              <td></td>
            </tr>
          </tbody>
        </table>
      </td>
      <td><strong>Web content</strong>

        <p>+ that's the real thing!</p>

        <p>- too many pages to check</p>

        <p>+ can be automatized for things like markup validation</p>

        <p></p>
      </td>
      <td><strong>Software</strong>

        <p>+ good lever effect for Web content</p>

        <p>- still lots of programs</p>

        <p>- manual testing often required</p>
      </td>
      <td><strong>Providers</strong>

        <p>+ good level effect as well</p>

        <p>- lots of people!</p>

        <p>+ good promotion for W3C brand</p>
      </td>
    </tr>
    <tr>
      <td><strong>W3C staff</strong> (potentially with help from Web
        community)

        <p>+ high quality testing</p>

        <p>+ low external coordination costs</p>

        <p>- high running test costs</p>

        <p>+ no (re)checking since done by us</p>
      </td>
      <td>. interesting concept of gallery

        <p>. do something special for W3C members' pages for visibility?</p>

        <p>. enforce a bit more our markup logo licence, e.g. use automatic
        checking to correct abuse</p>
      </td>
      <td>. Some WGs (ATAG, SMIL, SVG, etc) have produced self-assess tables

        <p>- publication of results is a liability, even legally forbiden
        sometimes</p>
      </td>
      <td>. we could gather our curricula, tutorials, and have more
        developped.

        <p>. staff could do training for trainers (maximized level)</p>
      </td>
    </tr>
    <tr>
      <td><strong>Third party</strong>

        <p>+ more professional approach</p>

        <p>- require more trust and financial relationship</p>

        <p>- high startup cost but better long-term business</p>

        <p>+ limited checking</p>
      </td>
      <td>. watch EuroAccessibility program for WCAG certification, which is
        a kind of third party certif for content

        <p></p>
      </td>
      <td>. would need to be trained to our (various) test harness

        <p>. could do the (spot) checking in the case of
        self-certification</p>
      </td>
      <td>. could run courses and exams

        <p>. could check if production of certified providers is ok</p>
      </td>
    </tr>
    <tr>
      <td><strong>Companies</strong>

        <p>- limited trust</p>

        <p>+ checking necessary</p>

        <p>+ low cost of running test for W3C</p>
      </td>
      <td>. continue to promote self-evaluation with no formal checking (e.g.
        valid HTML logo)

        <p></p>

        <p></p>
      </td>
      <td>. need for easily checkable test result (e.g. EARL)

        <p>. need to handle versioning, multiple technologies (HTML, SVG,
        SMIL, etc)</p>
      </td>
      <td>. people could pledge allegeance to a W3C Good practices chart.</td>
    </tr>
  </tbody>
</table>

<p></p>

<p></p>

<h2>8. Structure of the activity</h2>

<p>If it is organized as a new W3C Activity, we could create an IG for
overall discussion and one WG per certification program: one for products,
one for services, etc.</p>

<p>The initial duration could be 2 years.</p>

<p>We could start with a workshop to gather more detailed input from the
certification industry, the Web industry, and the Web community in general
(e.g. Open Source).</p>

<p>As for deliverables, the NIST white paper on Certification gives some very
good input as what they should be, see <a
href="/QA/2002/01/Note-qa-certif-20020102.html#Products">http://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html#Products</a></p>

<h2>9. Resource Statement/Revenue Model</h2>

<p>At least one more person would be needed to run this thing on the W3C QA
staff side (difficult to do with just volunteers/Members - need a dedicated
W3C or some contracted resources).</p>

<p>Eventually, this may become self-financed as product vendors pay for
certification stamp.</p>

<h3>Who would be the customers for these kind of services ?</h3>

<p><em>@@ Include here a report from the Business Scenario Workshop on
Certification held in Washington DC on October 25th 2003 (OpenGroup
conference)</em></p>

<h2>10. Liaison and dependencies</h2>

<p>We already had discussions with some of these groups, and we need more.</p>

<p><a href="http://ts.nist.gov/ts/htdocs/210/ncsci/primer.htm">NIST</a>, <a
href="http://www.opengroup.org/certification/">TheOpenGroup</a>, <a
href="http://www.uc-council.org/">UCC</a>, <a
href="http://www.veritest.com/">Veritest</a> (running the MS Windows
certified program), our members doing QA/Certif, Open Source community, ETSI,
ISO9000, <a
href="http://public.ansi.org/ansionline/Documents/News%20and%20Publications/Links%20Within%20Stories/NCAP.pdf">ANSI</a>,
etc.</p>

<p>See this very good paper submitted by NIST to W3C in 2002: <a
href="http://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html">http://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html</a></p>

<p>And this document as well, on <a
href="http://www.consortiuminfo.org/cb/">certification, testing and branding
at ConsortiumInfo.org</a>.</p>

<p></p>
<hr>

<div class="disclaimer">
<a href="http://validator.w3.org/check/referer"><img
src="http://validator.w3.org/images/vxhtml10" alt="Valid XHTML 1.0!"
height="31" width="88"></a>

<p>Created: April 4th 2003 - by danield</p>

<p class="policyfooter"><a
href="http://www.w3.org/Consortium/Legal/ipr-notice.html">Copyright</a> ©
2003-2005 <a href="http://www.w3.org">W3C</a> (<a
href="http://www.lcs.mit.edu">MIT</a>, <a
href="http://www.ercim.org">ERCIM</a>, <a
href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a
href="http://www.w3.org/Consortium/Legal/ipr-notice.html">liability,</a> <a
href="http://www.w3.org/Consortium/Legal/ipr-notice.html">trademark</a>, <a
href="http://www.w3.org/Consortium/Legal/copyright-documents.html">document
use</a> and <a
href="http://www.w3.org/Consortium/Legal/copyright-software.html">software
licensing</a> rules apply. Your interactions with this site are in accordance
with our <a
href="http://www.w3.org/Consortium/Legal/privacy-statement.html">public</a>
and <a
href="http://www.w3.org/Consortium/Legal/privacy-statement.html">Member</a>
privacy statements.</p>
</div>
</body>
</html>