index.html 79.7 KB

Raw Blame History Permalink

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8" />
    <title>The QA Handbook</title>
    <link title="W3C QA Glossary" href="http://www.w3.org/QA/glossary" rel="glossary" />
    <link title="Latest version of the QA Framework Specification Guidelines" href="http://www.w3.org/TR/qaframe-spec/" rel="bookmark" />
    <link title="Latest version of the QA Handbook" href="http://www.w3.org/TR/qa-handbook/" rel="chapter" />
    <meta name="Keywords" content="qa, quality assurance, conformance, specification" />
    <meta name="Description" content="The QA Handbook is a non-normative handbook about the process and operational aspects of certain quality assurance practices of W3C's Working Groups, with particular focus on testability and test topics." />
    <link rel="schema.DC" href="http://dublincore.org/" />
    <meta name="DC.Subject" xml:lang="en" lang="en" content="qa, quality assurance, conformance, specification" />
    <meta name="DC.Title" xml:lang="en" lang="en" content="The QA Handbook" />
    <meta name="DC.Description.Abstract" xml:lang="en" lang="en" content="The QA Handbook is a non-normative handbook about the process and operational aspects of certain quality assurance practices of W3C's Working Groups, with particular focus on testability and test topics." />
    <meta name="DC.Language" scheme="RFC1766" content="en" />
    <meta name="DC.Publisher" content="W3C - World Wide Web Consortium - http://www.w3.org" />
    <meta name="DC.Rights" content="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright" />
    <link rel="stylesheet" type="text/css" href="style/guideline.css" />
    <link rel="stylesheet" type="text/css" href="http://www.w3.org/StyleSheets/TR/W3C-WG-NOTE.css" />
</head>
<body>
    <div class="head">
        <a shape="rect" href="http://www.w3.org/"><img height="48" width="72" alt="W3C" src="http://www.w3.org/Icons/w3c_home" /></a>
        <h1><a shape="rect" id="spec-title" name="spec-title">The QA Handbook</a></h1>
        <h2><a shape="rect" id="dated-subtitle" name="dated-subtitle">W3C Working Group Note 6 September 2005</a></h2>
        <dl>
            <dt>This version:</dt>
            <dd><a shape="rect" href="http://www.w3.org/TR/2005/NOTE-qa-handbook-20050906/">http://www.w3.org/TR/2005/NOTE-qa-handbook-20050906/</a></dd>
            <dt>Latest version:</dt>
            <dd><a shape="rect" href="http://www.w3.org/TR/qa-handbook/">http://www.w3.org/TR/qa-handbook/</a></dd>
            <dt>Previous version:</dt>
            <dd><a shape="rect" href="http://www.w3.org/TR/2004/NOTE-qa-handbook-20041122/">http://www.w3.org/TR/2004/NOTE-qa-handbook-20041122/</a></dd>
            <dt>Editor:</dt>
            <dd>Lofton Henderson</dd>
            <dt>Contributors:</dt>
            <dd><a shape="rect" href="#acknowledgments">See Acknowledgments.</a></dd>
        </dl>
        <p class="copyright"><a href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a> © 2004-2005 <a href="http://www.w3.org/"><acronym title="World Wide Web Consortium">W3C</acronym></a><sup>®</sup> (<a href="http://www.csail.mit.edu/"><acronym title="Massachusetts Institute of Technology">MIT</acronym></a>, <a href="http://www.ercim.org/"><acronym title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>, <a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>, <a href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a> and <a href="http://www.w3.org/Consortium/Legal/copyright-documents">document use</a> rules apply.</p>
        <hr class="rule" />
    </div>
    <h2><a shape="rect" id="abstract" name="abstract">Abstract</a></h2>
    <p><cite>The QA Handbook</cite> (<acronym title="The QA Handbook">QAH</acronym>) is a non-normative handbook about the process and operational aspects of certain quality assurance practices of W3C's Working Groups, with particular focus on testability and test topics. It is intended for Working Group chairs and team contacts. It aims to help them to avoid known pitfalls and benefit from experiences gathered from the W3C Working Groups themselves. It provides techniques, tools, and templates that should facilitate and accelerate their work. This document is one of the QA Framework (QAF) family of documents of the <a shape="rect" href="http://www.w3.org/QA/">Quality Assurance (<acronym>QA</acronym>) Activity</a>. QAF includes the other in-progress specification, <cite>Specification Guidelines</cite>, <cite>Test Development FAQ</cite>, plus a handful of test- and other QA-related notes, advanced topics, and Wiki page collections.</p>
    <h2><a shape="rect" id="status" name="status">Status of this document</a></h2>
    <p><em>This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the <a href="http://www.w3.org/TR/">W3C technical reports index</a> at http://www.w3.org/TR/.</em></p>
    <p>This document is a W3C Working Group Note. It has been produced by the <a href="http://www.w3.org/QA/WG/">Quality Assurance Working Group</a>, which is part of the Quality Assurance Activity. It has been made available for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the <a href="http://www.w3.org/QA/Activity">QA Activity statement</a>. <a href="http://www.w3.org/2003/03/Translations/byTechnology?technology=qa-handbook">Translations</a> of this document may be available.</p>
    <p>This is the fourth publication of this document, and the second as a Working Group Note. This version harmonizes with the final document set of the <a href="http://www.w3.org/QA/WG/qaframe-primer">QA Framework</a>, including the new <cite>Test Development FAQ</cite>, and reflects some basic editorial cleanup.</p>
    <p>Publication as a Working Group Note does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.</p>
    <p>The QA Working Group does not expect this document to become a Recommendation. Rather, after further development, review and refinement, it may be updated and maintained as a Interest Group Note.</p>
    <p>You may email comments on this document to <a href="mailto:www-qa@w3.org">www-qa@w3.org</a>, the <a href="http://lists.w3.org/Archives/Public/www-qa/">publicly archived</a> list of the QA Interest Group.</p>
    <h2><a id="contents1" name="contents1">Table of contents</a></h2>
    <ol>
        <li>
            <a href="#introduction">Introduction &amp; roadmap</a>
            <ul>
                <li>1.1. <a href="#Five-stories">Five simple stories</a></li>
                <li>1.2. <a href="#why-qah">Why QAH?</a></li>
                <li>1.3. <a href="#Audience">Audience and purpose</a></li>
                <li>1.4. <a href="#scope">QAH scope</a></li>
                <li>1.5. <a href="#Other">Other QA Framework resources</a></li>
                <li>1.6. <a href="#Roadmap">Roadmap to the rest of QAH</a></li>
            </ul>
        </li>
        <li><a href="#Early">Early planning and commitment</a></li>
        <li>
            <a href="#Day-to-day">Day-to-day QA operations</a>
            <ul>
                <li>3.1. <a href="#Logistics">General modus operandi</a></li>
                <li>3.2. <a href="#Test">Test development framework and processes</a></li>
                <li>3.3. <a href="#Life">Life after Working Group &mdash; maintenance</a></li>
            </ul>
        </li>
        <li><a href="#IANAL">Licensing &amp; branding</a></li>
        <li><a href="#Acquire">Acquiring test materials</a></li>
        <li><a href="#acknowledgments">Acknowledgments</a></li>
        <li><a href="#References">References</a></li>
        <li><a href="#History">Change History</a></li>
    </ol>
    <p>List of Guidelines and Good Practices:</p>
    <ul>
        <li>
            <a href="#pr-early-planning">Guideline 1: Plan &amp; commit early.</a>
            <ul>
                <li><a href="#gp-build-acquire">Good Practice 1: Decide as soon as possible &mdash; will the Working Group build test materials or acquire them?</a></li>
                <li><a href="#gp-plan-deliverables">Good Practice 2: Think about and enumerate the quality-related deliverables that might help the Working Group through the Recommendation track.</a></li>
                <li><a href="#gp-sync-deliverables">Good Practice 3: Synchronize quality-related deliverables and their development milestones with specification milestones.</a></li>
                <li><a href="#gp-advancement-criteria">Good Practice 4: Consider whether the Working Group should bind any quality criteria to Rec-track advancement.</a></li>
                <li><a href="#gp-plan-staffing">Good Practice 5: Put some thought into how to staff the Working Group's test and other quality assurance plans.</a></li>
            </ul>
        </li>
        <li>
            <a href="#pr-collect-qap-info">Guideline 2: Document QA processes.</a>
            <ul>
                <li><a href="#gp-make-qapd">Good Practice 6: Put all of the Working Group's important test and other quality-related information in one place in a QA Process Document.</a></li>
                <li><a href="#gp-point-of-contact">Good Practice 7: Identify a Working Group point-of-contact for test materials or other quality-related business.</a></li>
                <li><a href="#gp-email-list">Good Practice 8: Specify an archived email list to use for quality-related communications</a>.</li>
                <li><a href="#gp-web-page">Good Practice 9: Identify Web page(s) for test suites, announcements, and other quality-related topics</a>.</li>
            </ul>
        </li>
        <li>
            <a href="#pr-ipr-agreement">Guideline 3: Resolve legal &amp; license issues.</a>
            <ul>
                <li><a href="#gp-submission-license">Good Practice 10: As early as possible, get agreement about acceptable license terms for submission of test materials.</a></li>
                <li><a href="#gp-publication-license">Good Practice 11: As soon as the nature of the Working Group's test materials becomes clear, get agreement about license terms for their publication.</a></li>
                <li><a href="#gp-branding-policy">Good Practice 12: Decide policy about having brands, logos, or conformance icons associated with the Working Group's test materials.</a></li>
            </ul>
        </li>
        <li>
            <a href="#pr-consider-acquire">Guideline 4: Consider acquiring test materials.</a>
            <ul>
                <li><a href="#gp-quality-assess">Good Practice 13: Do a quality assessment of proposed test materials before going any further.</a></li>
            </ul>
        </li>
    </ul>
    <p>This document has two separate external appendices, <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-root.html">QA Process Document template</a> and <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-charter.html">Charter template</a>, which contain templates to help Working Group Chairs and Staff Contacts to implement the document's good practice guidelines.</p>
    <hr class="rule" />
    <h2>1. <a shape="rect" id="introduction" name="introduction">Introduction &amp; roadmap</a></h2>
    <h3>1.1. <a shape="rect" name="Five-stories" id="Five-stories">Five simple stories</a></h3>
    <p>The following five use cases for <cite>The QA Handbook</cite> (<acronym title="The QA Handbook">QAH</acronym>), told as stories, show when and how <acronym title="The QA Handbook">QAH</acronym> could be helpful. They cover five different situations that might typically arise over the life of the Working Group.</p>
    <p><cite>The QA Handbook</cite> could be useful to Chairs and Staff Contacts at Charter time...</p>
    <div class="story">
        <p><span class="story-label">Story 1: Think about quality early</span><br />
        The Charter-drafting team of a new Working Group, led by Staff Contacts and co-Chairs, looks through <cite>The QA Handbook</cite>. Following the good practices in the <a shape="rect" href="#Early">early-commitment and planning</a> module, the team debates how much it can realistically commit to at this early stage. After deciding on some test materials deliverables, milestones, and specification synchronization that it considers to be safe commitments, the team uses the provided Charter Template to include these in the draft Charter.</p>
    </div>
    <p>Or it could help a Chair who is trying to jump-start a test suite project...</p>
    <div class="story">
        <p><span class="story-label">Story 2: Jump-starting a testing effort</span><br />
        An existing Working Group has just finished writing its Requirements and Use Cases documents, and has begun to draft its specification. At the same time, it is taking a first look at its test suite plans. As recommended in the <cite>The QA Handbook</cite>, the Chair jump starts the Test Suite project by <a shape="rect" href="#Logistics">appointing a point of contact and part-time Test Suite team</a>, whose first output is a quick QA Process Document (QAPD) using the provided QAPD template.</p>
    </div>
    <p>Or maybe the Chairs and Staff Contacts are pondering the transfer of a test suite from an external entity...</p>
    <div class="story">
        <p><span class="story-label">Story 3: Test-suite transfer</span><br />
        A Working Group has re-chartered to finish a Second Edition of its specification, and to develop the next functional version. The group did not develop a test suite during its first charter, but a collaboration of outside organizations and an industry consortium has developed one. The Chair and Staff Contacts have negotiated an agreement in principle to transfer the test suite to a stable home in W3C. In <cite>The QA Handbook</cite>, they find a <a shape="rect" href="#Acquire">handy checklist</a> of preliminary steps, requirements, and specific activities for a smooth transfer.</p>
    </div>
    <p>Or maybe they need to take preemptive action due to looming possible license-issue hassles...</p>
    <div class="story">
        <p><span class="story-label">Story 4: Test suite licensing</span><br />
        A Working Group is almost ready to request Candidate Recommendation (<acronym title="Candidate Recommendation">CR</acronym>), and has gotten a comprehensive test suite together for <acronym title="Candidate Recommendation">the CR</acronym>'s trial implementation period. As the Chair starts to arrange for publication of the test suite, she finds the Working Group split on which test suite distribution licenses to use. Consulting <cite>The QA Handbook</cite>, she finds discussion of the <a shape="rect" href="#IANAL">pros and cons of the W3C licenses</a> (the <a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231">Software License</a> and the <a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231">Document License</a>), and advice on devising an optimal licensing strategy.</p>
    </div>
    <p>Or maybe they can borrow the experience of other W3C Working Groups for various useful and common test suite processes...</p>
    <div class="story">
        <p><span class="story-label">Story 5: Test development processes</span><br />
        A Working Group has built and released a basic test suite for its specification. A Staff Contact has been given the Action Item to plan its expansion to a more comprehensive test suite, by leveraging and integrating the large test collections of several early implementors. Rather than figure out the issues and write a Test Contribution &amp; Review process from scratch, he looks at the <a shape="rect" href="#Test">summary advice</a> in <cite>The QA Handbook</cite>. <acronym title="The QA Handbook">QAH</acronym> points him both to some useful templates, and to more detailed stuff in collected test guidance &mdash; <cite>Test Development FAQ</cite> [<a href="#TEST-FAQ">TEST-FAQ</a>] and Wiki pages [<a shape="rect" href="#TEST-WIKI">T</a><a href="#TEST-WIKI">EST-WIKI</a>] &mdash; significantly shortening his effort to complete his action item.</p>
    </div>
    <h3>1.2. <a shape="rect" name="why-qah" id="why-qah">Why <acronym title="The QA Handbook">QAH</acronym>?</a></h3>
    <p>Chairs and Staff Contacts are overworked. They share a lot of the same concerns and pressures as <a shape="rect" href="http://esw.w3.org/topic/QaFramework_2fSpecGlUsability">specification editors</a> (see 2.1, "Specification editors").</p>
    <p>Common problems include:</p>
    <ul>
        <li>never enough motivated people to do the work;</li>
        <li>deliverables delivered late;</li>
        <li>licensing and legal hassles;</li>
        <li>too much to do, so little time</li>
    </ul>
    <p>A lot of groups have faced these issues. As a result, there is a growing body of experience in W3C about what works and what doesn't. <acronym title="The QA Handbook">QAH</acronym> aims to pull this together.</p>
    <h3>1.3. <a shape="rect" name="Audience" id="Audience">Audience and purpose</a></h3>
    <p><cite>The QA Handbook</cite> (<acronym title="The QA Handbook">QAH</acronym>) is intended for: Chairs and Staff Contacts of W3C Working Groups. These include both the process-savvy as well as the novices.</p>
    <p><acronym title="The QA Handbook">QAH</acronym> is a practical guide about applying good practices and quality assurance techniques to WG activities, especially developing Recommendations and test materials. It flags avoidable pitfalls, and provides techniques, tools, and templates that will facilitate and accelerate the work of the groups. Guidance in QAH is supported by real-world stories and examples.</p>
    <p>There is no normative content in <acronym title="The QA Handbook">QAH</acronym>, therefore conformance to <acronym title="The QA Handbook">QAH</acronym> is not an issue. <acronym title="The QA Handbook">QAH</acronym> takes this approach because:</p>
    <ul>
        <li>the diversity amongst W3C's working groups makes it difficult to craft simple normative rules for all;</li>
        <li>in diverse, people-centric processes, it is more effective to encourage and help than to mandate and proscribe.</li>
    </ul>
    <p>Advice is presented in the imperative voice as <em>Guidelines</em> and <em>Good Practice</em>s. Guidelines tend to be general or high level encapsulations of a given topic area, and good practices tend to be more low-level, actionable details.</p>
    <p>These suggestions will generally be helpful and enhance the quality of the work of a Working Group. However, each suggestion should be applied or not depending on the specific situation of the WG.</p>
    <h3>1.4. <a shape="rect" name="scope" id="scope"><acronym title="The QA Handbook">QAH</acronym> scope</a></h3>
    <p><cite>The QA Handbook</cite> (<acronym title="The QA Handbook">QAH</acronym>) is a non-normative handbook for integrating quality assurance techniques and good practices into the Working Group. It focuses especially on factors affecting development and implementation of specifications and tests, including:</p>
    <ul>
        <li>people resources;</li>
        <li>deliverables and their schedules;</li>
        <li>interaction with W3C and the world;</li>
        <li>W3C process, licenses, etc.</li>
    </ul>
    <p>Better specifications and effective test development are emphasized goals in QAH and its companion documents. In support of those goals, the scope of QAH goes beyond a narrow focus on authoring topics, and touches on many aspects of the broader context in which Working Groups operate -- from writing charters, to the logistics of managing the group's quality practices, to interacting with other groups and the public, to completion and maintenance of the Working Group's deliverables.</p>
    <h3>1.5. <a shape="rect" name="Other" id="Other">Other QA Framework resources</a></h3>
    <p><cite>The QA Handbook</cite> belongs to a document family collectively called the <cite>QA Framework</cite>. The other parts of the <cite>QA Framework</cite> are:</p>
    <ul>
        <li><cite>QA Framework: Specification Guidelines</cite> [<a shape="rect" href="#QAF-SPEC">QAF-SPEC</a>]</li>
        <li><cite>Variability in Specifications</cite> [<a href="#VIS">VIS</a>]: advanced topics in support of <cite>Specification Guidelines</cite></li>
        <li><cite>QA Framework Primer</cite>[<a href="#QAF-PRIM">QAF-PRIM</a>]</li>
        <li>Test guidance: <cite>Test Development FAQ</cite> [<a href="#TEST-FAQ">TEST-FAQ</a>] and Wiki pages [<a shape="rect" href="#TEST-WIKI">TEST-WIKI</a>] on test-related topics.</li>
    </ul>
    <p>Editors, test builders, and W3C members in general, in addition to Chairs and Staff Contacts, will find advice specific to their roles in these other framework parts. The <cite>Primer</cite> provides a brief orientation to the whole <cite>QA Framework</cite>.</p>
    <p>Besides the <cite>QA Framework,</cite>other useful documents about quality assurance techniques and good practices are maintained in a <a href="http://www.w3.org/QA/Library/">QA Library</a> by the QA Activity.</p>
    <h3>1.6. <a shape="rect" name="Roadmap" id="Roadmap">Roadmap to the rest of <acronym title="The QA Handbook">QAH</acronym></a></h3>
    <p>After this orientation section, <acronym title="The QA Handbook">QAH</acronym> contains four topical modules:</p>
    <ul>
        <li><a shape="rect" href="#Early">Early planning and commitment</a></li>
        <li><a shape="rect" href="#Day-to-day">Day-to-day QA operations</a></li>
        <li><a shape="rect" href="#IANAL">Licensing &amp; branding</a></li>
        <li><a shape="rect" href="#Acquire">Acquiring test materials</a></li>
    </ul>
    <p><a shape="rect">Each presents advice and guidance about tasks that contribute to the WG's realization of quality specifications and tests. These tend to have a chronological correspondence to phases (rec-track stages) of its activities.</a> Earlier is better, but later is better than never.</p>
    <hr />
    <h2>2. <a shape="rect" name="Early" id="Early">Early planning and commitment</a></h2><!-- This markup mimics WebArch, except this 'div' is their "boxedtext" for all such. -->
    <div class="story">
        <p><span class="story-label">Story: No functionality without tests.</span> <a shape="rect" name="test-policy" id="test-policy">The Xyz Working Group launched an intense test effort late in its specification development. During Candidate Recommendation (<acronym title="Candidate Recommendation">CR</acronym>), the still-growing Test Suite kept flushing out ambiguities and flaws in the specification. Xyz10 cycled multiple times in <acronym title="Candidate Recommendation">CR</acronym>, in part because of (beneficial!) changes from the test suite/specification-revision iteration. For Xyz20, the group decided up front that any functionality proposed for inclusion had to be accompanied by test cases.</a></p>
    </div>
    <div class="principle">
        <p><span class="principle-label">Guideline 1: Plan &amp; commit early.</span> <a shape="rect" id="pr-early-planning" name="pr-early-planning">Plan for and commit to the Working Group's <span class="vague-flag">test and other quality-related deliverables</span> as early as possible.</a></p>
    </div>
    <div class="why-care">
        <p><span class="care-label">Why care?</span>It's not always easy to anticipate Working Group needs during the rush and pressure of Chartering, or in the busyness of early Working Group life. But the earlier the group can accurately identify and commit to its test and other quality related deliverables, the less likely it is that the WG will get major surprises later on, or run into problems that will delay its progress towards Recommendation.</p>
        <p>Additional reason &mdash; binding decisions about participation (e.g., numbers per company) often are made at Charter time.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span> In practice, this means attention to a handful of points, which we enumerate in the following series of Good Practices.</p>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-build-acquire" name="gp-build-acquire">Good Practice 1:</a></span> Decide as soon as possible &mdash; will the Working Group build test materials or acquire them?</p>
    </div>
    <div class="why-care">
        <p><span class="care-label">Why care?</span> Clearly, it is going to make a big difference in a Working Group's staffing requirements &mdash; building test materials tends to be labor intensive (extremely so for some types of specifications). Even if the group imports them, some resources will have to be applied (see the <a shape="rect" href="#Acquire">final module</a>). The particular test-related activities and milestones in its program of work will in general be completely different for build versus acquire options.</p>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-plan-deliverables" name="gp-plan-deliverables">Good Practice 2:</a></span> Think about and enumerate the <span class="vague-flag">quality-related deliverables</span> that might help the Working Group through the Recommendation track.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span>Minimally, the Working Group should commit to assuring the timely existence of test materials. <strong>Test materials</strong> provide for the evaluation of an implementation against the requirements of a specification and/or provide information about the implementation. <strong>Conformance Test Materials</strong> are test materials that are used to indicate conformance of an implementation to a specification. Note that all test materials are not conformance test materials. For example, some Working Groups using a test-driven specification development process emphasize tests of new, changed, or contentious functionality.</p>
        <p>Test Materials include:</p>
        <ul>
            <li>test suites (tests, documentation and harness);</li>
            <li>validation tools;</li>
            <li>test assertions;</li>
            <li>sample code (in some cases);</li>
            <li>checklists;</li>
            <li>Implementation Conformance Statement (ICS) pro-forma.</li>
        </ul>
        <p><a name="qual-deliverables" id="qual-deliverables">Quality-related deliverables</a> include:</p>
    </div>
    <ul>
        <li>test materials;</li>
        <li>systematic, thorough reviews of specification;</li>
        <li>sample code (if not considered test material);</li>
        <li>model specification in formal language.</li>
    </ul>
    <div class="meaning">
        <p>Test materials are by far the most common <span class="vague-flag">quality-related deliverable</span>, and likely will be a key to quick and painless passage through Candidate Recommendation (<acronym title="Candidate Recommendation">CR</acronym>).</p>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a shape="rect" id="gp-sync-deliverables" name="gp-sync-deliverables">Good Practice 3:</a></span> Synchronize <span class="vague-flag">quality-related deliverables</span> and their development milestones with specification milestones.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span> This advice echoes that in <cite><a shape="rect" href="http://www.w3.org/2002/05/rec-tips">Tips for Getting to Recommendation Faster</a></cite> [<a shape="rect" href="#REC-TIPS">REC-TIPS</a>] for example see item #6 in section 2. Quality related deliverables include especially test materials, but could include as well such other items as are <a href="#qual-deliverables">enumerated above</a>.</p>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a shape="rect" id="gp-advancement-criteria" name="gp-advancement-criteria">Good Practice 4:</a></span> Consider whether the Working Group should bind any <span class="vague-flag">quality criteria</span> to Rec-track advancement.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span> For example, finishing test materials deliverables before requesting Candidate Recommendation (<acronym title="Candidate Recommendation">CR</acronym>) is important, in order to facilitate <acronym title="Candidate Recommendation">CR</acronym>'s Implementation Assessment. (Counter-example: as in the <a shape="rect" href="#test-policy">above Story</a>, if the group is still building them during <acronym title="Candidate Recommendation">CR</acronym>, it is likely to uncover things that will oblige it to cycle in <acronym title="Candidate Recommendation">CR</acronym>.)</p>
        <p>This good practice takes the synchronization of deliverables of the <a href="#gp-sync-deliverables">previous Good Practice</a> a step further &mdash; the specification may not advance unless the committed <span class="vague-flag">test or other quality criteria</span> are met.</p>
    </div>
    <div class="related">
        <p><span class="related-label">Related:</span><cite><a shape="rect" href="http://www.w3.org/2004/02/02-transitions">How to Organize a Recommendation Track Transition</a></cite> [<a shape="rect" href="#TRANSITION">TRANSITION</a>] talks some about the <a shape="rect" href="http://www.w3.org/2004/02/02-transitions#impl-reports">role of test suites</a> in advancement decisions.</p>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-plan-staffing" name="gp-plan-staffing">Good Practice 5:</a></span> Put some thought into how to staff the Working Group's <span class="vague-flag">test and other quality assurance plans</span>.</p>
    </div>
    <div class="why-care">
        <p><span class="care-label">Why care?</span>The earlier this is done, the more <a href="#Some">options</a> will be available. Charter-time <a href="#Staffing">staffing provisions of W3C Process</a>, by the way, provide another good reason to put some thought into test suite plans and other quality-related deliverables as early as possible.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span><a name="Some" id="Some">Some options include</a>:</p>
        <ul>
            <li>dedicated team from Working Group participants;</li>
            <li>part of everyone's time;</li>
            <li>Consider: asking in Call for Participation for extra people who would specifically be tasked to work on <span class="vague-flag">test and any other quality-related tasks</span>.</li>
        </ul>
        <p>The third option is not really different from the first. It's just a way of doing it. But notice that it's an option that is only available by looking into these questions at Charter time.</p>
        <p><a id="Staffing" name="Staffing">By the way</a>, there has been confusion about "W3C Process only allows each company to have two participants on the Working Group". In fact, that is not from W3C Process. W3C Process gives considerable freedom for this to be tailored to Working Group needs &mdash; <a shape="rect" href="http://www.w3.org/2004/02/Process-20040205/policies.html#member-rep">W3C Process says</a> it may be specified in the Charter. So, for example, the group could decide on these rules: allow two per company in technical discussion, issue resolution, voting, etc; and, allow additional dedicated test suite builders.</p>
        <p><cite><a shape="rect" href="http://www.w3.org/2002/05/rec-tips">Tips for Getting to Recommendation Faster</a></cite> [<a shape="rect" href="#REC-TIPS">REC-TIPS</a>] (section 3) also talks some about the value of (early) staffing decisions.</p>
    </div>
    <div class="technique">
        <h5 id="plan-commit-tech">Techniques</h5>
        <ul>
            <li>The <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-charter.html">Charter template</a> provides for a superset of the things that one might want to specify and commit to early (at chartering time). Use it for those items that are deemed appropriate to write into the charter.</li>
            <li>While often "earlier is better", Charter time may not be appropriate for each and every one of the above Good-Practice recommendations, for every Working Group. In some cases, it might be premature to commit to that level of detail at that early stage of the group's life.</li>
        </ul>
    </div>
    <div class="related">
        <p><span class="related-label">Related:</span></p>
        <ul>
            <li>A number of the topics in this module are mentioned also in <cite><a shape="rect" href="http://www.w3.org/2002/05/rec-tips">Tips for Getting to Recommendation Faster</a></cite> [<a shape="rect" href="#REC-TIPS">REC-TIPS</a>].</li>
            <li><cite>QA Framework: Specification Guidelines</cite> talks about utility of the Test-Assertion-list deliverable, as well as methodical specification reviews, for specification quality control</li>
            <li>Collected test-guidance, <cite>Test Development FAQ</cite> [<a href="#TEST-FAQ">TEST-FAQ</a>] and Wiki pages [<a shape="rect" href="#TEST-WIKI">TEST-WIKI</a>] on test-related topics, talk about types of test suites, automation, test assertions, etc &mdash; a shopping list especially of <span class="vague-flag">test-related deliverables</span>.</li>
            <li><cite><a shape="rect" href="http://www.w3.org/2004/02/02-transitions">How to Organize a Recommendation Track Transition</a></cite> [<a shape="rect" href="#TRANSITION">TRANSITION</a>] is related.</li>
        </ul>
    </div><!-- suppress until examples are integrated...
<div class="examples">
<p><span class="examples-label"><a id="exs-plan-commit"
name="exs-plan-commit">Examples:</a></span></p>
<ul>
  <li class="editorial"><span class="editorial">[<acronym
    title="To Be Done">TBD</acronym>. next version, get some existing
    examples from old OpsGL/ET and Intro]</span></li>
  <li class="editorial"><span class="editorial">[<acronym
    title="To Be Done">TBD</acronym>. during reviews, explicitly solicit good
    examples and especially some counter-examples]</span></li>
</ul>
</div>
-->
    <hr />
    <h2>3. <a shape="rect" name="Day-to-day" id="Day-to-day">Day-to-day QA operations</a></h2>
    <div class="story">
        <p><span class="story-label">Story: QA Process Document came first.</span> <a shape="rect" name="early-qapd" id="early-qapd">After doing it up front in one of its first test materials development efforts, a test development team was so convinced of the value of a good QA Process Document that, in its several subsequent W3C test development efforts, one of the first things it did was copy-paste-edit a previous instance of such a process document for the new effort.</a></p>
    </div>
    <div class="principle">
        <p><span class="principle-label">Guideline 2: Document QA processes.</span> <a shape="rect" id="pr-collect-qap-info" name="pr-collect-qap-info">In one place, collect all of the information needed to define the Working Group's work processes, inform others how to communicate with the Working Group on <span class="vague-flag">test and quality topics</span>, and direct everyone how to find and use its <span class="vague-flag">test and related resources</span>.</a></p>
    </div>
    <div>
        <p><span class="care-label">Why care?</span>Over and over we hear the message from successful test suite projects &mdash; define the work process early: test development framework, issues list, email list, faq, contribution &amp; review process, etc. Conversely, there are examples of disorganized collections of test cases with no one apparently in charge, no way to find their status, or verify their correctness.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span> The <a shape="rect" href="http://www.w3.org/2004/02/Process-20040205/">W3C Process Document</a> clearly organizes the way that a Working Group's specifications are progressed through the Rec track. But there is no similar template for the other aspects of a Working Group's life, particularly aspects related to <span class="vague-flag">test and other quality assurance processes</span> &mdash; logistical setup, communications with the outside, licensing and branding policy, test development process, etc.</p>
    </div>
    <div class="example">
        <h5 id="ex-Pos-qap-info">Examples</h5>
        <div class="indiv-example">
            <p><acronym title="Document Object Model">DOM</acronym> has a thorough <a href="http://www.w3.org/2002/06/DOMConformanceTS-Process-20020627">test suite process document</a> referenced from its <a href="http://www.w3.org/DOM/Test/">/Test/ Web page</a>. Starting at the Test page, one finds complete information about communications and logistics, licensing, submission and acceptance procedures, test materials location, and test suite status.</p>
        </div>
        <div class="indiv-example">
            <p>Starting at the <a href="http://www.w3.org/Style/CSS/Test/">CSS /Test/ page</a>, one quickly finds the location of CSS test materials, excellent <a href="http://www.w3.org/Style/CSS/Test/testsuitedocumentation.html">test suite documentation</a> for users and contributors (includes templates) alike, and an additional test <a href="http://www.w3.org/Style/CSS/Test/guidelines.html">authoring guidelines</a> document. (Some operational details -- e.g., communications and licensing -- are less easily located.)</p>
        </div>
    </div><!--
<div class="example">
<h5 id="ex-Pos-qap-info">Examples</h5>
<div class="indiv-example">
</div>
</div>
-->
    <div class="practice">
        <p><span class="practice-label"><a id="gp-make-qapd" name="gp-make-qapd">Good Practice 6:</a></span> Put all of the Working Group's important <span class="vague-flag">test and other quality-related</span> information in one place in a QA Process Document.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span> In practice, it means producing a QA Process Document recording at least those details of the Working Group's <span class="vague-flag">test and quality assurance work processes</span> that are outlined in the template and briefly discussed in the following subsections.</p>
    </div>
    <div class="technique">
        <h5 id="make-qapd-tech">Techniques</h5>
        <p>The technique is simple: Use the <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-root.html">QA Process Document template</a>. It will guide the user through everything needed, and then some. It is not only a template, but also a checklist of sorts, for the sort of things that the Working Group should consider having in its QA Process Document. The template has been assembled as the union of good practices seen in real QA Process Documents of W3C Working Groups.</p>
        <p>In the past, before this template, test efforts would often copy-edit an existing process document from another effort. There is not really any reason to do this anymore. Here are some examples from which the template has been assembled:</p>
    </div>
    <div class="example">
        <h5 id="ex-qapd-sources">Examples</h5>
        <div class="indiv-example">
            The best example of a comprehensive QA Process Document seen so far is the <a href="http://www.w3.org/2002/06/DOMConformanceTS-Process-20020627">DOM TS process document</a>.
        </div>
        <div class="indiv-example">
            The very similar <a href="http://www.w3.org/XML/Test/XMLConformanceTS-Process-20031210.html">XML TS process document</a> reflects a copy-edit by the XML TS development team (significant membership overlap with the DOM development team).
        </div>
    </div>
    <h3>3.1. <a shape="rect" name="Logistics" id="Logistics">General modus operandi</a></h3>
    <p>See the <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#QAcommunicate">corresponding section</a> in the template.</p>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-point-of-contact" name="gp-point-of-contact">Good Practice 7:</a></span> Identify a Working Group point-of-contact for test materials or other <span class="vague-flag">quality-related business.</span></p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span> This can be a special appointed "QA point-of-contact". Or it might be the Chair (or a co-chair), or Staff Contact. The important part is that there be an identified point-of-contact to whom others can address questions, bug reports, contributions, etc, and who will be accountable for responding.</p>
        <p>See also the <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#QAcommunicate">QA Process Document subsection</a> in the template.</p>
    </div>
    <div class="example">
        <h5 id="ex-point-of-contact">Examples</h5>
        <div class="indiv-example">
            See the already-mentioned <a href="http://www.w3.org/2002/06/DOMConformanceTS-Process-20020627">DOM TS process document</a>.
        </div>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-email-list" name="gp-email-list">Good Practice 8:</a></span> Specify an archived email list to use for <span class="vague-flag">quality-related communications</span>.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span>It can be a dedicated "Test" list. Or it can be a public Interest Group list. Or it can be a public comment list separate from the Working Group list, in the case that there isn't an Interest Group. Because of the variety of public/private mixes amongst W3C's Working Groups, "one size fits all" does not work well here.</p>
    </div>
    <p>The list should be available for all quality-related topics, including test suite questions, bug reports, contributions, etc.</p>
    <p>See also the <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#QAcommunicate">corresponding QA Process Document subsection</a> in the template.</p>
    <div class="example">
        <h5 id="ex-email-list">Examples</h5>
        <div class="indiv-example">
            The <a href="http://www.w3.org/XML/Test/">XML /Test/ page</a> identifies a public <a>mail list for test topics</a> (public-xml-testsuite@w3.org).
        </div>
        <div class="indiv-example">
            The <a href="http://www.w3.org/Graphics/SVG/Test/">SVG test pages</a> identify a public test-topics mail list (svg-testsuite-comments@w3.org) prominently at the beginning.
        </div>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-web-page" name="gp-web-page">Good Practice 9:</a></span> Identify Web page(s) for test suites, announcements, and other <span class="vague-flag">quality-related topics</span>.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span>Obviously, this needs to be publicly accessible. Doing it all in the public portion of the Working Group's Web space is one way to achieve that. In particular, that provides a good, secure repository location for test materials.</p>
        <p>Some groups, for example <acronym title="Scalable Vector Graphics">SVG</acronym>, have two locations &mdash; a private CVS repository for the in-development test materials, and a simplified <a href="http://www.w3.org/Graphics/SVG/Test/">public location</a> for periodic public releases of test materials.</p>
    </div>
    <div class="technique">
        <h5 id="web-pages-tech">Techniques</h5>
        <p>Link the test pages from the Working Group's home page, and use the common convention of putting them in the /Test/ directory under the Working Group's home page.</p>
        <p>See also the <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#QAcommunicate">corresponding QA Process Document subsection</a> in the template.</p>
        <p>The Working Group's QA Process Document is also a handy central location to record its licensing policies and (any) branding policies. These are sufficiently important that they have their <a shape="rect" href="#IANAL">own <acronym title="The QA Handbook">QAH</acronym> module</a>, which addresses:</p>
        <ul>
            <li>license terms applicable to test materials, both submitted and published;</li>
            <li>any branding policy &mdash; logos, conformance icons, etc &mdash; and associated rules and restrictions.</li>
        </ul>
    </div>
    <div class="example">
        <h5 id="ex-web-page">Examples</h5>
        <div class="indiv-example">
            SVG uses a helpful convention for its <a href="http://www.w3.org/Graphics/SVG/Test/">test suite pages</a> -- the URL is the <a href="http://www.w3.org/Graphics/SVG/">SVG home page</a> URL (http://www.w3.org/Graphics/SVG/) with /Test/ appended. Additionally, the test page is prominently linked from the <a href="http://www.w3.org/Graphics/SVG/">SVG home page</a>.
        </div>
        <div class="indiv-example">
            The <a href="http://www.w3.org/XML/Test/">XML test pages</a> follow the same useful convention -- their location is a /Test/ directory under the WG home page.
        </div>
    </div>
    <h3>3.2. <a shape="rect" name="Test" id="Test">Test development framework and processes</a></h3>
    <p>If the Working Group itself is developing test materials, then there are several topics associated with test materials development process that are going to impact its operations and management. Useful guidance and discussion for these may be found in QA's <cite>Test Development FAQ</cite> [<a href="#TEST-FAQ">TEST-FAQ</a>] and Wiki pages [<a shape="rect" href="#TEST-WIKI">TEST-WIKI</a>] on test-related topics. These topics, which might conveniently be documented in the Working Group's QA Process Document (<acronym title="QA Process Document">QAPD</acronym>), include definitions of:</p>
    <ul>
        <li>development framework <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#DevelopFramework">[<acronym title="QA Process Document">QAPD</acronym> template]</a></li>
        <li>test materials contribution and review process <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#DevelopFramework">[<acronym title="QA Process Document">QAPD</acronym> template]</a></li>
        <li>test materials publication <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#DevelopFramework">[<acronym title="QA Process Document">QAPD</acronym> template]</a></li>
        <li>testing and test results promotion <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#DevelopFramework">[<acronym title="QA Process Document">QAPD</acronym> template]</a></li>
    </ul>
    <h3>3.3. <a shape="rect" name="Life" id="Life">Life after Working Group &mdash; maintenance</a></h3>
    <p><a shape="rect" name="Finally" id="Finally">Finally</a>, as a part of planning about "life after Working Group", the group will need to decide what happens to both its test materials and associated processes. Discussion of maintenance-related topics may be found in QA's <cite>Test Development FAQ</cite> [<a href="#TEST-FAQ">TEST-FAQ</a>] and Wiki pages [<a shape="rect" href="#TEST-WIKI">TEST-WIKI</a>] on test-related topics. Again the information might conveniently be located in the group's QA Process Document:</p>
    <ul>
        <li>what happens to the test materials themselves?</li>
        <li>what happens to the Working Group's contribution and review processes? <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#maintenance">[<acronym title="QA Process Document">QAPD</acronym> template]</a></li>
        <li>how to track errata and (for example) new specification editions? <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#maintenance">[<acronym title="QA Process Document">QAPD</acronym> template]</a></li>
        <li>what happens with bug reports about the test materials? <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#maintenance">[<acronym title="QA Process Document">QAPD</acronym> template]</a></li>
    </ul>
    <div class="related">
        <p><span class="related-label">Related:</span></p>
        <ul>
            <li><cite>Test Development FAQ</cite> [<a href="#TEST-FAQ">TEST-FAQ</a>] and Wiki pages [<a shape="rect" href="#TEST-WIKI">TEST-WIKI</a>] on test-related topics. &mdash; talks about test-specific processes, so <acronym title="The QA Handbook">QAH</acronym> mostly doesn't. Placeholders for these are found in the <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-root.html">QA Process Document template</a></li>
            <li><a shape="rect" href="#IANAL">Licensing &amp; branding module</a></li>
        </ul>
    </div>
    <div class="technique">
        <h5 id="qapd-tech">Technique</h5>
        <ul>
            <li>The <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-root.html">QA Process Document template</a> is filled out and posted on the Working Group's web site.</li>
        </ul>
    </div>
    <hr />
    <h2>4. <a shape="rect" name="IANAL" id="IANAL">Licensing &amp; branding</a></h2>
    <div class="story">
        <p><span class="story-label">Story: License issues delayed this project.</span> <a shape="rect" name="license-disagree" id="license-disagree">Xxxxxy <acronym title="Working Group">WG</acronym> has a great collection of test cases, but they couldn't agree now on publication licenses that are acceptable to everyone, and so they were stalled indefinitely while they worked it out.</a></p>
    </div>
    <div class="principle">
        <p><span class="principle-label"><a id="pr-ipr-agreement" name="pr-ipr-agreement">Guideline 3: Resolve legal &amp; license issues.</a></span> Get agreement up front about license and any other legal issues around planned test materials.</p>
    </div>
    <div class="why-care">
        <p><span class="care-label">Why care?</span>Get it right early, or it may stall the Working Group's Rec-track progress indefinitely. While this might seem to be a routine piece of <a shape="rect" href="#Day-to-day">Day-to-day operations</a>, it has proved to be sufficiently troublesome within W3C that it deserves to be a Guideline by itself.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span> There are two kinds of licensing issues: submission license, and publication license. Both of them can be problematic and can interrupt the Working Group's progress on Rec-track if not early addressed and carefully handled. Test materials license issues are the subject of ongoing debate and discussion within W3C, but there are some tactics to minimize potential problems.</p>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-submission-license" name="gp-submission-license">Good Practice 10:</a></span> As early as possible, get agreement about acceptable license terms for submission of test materials.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span>W3C now has an official set of comprehensive <a href="http://www.w3.org/2004/10/27-testcases.html">policies for contribution of test cases</a> to W3C. (Formerly, it had a more or less routine submission license, <cite><a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/contribution-grant-20021231">Contribution of Software or Test Materials to W3C</a></cite> [<a shape="rect" href="#CONTRIB">CONTRIB</a>], but without policy.)</p>
        <p>By early attention, it should become clear whether any potential test sources have problems with the standard terms, before possible disputes can impact the Working Group's deliverables and schedules. There have been cases where potential submitters would not accept the standard terms.</p>
    </div>
    <div class="example">
        <h5 id="ex-custom-submit">Examples</h5>
        <div class="indiv-example">
            XML Schema <a href="http://www.w3.org/2001/05/xmlschema-test-collection.html">introduction to test collection</a> contains a reference to <a href="http://www.w3.org/2001/05/test-materials-license.html">contribution terms</a> (this example pre-dates recent W3C release of more detailed policies).
        </div>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-publication-license" name="gp-publication-license">Good Practice 11:</a></span> As soon as the nature of the Working Group's test materials becomes clear, get agreement about license terms for publication of the test materials.</p>
    </div>
    <div class="why-care">
        <p><span class="care-label">Why care?</span>Any W3C-hosted materials must have approved license and use terms. Experience has shown that there is no single license that is appropriate for all parts of all test materials, so the Working Group needs to address this after it has come to an understanding of the structure, content, and intended use of its test materials.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span>Currently approved W3C licenses that may be applied to test materials are the <a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231">Document License</a> and the <a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231">Software License</a>. The Document License has two characteristics that are attractive for test materials:</p>
        <ul>
            <li>It prohibits the test materials from being modified by the user upon download, therefore guaranteeing the integrity of the test materials;</li>
            <li>It requires that if the test materials are copied or redistributed, they must contain the link to the original test materials and their status.</li>
        </ul>
        <p>On the other hand, there are situations in which the <a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231">Document License</a> is inappropriate, because (for example) some parts of the test materials require modification or completion in order to apply them.</p>
        <p>The W3C <a href="http://lists.w3.org/Archives/Member/chairs/2004AprJun/0062.html">Advisory Board feels</a> (member-only link) that the Document License is the appropriate license for <strong>test cases</strong>.</p>
        <p>That ruling notwithstanding, test materials might contain some mixture of different components: test software, test documentation, and test cases. In some cases, one license may be appropriate for some component(s) of the test materials, but another license may be better for other parts. A careful look at the test materials' composition and use cases should reveal what is the best solution.</p>
        <p>The Working Group should consult with W3C Legal if it believes that:</p>
        <ul>
            <li>neither the <a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231">Document License</a> nor the <a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231">Software License</a> is applicable, not even piecewise as just described,</li>
            <li>or, contrary to the <a href="http://lists.w3.org/Archives/Member/chairs/2004AprJun/0062.html">Advisory Board's advice</a>, the Document License is inappropriate for the <strong>test cases</strong> component of its test materials</li>
        </ul>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-branding-policy" name="gp-branding-policy">Good Practice 12:</a></span> Decide policy about having brands, logos, or conformance icons associated with the Working Group's test materials.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span>There are two questions that the Working Group will face. First, whether to have logos, icons, or brands associated with its test materials. If that answer is yes, then it will have to address a series issues about proper usage of the logos.</p>
        <p><cite><a shape="rect" href="http://www.w3.org/Consortium/Legal/logo-usage-20000308.html">W3C Logo and Icon Usage</a></cite> contains a complete catalog of the logos in use within W3C. Of particular interest (related to conformance) are the logos associated with content validation (HTML, CSS, etc), Web Accessibility conformance, etc. These logos have associated issues of veracity and enforceability of conformance claims. A Working Group considering its own logo or brand will face similar issues, amongst which are:</p>
        <ul>
            <li>terms for valid application of the logo;</li>
            <li>relationship of the logo to conformance claims;</li>
            <li>challenge procedures for disputed logo usage;</li>
            <li>policing &amp; enforcement policy.</li>
        </ul>
    </div>
    <div class="technique">
        <h5 id="logos-tech">Techniques</h5>
        <p>Two final comments about both the topics of licenses and logos/brands:</p>
        <ul>
            <li>There are places in <a shape="rect" href="http://www.w3.org/QA/2004/08/QAH-qapd-text.html#licenses">QA Process Document template</a> to document the Working Group's decisions, about licenses as well as branding/logos.</li>
            <li>Don't forget to make license terms obvious and visible in all deliverables (and all their bits and pieces)!</li>
        </ul>
    </div>
    <div class="example">
        <h5 id="exs-logos-brands">Examples</h5>
        <div class="indiv-example">
            See <a href="http://www.w3.org/Consortium/Legal/logo-usage-20000308.html">W3C Logo and Icon Usage</a>. It contains numerous W3C logo/icon examples.
        </div>
    </div>
    <div class="related">
        <p><span class="related-label">Related:</span>(References for all of the legal and licensing Good Practices above.)</p>
        <ul>
            <li>Minuted <a href="http://lists.w3.org/Archives/Member/chairs/2004AprJun/0062.html">Advisory Board opinion</a> on appropriate license for test materials.</li>
            <li>Relationship of Patent Policy to test materials' submission (in a <a href="http://www.w3.org/2003/12/22-pp-faq.html#testcases">Patent Policy FAQ</a> from W3C Legal)</li>
            <li><cite><a shape="rect" href="http://www.w3.org/Consortium/Legal/logo-usage-20000308.html">W3C Logo and Icon Usage</a></cite> summarizes key policy details for existing W3C conformance icons.</li>
            <li><a href="http://www.w3.org/TR/qaframe-spec/"><cite>Specification Guidelines</cite></a> deals with the related topic of conformance claims.</li>
        </ul>
    </div>
    <div class="history">
        <p><span class="related-label">Additional History:</span>For those who are interested in more background and detail on the license issues, there is an extensive history:</p>
        <ul>
            <li>a long running <a href="http://www.w3.org/QA/WG/qawg-issues-html#x49">QA Working Group issue</a>;</li>
            <li>A 2003 ad-hoc task group generated a <a shape="rect" href="http://www.w3.org/2003/02/test-suite-copyright.html">succinct issue summary</a> (Member-only);</li>
            <li><a shape="rect" href="http://www.w3.org/2003/06/11-test-copyright.html">final report</a> of sorts (Member-only) documented issue resolutions acceptable to the ad-hoc task group participants.</li>
        </ul>
    </div>
    <div class="principle">
        <p><span class="principle-label">Caveat.</span> These legal and licensing topics have been very active within W3C. Some of these references in this present <acronym title="The QA Handbook">QAH</acronym> version may be superseded by future, newer advice or policies.</p>
    </div>
    <hr />
    <h2>5. <a shape="rect" name="Acquire" id="Acquire">Acquiring test materials</a></h2>
    <div class="story">
        <p><span class="story-label">Story: Saved a ton of work by acquiring.</span> <a name="xml-ts-xfer" id="xml-ts-xfer">An external organization built a test suite for ABC 1.0. The ABC Working Group had no test suite, no effort in progress, and no resources to staff a from-scratch effort. The external organization had no resources to continue maintaining the test suite. With a QA task-force moderating, several months of sorting the details led to a win-win agreement to transfer the ABC 1.0 test suite to the ABC working group. The test suite got a secure repository within W3C, was published for public use, and was given adequate maintenance resources.</a></p>
    </div>
    <div class="principle">
        <p><span class="principle-label"><a id="pr-consider-acquire" name="pr-consider-acquire">Guideline 4: Consider acquiring test materials.</a></span> The Working Group can save lots of time and work by acquiring a test suite, but be ready to address and resolve many of the same issues as build-your-own scenarios.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span> The three main problems are:</p>
        <ul>
            <li>license issues around test materials</li>
            <li>human resources</li>
            <li>quality of the test materials</li>
        </ul>
        <p>The license and human resources issues are similar in concept to what the Working Group faces when building test materials, but probably lesser in degree. A pre-transfer quality assessment might seem unique to the acquire option, but the actual steps will probably look similar to a test case contribution/review process in a build-your-own scenario.</p>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-quality-assess" name="gp-quality-assess">Good Practice 13:</a></span> Do a quality assessment of proposed test materials before going any further.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span>Basic things that a good quality assessment might cover would for example include:</p>
        <ul>
            <li>clear declaration of scope,</li>
            <li>traceability,</li>
            <li>atomicity,</li>
            <li>user documentation.</li>
        </ul>
        <p>A more comprehensive list of things that an assessment process could add to that basic list:</p>
        <ul>
            <li>maintainer documentation,</li>
            <li>correctness,</li>
            <li>completeness (vis à vis declared scope),</li>
            <li>harnesses or interfaces for application of the test materials,</li>
            <li>reconfigurability,</li>
            <li>results assessment,</li>
            <li>results recording &amp; reporting,</li>
            <li>automation features,</li>
            <li>versioning/errata support,</li>
            <li>declaration of publication licenses,</li>
            <li>integrated submission procedures.</li>
        </ul>
    </div>
    <div class="related">
        <p><span class="related-label">Related</span></p>
        <ul>
            <li>Further detailed information on some of these topics is available in the <cite>Test Development FAQ</cite> [<a href="#TEST-FAQ">TEST-FAQ</a>] and Wiki pages [<a shape="rect" href="#TEST-WIKI">TEST-WIKI</a>] on test-related topics.</li>
        </ul>
    </div>
    <div class="example">
        <h5 id="ex-assess-criteria">Examples</h5>
        <div class="indiv-example">
            The <acronym title="Scalable Vector Graphics">SVG</acronym> test suite manual contains a chapter listing <a shape="rect" href="http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#TestReviewGuidelines">test review guidelines</a>.
        </div>
        <div class="indiv-example">
            Some of the <a shape="rect" href="http://www.w3.org/Style/CSS/Test/guidelines.html">CSS test authoring guidelines</a> could also be turned around and used as assessment criteria.
        </div>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-ensure-staff" name="gp-ensure-staff">Good Practice 14:</a></span> Ensure there are enough human resources to support the transferred test materials.</p>
    </div>
    <div class="meaning">
        <p><span class="meaning-label">What does it mean?</span>A test materials manager (possibly one of the job functions of the QA point of contact) is still needed, but total human resources ought to be considerably less than build-your-own scenarios. With luck, the original manager of the external test materials source might become a participant in the Working Group after the test materials are transferred.</p>
    </div>
    <div class="practice">
        <p><span class="practice-label"><a id="gp-resolve-ipr" name="gp-resolve-ipr">Good Practice 15:</a></span> Sort out licensing issues with the external party that produced the test materials.</p>
    </div>
    <div class="technique">
        <h5 id="transfer-tech">Techniques</h5>
        <p>The following is a virtualization of an <a shape="rect" href="#xml-ts-xfer">actual transfer scenario</a> that QA helped to moderate. It could serve as a checklist of steps to consider for Working Group's taking the <em>acquire</em> route.</p>
        <div class="story">
            <div class="story">
                <p>Legend: <strong><acronym title="External Group">EG</acronym></strong> the external group or entity; <strong><acronym title="QA Working Group">QAWG</acronym></strong> the QA Working Group; <strong><acronym title="Test Materials">TM</acronym></strong> the test materials; <strong><acronym title="Working Group">WG</acronym></strong> the Working Group acquiring the test materials.</p>
            </div>
            <ol>
                <li>Before the transfer, <acronym title="Working Group">WG</acronym> with the help of <acronym title="QA Working Group">QAWG</acronym>:
                    <ul>
                        <li>performs an assessment of the quality of candidate <acronym title="Test Materials">test materials</acronym> (by <acronym title="Working Group">WG</acronym>, <acronym title="QA Working Group">QAWG</acronym>)</li>
                        <li>identifies and commits to a set of test-related deliverables from the candidate <acronym title="Test Materials">test materials</acronym>. These could be: releases, appeal/challenge processes, maintenance plan, submission/review process, Web site, mail list, etc. (by <acronym title="Working Group">WG</acronym>)</li>
                        <li>identifies sufficient staffing, including at least a <acronym title="Test Materials">test materials</acronym> manager. <em>Recommendation:</em> recruit the <acronym title="Test Materials">test materials</acronym> manager from the <acronym title="External Group">EG</acronym> (if one exists) to become a <acronym title="Working Group">WG</acronym> participant after the transfer. (by <acronym title="Working Group">WG</acronym>)</li>
                        <li>makes the decision to seek/accept the transfer. (by <acronym title="Working Group">WG</acronym>)</li>
                        <li>(potentially) initiates Charter amendment (by <acronym title="Working Group">WG</acronym>, <acronym title="QA Working Group">QAWG</acronym> may consult), if the <acronym title="Test Materials">test materials</acronym> acquisition doesn't fit within the current Charter.</li>
                    </ul>
                </li>
                <li>During the transfer:
                    <ul>
                        <li><acronym title="External Group">EG</acronym> and W3C reach agreement to transfer the <acronym title="Test Materials">test materials</acronym> (by <acronym title="Working Group">WG</acronym>, <acronym title="QA Working Group">QAWG</acronym>, <acronym title="External Group">EG</acronym>)</li>
                        <li><acronym title="Working Group">WG</acronym> and <acronym title="External Group">EG</acronym> perform the actual transfer of the <acronym title="Test Materials">test materials</acronym>, <acronym title="Working Group">WG</acronym> creates an initial repository (by <acronym title="Working Group">WG</acronym>, <acronym title="External Group">EG</acronym>)</li>
                    </ul>
                </li>
                <li>After transfer, initial test development/framework process setup:
                    <ul>
                        <li><acronym title="Working Group">WG</acronym> appoints a <acronym title="Test Materials">test materials</acronym> manager.</li>
                        <li>The <acronym title="Test Materials">test materials</acronym> manager creates a QA Process Document for <acronym title="Working Group">WG</acronym> (by <acronym title="Working Group">WG</acronym>, <acronym title="Test Materials">test materials</acronym> manager, <acronym title="QA Working Group">QAWG</acronym> may consult)</li>
                        <li>set up the <acronym title="Test Materials">test materials</acronym> home page, a <acronym title="Test Materials">test materials</acronym> issues mailing list (by <acronym title="Working Group">WG</acronym>, <acronym title="Test Materials">test materials</acronym> manager)</li>
                        <li>determine the appropriate W3C publication license (by <acronym title="Working Group">WG</acronym>, <acronym title="QA Working Group">QAWG</acronym>)</li>
                    </ul>
                </li>
                <li>First W3C public release of the new <acronym title="Test Materials">test materials</acronym>:
                    <ul>
                        <li>make any needed enhancements prior to the first public release: fix known/reported errors, produce documentation (by <acronym title="Working Group">WG</acronym>), W3C license labelling, etc.</li>
                        <li>announce the first public release of <acronym title="Test Materials">test materials</acronym> (by <acronym title="Test Materials">test materials</acronym> manager, Communications Group)</li>
                        <li>joint W3C/<acronym title="External Group">EG</acronym> press release (by <acronym title="Working Group">WG</acronym>, <acronym title="QA Working Group">QAWG</acronym>, Communications Group, <acronym title="External Group">EG</acronym>)</li>
                    </ul>
                </li>
                <li>After the first public release, the <acronym title="Test Materials">test materials</acronym> enter the maintenance phase.</li>
            </ol>
        </div>
    </div><!-- ending how-to section -->
    <hr class="rule" />
    <hr class="rule" />
    <h2>7. <a shape="rect" name="acknowledgments" id="acknowledgments">Acknowledgments</a></h2>
    <p>The following QA Working Group and Interest Group participants have contributed significantly to this document:</p>
    <ul>
        <li>Daniel Dardailler (W3C),</li>
        <li>Dimitris Dimitriadis (Ontologicon),</li>
        <li>Karl Dubost (W3C),</li>
        <li>Dominique Hazaël-Massieux (W3C),</li>
        <li>Lofton Henderson (CGM Open),</li>
        <li>David Marston (Lotus Development Corp),</li>
        <li>Patrick Curran (Sun Microsystems),</li>
        <li>Lynne Rosenthal (NIST),</li>
        <li>Mark Skall (NIST),</li>
        <li>Andrew Thackrah (Open Group),</li>
        <li>Olivier Thereaux (W3C),</li>
        <li>Jeremy Carroll (Hewlett-Packard),</li>
        <li>Richard Kennedy (Boeing Commercial Airplanes),</li>
        <li>Tim Boland (NIST)</li>
    </ul>
    <h2>8. <a shape="rect" name="References" id="References">References</a></h2>
    <dl>
        <dt class="label"><a shape="rect" id="PROCESS" name="PROCESS">[PROCESS]</a></dt>
        <dd><a shape="rect" href="http://www.w3.org/2004/02/Process-20040205/"><cite>World Wide Web Consortium Process Document</cite></a>, I. Jacobs, Ed., 05 February 2004, available at http://www.w3.org/2004/02/Process-20040205/ .</dd>
        <dt class="label"><a shape="rect" id="CONTRIB" name="CONTRIB">[CONTRIB]</a></dt>
        <dd><cite><a shape="rect" href="http://www.w3.org/Consortium/Legal/2002/contribution-grant-20021231">Contribution of Software or Test Materials to W3C</a></cite>, which defines W3C-approved procedures and terms for submission of Software and Test Materials, available at http://www.w3.org/Consortium/Legal/2002/contribution-grant-20021231 .</dd>
        <dt><a shape="rect" name="PROTOCOL-WG-TS" id="PROTOCOL-WG-TS">[PROTOCOL-WG-TS]</a></dt>
        <dd>The XML Protocol WG has a <a shape="rect" href="http://www.w3.org/2000/xp/Group/1/10/ts-contribution"><acronym title="Test Suite">TS</acronym> process document</a>, available at http://www.w3.org/2000/xp/Group/1/10/ts-contribution, and a <a shape="rect" href="http://www.w3.org/2001/10/test-materials-license.html">contribution/submission license</a> (example of a submission legal notice), available at http://www.w3.org/2001/10/test-materials-license.html .</dd>
        <dt><a shape="rect" name="REC-TIPS" id="REC-TIPS">[REC-TIPS]</a></dt>
        <dd><cite><a shape="rect" href="http://www.w3.org/2002/05/rec-tips">Tips for Getting to Recommendation Faster</a></cite>, a public part of the (Member-only) <cite><a shape="rect" href="http://www.w3.org/Guide/">W3C Guidebook</a></cite>.</dd>
        <dt><a shape="rect" name="QA-GLOSSARY" id="QA-GLOSSARY">[QA-GLOSSARY]</a></dt>
        <dd>A comprehensive glossary of QA terms, maintained by the QA Working Group. (<a shape="rect" href="http://www.w3.org/QA/glossary">Incomplete version</a> under construction at http://www.w3.org/QA/glossary .)</dd>
        <dt class="label"><a shape="rect" id="QAF-PRIM" name="QAF-PRIM">[QAF-PRIM]</a></dt>
        <dd><cite><a href="http://www.w3.org/QA/WG/qaframe-primer">QA Framework Primer</a></cite>, an informative supplement to <cite>The QA Handbook</cite> that provides an orientation to whole <cite>QA Framework</cite>, including a primer and some user scenarios. Published as a QA note at http://www.w3.org/QA/WG/qaframe-primer .</dd>
        <dt class="label"><a shape="rect" id="QAF-SPEC" name="QAF-SPEC">[QAF-SPEC]</a></dt>
        <dd><cite><a href="http://www.w3.org/TR/2004/WD-qaframe-spec-20041122/">QA Framework: Specification Guidelines</a></cite>, Karl Dubost, L. Rosenthal, Dominique Hazaël-Massieux, Lofton Henderson, Eds., W3C Working Draft companion to this document, 22 November 2004. A light-weight revision of the <a href="http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/">previous (Candidate Recommendation) <cite>Specification Guidelines</cite></a> collection of documents. Latest version available at http://www.w3.org/TR/qaframe-spec/ .</dd>
        <dt class="label"><a shape="rect" id="TEST-WIKI" name="TEST-WIKI">[TEST-WIKI]</a></dt>
        <dd>A collection of discussion topics and guidance on test building and other test-related subjects is found on the <a href="http://esw.w3.org/topic/QA#head-8dc1a611f322110d4051f570a734454f1a6463f7">test-related Wiki pages</a>.</dd>
        <dt class="label"><a id="TEST-FAQ" name="TEST-FAQ">[TEST-FAQ]</a></dt>
        <dd><cite><a href="http://www.w3.org/QA/WG/2005/01/test-faq">Test Development FAQ</a></cite> provides introductory information about the purpose of testing, how to get started on test materials development, and what the testing process involves. Published as a QA note at http://www.w3.org/QA/WG/2005/01/test-faq .</dd>
        <dt class="label"><a shape="rect" id="QAWG" name="QAWG"><acronym title="QA Working Group">[QAWG]</acronym></a></dt>
        <dd><a shape="rect" href="http://www.w3.org/QA/WG/">Quality Assurance Working Group</a> of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/ .</dd>
        <dt><a name="VIS" id="VIS">[VIS]</a></dt>
        <dd><cite><a href="http://www.w3.org/TR/2004/WD-spec-variability-20040830/">Variability in Specifications</a></cite> , Dominique Hazaël-Massieux, Lynne Rosenthal, W3C Working Draft 30 August 2004, http://www.w3.org/TR/2004/WD-spec-variability-20040830/ . Latest version available at http://www.w3.org/TR/spec-variability/ .</dd>
        <dt><a shape="rect" name="TRANSITION" id="TRANSITION">[TRANSITION]</a></dt>
        <dd><cite><a shape="rect" href="http://www.w3.org/2004/02/02-transitions">How to Organize a Recommendation Track Transition</a></cite>, a part of the Member Guide (Member-only), is available at http://www.w3.org/2004/02/02-transitions .</dd>
    </dl>
    <h2>9. <a shape="rect" name="History" id="History">Change history</a></h2>
    <dl>
        <dt>2005-08-31, Fourth publication (WG Note)</dt>
        <dd>
            <ul>
                <li>updated "Acknowledgements"</li>
                <li>proper title for <cite>QA Framework Primer</cite> references.</li>
                <li>tidied up references to the QAPD template.</li>
                <li>added GL &amp; GP index after TOC.</li>
                <li>changed TEST-GUIDE references to (new) TEST-FAQ plus TEST-WIKI.</li>
                <li>fixed link address / reference in "legal" section.</li>
            </ul>
        </dd>
        <dt>2004-11-22, Third publication (WG Note)</dt>
        <dd>
            <ul>
                <li>incorporated new W3M policies on contribution of test materials into Chapter 4.</li>
                <li>editorial changes per <a href="http://lists.w3.org/Archives/Public/www-qa-wg/2004Sep/0045.html">L.Rosenthal comments</a> and editor's reply.</li>
                <li>removed chapter 1 reference to "chronology diagram" (no one volunteered a new diagram).</li>
                <li>changed the 4 occurrences of "Principle" to "Guideline" (but markup still is principle-oriented).</li>
                <li>numbered (1..4) and labelled Guidelines.</li>
                <li>numbered (1..15) Good Practices (not yet labelled).</li>
                <li>labelled the stories at start of chapters 2..5.</li>
                <li>reworked all references to <cite>Test Guidelines</cite>, in some cases substituting test-guidance Wiki collection [TEST-GUIDE].</li>
                <li>Removed Test Guidelines from components of QA Framework, added ViS, Primer, and test-guidance Wiki pages.</li>
                <li>Shortened some (but not all) long Guidelines and Good Practices.</li>
                <li>Made markup and topic sub-headings more consistent with SpecGL.</li>
                <li>Removed editorial notes to self.</li>
                <li>Integrated into GP2 LR/MS break-out stuff (Reading) on test materials and quality-related deliverables.</li>
            </ul>
        </dd>
        <dt>2004-08-30, Second Published Draft</dt>
        <dd>
            <ul>
                <li>edits for improved language, removal of excessive use of abbreviations, etc (per <a href="http://www.w3.org/QA/WG/2004/06/QAH-issues#Jeremy">J.Carroll comments</a>).</li>
                <li>changed vague usage of "IPR", to be more specific (per <a href="http://www.w3.org/QA/WG/2004/06/QAH-issues#Connolly">D.Connolly comments</a>).</li>
                <li>anchors on all PRs and GPs, plus at all example-placeholder locations.</li>
                <li>indicate "back burner" status of TestGL (in SoTD, and in References -- leave alone the body text for now).</li>
                <li>split out the "Orientation to QA Framework" appendix into separate QA Framework Roadmap document.</li>
                <li>integrated <a href="http://lists.w3.org/Archives/Public/www-qa-wg/2004Jun/0064.html">K.Dubost comments</a>.</li>
                <li>integrated <a href="http://lists.w3.org/Archives/Public/www-qa-wg/2004Jun/0089.html">M.Skall comments</a>.</li>
                <li>integrated <a href="http://lists.w3.org/Archives/Public/www-qa-wg/2004Jun/0098.html">L.Rosenthal comments</a>.</li>
                <li>integrated changes to the legal/license Good Practices, per recent AB rulings.</li>
                <li>sorted out JC's scope comment ("test &amp; testability") and implemented solution.</li>
                <li>first cut at sorting out class="vague" stuff (needs critical review)</li>
                <li>drafted icon/logo section (but more needed next version)</li>
                <li>fleshed out examples in section 3.1 (General modus operandi)</li>
            </ul>
        </dd>
        <dt>2004-05-10, First Published Working Draft</dt><!-- Any general comment(s) about the version. -->
        <dd>Combines the best features of the former <cite>QA Framework: Introduction</cite> and <cite>QA Framework: Operational Guidelines</cite> into a lightweight, non-normative handbook for W3C Working Group Chairs and Staff contacts. <!-- Specific change items as 'li' in a 'ul' --></dd>
    </dl>
</body>
</html>