qaframe-primer 29.7 KB
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
       "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <title>QA Framework Primer</title>
<!--  <link rel="Stylesheet" href="/QA/2002/12/qa4.css" />-->
<link rel="Stylesheet" href="/QA/2005/08/qa-style.css" />
</head>

<body>

<!-- Header -->
<div id="Logo">
<a href="http://www.w3.org/"><img alt="W3C" src="/Icons/WWW/w3c_home" width="72" height="48" /></a>
<a href="http://www.w3.org/QA/"><img alt="QA" src="/QA/images/qa" width="161" height="48" /></a>

<!-- <div id="Header">Be strict to be cool</div> -->
 <div><map name="introLinks" id="introLinks" title="Introductory Links">
<div class="banner"> <a
class="bannerLink" title="W3C Activities" accesskey="A"
href="/Consortium/Activities">Activities</a> | <a class="bannerLink"
title="Technical Reports and Recommendations" accesskey="T"
href="/TR/">Technical Reports</a> | <a class="bannerLink"
title="Alphabetical Site Index" accesskey="S"
href="/Help/siteindex">Site Index</a> | <a class="bannerLink"
title="Help for new visitors" accesskey="N"
href="/2002/03/new-to-w3c">New Visitors</a> | <a
class="bannerLink" title="About W3C" accesskey="B"
href="/Consortium/">About W3C</a> | <a class="bannerLink"
title="Join W3C" accesskey="J"
href="/Consortium/Prospectus/Joining">Join W3C</a></div>
</map></div>
</div>


<!-- menuRight -->

<div id="Menu">
<p><a href="#Introduction">Introduction</a><span class="dot">·</span> <a
href="#qaf-audience">QA Framework audience</a><span class="dot">·</span> <a
href="#qaf-primer">Primer</a><span class="dot">·</span></p>
<hr />

<p class="navhead">Nearby:</p>

<p><a href="/QA/"><abbr
title="Quality Assurance">QA</abbr>&nbsp;Homepage</a><span
class="dot">·</span> <a href="/QA/#latest">Latest News</a><span
class="dot">·</span> <a href="/QA/#resources">QA&nbsp;Resources</a><span
class="dot">·</span> <a href="/QA/WG/">QA&nbsp;<abbr
title="Interest Group">WG</abbr></a><span class="dot">·</span> <a
href="/TR/qa-handbook/">QA&nbsp;Handbook</a><span class="dot">·</span> <a
href="/TR/qaframe-spec/">Spec Guidelines</a><span class="dot">·</span></p>
</div>

<!-- content -->
<div id="Content">

<!-- Your content is starting after this -->

<h1><a id="spec-title" name="spec-title">QA Framework Primer</a></h1>

<dl>
<dt>Editors:</dt>
    <dd>Lofton Henderson, CGM Open</dd>
    <dd>Lynne Rosenthal, NIST</dd>
    <dd>Mark Skall, NIST</dd>
<dt>Last Modified:</dt>
    <dd>24 August 2005</dd>
</dl>

<h2><a id="abstract" name="abstract">Abstract</a></h2>

<p><cite>QA Framework Primer </cite> provides
a general orientation to  the QA
Framework. The QA Framework is a set of related documents (a
Recommendation, WG Notes, FAQ, Matrix, and Wiki) whose goal is to improve the
quality of specifications and the implementation of these specifications.
This Primer provides a roadmap to these documents. It introduces and orients
the reader by presenting the QA Framework from three different
prespectives - a document view, a role-based view, and a Working Group
milestones view.</p>

<h2><a id="status" name="status">Status of this document</a></h2>

<p>This document was previously published as a part of <cite>The QA
Handbook</cite> — see <a
href="http://www.w3.org/TR/2004/WD-qa-handbook-20040510/#primer-scenario">section
6 of first published working draft</a> of that document.</p>


<p><cite>QA Framework Primer</cite> is closely linked with and will continue
to be coordinated with the other parts of the QA Framework: <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>; <cite><a
href="http://www.w3.org/TR/qaframe-spec/">QA Framework: Specification
Guidelines</a></cite>; <cite><a href="2005/01/test-faq">Matrix, Test
FAQ.</a></cite>; and some more informal materials about testing in the <a
href="http://esw.w3.org/topic/QA">QA Activity Wiki</a>.</p>

<p>Comments on this document may be emailed to <a
href="mailto:www-qa-wg@w3.org">www-qa-wg@w3.org</a>, the <a
href="http://lists.w3.org/Archives/Public/www-qa-wg/">publicly archived</a>
list of the <a href="http://www.w3.org/QA/WG/">QA Working Group</a>.
Commenters please note that your comments will be <strong>publicly</strong>
archived and available, do not send information that should not be
distributed, such as private data.</p>

<h2><a id="contents1" name="contents1">Table of contents</a></h2>
<ol>
  <li><a href="#Introduction">Introduction</a></li>
  <li><a href="#qaf-document">Document View of the QA Framework</a></li>
  <li><a href="#qaf-role">Role-based View of the QA Framework</a></li>
  <li><a href="#qaf-milestone">Milestones View of the QA Framework</a></li>
  <li><a href="#acknowledgments">Acknowledgments</a></li>
  <li><a href="#History">Change History</a></li>
</ol>
<hr class="rule" />

<h2><a id="Introduction" name="Introduction">1. Introduction</a></h2>

<p><cite>QA Framework Primer</cite> provides a general orientation to
 the QA Framework family of documents. The QA Framework
consists of a set of related documents (a Recommendation, WG Notes, FAQs,
Matrix, and Wiki) whose goal is to improve the quality of specifications and
the implementation of these specifications. This Primer provides a roadmap of
these documents. It introduces and orients the reader by presenting the QA
Framework from three different prespectives - a document view, a role-based
view, and a Working Group milestones view.</p>
<ul>
  <li>The document view provides a brief summary of each QA Framework
    document,</li>
  <li>The role-based view identifies the documents in the QA Framework that
    might interest those who fill the various roles within a Working
  Group,</li>
  <li>The milestones view associates the QA Framework documents with the
    Working Group's progression of events and milestones.</li>
</ul>

<p>By presenting three views into the QA Framework, the reader is able to
read and focus on areas of interest. The reader is encouraged to jump into
any section of the document, reading only those sections relevant to the
reader or reading the Primer in its entirety.</p>

<p>The target audience for this Primer is anyone and everyone with an
interest in finding out about the QA Framework documents - to get a general
understanding of the documents, to identify which documents are of interest,
and/or to know when to use which document.</p>

<div id="Content1">
<p>The QA Framework consists of these existing documents:</p>
<ul>
  <li><cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
    Handbook</a></cite>, a set of good practices that help Working Groups
    improve their deliverables and their schedules</li>
  <li><cite><a href="http://www.w3.org/TR/qaframe-spec/">QA Framework:
    Specification Guidelines</a></cite>, documenting the most important
    points to be addressed when designing and writing a specification</li>
  <li><cite><a href="http://www.w3.org/TR/spec-variability/">Variability in
    Specifications</a></cite>, which models how design decisions affecting
    conformance change the interoperability landscape for a specification</li>
  <li><cite><a href="2005/01/test-faq">Test Development FAQ</a></cite>, which
    provides introductory information about the purpose of testing, how to
    get started, and what the testing process involves.</li>
  <li><a href="http://www.w3.org/QA/TheMatrix">Matrix of W3C
    Specifications</a>, gathers and formalizes QA efforts for W3C
    specifications.</li>
  <li>Various pages in the <a
    href="http://esw.w3.org/topic/QA">QA Wiki</a> on developing a test
  suite</li>
</ul>

<p>In addition to these documents, the QAWorking Group also developed several
templates to help implement good practices described in these documents.
These templates are available (via links) from the QA Framework document to
which it applies. In particular, there is a Charter template available from
the QA Handbook and How to write a Conformance Clause template available from
the Specification Guidelines document.</p>

<h2><a id="qaf-document" name="qaf-document">2. Document View of the <cite>QA
Framework</cite></a></h2>

<p>The following provides an overview of each document in the <cite>The QA
Framework</cite>.</p>
</div>

<h3><a href="http://www.w3.org/TR/qa-handbook/">QA Handbook (QAH)</a></h3>

<p>This document is a non-normative handbook about the process and
operational aspects of certain quality assurance practices of W3C's Working
Groups, with particular focus on testability and test topics. It is a a
practical guide about applying good practices and quality assurance
techniques to WG activities, especially developing Recommendations and test
materials. It flags avoidable pitfalls, and provides techniques, tools, and
templates that will facilitate and accelerate the work of the groups.
Guidance in QAH is supported by real-world stories and examples. QAH will
help Working Groups avoid known pitfalls and benefit from experiences
gathered from the W3C Working Groups.</p>

<h3><a
href="http://www.w3.org/TR/2005/REC-qaframe-spec-20050817/">Specification
Guidelines (Spec GL)</a></h3>

<p>This document is a guide for editors of W3C specifications. It helps W3C
editors write better specifications, by making a specification easier to
interpret without ambiguity and clearer as to what is required in order to
conform. It focuses on how to define and specify conformance. It addresses
the most basic and critical topics with respect to conformance, including how
to convey what is required for an implementation in order to conform. It also
addresses how a specification might allow variation among conforming
implementations. The document presents guidelines or requirements,
supplemented with good practices, examples and techniques. As a result of
using SpecG L, Working Groups will be able to produce specifications that are
precise, easier to interpret without ambiguity or confusion, and clearer as
to what is required in order to conform.</p>

<h3><a
href="http://www.w3.org/TR/2005/WD-spec-variability-20050629/">Variability in
Specifications</a></h3>

<p>This document details some of the most important conformance-related
concepts evoked in the QA Specification Guidelines, by developing some of the
analysis that need to be considered while designing a specification and
providing advanced techniques, particularly for dealing with conformance
variability and complexity. This document analyzes how design decisions of a
specification's conformance model may affect its implementability and the
interoperability of its implementations. To do so, it introduces the concept
of variability - how much implementations conforming to a given specification
may vary among themselves - and presents a set of well-known dimensions of
variability. Its goal is to raise awareness of the potential cost that some
benign-looking decisions may have on interoperability and to provide guidance
on how to avoid these pitfalls by better understanding the mechanisms
affected by variability.</p>

<h3><a href="http://www.w3.org/QA/WG/2005/01/test-faq">Test FAQ</a></h3>

<p>Test FAQ provides introductory information about the purpose of testing,
how to get started, and what the testing process involves. This FAQ primarily
documents what is already considered good testing practice or the norm, but
it also includes a number of advanced testing goals that have not yet been
fully achieved by any Working Group. The Test FAQ document is addressed to
those who develop tests or organize testing efforts. It should also be useful
to those who develop specifications or who run tests. It stresses early
planning for testing, defines what makes a good test and examines test
reporting, publishing and packaging of tests.</p>


<h3><a href="http://www.w3.org/QA/TheMatrix">Matrix of W3C
Specifications</a></h3>

<p>The Matrix is a table of W3C Specifications that are in at least the Last Call
stage (i.e., at Last Call, Candidate Recommendation, Proposed Recommendation
or Recommendation). For each specification entry, a symbol indicates if the
specification has a conformance clause and if there are conformance tools or
test suites associated with the specification.</p>


<h3><a href="http://esw.w3.org/topic/QA">QA Wiki</a></h3>

<p>The Wiki is an open forum that allows anyone to contribute, share and
develop QA ideas, tools and methods at W3C. Anyone is encouraged to add
individual opinions, comments, and solutions to problems directly on the
pages. Topics focus on writing good specifications and building test
suites.</p>


<div id="Content11">
<h2><a id="qaf-role" name="qaf-role">3. Role-based View of the <cite>QA
Framework</cite></a></h2>

<p>This section identifies the parts of the <cite>The QA Framework</cite>
that might interest those who fill the various roles within a Working Group
or that interact with a Working Group. While each part of framework is
targeted itself at a specific principal audience, the various parts might
have somewhat broader interest and applicability.</p>
<dl>
  <dt>all participants</dt>
    <dd>For any (potential) Working Group participant, the <a
      href="qa-handbook.html#Early">early planning and commitment parts</a>
      of <cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
      Handbook</a></cite> might provide helpful context for understanding
      what the group has committed to deliver. Familiarity with the <cite><a
      href="http://www.w3.org/TR/qaframe-spec/">Specification
      Guidelines</a></cite> will be helpful to any participant who actively
      participates in the advancement of the specifications to
    Recommendation.</dd>
  <dt>spec editors and authors</dt>
    <dd>As for all participants, <cite><a
      href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>
      might be interesting, for shedding some light on the context in which
      the group is operating. A good working understanding the Principles and
      Good Practices of <cite>Specification Guidelines</cite>, together with
      its collected examples, tools, and templates, should be a valuable
      resource in choosing document structure, formats, techniques and
      conformance-related designs that will facilitate production of a
      high-quality specification.</dd>
  <dt>Chair</dt>
    <dd>As the person ultimately responsible for all aspects of the group's
      work, a familiarity with the guidance for operations and process of
      <cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
      Handbook</a></cite> should be helpful — Chairs and Staff Contacts
      are the principal intended audience. Some familiarity
      with the advice and guidance of <cite><a
      href="http://www.w3.org/TR/qaframe-spec/">Specification
      Guidelines</a></cite> should be helpful as well, as the Chairs
      ultimately oversees the advancement of the specifications.</dd>
  <dt>Test Task Force participant</dt>
    <dd>Those who are active in building the test materials of the Working
      Group should benefit from reading the <a href="2005/01/test-faq">Test
      FAQ</a> as well as the articles regarding test materials in the <a
      href="http://esw.w3.org/topic/QA">QA Wiki</a>, and from their
      associated examples, techniques, and tools. Because of the close
      dependency of test materials on the functional specifications, a
      familiarity with the <cite><a
      href="http://www.w3.org/TR/qaframe-spec/">Specification
      Guidelines</a></cite> could be useful as well.</dd>
  <dt>Test Task Force lead</dt>
    <dd>The person who manages the Working Group's test suite projects should
      have working understandings of guidance and techniques for
      specifications of <cite><a
      href="http://www.w3.org/TR/qaframe-spec/">Specification
      Guidelines</a></cite> , as well as the main points of the <a
      href="2005/01/test-faq">Test FAQ</a>.</dd>
  <dt>Non-Working Group spec reviewers</dt>
    <dd>Whether from other Working Groups, or the public at large, the
      <cite><a href="http://www.w3.org/TR/qaframe-spec/">Specification
      Guidelines</a></cite> will be helpful to those who review a Working
      Group's specifications, by providing some objective metrics by which to
      measure the specifications.</dd>
  <dt>Reviewers of Activity proposals &amp; charters</dt>
    <dd>For those W3C Members who will be reviewing Activity proposals and
      proposed Working Group charters, and helping to form their Advisory
      Committee Representative's positions, the <a
      href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
      commitment parts</a> of <cite><a
      href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>
      might be helpful in evaluating whether or not the group's attention to
      test and quality deliverables is appropriate and consistent with the
      Working Group's overall program of work.</dd>
</dl>

<h2><a id="qaf-milestone" name="qaf-milestone">4. Milestone View of the
<cite>QA Framework</cite></a></h2>

<p>This section identifies which parts (sections and documents) of the QA Framework are useful during the life of a Working Group. It progresses through significant milestones in a Working Group's life, from writing a Charter through publishing Recommendations, and looks at associated test suite and other quality
assurance activities.</p>

<h3><a id="qa-commit" name="qa-commit">First Step — QA
Commitment</a></h3>

<p>Because QA is properly an integral part of the activities of each Working
Group, each Working Group has to consider and commit to a set of QA
deliverables appropriate to its work. A spectrum of possibilities are
discussed and illustrated in the <a
href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
commitment</a> module of <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>.</p>

<p>If a Working Group is being newly formed, and if the it is able to
anticipate and agree at Charter time on deliverables like test suites, then
it should consider documenting those QA deliverables in its Charter, just as
it does all other deliverables. Again, see the <a
href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
commitment</a> section of <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>. A
Working Group being re-chartered is a similar case to a newly formed group,
although the scope and direction of its work might be clearer.</p>

<p>For an already-chartered Working Group undertaking new test and other
QA-related activities, if these deliverables are not documented in the
Charter already, then there are a couple of options. The <a
href="http://www.w3.org/Consortium/Process">W3C Process Document</a>
describes how to amend a Charter to accommodate significant new deliverables,
if it wishes to take this route.</p>

<h3><a id="setup-proc" name="setup-proc">Set Up Processes and
Logistics</a></h3>

<p>Once the Working Group is off and running, and assuming that it has
planned on some test- or other quality-related deliverables, the next step is
to chose and document the processes and logistics that it will use for its QA
activities. These include such typical details as:</p>
<ul>
  <li>QA point-of-contact;</li>
  <li>"/Test/" home page in the group's Web space;</li>
  <li>test materials email discussion list;</li>
  <li>Working Group process document for QA;</li>
  <li>decisions about test materials licenses.</li>
</ul>

<p>The sections within the <a
href="http://www.w3.org/TR/qa-handbook/#Day-to-day">Day-to-Day operations</a>
module of <cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
Handbook</a></cite> give good-practice advice about how to do this, plus
examples and a handy template for writing a QA Process Document.</p>

<h3><a id="plan-write" name="plan-write">Planning and Writing the
Specification</a></h3>

<p>There is a tight bond between how a specification is written on the one
hand, and on the other hand its testability and its suitability as a basis
for interoperable implementations.</p>

<h4>New specification</h4>

<p><cite><a href="http://www.w3.org/TR/qaframe-spec/">QA Framework:
Specification Guidelines</a></cite> should be applied from very beginning.
Among the key topics that it addresses are:</p>
<ul>
  <li>conformance model;</li>
  <li>normative language usage;</li>
  <li>writing with test assertions;</li>
  <li>optional features, extensibility;</li>
  <li>profiles, levels, conformance claims.</li>
</ul>

<p>Consider the advice of <cite><a
href="http://www.w3.org/TR/qaframe-spec/">QA Framework: Specification
Guidelines</a></cite> even at the stage of planning the structure and
presentation style of the spec. Along with W3C "pubrules" and <cite>W3C
Manual of Style</cite>, spec editors should refer to the spec guidelines
throughout their work, on topics like testable language, clarity,
conciseness.</p>

<h4>New Edition of specification</h4>

<p>A new Edition of the same functional level of a specification is typically
used for incorporation of errata (e.g., <cite>XML 1.0 Second Edition</cite>).
Normally, this should not be considered a good time to align a specification
to <cite>QA Framework: Specification Guidelines</cite> — the changes
associated with such alignment could significantly disrupt and restructure
the specification.</p>

<h4>New Version of specification</h4>

<p>A new Version of the specification refers to a significant functional
change and enhancement. This presents a good opportunity to improve the
testability and implementability of the specification, as just described for
new specifications.</p>

<h3><a id="progress-spec" name="progress-spec">Reviewing and Progressing the
Specification</a></h3>

<p>This stage in the specification's life has two significant aspects:</p>
<ul>
  <li>the advancement of the specification to First Public Working Draft and
    beyond;</li>
  <li>and, the need to synchronize production of test suites and tools more
    closely with the spec progression.</li>
</ul>

<p>When the specification is published in <a href="http://www.w3.org/TR/"
shape="rect">TR space</a>, then W3C Members not participating to the Working
Group and the general public begin to review and comment. It would be
valuable that such reviewers consult and understand the material in <cite>QA
Framework: Specification Guidelines</cite> — it gives and informed set
of evaluation criteria about the conformance, testability, and
interoperability aspects of the specification.</p>

<p>Working Group participants and especially Test Task Force participants
should refer to the good-practice pieces about <a
href="http://www.w3.org/TR/qa-handbook/#gp-sync-deliverables">advancement
criteria and synchronization</a> (between specs and test materials) of
<cite>The QA Handbook</cite>. Projects enter <cite><a
href="/QA/TheMatrix">The Matrix</a></cite> at Last Call Working Draft (if not
sooner). A de-facto process convention is emerging, that there should be
significant conformance test materials before finishing Candidate
Recommendation phase. This timing coordinates with the explicit process
requirement of two interoperable implementations.</p>

<h3><a id="build-tm" name="build-tm">Designing and Building Test
Materials</a></h3>

<p>There are several scenarios for how the Working Group "builds" its
conformance test materials:</p>
<ul>
  <li>design and build test cases and test tools in the group, from
  scratch;</li>
  <li>import completed test materials from an outside group or
  organization;</li>
  <li>assemble the test materials from contributed test cases and
  materials.</li>
</ul>

<p>All these topics are addressed in more details in the <a
href="2005/01/test-faq">Test FAQ</a>.</p>

<h4>Intra-group build</h4>

<p>Before starting the development, the Test Task Force participants would
benefit from a familiarity with the material in the <a
href="2005/01/test-faq">Test FAQ</a> and optionally in the articles of the
<a href="http://esw.w3.org/topic/QA">QA Wiki</a> regarding test suites.
There is useful information for both high-level planning — e.g., does
the Working Group want breadth-first Basic Effectivity or a fully detailed
suite? — as well as specific details for building the individual test
cases.</p>

<p>Another aspect of building test materials is an acceptance procedure for
the individual bits, as they are built. This is addressed in the <a
href="http://www.w3.org/QA/WG/2005/01/test-faq#review">Test FAQ.</a> with
examples of review guidelines..</p>

<h4><a name="AssembleContributions" id="AssembleContributions">Imported test
materials</a></h4>

<p>Several high-quality test suites have been developed outside of the
relevant W3C Working Group, and then transferred to the group, or have been
the result of assembling pieces from various contributions. Groups which are
considering such a transfer should refer to <a
href="http://www.w3.org/TR/qa-handbook/#Acquire">test materials
acquisition</a> module of <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>. Clearly,
the <a href="http://esw.w3.org/topic/TestMaterialsQualities">quality of the
candidate test materials</a> should be carefully assessed, but also, the
licensing issues (as discussed in the <a
href="http://www.w3.org/TR/qa-handbook/#IANAL">QA Handbook</a> needs careful
planning.</p>

<h3><a id="publish-tm" name="publish-tm">Publication of Test
Materials</a></h3>

<p>Typically, a Working Group Test Task Force will want to publish releases
of test materials, particularly as the specification advances through its
final maturity levels (e.g., Proposed Recommendation) towards Recommendation. </p>

<p>One hurdle on the way to publication is legal — deciding and
agreeing on suitable publication licenses. Advice on navigating this
potential quagmire is presented in the <a
href="http://www.w3.org/TR/qa-handbook/#IANAL">licensing module</a> of
<cite>The QA Handbook</cite>.</p>

<p>As soon as the test materials become public, then there is definite need
for a procedure to process challenges to correctness, make determinations,
and appeal decisions.</p>

<p>Publication of test materials often comprises an implicit (or explicit)
invitation for contributions. The considerations described in <a
href="#AssembleContributions">"Imported test materials"</a> are equally
applicable here.</p>

<h3><a id="rec-n-beyond" name="rec-n-beyond">Recommendation stage and
Beyond</a></h3>

<p>When the specification reaches Recommendation, there is typically a
concurrent publication of the test materials. This might be considered a
"final" publication, or ongoing development may still be planned. In any case, a maintenance procedure must be in place for the test
materials. Firstly, there are tie-ins between approved specification errata
and applicability of particular tests. Secondly, there is the above-discussed need for both
challenge/review/appeal processes. Finally, even if the Working Group ceases
active development of test materials, it may want to continue to consider
submissions, and review and integrate them.</p>

<h3><a id="life-after" name="life-after">Life after Working Group</a></h3>

<p>It is possible that the Working Group and its Test Task Force may disband
after its charter expires. The various aspects of this situation are <a
shape="rect" href="http://www.w3.org/TR/qa-handbook/#Life">introduced</a> in
<cite>The QA Handbook</cite>.</p>

<h2><a name="acknowledgments" id="acknowledgments">5. Acknowledgments</a></h2>

<p>The following QA Working Group and Interest Group participants have
contributed  to this document:</p>
<ul>
  <li>Daniel Dardailler (W3C),</li>
  <li>Dimitris Dimitriadis (Ontologicon),</li>
  <li>Karl Dubost (W3C),</li>
  <li>Dominique Hazaël-Massieux (W3C),</li>
  <li>Lofton Henderson (CGM Open),</li>
  <li>Richard Kennedy (Boeing),</li>
  <li>David Marston (IBM Research),</li>
  <li>Patrick Curran (Sun Microsystems),</li>
  <li>Lynne Rosenthal (NIST),</li>
  <li>Mark Skall (NIST),</li>
  <li>Andrew Thackrah (Open Group),</li>
  <li>Olivier Théreaux (W3C),</li>
  <li>Jeremy Carroll (Hewlett-Packard)</li>
</ul>

<h2><a name="History" id="History">6. Change history</a></h2>
<dl>
  <dt>2005-08-18</dt>
    <dd><ul>
        <li>revised Introduction to reflect 3 views</li>
        <li>added document overview section (view 1)</li>
        <li>renamed sections for views 2 and 3</li>
        <li>updated entire document to read better and to reflect changes to
          Framework documents</li>
      </ul>
    </dd>
  <dt>2004-11-04</dt>
    <dd><ul>
        <li>removed TestGL references, since its development has stopped</li>
        <li>referred to Wiki pages when relevant</li>
        <li>updated with ref to Variability in Spec</li>
      </ul>
    </dd>
  <dt>2004-08-30</dt>
    <dd><ul>
        <li>Initial version created by separation from QAH.</li>
      </ul>
    </dd>
</dl>
</div>

<!-- Your content is finishing before this -->
</div>
<!-- Footer -->

<div class="disclaimer">
<a href="http://validator.w3.org/check/referer"><img
src="http://validator.w3.org/images/vxhtml10" alt="Valid XHTML 1.0!"
height="31" width="88" /></a>

<p>Last modified $Date: 2005/09/12 17:45:53 $ by $Author dom
$</p>

<p class="policyfooter"><a rel="Copyright"
href="/Consortium/Legal/ipr-notice#Copyright">Copyright</a> © 2000-2004 <a
href="/"><acronym
title="World Wide Web Consortium">W3C</acronym></a><sup>®</sup> (<a
href="http://www.csail.mit.edu/"><acronym
title="Massachusetts Institute of Technology">MIT</acronym></a>, <a
href="http://www.ercim.org/"><acronym
title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>,
<a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a
href="/Consortium/Legal/ipr-notice#Legal—Disclaimer">liability</a>, <a
href="/Consortium/Legal/ipr-notice#W3C—Trademarks">trademark</a>, <a
rel="Copyright" href="/Consortium/Legal/copyright-documents">document use</a>
and <a rel="Copyright" href="/Consortium/Legal/copyright-software">software
licensing</a> rules apply. Your interactions with this site are in accordance
with our <a href="/Consortium/Legal/privacy-statement#Public">public</a> and
<a href="/Consortium/Legal/privacy-statement#Members">Member</a> privacy
statements.</p>
</div>
</body>
</html>