qaframe-primer
29.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>QA Framework Primer</title>
<!-- <link rel="Stylesheet" href="/QA/2002/12/qa4.css" />-->
<link rel="Stylesheet" href="/QA/2005/08/qa-style.css" />
</head>
<body>
<!-- Header -->
<div id="Logo">
<a href="http://www.w3.org/"><img alt="W3C" src="/Icons/WWW/w3c_home" width="72" height="48" /></a>
<a href="http://www.w3.org/QA/"><img alt="QA" src="/QA/images/qa" width="161" height="48" /></a>
<!-- <div id="Header">Be strict to be cool</div> -->
<div><map name="introLinks" id="introLinks" title="Introductory Links">
<div class="banner"> <a
class="bannerLink" title="W3C Activities" accesskey="A"
href="/Consortium/Activities">Activities</a> | <a class="bannerLink"
title="Technical Reports and Recommendations" accesskey="T"
href="/TR/">Technical Reports</a> | <a class="bannerLink"
title="Alphabetical Site Index" accesskey="S"
href="/Help/siteindex">Site Index</a> | <a class="bannerLink"
title="Help for new visitors" accesskey="N"
href="/2002/03/new-to-w3c">New Visitors</a> | <a
class="bannerLink" title="About W3C" accesskey="B"
href="/Consortium/">About W3C</a> | <a class="bannerLink"
title="Join W3C" accesskey="J"
href="/Consortium/Prospectus/Joining">Join W3C</a></div>
</map></div>
</div>
<!-- menuRight -->
<div id="Menu">
<p><a href="#Introduction">Introduction</a><span class="dot">·</span> <a
href="#qaf-audience">QA Framework audience</a><span class="dot">·</span> <a
href="#qaf-primer">Primer</a><span class="dot">·</span></p>
<hr />
<p class="navhead">Nearby:</p>
<p><a href="/QA/"><abbr
title="Quality Assurance">QA</abbr> Homepage</a><span
class="dot">·</span> <a href="/QA/#latest">Latest News</a><span
class="dot">·</span> <a href="/QA/#resources">QA Resources</a><span
class="dot">·</span> <a href="/QA/WG/">QA <abbr
title="Interest Group">WG</abbr></a><span class="dot">·</span> <a
href="/TR/qa-handbook/">QA Handbook</a><span class="dot">·</span> <a
href="/TR/qaframe-spec/">Spec Guidelines</a><span class="dot">·</span></p>
</div>
<!-- content -->
<div id="Content">
<!-- Your content is starting after this -->
<h1><a id="spec-title" name="spec-title">QA Framework Primer</a></h1>
<dl>
<dt>Editors:</dt>
<dd>Lofton Henderson, CGM Open</dd>
<dd>Lynne Rosenthal, NIST</dd>
<dd>Mark Skall, NIST</dd>
<dt>Last Modified:</dt>
<dd>24 August 2005</dd>
</dl>
<h2><a id="abstract" name="abstract">Abstract</a></h2>
<p><cite>QA Framework Primer </cite> provides
a general orientation to the QA
Framework. The QA Framework is a set of related documents (a
Recommendation, WG Notes, FAQ, Matrix, and Wiki) whose goal is to improve the
quality of specifications and the implementation of these specifications.
This Primer provides a roadmap to these documents. It introduces and orients
the reader by presenting the QA Framework from three different
prespectives - a document view, a role-based view, and a Working Group
milestones view.</p>
<h2><a id="status" name="status">Status of this document</a></h2>
<p>This document was previously published as a part of <cite>The QA
Handbook</cite> — see <a
href="http://www.w3.org/TR/2004/WD-qa-handbook-20040510/#primer-scenario">section
6 of first published working draft</a> of that document.</p>
<p><cite>QA Framework Primer</cite> is closely linked with and will continue
to be coordinated with the other parts of the QA Framework: <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>; <cite><a
href="http://www.w3.org/TR/qaframe-spec/">QA Framework: Specification
Guidelines</a></cite>; <cite><a href="2005/01/test-faq">Matrix, Test
FAQ.</a></cite>; and some more informal materials about testing in the <a
href="http://esw.w3.org/topic/QA">QA Activity Wiki</a>.</p>
<p>Comments on this document may be emailed to <a
href="mailto:www-qa-wg@w3.org">www-qa-wg@w3.org</a>, the <a
href="http://lists.w3.org/Archives/Public/www-qa-wg/">publicly archived</a>
list of the <a href="http://www.w3.org/QA/WG/">QA Working Group</a>.
Commenters please note that your comments will be <strong>publicly</strong>
archived and available, do not send information that should not be
distributed, such as private data.</p>
<h2><a id="contents1" name="contents1">Table of contents</a></h2>
<ol>
<li><a href="#Introduction">Introduction</a></li>
<li><a href="#qaf-document">Document View of the QA Framework</a></li>
<li><a href="#qaf-role">Role-based View of the QA Framework</a></li>
<li><a href="#qaf-milestone">Milestones View of the QA Framework</a></li>
<li><a href="#acknowledgments">Acknowledgments</a></li>
<li><a href="#History">Change History</a></li>
</ol>
<hr class="rule" />
<h2><a id="Introduction" name="Introduction">1. Introduction</a></h2>
<p><cite>QA Framework Primer</cite> provides a general orientation to
the QA Framework family of documents. The QA Framework
consists of a set of related documents (a Recommendation, WG Notes, FAQs,
Matrix, and Wiki) whose goal is to improve the quality of specifications and
the implementation of these specifications. This Primer provides a roadmap of
these documents. It introduces and orients the reader by presenting the QA
Framework from three different prespectives - a document view, a role-based
view, and a Working Group milestones view.</p>
<ul>
<li>The document view provides a brief summary of each QA Framework
document,</li>
<li>The role-based view identifies the documents in the QA Framework that
might interest those who fill the various roles within a Working
Group,</li>
<li>The milestones view associates the QA Framework documents with the
Working Group's progression of events and milestones.</li>
</ul>
<p>By presenting three views into the QA Framework, the reader is able to
read and focus on areas of interest. The reader is encouraged to jump into
any section of the document, reading only those sections relevant to the
reader or reading the Primer in its entirety.</p>
<p>The target audience for this Primer is anyone and everyone with an
interest in finding out about the QA Framework documents - to get a general
understanding of the documents, to identify which documents are of interest,
and/or to know when to use which document.</p>
<div id="Content1">
<p>The QA Framework consists of these existing documents:</p>
<ul>
<li><cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
Handbook</a></cite>, a set of good practices that help Working Groups
improve their deliverables and their schedules</li>
<li><cite><a href="http://www.w3.org/TR/qaframe-spec/">QA Framework:
Specification Guidelines</a></cite>, documenting the most important
points to be addressed when designing and writing a specification</li>
<li><cite><a href="http://www.w3.org/TR/spec-variability/">Variability in
Specifications</a></cite>, which models how design decisions affecting
conformance change the interoperability landscape for a specification</li>
<li><cite><a href="2005/01/test-faq">Test Development FAQ</a></cite>, which
provides introductory information about the purpose of testing, how to
get started, and what the testing process involves.</li>
<li><a href="http://www.w3.org/QA/TheMatrix">Matrix of W3C
Specifications</a>, gathers and formalizes QA efforts for W3C
specifications.</li>
<li>Various pages in the <a
href="http://esw.w3.org/topic/QA">QA Wiki</a> on developing a test
suite</li>
</ul>
<p>In addition to these documents, the QAWorking Group also developed several
templates to help implement good practices described in these documents.
These templates are available (via links) from the QA Framework document to
which it applies. In particular, there is a Charter template available from
the QA Handbook and How to write a Conformance Clause template available from
the Specification Guidelines document.</p>
<h2><a id="qaf-document" name="qaf-document">2. Document View of the <cite>QA
Framework</cite></a></h2>
<p>The following provides an overview of each document in the <cite>The QA
Framework</cite>.</p>
</div>
<h3><a href="http://www.w3.org/TR/qa-handbook/">QA Handbook (QAH)</a></h3>
<p>This document is a non-normative handbook about the process and
operational aspects of certain quality assurance practices of W3C's Working
Groups, with particular focus on testability and test topics. It is a a
practical guide about applying good practices and quality assurance
techniques to WG activities, especially developing Recommendations and test
materials. It flags avoidable pitfalls, and provides techniques, tools, and
templates that will facilitate and accelerate the work of the groups.
Guidance in QAH is supported by real-world stories and examples. QAH will
help Working Groups avoid known pitfalls and benefit from experiences
gathered from the W3C Working Groups.</p>
<h3><a
href="http://www.w3.org/TR/2005/REC-qaframe-spec-20050817/">Specification
Guidelines (Spec GL)</a></h3>
<p>This document is a guide for editors of W3C specifications. It helps W3C
editors write better specifications, by making a specification easier to
interpret without ambiguity and clearer as to what is required in order to
conform. It focuses on how to define and specify conformance. It addresses
the most basic and critical topics with respect to conformance, including how
to convey what is required for an implementation in order to conform. It also
addresses how a specification might allow variation among conforming
implementations. The document presents guidelines or requirements,
supplemented with good practices, examples and techniques. As a result of
using SpecG L, Working Groups will be able to produce specifications that are
precise, easier to interpret without ambiguity or confusion, and clearer as
to what is required in order to conform.</p>
<h3><a
href="http://www.w3.org/TR/2005/WD-spec-variability-20050629/">Variability in
Specifications</a></h3>
<p>This document details some of the most important conformance-related
concepts evoked in the QA Specification Guidelines, by developing some of the
analysis that need to be considered while designing a specification and
providing advanced techniques, particularly for dealing with conformance
variability and complexity. This document analyzes how design decisions of a
specification's conformance model may affect its implementability and the
interoperability of its implementations. To do so, it introduces the concept
of variability - how much implementations conforming to a given specification
may vary among themselves - and presents a set of well-known dimensions of
variability. Its goal is to raise awareness of the potential cost that some
benign-looking decisions may have on interoperability and to provide guidance
on how to avoid these pitfalls by better understanding the mechanisms
affected by variability.</p>
<h3><a href="http://www.w3.org/QA/WG/2005/01/test-faq">Test FAQ</a></h3>
<p>Test FAQ provides introductory information about the purpose of testing,
how to get started, and what the testing process involves. This FAQ primarily
documents what is already considered good testing practice or the norm, but
it also includes a number of advanced testing goals that have not yet been
fully achieved by any Working Group. The Test FAQ document is addressed to
those who develop tests or organize testing efforts. It should also be useful
to those who develop specifications or who run tests. It stresses early
planning for testing, defines what makes a good test and examines test
reporting, publishing and packaging of tests.</p>
<h3><a href="http://www.w3.org/QA/TheMatrix">Matrix of W3C
Specifications</a></h3>
<p>The Matrix is a table of W3C Specifications that are in at least the Last Call
stage (i.e., at Last Call, Candidate Recommendation, Proposed Recommendation
or Recommendation). For each specification entry, a symbol indicates if the
specification has a conformance clause and if there are conformance tools or
test suites associated with the specification.</p>
<h3><a href="http://esw.w3.org/topic/QA">QA Wiki</a></h3>
<p>The Wiki is an open forum that allows anyone to contribute, share and
develop QA ideas, tools and methods at W3C. Anyone is encouraged to add
individual opinions, comments, and solutions to problems directly on the
pages. Topics focus on writing good specifications and building test
suites.</p>
<div id="Content11">
<h2><a id="qaf-role" name="qaf-role">3. Role-based View of the <cite>QA
Framework</cite></a></h2>
<p>This section identifies the parts of the <cite>The QA Framework</cite>
that might interest those who fill the various roles within a Working Group
or that interact with a Working Group. While each part of framework is
targeted itself at a specific principal audience, the various parts might
have somewhat broader interest and applicability.</p>
<dl>
<dt>all participants</dt>
<dd>For any (potential) Working Group participant, the <a
href="qa-handbook.html#Early">early planning and commitment parts</a>
of <cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
Handbook</a></cite> might provide helpful context for understanding
what the group has committed to deliver. Familiarity with the <cite><a
href="http://www.w3.org/TR/qaframe-spec/">Specification
Guidelines</a></cite> will be helpful to any participant who actively
participates in the advancement of the specifications to
Recommendation.</dd>
<dt>spec editors and authors</dt>
<dd>As for all participants, <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>
might be interesting, for shedding some light on the context in which
the group is operating. A good working understanding the Principles and
Good Practices of <cite>Specification Guidelines</cite>, together with
its collected examples, tools, and templates, should be a valuable
resource in choosing document structure, formats, techniques and
conformance-related designs that will facilitate production of a
high-quality specification.</dd>
<dt>Chair</dt>
<dd>As the person ultimately responsible for all aspects of the group's
work, a familiarity with the guidance for operations and process of
<cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
Handbook</a></cite> should be helpful — Chairs and Staff Contacts
are the principal intended audience. Some familiarity
with the advice and guidance of <cite><a
href="http://www.w3.org/TR/qaframe-spec/">Specification
Guidelines</a></cite> should be helpful as well, as the Chairs
ultimately oversees the advancement of the specifications.</dd>
<dt>Test Task Force participant</dt>
<dd>Those who are active in building the test materials of the Working
Group should benefit from reading the <a href="2005/01/test-faq">Test
FAQ</a> as well as the articles regarding test materials in the <a
href="http://esw.w3.org/topic/QA">QA Wiki</a>, and from their
associated examples, techniques, and tools. Because of the close
dependency of test materials on the functional specifications, a
familiarity with the <cite><a
href="http://www.w3.org/TR/qaframe-spec/">Specification
Guidelines</a></cite> could be useful as well.</dd>
<dt>Test Task Force lead</dt>
<dd>The person who manages the Working Group's test suite projects should
have working understandings of guidance and techniques for
specifications of <cite><a
href="http://www.w3.org/TR/qaframe-spec/">Specification
Guidelines</a></cite> , as well as the main points of the <a
href="2005/01/test-faq">Test FAQ</a>.</dd>
<dt>Non-Working Group spec reviewers</dt>
<dd>Whether from other Working Groups, or the public at large, the
<cite><a href="http://www.w3.org/TR/qaframe-spec/">Specification
Guidelines</a></cite> will be helpful to those who review a Working
Group's specifications, by providing some objective metrics by which to
measure the specifications.</dd>
<dt>Reviewers of Activity proposals & charters</dt>
<dd>For those W3C Members who will be reviewing Activity proposals and
proposed Working Group charters, and helping to form their Advisory
Committee Representative's positions, the <a
href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
commitment parts</a> of <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>
might be helpful in evaluating whether or not the group's attention to
test and quality deliverables is appropriate and consistent with the
Working Group's overall program of work.</dd>
</dl>
<h2><a id="qaf-milestone" name="qaf-milestone">4. Milestone View of the
<cite>QA Framework</cite></a></h2>
<p>This section identifies which parts (sections and documents) of the QA Framework are useful during the life of a Working Group. It progresses through significant milestones in a Working Group's life, from writing a Charter through publishing Recommendations, and looks at associated test suite and other quality
assurance activities.</p>
<h3><a id="qa-commit" name="qa-commit">First Step — QA
Commitment</a></h3>
<p>Because QA is properly an integral part of the activities of each Working
Group, each Working Group has to consider and commit to a set of QA
deliverables appropriate to its work. A spectrum of possibilities are
discussed and illustrated in the <a
href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
commitment</a> module of <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>.</p>
<p>If a Working Group is being newly formed, and if the it is able to
anticipate and agree at Charter time on deliverables like test suites, then
it should consider documenting those QA deliverables in its Charter, just as
it does all other deliverables. Again, see the <a
href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
commitment</a> section of <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>. A
Working Group being re-chartered is a similar case to a newly formed group,
although the scope and direction of its work might be clearer.</p>
<p>For an already-chartered Working Group undertaking new test and other
QA-related activities, if these deliverables are not documented in the
Charter already, then there are a couple of options. The <a
href="http://www.w3.org/Consortium/Process">W3C Process Document</a>
describes how to amend a Charter to accommodate significant new deliverables,
if it wishes to take this route.</p>
<h3><a id="setup-proc" name="setup-proc">Set Up Processes and
Logistics</a></h3>
<p>Once the Working Group is off and running, and assuming that it has
planned on some test- or other quality-related deliverables, the next step is
to chose and document the processes and logistics that it will use for its QA
activities. These include such typical details as:</p>
<ul>
<li>QA point-of-contact;</li>
<li>"/Test/" home page in the group's Web space;</li>
<li>test materials email discussion list;</li>
<li>Working Group process document for QA;</li>
<li>decisions about test materials licenses.</li>
</ul>
<p>The sections within the <a
href="http://www.w3.org/TR/qa-handbook/#Day-to-day">Day-to-Day operations</a>
module of <cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
Handbook</a></cite> give good-practice advice about how to do this, plus
examples and a handy template for writing a QA Process Document.</p>
<h3><a id="plan-write" name="plan-write">Planning and Writing the
Specification</a></h3>
<p>There is a tight bond between how a specification is written on the one
hand, and on the other hand its testability and its suitability as a basis
for interoperable implementations.</p>
<h4>New specification</h4>
<p><cite><a href="http://www.w3.org/TR/qaframe-spec/">QA Framework:
Specification Guidelines</a></cite> should be applied from very beginning.
Among the key topics that it addresses are:</p>
<ul>
<li>conformance model;</li>
<li>normative language usage;</li>
<li>writing with test assertions;</li>
<li>optional features, extensibility;</li>
<li>profiles, levels, conformance claims.</li>
</ul>
<p>Consider the advice of <cite><a
href="http://www.w3.org/TR/qaframe-spec/">QA Framework: Specification
Guidelines</a></cite> even at the stage of planning the structure and
presentation style of the spec. Along with W3C "pubrules" and <cite>W3C
Manual of Style</cite>, spec editors should refer to the spec guidelines
throughout their work, on topics like testable language, clarity,
conciseness.</p>
<h4>New Edition of specification</h4>
<p>A new Edition of the same functional level of a specification is typically
used for incorporation of errata (e.g., <cite>XML 1.0 Second Edition</cite>).
Normally, this should not be considered a good time to align a specification
to <cite>QA Framework: Specification Guidelines</cite> — the changes
associated with such alignment could significantly disrupt and restructure
the specification.</p>
<h4>New Version of specification</h4>
<p>A new Version of the specification refers to a significant functional
change and enhancement. This presents a good opportunity to improve the
testability and implementability of the specification, as just described for
new specifications.</p>
<h3><a id="progress-spec" name="progress-spec">Reviewing and Progressing the
Specification</a></h3>
<p>This stage in the specification's life has two significant aspects:</p>
<ul>
<li>the advancement of the specification to First Public Working Draft and
beyond;</li>
<li>and, the need to synchronize production of test suites and tools more
closely with the spec progression.</li>
</ul>
<p>When the specification is published in <a href="http://www.w3.org/TR/"
shape="rect">TR space</a>, then W3C Members not participating to the Working
Group and the general public begin to review and comment. It would be
valuable that such reviewers consult and understand the material in <cite>QA
Framework: Specification Guidelines</cite> — it gives and informed set
of evaluation criteria about the conformance, testability, and
interoperability aspects of the specification.</p>
<p>Working Group participants and especially Test Task Force participants
should refer to the good-practice pieces about <a
href="http://www.w3.org/TR/qa-handbook/#gp-sync-deliverables">advancement
criteria and synchronization</a> (between specs and test materials) of
<cite>The QA Handbook</cite>. Projects enter <cite><a
href="/QA/TheMatrix">The Matrix</a></cite> at Last Call Working Draft (if not
sooner). A de-facto process convention is emerging, that there should be
significant conformance test materials before finishing Candidate
Recommendation phase. This timing coordinates with the explicit process
requirement of two interoperable implementations.</p>
<h3><a id="build-tm" name="build-tm">Designing and Building Test
Materials</a></h3>
<p>There are several scenarios for how the Working Group "builds" its
conformance test materials:</p>
<ul>
<li>design and build test cases and test tools in the group, from
scratch;</li>
<li>import completed test materials from an outside group or
organization;</li>
<li>assemble the test materials from contributed test cases and
materials.</li>
</ul>
<p>All these topics are addressed in more details in the <a
href="2005/01/test-faq">Test FAQ</a>.</p>
<h4>Intra-group build</h4>
<p>Before starting the development, the Test Task Force participants would
benefit from a familiarity with the material in the <a
href="2005/01/test-faq">Test FAQ</a> and optionally in the articles of the
<a href="http://esw.w3.org/topic/QA">QA Wiki</a> regarding test suites.
There is useful information for both high-level planning — e.g., does
the Working Group want breadth-first Basic Effectivity or a fully detailed
suite? — as well as specific details for building the individual test
cases.</p>
<p>Another aspect of building test materials is an acceptance procedure for
the individual bits, as they are built. This is addressed in the <a
href="http://www.w3.org/QA/WG/2005/01/test-faq#review">Test FAQ.</a> with
examples of review guidelines..</p>
<h4><a name="AssembleContributions" id="AssembleContributions">Imported test
materials</a></h4>
<p>Several high-quality test suites have been developed outside of the
relevant W3C Working Group, and then transferred to the group, or have been
the result of assembling pieces from various contributions. Groups which are
considering such a transfer should refer to <a
href="http://www.w3.org/TR/qa-handbook/#Acquire">test materials
acquisition</a> module of <cite><a
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>. Clearly,
the <a href="http://esw.w3.org/topic/TestMaterialsQualities">quality of the
candidate test materials</a> should be carefully assessed, but also, the
licensing issues (as discussed in the <a
href="http://www.w3.org/TR/qa-handbook/#IANAL">QA Handbook</a> needs careful
planning.</p>
<h3><a id="publish-tm" name="publish-tm">Publication of Test
Materials</a></h3>
<p>Typically, a Working Group Test Task Force will want to publish releases
of test materials, particularly as the specification advances through its
final maturity levels (e.g., Proposed Recommendation) towards Recommendation. </p>
<p>One hurdle on the way to publication is legal — deciding and
agreeing on suitable publication licenses. Advice on navigating this
potential quagmire is presented in the <a
href="http://www.w3.org/TR/qa-handbook/#IANAL">licensing module</a> of
<cite>The QA Handbook</cite>.</p>
<p>As soon as the test materials become public, then there is definite need
for a procedure to process challenges to correctness, make determinations,
and appeal decisions.</p>
<p>Publication of test materials often comprises an implicit (or explicit)
invitation for contributions. The considerations described in <a
href="#AssembleContributions">"Imported test materials"</a> are equally
applicable here.</p>
<h3><a id="rec-n-beyond" name="rec-n-beyond">Recommendation stage and
Beyond</a></h3>
<p>When the specification reaches Recommendation, there is typically a
concurrent publication of the test materials. This might be considered a
"final" publication, or ongoing development may still be planned. In any case, a maintenance procedure must be in place for the test
materials. Firstly, there are tie-ins between approved specification errata
and applicability of particular tests. Secondly, there is the above-discussed need for both
challenge/review/appeal processes. Finally, even if the Working Group ceases
active development of test materials, it may want to continue to consider
submissions, and review and integrate them.</p>
<h3><a id="life-after" name="life-after">Life after Working Group</a></h3>
<p>It is possible that the Working Group and its Test Task Force may disband
after its charter expires. The various aspects of this situation are <a
shape="rect" href="http://www.w3.org/TR/qa-handbook/#Life">introduced</a> in
<cite>The QA Handbook</cite>.</p>
<h2><a name="acknowledgments" id="acknowledgments">5. Acknowledgments</a></h2>
<p>The following QA Working Group and Interest Group participants have
contributed to this document:</p>
<ul>
<li>Daniel Dardailler (W3C),</li>
<li>Dimitris Dimitriadis (Ontologicon),</li>
<li>Karl Dubost (W3C),</li>
<li>Dominique Hazaël-Massieux (W3C),</li>
<li>Lofton Henderson (CGM Open),</li>
<li>Richard Kennedy (Boeing),</li>
<li>David Marston (IBM Research),</li>
<li>Patrick Curran (Sun Microsystems),</li>
<li>Lynne Rosenthal (NIST),</li>
<li>Mark Skall (NIST),</li>
<li>Andrew Thackrah (Open Group),</li>
<li>Olivier Théreaux (W3C),</li>
<li>Jeremy Carroll (Hewlett-Packard)</li>
</ul>
<h2><a name="History" id="History">6. Change history</a></h2>
<dl>
<dt>2005-08-18</dt>
<dd><ul>
<li>revised Introduction to reflect 3 views</li>
<li>added document overview section (view 1)</li>
<li>renamed sections for views 2 and 3</li>
<li>updated entire document to read better and to reflect changes to
Framework documents</li>
</ul>
</dd>
<dt>2004-11-04</dt>
<dd><ul>
<li>removed TestGL references, since its development has stopped</li>
<li>referred to Wiki pages when relevant</li>
<li>updated with ref to Variability in Spec</li>
</ul>
</dd>
<dt>2004-08-30</dt>
<dd><ul>
<li>Initial version created by separation from QAH.</li>
</ul>
</dd>
</dl>
</div>
<!-- Your content is finishing before this -->
</div>
<!-- Footer -->
<div class="disclaimer">
<a href="http://validator.w3.org/check/referer"><img
src="http://validator.w3.org/images/vxhtml10" alt="Valid XHTML 1.0!"
height="31" width="88" /></a>
<p>Last modified $Date: 2005/09/12 17:45:53 $ by $Author dom
$</p>
<p class="policyfooter"><a rel="Copyright"
href="/Consortium/Legal/ipr-notice#Copyright">Copyright</a> © 2000-2004 <a
href="/"><acronym
title="World Wide Web Consortium">W3C</acronym></a><sup>®</sup> (<a
href="http://www.csail.mit.edu/"><acronym
title="Massachusetts Institute of Technology">MIT</acronym></a>, <a
href="http://www.ercim.org/"><acronym
title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>,
<a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a
href="/Consortium/Legal/ipr-notice#Legal—Disclaimer">liability</a>, <a
href="/Consortium/Legal/ipr-notice#W3C—Trademarks">trademark</a>, <a
rel="Copyright" href="/Consortium/Legal/copyright-documents">document use</a>
and <a rel="Copyright" href="/Consortium/Legal/copyright-software">software
licensing</a> rules apply. Your interactions with this site are in accordance
with our <a href="/Consortium/Legal/privacy-statement#Public">public</a> and
<a href="/Consortium/Legal/privacy-statement#Members">Member</a> privacy
statements.</p>
</div>
</body>
</html>