Note-qa-certif-20020102.html 30.2 KB

Raw Blame History Permalink

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
  <title>Conformance Testing and Certification Model for W3C
  Specifications</title>
  <link rel="stylesheet" type="text/css"
  href="http://www.w3.org/StyleSheets/TR/W3C-NOTE" />
</head>

<body>

<div class="head">
<a href="http://www.w3.org/"><img height="48" width="72" alt="W3C"
src="http://www.w3.org/Icons/w3c_home" /></a>

<h1>Conformance Testing and Certification Model for W3C Specifications</h1>

<h2>W3C Working Draft 2 January 2002</h2>
<dl>
  <dt>This version:</dt>
    <dd><a
      href="http://www.w3.org/QA/2002/01/Note-qa-certif-20020102">http://www.w3.org/QA/2002/01/Note-qa-certif-20020102</a></dd>
  <dt>Latest version:</dt>
    <dd>http://www.w3.org/QA/qa-certif</dd>
  <dt>Previous version:</dt>
    <dd>This is the first draft from the W3C QA Activity. <br />
      It it based on a NIST White Paper entitled: Conformance Testing and
      Certification Model for Software Specifications, by Lynne Rosenthal,
      Mark Skall, and Lisa Carnahan</dd>
  <dt>Editors:</dt>
    <dd>Daniel Dardailler (<a
      href="mailto:danield@w3.org">danield@w3.org</a>)
      <p></p>
    </dd>
</dl>

<p class="copyright"><a
href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a>
©2001 <a href="http://www.w3.org/"><abbr
title="World Wide Web Consortium">W3C</abbr></a><sup>®</sup> (<a
href="http://www.lcs.mit.edu/"><abbr
title="Massachusetts Institute of Technology">MIT</abbr></a>, <a
href="http://www.inria.fr/"><abbr xml:lang="fr" lang="fr"
title="Institut National de Recherche en Informatique et Automatique">INRIA</abbr></a>,
<a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a
href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>,
<a
href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a>,
<a
href="http://www.w3.org/Consortium/Legal/copyright-documents-19990405">document
use</a> and <a
href="http://www.w3.org/Consortium/Legal/copyright-software-19980720">software
licensing</a> rules apply.</p>
<hr />
</div>

<h2><a id="abstract" name="abstract">Abstract</a></h2>

<p>The use of conformity assessment as a means by which buyers and sellers
can communicate requirements will increase as information technology systems
and applications grow more complex. Models for conformance testing and
certification programs are necessary to understand principles and issues that
are essential for successful conformity assessment programs. This paper
presents one such model by identifying key roles, activities and products
involved in any conformance testing and certification program. This model has
been successfully used by NIST in helping private-sector organizations
establish their certification programs.</p>

<h2><a id="status" name="status">Status of this document</a></h2>

<p>This document is a Note, made available by the W3C Quality Assurance
Activity (QAA) for discussion on the QA email discussion list. It is
submitted as to stimulate discussion within the <a
href="http://www.w3.org/QA/IG">W3C Quality Assurance Interest Group</a>
regarding guidance to external organizations that may wish to establish
certification programs.</p>

<p>It may form the basis of a W3C Note and as such may be modified, replaced
or obsoleted by other documents. Publication of this document does not imply
endorsement by the W3C, its membership or its staff. It is inappropriate to
use W3C Working Drafts as reference material or to cite them as other than
"work in progress". Please send comments on the publicly archived list <a
href="mailto:www-qa@w3.org">www-qa@w3.org</a>.</p>

<h2>Table of Content</h2>

<p>1. <a href="#Communicat">Introduction</a></p>

<p>2. <a href="#Conformanc">Conformance Testing and Certification
Model</a></p>

<p>3. <a href="#Roles">Roles</a></p>

<p>4. <a href="#Activities">Activities</a></p>

<p>5. <a href="#Products">Products</a></p>

<p>6. <a href="#Examples">Examples</a></p>

<p>7. <a href="#Conclusion">Conclusion</a></p>

<p><a href="#References">References</a></p>

<p></p>
<hr />

<h2><a name="Communicat" id="Communicat">1. Introduction</a></h2>

<p>As the pervasiveness of information technology increases, so does the
importance of ensuring the quality of products (i.e., software and systems).
Conformance testing is defined in ISO/IEC Guide 2, "as any activity concerned
with determining directly or indirectly that relevant requirements are
fulfilled". The W3C QA activity's goal is to make sure that all W3C
specifications are covered by adequate tools supporting their conformance
testing, but it is not chartered, nor is the W3C itself, to run certification
program. The QA activity on the other hand is interested in understanding
principles and issues that are essential for the development of successful
conformity assessment programs run externally to W3C. The goal of this
document is to describe a model by identifying key roles, activities and
products involved in any conformance testing and certification program.</p>

<h3>Communication between Buyers and Sellers</h3>

<p>In the marketplace, conformance testing provides a vehicle for exchanging
information between buyer and seller. It increases a buyer's (and/or user's)
confidence in a product and its ability to meet their needs. It provides an
independent, objective method for evaluating products and not becoming
locked-into a single vendor. For sellers (and developers), conformance
testing can help to substantiate claims that a product meets the given
specification.</p>

<p>Conformance testing is a means of measuring whether a product faithfully
implements a specification. The level and formality of the testing are
determined by the market - the requirements of the buyer directly or an
organization acting on behalf of a community of buyers, or by regulation
(e.g., safety, health, national security concerns). For example, some
programs may require a very formal testing and certification approach
consisting of independent (i.e., third party), nationally accredited testing
laboratories while others may be more appropriate for self declaration and
demonstration testing.</p>

<p>The sections below describe a generic model for establishing a conformance
testing and certification program. It describes the processes and procedures
for establishing, administering a testing program. While much has changed
regarding conformity assessment given the growth and changes in the software
industry, the conformance and certification model has not. Examples are used
to describe how the model is applied to support the changes in the software
industr</p>

<h2><a name="Conformanc" id="Conformanc">2. Conformance Testing and
Certification Model</a></h2>

<p>It is well recognized that conformance testing and certification is a way
to ensure that "standard-based" products are implemented. The advantage
afforded by testing and certification are fairly obvious: quality products,
competitive markets with more choices, commodity pricing, and less
opportunity to become "locked in" to a particular vendor. Moreover, a testing
and certification program based on well understood and sound principles will
be acceptable and credible to its community of users.</p>

<p>The conformance testing and certification model described herein contains
the fundamental roles, activities, and products that are necessary in
administering and operating a testing and certification program (see Table
1). By adjusting and modifying the various activities, roles and products,
the model can be applied and used in establishing any testing and
certification program. Figure 1 highlights the interactions between the roles
and activities. The model allows for roles, activities and/or products to be
consolidated or further partitioned.</p>

<p> </p>

<table width="90%" cellspacing="0" cellpadding="0" border="2">
  <tbody>
    <tr>
      <th>Roles</th>
      <th>Activities</th>
      <th>Products</th>
    </tr>
    <tr>
      <td>Buyer</td>
      <td>Require Certification</td>
      <td>Specification</td>
    </tr>
    <tr>
      <td>Seller</td>
      <td>Test Implementation</td>
      <td>Implementation Under Test (IUT)</td>
    </tr>
    <tr>
      <td>Test Laboratory (TL)</td>
      <td>Recognize Test Method</td>
      <td>Test Method</td>
    </tr>
    <tr>
      <td>Certificate Issuer</td>
      <td>Recognize Tester</td>
      <td>Test Report</td>
    </tr>
    <tr>
      <td>Control Board (CB)</td>
      <td>Validate Results</td>
      <td>Certification Program Policy</td>
    </tr>
    <tr>
      <td></td>
      <td>Anser Programmatic Queries</td>
      <td>Testing Laboratory Criteria</td>
    </tr>
    <tr>
      <td></td>
      <td>Answer Test Method Queries</td>
      <td>Certificate of Conformance</td>
    </tr>
    <tr>
      <td></td>
      <td>Resolve Test Method Disputes</td>
      <td></td>
    </tr>
    <tr>
      <td></td>
      <td>Validate Conformance</td>
      <td></td>
    </tr>
    <tr>
      <td></td>
      <td>Issue Certificate</td>
      <td></td>
    </tr>
  </tbody>
</table>

<div align="center">
Table 1: Roles, Activities, and Products</div>
<br />
<br />


<div align="center">
<img src="interact.gif" border="0"
title="Figure 1: Interactions Among Roles and Activities"
alt="Interaction among roles and activities" /></div>

<div align="center">
Figure 1: Interaction Among Roles and Activities</div>
<br />
<br />


<p>While actual testing and certification can be carried out by various
organizations, it is essential that there be a centralized sponsor or owner
of the testing and certification program. The sponsor has a fundamental
interest in ensuring the success of the program. Typically, the sponsor
establishes and maintains the conformance testing and certification program.
It assumes responsibility for insuring that the components of the program are
in place and becomes the centralized source for information about the
program. The sponsor may be composed of one or more organizations. Examples
of sponsors are consortia, trade associations, standards groups, or a
government agency. More often than not, the sponsor of the program is also
the Certificate Issuer.</p>

<h2><a name="Roles" id="Roles">3. Roles</a></h2>

<p>To execute the activities of the model, five roles are defined. In the
realization of this model, some roles may be combined and performed by a
single organization or further distributed among several organizations.</p>
<ul>
  <li><em>Buyer</em> requires conformance to the Specification.</li>
  <li><em>Seller</em> builds the product with the intent of meeting the
    conformance requirement of the purchaser. Products that undergo testing
    are called Implementation Under Test (IUT).</li>
  <li><em>Test Laboratory (TL)</em> performs the operational testing of the
    IUT .</li>
  <li><em>Certificate Issuer (CI)</em>, issues a Certificate of Conformance
    for IUTs that have successfully completed the testing process.</li>
  <li><em>Control Board (CB)</em>, resolves dispute and answers queries on
    behalf of the CI.</li>
</ul>

<h3>Buyer</h3>

<p>The Buyer requires that a product be tested for conformance. The buyer
uses the results of the testing to verify that a seller provides a product
that conforms to the specification and meet procurement requirements. In
general, the buyer is the impetus for sellers to undergo conformance testing.
Specifically, if buyers don't demand that a product be tested and show
evidence of that testing, it is most likely that sellers will not undertake
having their products tested.</p>

<h3>Seller</h3>

<p>The Seller or developer uses the conformance tests and undergoes testing
to demonstrate that the product adheres to the specification and thus, meets
established conformance requirements. Additionally, developers may use the
tests to debug their products prior to market</p>

<h3>Test Laboratory</h3>

<p>The Test Laboratory (TL) conducts the conformance testing using the
prescribed test method. The testing is performed on the seller/developer's
product. A TL can be an organization or individual. A TL can be accredited
from a formal accreditation organization such as NIST's National Voluntary
Laboratory Accreditation Program (NVLAP) or recognized by the buyer, seller,
and certificate issuer, as qualified to perform the testing.</p>

<h3>Certificate Issuer</h3>

<p>The Certificate Issuer (CI) is responsible for issuing certificates for
conforming products. The decision to issue a certificate is based on the
testing results and established criteria for issuing certificates</p>

<h3>Control Board</h3>

<p>The Control Board (CB) is an impartial body of experts who function on
behalf of the CI. The CB is responsible for resolving queries and disputes
related to the testing process.</p>

<h2><a name="Activities" id="Activities">4. Activities</a></h2>

<p>The activities comprising the model can be categorized into one of four
areas:</p>
<ul>
  <li><em>Recognition</em> of competent testing laboratories,</li>
  <li><em>Testing</em>with an approved test method,</li>
  <li><em>Testing process</em>,</li>
  <li><em>Resolution</em> of queries and disputes.</li>
</ul>

<h3>Recognition of Competent Testing Laboratory</h3>

<p>A Testing Laboratory (TL) is an entity that provides services to measure,
examine, test, or otherwise assess conformance of an implementation with its
specification. Within the buyer/seller model, a TL can be either a
first-party, (the seller performs the testing), second-party (the buyer
performs the testing), or third-party (an independent organization performs
the testing) testing organization. All three types of testing are used in the
software industry. Often there will be multiple TLs for a conformance testing
and certification program</p>

<p>The Certificate Issuer (CI) as well as Sellers and other interested
parties, must have confidence in the competency of the TL. Competence is
based on three concepts,</p>
<ol>
  <li>the ability to apply the test method correctly,</li>
  <li>the ability to repeat a given test and generate the same results,</li>
  <li>the ability to operate the TL in a manner that maintains objectivity
    and neutrality (obviously, first and second party testing organizations
    are not neutral).</li>
</ol>

<p>The CI defines competence through requirements and criteria. The CI can
then apply the criteria to a TL, determine its level of competency and, if
appropriate, recognize the TL as competent to perform testing. This practical
approach to identifying and recognizing qualified testing organizations is
appropriate when costs, time and efforts do not warrant seeking accreditation
from a formal accreditation organization.</p>

<p>If a more formal and rigorous approach is appropriate, there exists many
accreditation bodies exist that are capable of performing this function. The
National Voluntary Laboratory Accreditation Program (NVLAP) is a NIST
organization that accredits testing organizations based on the requirements
of ISO Guide 17025 and additional subject-matter requirements.</p>

<p>The purpose of the recognition criteria or accreditation is to assure that
TLs are capable and competent to meet the needs of the testing and
certification program. The basic activities to make this determination
include:</p>
<ul>
  <li>proficiency testing - demonstration of a TL's competency to
    successfully perform the conformance testing using the test method,</li>
  <li>on-site assessment - visit by a technical expert to determine
    compliance with the recognition criteria and ensure the TL is a legally
    identifiable organization with staff and resource to discharge their
    duties,</li>
  <li>quality assurance - documentation and practices to ensure technical
    integrity of testing and analyses and adherence to quality practices
    appropriate to the testing and certification program.</li>
</ul>

<p>Additional attributes required of a third-party TL include that it:</p>
<ul>
  <li>ensure that its personnel are free from any commercial, financial and
    other pressures which might adversely affect the quality of their
  work,</li>
  <li>ensure that the protection of sellers' confidential information and
    proprietary rights are protected,</li>
  <li>ensure that sellers are served with impartiality and integrity,</li>
  <li>maintain a functional record keeping system for each seller testing
    process,</li>
  <li>have the adequate facilities and equipment to fulfill the requirements
    of a TL.</li>
</ul>

<h3>Testing with an Approved Test Method</h3>

<p>For a Certificate of Conformance to be meaningful, all implementations
must be tested in the same manner. Testing reflects the essence of technical
requirements of specifications and measures whether a product faithfully
implements the specification. A <em>test method</em> is a defined technical
procedure for performing a test. A test is the technical operation that
consists of the determination of one or more characteristics of a given
product, process or service according to a specified procedure. A test suite
is the collection of tests. Critical to the success of any conformance
testing and certification program is an appropriate and adequate test
method.</p>

<p>An adequate test method is one that provides test results that give enough
information for the CI to be satisfied that conformance can be measured. An
adequate test method meets the requirement of rigor. An appropriate test
method is one that, while adequate, does not place undue requirements on the
IUT and is cost justifiable. If the test method is too expensive to employ
then it will not be used. The definition of adequate and appropriate is left
to the CI to determine.</p>

<h3>Testing Process</h3>

<p>The Testing Process is described in a conformance testing and
certification policy and procedures document. The document identifies the
administrative as well as testing processes.</p>

<p>The testing process initiates with a seller (or anyone desiring to be
tested) contracting with the TL to have an implementation tested for
conformance. The seller and TL negotiate the scope of testing, the cost of
testing, and the timeliness of testing. For a given seller, the TL must not
be in a position to benefit nor suffer (beyond the testing fees) from the
resulting pass or failure of the implementation under test (IUT).</p>

<p>Using the approved Test Method, the TL tests the IUT for conformance and
reports the results in a <em>Test Report</em>. The TL forwards the Test
Report and an indication of pass/fail to the CI. If the IUT successfully
completes all the tests and meets the criteria for issuing certificates, the
CI issues a <em>Certificate of Conformance</em>to the seller. Typically, the
CI maintains and makes available to the public, a listing of products that
have received certificates of conformance.</p>

<h3>Resolution of Queries and Disputes</h3>

<p>Queries and disputes involving the test method, procedures, test results,
and program administration are directed to the Control Board (CB). The
purpose of the CB is to resolve these issues and communicate the decision to
all parties involved. The CB acts on behalf of the CI. A query or dispute can
be initiated by a seller, TL or entity (e.g., developer) at any point in the
testing process. Queries and disputes should contain a statement of the
problem, rationale for dispute, and desired resolution. All matters to be
resolved by the CB should be determined by consensus or as determined by
documented CB policy and procedures.</p>

<p>Additional activities that may be under the auspices of the CB include:</p>
<ul>
  <li>maintain liaison with appropriate standards bodies and test
    laboratories,</li>
  <li>participate in the assessment of TL's seeking recognition status,</li>
  <li>recommend changes to new versions of the test method or test laboratory
    recognition criteria,</li>
  <li>serve as technical advisor to the CI and TLs,</li>
  <li>maintain the test suite,</li>
  <li>control changes to the conformance testing process.</li>
</ul>

<h2><a name="Products" id="Products">5. Products</a></h2>

<p>The following products are used in the model:</p>
<ul>
  <li>Certification Program Policy,</li>
  <li>Testing Laboroatory Criteria,</li>
  <li>Specification,</li>
  <li>Implementation Under Test (IUT)</li>
  <li>Test Method,</li>
  <li>Test Report,</li>
  <li>Certificate of Conformance</li>
</ul>

<h3>Certification Program Policy</h3>

<p>The Certification Program Policy (CPP) defines the certification system.
ISO/IEC Guide 2 defines a certification system as a system having its own
rules of procedure and management for carrying out conformity certifications.
The CPP addresses the following:</p>
<ul>
  <li>responsibilities of the CI,</li>
  <li>responsibilities of the TLs,</li>
  <li>responsibilities of the seller (the IUT owner),</li>
  <li>policy and procedures for test laboratory recognition,</li>
  <li>policy and procedures for testing process,</li>
  <li>policy and procedures for handling queies and disputes,</li>
  <li>complete deinition of the certificate of conformance.</li>
</ul>

<h3>Test Laboratory Criteria</h3>

<p>Testing Laboratory Criteria serves three purposes. The first purpose is to
define the competence and quality-related requirements that a testing
laboratory must possess to be designated as a recognized testing laboratory.
The second purpose is to describe the manner in which the laboratory will be
assessed against the requirements. The third purpose is to show those who
want to use the testing laboratory (e.g., sellers), or those who want to
accept the conformance certificate as evidence of conformance (e.g., buyers)
the rigor under which the testing laboratory operates</p>

<h3>Specification</h3>

<p>First and foremost to conformance testing and certification is the
specification. This paper delineates "standards-based" software specification
from other types of specification. This is because not all specifications can
be objectively tested for conformance. We recognize that not all
"standards-based" specifications can be objectively tested. However objective
measurement (not necessarily conformance testing per se) is usually a goal in
these specification development efforts</p>

<p>If the specification can not be objectively tested, then a alternate
approach to conformance testing should be used to measure whether a produce
faithfully implements the specification. This is because an accepted test
method cannot be developed, thus repeatability and reproducibility cannot be
ensured.</p>

<h3>Implementation Under Test</h3>

<p>The implementation under test (IUT) is the object that is being tested for
conformance. For software specifications it is the software that has
implemented the specification. For any certification program, the scope of
the IUT must be defined and delineated from the rest of the supporting
software and hardware of the total system (referred to as the system under
test). In many current certification programs the hardware that is used by
the software must also be defined. The software and supporting hardware
constitute the IUT and are listed in both the test report and certificate of
conformance.</p>

<h3>Test Method</h3>

<p>The test method must be adequate and appropriate within the conformance
testing and certification program in which it is used. Beyond these
properties, test methods (and thus the tests) should be objective, have
adequate coverage, and correctly implement the specification. In trying to
meet these requirements, those using and applying the test method should not
make the common mistake of allowing the test method to become the
specification. This means that sellers (builders of IUTs) will build the IUT
to pass the conformance tests, rather than building to the specification.</p>

<p>An objective test method allows for test results to be reproducable by the
same testing laboratory and to be repeatable by a different laboratory.
Initially some test methods do not quite achieve a sufficient level of
objectivity. However objectivity should be something that is always strived
for in the development and ongoing refinement of a test method.</p>

<h3>Test Report</h3>

<p>A test report contains the results of the testing effort, along with any
additional information required by the CI. The test report should provide
enough information that, if necessary, the testing effort could be
duplicated. The testing report should contain:</p>
<ul>
  <li>a complete description of the IUT,</li>
  <li>the name of the testing laboratory,</li>
  <li>the signature of a testing laboratory official,</li>
  <li>the date that the testing was completed,</li>
  <li>the name and version number of the test method (and test suite),</li>
  <li>the results of the test method,</li>
  <li>an unambiguous statement indicateing pass or fail.</li>
</ul>

<h3>Certificate of Conformance</h3>

<p>The certificate of conformance is typically a summation of the test
report. Since it is often used in the procurement process, it includes
information most pertinent between the busyer and the seller.</p>

<p>The certificate includes statements made by the CI. These statements
articulate what the CI is asserting as being conformant. Typically these
statements indicate that "this IUT was tested in this environment, on this
day, using this test method: the test results produced were consistent with
expected test results". The certificate also includes the signature of a CI
official.</p>

<h2><a name="Examples" id="Examples">6. Examples</a></h2>

<h3>ATA Computer Graphics Metafile (CGM) Conformance Testing Program</h3>

<p>The <a href="http://www.air-transport.org">Air Transport Association
(ATA)</a> CGM Program was originally established and operated by NIST to
support the ATA 2100 Specification, Graphics Exchange (a.k.a. ATA CGM
profile). The testing program is a critical component of the ATA's program to
represent maintenance manuals in digital form and move to completely on-line
maintenance manuals. Testing is done to ensure that the fidelity and quality
of the digital information is sufficient to satisfy the airline companies'
safety and quality concerns. The program is a means whereby a seller of a CGM
implementation can formally demonstrate conformance to the ATA CGM
profile.</p>

<p>NIST is currently working with the ATA in its assumption of the testing
program. The ATA CGM Conformance Testing Program will consist of recognized
Testing Laboratories to conduct the testing and a Control Board to handle
disputes and serve as an advisor to the ATA. The ATA will act as the sponsor
and administrator of the program. The ATA or an ATA designate will issue
certificates of conformance. The roles, activities, and products as described
in the generic model apply here with little modification. The Control Board
takes on the additional activity of assessing the testing laboratories
according to pre-established criteria. Additionally, the ATA Technical
Information Communication Committee's Graphics Working Group serves as a
technical advisor to both the ATA and the control board.</p>

<p>The test method consists of a NIST developed test suite and test
procedures. The test method has been accepted and used by the community. It
is publicly available along with other program documents.</p>

<h3>IEEE POSIX Validation Service</h3>

<p>The IEEE established a validation service for the POSIX (Portable
Operating System Interface). The <a
href="http://standards.ieee.org/regauth/posix/index.html">IEEE Validation
Service</a> uses accredited POSIX testing laboratories, issue certification
of validated test results, and maintains a register of accredited
laboratories and successfully tested products. The laboratories are
accredited by the NVLAP under its POSIX program</p>

<p>The requirement for testing is buyer driven. Initially, federal agencies
in their requests for procurement (RFP) of POSIX systems required
certificates of validation prior to purchase. However, the benefits of POSIX
testing and its acceptance in the industry has resulted in sellers requesting
to be tested as a matter of course, rather than a procurement requirement.</p>

<p>The test suite was produced in a joint effort between NIST and several
computer vendors. The original testing policy and procedures produced by NIST
have been adopted by the IEEE.</p>

<p></p>
<hr />

<h2><a name="Conclusion" id="Conclusion">7. Conclusion</a></h2>

<p>This model describing the conformance testing and certification process
has been used many times over in certification programs for standards-based
software specifications. The examples above illustrate just a few of these
programs. It will continue to be used as a communication mechanism between
buyers and sellers.</p>

<p>Test method developers must continue to develop test methods that have
adequate coverage with regard to the specification; are well defined in terms
measurement (i.e., what does each test case prove); and be adequate and
appropriate as defined by the Certificate Issuer.</p>

<p>As the industry moves toward component based software, the challenge will
be to develop test methods and associated certification programs that can
provide meaningful measurement in this environment</p>

<h2><a name="References" id="References">References</a></h2>
<dl>
    <dd>Breitenberg, Maureen, The ABC's of the U.S. Conformity Assessment
      System, NISTIR 6014, April 1997.</dd>
    <dd>Breitenberg, Maureen, The U.S. Certification System from a Government
      Perspective, NISTIR 6077, October, 1997.</dd>
    <dd>Carnahan, Lisa, Developing Federal Standards and Accreditations for
      Data Protection Products, Proceeding of SPIE Conference, October,
    1995.</dd>
    <dd>Dashiell, William H., L. Arnold Johnson and Lynne S. Rosenthal,
      Overview of Model for United States Geological Survey Recognition of
      Spatial Data Transfer Standard Certification System, NIST IR 6124, May
      1998.</dd>
    <dd>Horlick Jeffrey, and Lisa Carnahan, Cryptographic Module Testing,
      Handbook 150-17, April, 1995.</dd>
    <dd>ISO/IEC Guide 2: 1996, Standardization and Related Activities:
      General Vocabulary.</dd>
    <dd>ISO/IEC Guide 17025: 1999, General Requirements for the Competence of
      Calibration and Testing Laboratories.</dd>
    <dd>NIST, Derived Test Requirements for FIPS 140-1, Security Requirements
      for Cryptographic Modules, March, 1995.</dd>
    <dd>NIST, Procedures and Requirements, NIST Handbook 150, March 1994.</dd>
</dl>
</body>
</html>