index.html 15.2 KB

Raw Blame History Permalink

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
 <head>
  <meta http-equiv="content-type" content="text/html;charset=utf-8" />
  <title>Guidelines for writing device independent tests</title>
  <link href="http://www.w3.org/StyleSheets/TR/W3C-WG-NOTE" rel="stylesheet" type="text/css" />
  <style type='text/css'>
  .figure { float: left; padding-right: 1em; }
  h3 { clear: left; }
  .guidelines li {
padding:1em;
background-color:#7fc380;
margin-bottom:0.5em;
  }
  </style>
</head>

<body>

  <div class="head">
  <p><a href="http://www.w3.org/"><img alt="W3C" src="http://www.w3.org/Icons/w3c_home" width="72" height="48" /></a></p>
    <h1>Guidelines for writing device independent tests</h1>
    <h2>W3C Working Group Note 12 May 2009</h2>
    <dl>
      <dt>This version:</dt>
<dd><a href=
"http://www.w3.org/TR/2009/NOTE-di-testing-20090512/">http://www.w3.org/TR/2009/NOTE-di-testing-20090512/</a></dd>
<dt>Latest version:</dt>
<dd><a href=
"http://www.w3.org/TR/di-testing/">http://www.w3.org/TR/di-testing/</a></dd>
      <dt>Editors:</dt>
      <dd>Dominique Hazaël-Massieux, W3C</dd>
      <dd>Carmelo Montanez, National Institute of Standards and Technology</dd>
    </dl><!--begin-include "../copyright.inc"-->

<p class="copyright"><a href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a> &copy; 2008-2009 <a href="http://www.w3.org/"><acronym title="World Wide Web Consortium">W3C</acronym></a><sup>&reg;</sup> (<a href="http://www.csail.mit.edu/"><acronym title="Massachusetts Institute of Technology">MIT</acronym></a>, <a href="http://www.ercim.org/"><acronym title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>, <a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>, <a href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a> and <a href="http://www.w3.org/Consortium/Legal/copyright-documents">document use</a> rules apply.</p>
    <!--end-include-->
    <hr title="Separator for header" />
  </div>

  <h2 class="no-num no-toc" id="abstract">Abstract</h2>

<p>As support for Web technologies grows, it is important that tests writers
develop test suites that will work as well as possible across devices. This
document offers guidance in the form of simple guidelines to follow to create
device-independent tests.</p>


  <h2 class="no-num no-toc" id="status">Status of this document</h2>

<p><em>This section describes the status of this document at the
time of its publication. Other documents may supersede this
document. A list of current W3C publications and the latest
revision of this technical report can be found in the <a href="http://www.w3.org/TR/">W3C technical reports index</a> at
http://www.w3.org/TR/.</em></p>

<p>This is the First Public Working Group Note of the Device Independent Testing Guidelines that represent the consensus of the participants of the  by the <a href="http://www.w3.org/2005/MWI/Tests/">Mobile Web Initiative Test Suite Working Group</a>, part of the <a href=
  "http://www.w3.org/Mobile/">Mobile Web Initiative</a>. The Working Group is seeking feedback on the level of details in which these guidelines should go, and what other guidelines should be included as part of the document.</p>


  <p>Comments on, and discussions of this document can be sent on the (<a href=
  "http://lists.w3.org/Archives/Public/public-mwts/">archived</a>) public mailing list <a href=
  "mailto:public-mwts@w3.org">public-mwts@w3.org</a> (see <a href="http://www.w3.org/Mail/Request">instructions</a>).</p>


<p>
Publication as
      a
 Working Group Note
does not imply endorsement by the W3C
Membership. This is a draft document and may be updated, replaced or
obsoleted by other documents at any time. It is inappropriate to cite
this document as other than work in progress.
</p>


<p>This document was produced by a group operating under the <a href="http://www.w3.org/Consortium/Patent-Policy-20040205/">5
  February 2004 W3C Patent Policy</a>.


    W3C maintains a <a rel="disclosure" href="http://www.w3.org/2004/01/pp-impl/40010/status">public list of any
    patent disclosures</a> made in connection with the
    deliverables of the group; that page also includes
    instructions for disclosing a patent.
  An individual who has actual knowledge of a patent which the
  individual believes contains <a href="http://www.w3.org/Consortium/Patent-Policy-20040205/#def-essential">Essential Claim(s)</a> must disclose the
  information in accordance with <a href="http://www.w3.org/Consortium/Patent-Policy-20040205/#sec-Disclosure">section 6 of the W3C Patent Policy</a>.
</p>


<h2 id="toc">Table of Contents</h2>
<ul>
<li><a href="#overview">1. Introduction</a>
<ul><li><a href="#existing">1.1 Existing Work</a></li>
<li><a href="#limitations">1.2 Device Limitations</a></li>
<li><a href="#target">1.3 Target Devices</a></li>
</ul></li>
<li><a href="#guidelines">2. Guidelines</a>
<ul>
<li><a href="#screen">2.1 Screen Limitations</a></li>
<li><a href="#memory">2.2 Memory Limitations</a></li>
<li><a href="#network">2.3 Network Bandwidth, Latency and Cost</a></li>
<li><a href="#cpu">2.4 CPU Power</a></li>
<li><a href="#ext">2.5 Extension Capabilities</a></li>
<li><a href="#keyboard">2.6 Keyboard or Pointing Device Access</a></li>
<li><a href="#prereq">2.7 Prerequisites</a></li></ul>
</li>
<li><a href="#ref">3. References</a></li>
</ul>

  <h2 id="overview">1. Introduction</h2>
  <p>This document provide a set of guidelines for writing test cases that can be run effectively across devices, in particular on mobile devices.</p>

  <h3 id="existing">1.1 Existing Work</h3>

  <p>The <a href="http://www.w3.org/TR/2003/NOTE-acdi-20030901/">Authoring Challenges for Device Independence</a> [<a href="#ACDI">ACDI</a>] explore these different limitations in the general context of writing device independent content, and the <a href="http://www.w3.org/TR/mobile-bp/">Mobile Web Best Practices 1.0</a> [<a href="#MWBP">MWBP</a>] give specific guidance on writing content targeted at mobile devices.</p>

<p>The <a href="http://www.w3.org/Style/CSS/Test/guidelines.html">CSS2.1 Test Case Authoring Guidelines</a> [<a href="#CSSTCAG">CSSTCAG</a>] provides guidance on how to write test cases, and sets device-independence as a goal. The <a href="http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#TestReviewGuidelines">SVG Test Suite Manual</a> [<a href="#SVGTS">SVGTS</a>] also offers advices on writing test cases.</p>

  <p>Inspired by this existing work, and based on the experience gathered by the Mobile Web Test Suites Working Group while reviewing test cases and their fitness to mobile devices [<a href="#MWTSSURVEY">MWTSSURVEY</a>], this document explores the specific aspects to take into account when writing test cases to ensure greater device independence.</p>


  <h3 id="limitations">1.2 Devices Limitations</h3>


<p>Consider recording the browser identifier with the widely implemented
<code>window.navigator.userAgent</code>, as there might be several browsers
on one device.</p>

<p>When designing device-independent test cases, it is important to
acknowledge the limitations of most devices:</p>

  <ol>
    <li>Screen</li>
    <li>Memory available</li>
    <li>Network bandwidth, latency and cost</li>
    <li>CPU power</li>
    <li>Extensions capabilities</li>
  </ol>

<p>For tests that require interaction (either for running the test or for submitting results), consider:</p>
  <ol>
<li>Keyboard or pointing device access and ease of use</li>
<li>Human cost of correctly submitting the results</li>
<li>Automatic start of the test automatically (e.g. through an onload event)</li>
  </ol>

  <h3 id="target">1.3 Target Devices</h3>
  <p>Trying to take into account all the possible constraints of all possible devices would make writing device-independent tests impossible.</p>
  <p>The first step towards writing device-independent test cases is thus to determine the range of devices on which the test cases would need or are likely to be run. To make that assessment:</p>
  <ol>
    <li>if the technology is already deployed, consider on what devices it is running</li>
    <li>if the technology can only be deployed on devices with a certain level of hardware characteristics, adapt the constraints in the guidelines below to that level</li>
    <li>if it is not possible to create a single test that will work well across devices, consider creating several versions of the test, or using server-based content adaptation to adapt it based on the requesting devices.</li>
  </ol>

  <p>In any case, it is always a good idea to document in the test suites what are the minimal expected requirements to run the tests, and possibly which test cases require a certain level of support.</p>

  <h2 id="guidelines">2. Guidelines</h2>

  <h3 id="screen">2.1 Screen Limitations</h3>

  <p>Screen size matters when designing visual test cases - e.g. where the tester needs to assess whether the rendering of a test cases matches a reference rendering.</p>

  <p>Across devices, the following screen parameters vary widely:</p>
  <ol>
    <li>screen resolution</li>
    <li>page zooming capability, scrolling</li>
    <li>physical dimensions</li>
    <li>number of colors</li>
    <li>contrast</li>
  </ol>

  <p>In general, to avoid problems when running test across devices:</p>
  <ol class='guidelines'>
    <li>Keep your tests as simple as possible. Do not include any visible meta data that might obfuscate or get in the way of the result. Stating prerequisites, pass conditions and test result is all that is needed.</li>
    <li>Avoid tests based on absolute dimensions, or provide several versions for different screen resolutions.</li>
    <li>Keep non-essential content off the top of the rendered content to avoid scrolling.</li>
    <li>Be as concise as possible to avoid scrolling.</li>
    <li>When using colors to express the result of the test, convey the result also with text.</li>
    <li>On color-based tests, use well-contrasted colors for parts that need to be distinguished.</li>
  </ol>

  <div class='figure'>
  <p>Bad test:</p>
  <p><img src='bad-test.png' alt='Image depcting how a test from the CSS1 test suite looks on a mobile browser. Due to the amount of visible meta data included, it is impossible to see the actual test result.' /></p>
  </div>

  <div class='figure'>
  <p>Good test:</p>
  <p><img src='good-test.png' alt='Image depicting a simple CSS test case consisting of a pass condition (“There should be a green block under this paragraph”) and a green block indicating that the test has passed.' /></p>
  </div>


  <h3 id="memory">2.2 Memory Limitations</h3>

  <p>To avoid hitting memory limitations of the devices on which the technology under test run:</p>
  <ol class='guidelines'>
    <li>Keep the number of style sheets, images, objects or scripts to a minimum.</li>
    <li>On markup based documents, keep the document object model (DOM) tree as small as possible.</li>
    <li>Keep the number of data structure created dynamically (e.g. by scripts) to a minimum.</li>
  </ol>

  <h3 id="network">2.3 Network Bandwidth, Latency and Cost</h3>

  <p>The characteristics of network access across devices vary greatly, in particular in terms of bandwidth available, the latency induced by network requests, and the cost of transferring content over the network.</p>

  <p>To cater for test environments where network is slow or costly:</p>
  <ol class='guidelines'>
    <li>Keep the number of external resources loaded along with the test case to a minimum; prefer including the content (e.g. using internal style sheets instead of external ones) rather than loading it, except if it can be cached and re-used across test cases.</li>
    <li>Keep the number of network requests originated from scripts to a minimum to speed up data transfer.</li>
    <li>Take care when triggering DOM operations that they will not require downloading DTDs.</li>
      <li>Be aware that mobile networks are can be aggressively cached or transformed, therefore you might need to adjust originator HTTP headers depending on the nature of your test.</li>
      <li>Eliminate any white space in your tests.</li>
  </ol>

  <h3 id="cpu">2.4 CPU Power</h3>
  <p>CPU-intensive operations on mobile devices can drain the battery, and are likely to be much slower than on larger hardware. As a result:</p>
  <ol class='guidelines'>
    <li>Avoid unneeded images processing operations, such as scaling up or down raster images.</li>
  </ol>

  <h3 id="ext">2.5 Extension Capabilities</h3>
  <p>Most mobile devices have limited capabilities when it comes to plugins, additional fonts, or software extensions in general. As a result:</p>
  <ol class="guidelines">
    <li>Avoid relying on effects based on specific fonts.</li>
    <li>Avoid tests that rely on a specific plugin to be installed.</li>
  </ol>

  <h3 id="keyboard">2.6 Keyboard or Pointing Device Access</h3>
  <p>Many mobile devices don't provide a full keyboard, and thus require several key press to enter a given character.</p>
  <ol class="guidelines">
       <li>Minimise interactive tests</li>

      <li>Provide helpers to reach the test suites using short URLs, 2D-barcodes, etc.</li>
    <li>If possible, avoid making your tests dependent on any specific input device.</li>

      <li>Provide simple navigation among test cases</li>
  </ol>


  <h3 id="prereq">2.7 Prerequisites</h3>
  <p>A number of tests <em>do</em> require specific features to be available for the test to work. These requirements could be specific input devices or device APIs, XmlHttpRequest, SVG or similar.</p>

  <ol class="guidelines">
    <li>If your test depends on any specific feature to be available on the tested device, you should explicitly state so before the pass condition.</li>
  </ol>

<h2 id="ref">3. References</h2>

<dl>
<dt id="ACDI">ACDI</dt>
<dd>Authoring Challenges for Device Independence, Rhys Lewis, Editor, W3C Working Group Note, 1 September 2003 (See <a href="http://www.w3.org/TR/2003/NOTE-acdi-20030901/">http://www.w3.org/TR/2003/NOTE-acdi-20030901/</a>)</dd>
<dt id="CSSTCAG">CSSTCAG</dt>
<dd>CSS2.1 Test Case Authoring Guidelines,     Tantek Çelik, Ian Hickson, Elika J. Etemad, Editors (See <a href="http://www.w3.org/Style/CSS/Test/guidelines.html">http://www.w3.org/Style/CSS/Test/guidelines.html</a>)</dd>
<dt id="MWTSSURVEY">MWTSSURVEY</dt>
<dd>Conformance Test Suites for mobile web technologies (See <a href="http://www.w3.org/2005/MWI/Tests/matrix">http://www.w3.org/2005/MWI/Tests/matrix</a>)</dd>
<dt id="MWBP">MWBP</dt>
<dd>Mobile Web Best Practices 1.0, Basic Guidelines, Jo Rabin, Charles McCathieNeville, Editors, W3C Recommendation, 29 July 2008 (See <a href="http://www.w3.org/TR/mobile-bp/">http://www.w3.org/TR/mobile-bp/</a>)</dd>
<dt id="SVGTS">SVGTS</dt>
<dd>SVG Test Suite Manual, Lofton Henderson, Editor, 1 September 2001 (See <a href="http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#TestReviewGuidelines">http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm#TestReviewGuidelines</a>)</dd>
<dt id="testfaq">TESTFAQ</dt>
<dd>Test Development FAQ, Patrick Curran, Editor (See <a href="http://www.w3.org/QA/WG/2005/01/test-faq">http://www.w3.org/QA/WG/2005/01/test-faq</a>)</dd>
</dl>

</body>
</html>