how_do_we_test_a_web_browser.html 17.4 KB

Raw Blame History Permalink

<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <style type="text/css" media="all">
    @import "/QA/2006/01/blogstyle.css";
    </style>
    <meta name="keywords" content='api, css, html, html5, svg, test, test suite' />
    <meta name="description" content="Testing all possible Web browsers out there is hard and requires more effort than one organization can afford by itself. The idea of increasing the level of Web browser testing done in W3C is to involve the community at large as much as possible. If we really want an interoperable Web, that's what W3C should move to." />
    <meta name="revision" content="$Id: how_do_we_test_a_web_browser.html,v 1.35 2011/12/15 23:20:55 mirror Exp $" />
   <link rel="alternate" type="application/atom+xml" title="Atom" href="http://www.w3.org/QA/atom.xml" />
   <link rel="alternate" type="application/rss+xml" title="RSS 1.0" href="http://www.w3.org/QA/news.rss" />
   <title>How do we test a Web browser? - W3C Blog</title>

   <link rel="start" href="http://www.w3.org/QA/" title="Home" />
   <link rel="prev" href="http://www.w3.org/QA/2009/09/linked_government_data.html" title="Linked Government Data" />
   <link rel="next" href="http://www.w3.org/QA/2009/09/first_ever_developer_gathering.html" title="First Ever Developer Gathering during W3C Technical Plenary Week" />

   <!--
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/"
         xmlns:dc="http://purl.org/dc/elements/1.1/">
<rdf:Description
    rdf:about="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html"
    trackback:ping="http://www.w3.org/QA/sununga/mt-tb.cgi/284"
    dc:title="How do we test a Web browser?"
    dc:identifier="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html"
    dc:subject="HTML"
    dc:description="Testing all possible Web browsers out there is hard and requires more effort than one organization can afford by itself. The idea of increasing the level of Web browser testing done in W3C is to involve the community at large as much as possible. If we really want an interoperable Web, that&apos;s what W3C should move to."
    dc:creator="Philippe Le Hégaret"
    dc:date="2009-09-17T21:51:53+00:00" />
</rdf:RDF>
-->

    <!-- <script type="text/javascript" src="http://www.w3.org/QA/mt.js"></script>-->

</head>
<body class="layout-one-column">
      <div id="banner">
      <h1 id="title">
	<a href="http://www.w3.org/"><img height="48" alt="W3C" id="logo" src="http://www.w3.org/Icons/WWW/w3c_home_nb" /></a>
W3C Blog
</h1>
    </div>

    <ul class="navbar" id="menu">
        <li><strong><a href="/QA/" title="W3C Blog Home">[ W3C Blog ]</a></strong></li>
        <li><a href="/QA/Library/" title="Documents and Publications on Web and Quality">Documents</a></li>
        <li><a href="/QA/Tools/" accesskey="3" title="Validators and other Tools">Tools</a></li>
        <li><a href="/2007/12/qa-blog-help/index#feedback">Feedback</a></li>
    </ul>
<div id="searchbox">
<form method="get" action="http://www.google.com/custom" enctype="application/x-www-form-urlencoded">
<p id="formbox"><input type="text" size="15" class="textfield" name="q" accesskey="E" maxlength="255" /> <input type="submit" class="submitfield" value="Search" id="goButton" name="sa" accesskey="G" /> <input type="hidden" name="cof" value="T:black;LW:72;ALC:#ff3300;L:http://www.w3.org/Icons/w3c_home;LC:#000099;LH:48;BGC:white;AH:left;VLC:#660066;GL:0;AWFID:0b9847e42caf283e;" /><input type="hidden" id="searchW3C" name="sitesearch" checked="checked" value="www.w3.org/QA" /><input type="hidden" name="domains" value="www.w3.org/QA" /></p>
</form>
</div>


    <div id="main"><!-- This DIV encapsulates everything in this page - necessary for the positioning -->

                     <p class="content-nav">
                        <a href="http://www.w3.org/QA/2009/09/linked_government_data.html">&laquo; Linked Government Data</a> |
                        <a href="http://www.w3.org/QA/">Main</a>
                        | <a href="http://www.w3.org/QA/2009/09/first_ever_developer_gathering.html">First Ever Developer Gathering during W3C Technical Plenary Week &raquo;</a>
                     </p>

                        <h2 class="entry-header">How do we test a Web browser?</h2>
                           <div class="entry-body">

  <p>The idea started with the fact that we have a number of Working Groups who
 are trying to review the way they do testing, but also
 increase the number of tests they are doing as well.</p>

 <p> The CSS Working Group was foremost in mind when it comes to testing. The
 Group has several documents in Candidate Recommendation stage that are waiting tests and
 testing. The HTML Working Group is starting to look into testing as well
 and a key component of ensure the proper success of HTML 5 is through
 testing. The specification is quite big to say the least and,
 when it comes to testing, it's going to require a lot of work.
 We also have more and more APIs within the Web Apps group, Device API,
 Geolocation, etc. The SVG Working Group has a test suite for
 1.2, but they're looking at different ways of testing as well.
 The framework produced by the MWI Test Suites framework allow two methods. One
 requires a human to look at it and select pass/fail. The other
 one is more suitable for script tests, ie APIs testing.</p>

 <p> A bunch of us, namely Mike Smith, Fantasai, Jonathan Watt, Doug
 Schepers, and myself, decided to get together to discuss this
 and figure out how to improve the situation. We focused on three
 axes: test submissions, test reviews and how to run a test.</p>

 <p> First, we'd like ideally every single Web author to be able to
 submit tests, so when they run into a browser bug based on a
 specification, it should be easy for them to submit a test to
 W3C. It should also allow browser vendors to submit thousands of
 tests at once. There is the question of how much metadata do you
 require when submitting a test. For example, we do need to know
 at some point which feature/part of a spec is being tested. We
 should also as many format as possible for tests. Reftests,
 mochitests, DOM-only tests, human tests, etc. The importance
 aspect here is to be able to run those tests on many
 platforms/browsers as possible. A test format that can only be
 ran on one browser is of no use for us.</p>

 <p> Once a test has been submitted, it needs to be reviewed. The
 basic idea behind improving test reviews is to allow more
 individuals to contribute. The resources inside W3C aren't
 enough to review ten of thousands of tests. We need to involve
 the community at large by doing crowd reviews. It will allow the
 working groups to only focus on the controversial tests.</p>

 <p> Once the test got reviewed, we need to run them on the browsers, as many
 as possible. Human tests for example are easy to run on all of
 them, but it does require a lot of humans. Automatic layout
 tests are a lot trickier, especially on mobiles. We focused on
 one method during our gathering: screenshot based approach. The basic
 idea here is that a screenshot of the page is compared to a
 reference. Mozilla developed a technology called ref-tests that
 compares Web pages themselves. You write two pages differently that are
 supposed the exact same rendering and compare their
 screenshots. It avoids a lot of cross-platforms issues one can.
 The way Mozilla is doing that is via the mozPaint API in debug
 mode. That works well, but only works in Mozilla. You can guess
 that other browser vendors have a similar to automatically take
 screenshots as well. We wanted to find a way to do this with all
 browsers without forcing them or us to write significant amounts
 of code. We found a Web site called browsertests.org and we got
 in touch with that Sylvain Pasche and, with his help, we started
 to make some improvements on his application. It works well on
 desktops at least. Once again, we don't think W3C is big enough
 to replicate all types of browser environments, so we should
 make it easy for people to run the tests in their browser and
 report the results back to us. Plenty of testing frameworks have
 been done already and we should try to leverage them as much as
 possible.</p>

 <p> We started to set up a database for receiving the tests and
 their results. We'd like to continue the efforts on the
 server/database side, as well as continuing to improve Sylvain's
 application, allowing more tests methods and formats. Testing
 the CSS or HTML5 parser should be allowed for example.</p>

 <p> You'll find more information at our <a href="http://omocha.w3.org/">unstable server</a> but keep in mind that:</p>
<ol><li>we're in the very early stages</li>
<li>this server is a temporary one that I managed to steal for a
    few days from our system folks. They'll want it back one of
    those days and I need to find a more stable home prior to
    that event. I'll update the link once this happens but expect it to break if you bookmark it.</li>
<li>Unless I can secure more resources for the project, we won't
    go far by ourselves.</li>
</ol>

 <p>The server also contains links to more resources on the Web related to
 various testing efforts, as well as a more complete of what we
 wish the testing framework to accomplish.</p>

<p> For the conclusion, I'd like to thank Mike Smith and Doug
 Schepers, and especially Jonathan Watt and Fantasai from the
 Mozilla Foundation. They all accepted to argue and code for 8
 days around the simple idea of improving the state of testing at
 W3C. I hope we're going to be able to take this project off the
 ground in the near future. If you're interested in contributing,
 got ideas and time, don't hesitate to contact me.</p>

                           </div>
                           <div id="more" class="entry-more">

                           </div>
                       <p class="postinfo">Filed by <a href="http://www.w3.org/People/LeHegaret/">Philippe Le Hégaret</a> on September 17, 2009  9:51 PM in <a href="http://www.w3.org/QA/archive/technology/css/">CSS</a>, <a href="http://www.w3.org/QA/archive/technology/html/">HTML</a>, <a href="http://www.w3.org/QA/archive/technology/svg/">SVG</a>, <a href="http://www.w3.org/QA/archive/w3cqa_news/tools/">Tools</a><br />
<span class="separator">|</span> <a class="permalink" href="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html">Permalink</a>
                                 | <a href="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html#comments">Comments (4)</a>
                                 | <a href="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html#trackback">TrackBacks (0)</a>
</p>


<h3 class="comments-header" id="comments">Comments</h3>
<div class="comment" id="comment-183584">
<p class="comment-meta" id="c183584">
<span class="comment-meta-author"><strong>Doug Schepers </strong></span>
<span class="comment-meta-date"><a href="#c183584">#</a> 2009-09-17</span>
</p>
<div class="comment-bulk">
<p>I also want to thank the other participants of the SVG WG here... jwatt and myself were the only ones in the group who could attend the Test Hackathon, but Anthony Grasso, Chris Lilley, Cameron McCormack, and Erik Dahlström have also been actively involved in the process of improving our test framework most recently, and the rest of the SVG WG laid the groundwork in years past.</p>

<p>We know from first hand experience how time consuming and draining building a comprehensive test suite is, and we are confident that with more community participation, and more direct implementer contributions, the process of bringing features to the market (and the hands of developers) will be much faster and smoother.</p>

<p>The more complete the test suite, the easier it is to implement (and implement right), and the better the interoperability will be.  This is a high priority for the SVG WG, and we are eager to use what comes out of this effort.</p>

</div>
</div>


<div class="comment" id="comment-183608">
<p class="comment-meta" id="c183608">
<span class="comment-meta-author"><strong>Chris Macdonald </strong></span>
<span class="comment-meta-date"><a href="#c183608">#</a> 2009-09-19</span>
</p>
<div class="comment-bulk">
<p>In the smaller unexpanded text description I think this change should be made to be more easily read. From</p>

<p>"Testing all possible of Web browsers out there is hard and require more effort than one organization can afford by itself. The idea increasing the level of Web browser testing done in W3C is to involve the community at large as much as possible. If we really want an interoperable Web, that's what W3C should move to."</p>

<p>to...</p>

<p>Testing all possible Web browsers out there is hard and requires more effort than one organization can afford by itself. The idea of increasing the level of Web browser testing done in W3C is to involve the community at large as much as possible. If we really want an interoperable Web, that's what W3C should move to.</p>

</div>
</div>


<div class="comment" id="comment-186928">
<p class="comment-meta" id="c186928">
<span class="comment-meta-author"><strong>Emma </strong></span>
<span class="comment-meta-date"><a href="#c186928">#</a> 2010-02-15</span>
</p>
<div class="comment-bulk">
<p>This sounds like a huge project - do you really think it's feasible?</p>

</div>
</div>


<div class="comment" id="comment-222415">
<p class="comment-meta" id="c222415">
<span class="comment-meta-author"><strong>Shanti Bhushan </strong></span>
<span class="comment-meta-date"><a href="#c222415">#</a> 2011-03-24</span>
</p>
<div class="comment-bulk">
<p>how can i contribute in this? please do mail the process once you finalize the execution.
Already i am in to browser testing...
i feel 1st of all basic functionality of the browser should be tested, then we should move for w3c standard testing like HTM5 ,Dom etc. </p>

</div>
</div>


  <div class="comments-open" id="comments-open">
<h3 class="comments-open-header">Leave a comment</h3>

<div class="comments-open-moderated">
   <p>
   Note: this blog is intended to foster <strong>polite
   on-topic discussions</strong>. Comments failing these
   requirements and spam will not get published. Please,
   enter your real name and email address. Every
   individual comment is reviewed by the W3C staff.
   This may take some time, thank you for your patience.
   </p>
   <p>
   You can use the following HTML markup (a href, b, i,
   br/, p, strong, em, ul, ol, li, blockquote, pre)
   and/or <a href="http://daringfireball.net/projects/markdown/syntax">Markdown syntax</a>.</p>
</div>

<div id="comments-open-data">
<form method="post" action="http://www.w3.org/QA/sununga/beach.pl" id="comments-form">
<h4>Your comment</h4>
<div id="comments-open-text">
  <textarea id="comment-text" name="text" rows="20" cols="100"></textarea><br />
<label for="comment-text">Write your comment text here. Remember, keep the discussion on topic and courteous.</label>
</div>

<h4>About you</h4>
<div id="comment-form-name">
  <input type="hidden" name="static" value="1" />
<input type="hidden" name="entry_id" value="6478" />
<input type="hidden" name="__lang" value="en" />
<label for="comment-author">Your Name</label>
<input id="comment-author" name="author" size="30" value="" />
</div>
<div id="comment-form-email">
<label for="comment-email">Your Email Address</label>
<input id="comment-email" name="email" size="30" value="" />
</div>

<div id="comments-open-footer">
<input type="submit" accesskey="s" name="post" id="comment-submit" value="Submit" />

</div>
</form>
</div>
</div>


<p id="gentime">This page was last generated on $Date: 2011/12/15 23:20:55 $</p>

      </div><!-- End of "main" DIV. -->

<address>

This blog is written by W3C staff and working group participants,<br />
&nbsp;and maintained by <a href="/People/CMercier/">Coralie Mercier</a>.<br />
Authorized parties may <a href="/QA/new">log in</a> to create a new entry.<br/>
<span id="poweredby">Powered by Movable Type, magpierss and a lot of Web Technology</span>
    </address>


    <p class="copyright">
      <a rel="Copyright" href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a> &copy; 1994-2011
      <a href="http://www.w3.org/"><acronym title="World Wide Web Consortium">W3C</acronym></a>&reg;
      (<a href="http://www.csail.mit.edu/"><acronym title="Massachusetts Institute of Technology">MIT</acronym></a>,
      <a href="http://www.ercim.eu/"><acronym title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>,
      <a href="http://www.keio.ac.jp/">Keio</a>),
      All Rights Reserved.
      W3C <a href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>,
      <a href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a>,
      <a rel="Copyright" href="http://www.w3.org/Consortium/Legal/copyright-documents">document use</a>
      and <a rel="Copyright" href="http://www.w3.org/Consortium/Legal/copyright-software">software licensing</a>
      rules apply. Your interactions with this site are in accordance
      with our <a href="http://www.w3.org/Consortium/Legal/privacy-statement#Public">public</a> and
      <a href="http://www.w3.org/Consortium/Legal/privacy-statement#Members">Member</a> privacy
      statements.
    </p>

  </body>
</html>