how_do_we_test_a_web_browser.html
17.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css" media="all">
@import "/QA/2006/01/blogstyle.css";
</style>
<meta name="keywords" content='api, css, html, html5, svg, test, test suite' />
<meta name="description" content="Testing all possible Web browsers out there is hard and requires more effort than one organization can afford by itself. The idea of increasing the level of Web browser testing done in W3C is to involve the community at large as much as possible. If we really want an interoperable Web, that's what W3C should move to." />
<meta name="revision" content="$Id: how_do_we_test_a_web_browser.html,v 1.35 2011/12/15 23:20:55 mirror Exp $" />
<link rel="alternate" type="application/atom+xml" title="Atom" href="http://www.w3.org/QA/atom.xml" />
<link rel="alternate" type="application/rss+xml" title="RSS 1.0" href="http://www.w3.org/QA/news.rss" />
<title>How do we test a Web browser? - W3C Blog</title>
<link rel="start" href="http://www.w3.org/QA/" title="Home" />
<link rel="prev" href="http://www.w3.org/QA/2009/09/linked_government_data.html" title="Linked Government Data" />
<link rel="next" href="http://www.w3.org/QA/2009/09/first_ever_developer_gathering.html" title="First Ever Developer Gathering during W3C Technical Plenary Week" />
<!--
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/"
xmlns:dc="http://purl.org/dc/elements/1.1/">
<rdf:Description
rdf:about="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html"
trackback:ping="http://www.w3.org/QA/sununga/mt-tb.cgi/284"
dc:title="How do we test a Web browser?"
dc:identifier="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html"
dc:subject="HTML"
dc:description="Testing all possible Web browsers out there is hard and requires more effort than one organization can afford by itself. The idea of increasing the level of Web browser testing done in W3C is to involve the community at large as much as possible. If we really want an interoperable Web, that's what W3C should move to."
dc:creator="Philippe Le Hégaret"
dc:date="2009-09-17T21:51:53+00:00" />
</rdf:RDF>
-->
<!-- <script type="text/javascript" src="http://www.w3.org/QA/mt.js"></script>-->
</head>
<body class="layout-one-column">
<div id="banner">
<h1 id="title">
<a href="http://www.w3.org/"><img height="48" alt="W3C" id="logo" src="http://www.w3.org/Icons/WWW/w3c_home_nb" /></a>
W3C Blog
</h1>
</div>
<ul class="navbar" id="menu">
<li><strong><a href="/QA/" title="W3C Blog Home">[ W3C Blog ]</a></strong></li>
<li><a href="/QA/Library/" title="Documents and Publications on Web and Quality">Documents</a></li>
<li><a href="/QA/Tools/" accesskey="3" title="Validators and other Tools">Tools</a></li>
<li><a href="/2007/12/qa-blog-help/index#feedback">Feedback</a></li>
</ul>
<div id="searchbox">
<form method="get" action="http://www.google.com/custom" enctype="application/x-www-form-urlencoded">
<p id="formbox"><input type="text" size="15" class="textfield" name="q" accesskey="E" maxlength="255" /> <input type="submit" class="submitfield" value="Search" id="goButton" name="sa" accesskey="G" /> <input type="hidden" name="cof" value="T:black;LW:72;ALC:#ff3300;L:http://www.w3.org/Icons/w3c_home;LC:#000099;LH:48;BGC:white;AH:left;VLC:#660066;GL:0;AWFID:0b9847e42caf283e;" /><input type="hidden" id="searchW3C" name="sitesearch" checked="checked" value="www.w3.org/QA" /><input type="hidden" name="domains" value="www.w3.org/QA" /></p>
</form>
</div>
<div id="main"><!-- This DIV encapsulates everything in this page - necessary for the positioning -->
<p class="content-nav">
<a href="http://www.w3.org/QA/2009/09/linked_government_data.html">« Linked Government Data</a> |
<a href="http://www.w3.org/QA/">Main</a>
| <a href="http://www.w3.org/QA/2009/09/first_ever_developer_gathering.html">First Ever Developer Gathering during W3C Technical Plenary Week »</a>
</p>
<h2 class="entry-header">How do we test a Web browser?</h2>
<div class="entry-body">
<p>The idea started with the fact that we have a number of Working Groups who
are trying to review the way they do testing, but also
increase the number of tests they are doing as well.</p>
<p> The CSS Working Group was foremost in mind when it comes to testing. The
Group has several documents in Candidate Recommendation stage that are waiting tests and
testing. The HTML Working Group is starting to look into testing as well
and a key component of ensure the proper success of HTML 5 is through
testing. The specification is quite big to say the least and,
when it comes to testing, it's going to require a lot of work.
We also have more and more APIs within the Web Apps group, Device API,
Geolocation, etc. The SVG Working Group has a test suite for
1.2, but they're looking at different ways of testing as well.
The framework produced by the MWI Test Suites framework allow two methods. One
requires a human to look at it and select pass/fail. The other
one is more suitable for script tests, ie APIs testing.</p>
<p> A bunch of us, namely Mike Smith, Fantasai, Jonathan Watt, Doug
Schepers, and myself, decided to get together to discuss this
and figure out how to improve the situation. We focused on three
axes: test submissions, test reviews and how to run a test.</p>
<p> First, we'd like ideally every single Web author to be able to
submit tests, so when they run into a browser bug based on a
specification, it should be easy for them to submit a test to
W3C. It should also allow browser vendors to submit thousands of
tests at once. There is the question of how much metadata do you
require when submitting a test. For example, we do need to know
at some point which feature/part of a spec is being tested. We
should also as many format as possible for tests. Reftests,
mochitests, DOM-only tests, human tests, etc. The importance
aspect here is to be able to run those tests on many
platforms/browsers as possible. A test format that can only be
ran on one browser is of no use for us.</p>
<p> Once a test has been submitted, it needs to be reviewed. The
basic idea behind improving test reviews is to allow more
individuals to contribute. The resources inside W3C aren't
enough to review ten of thousands of tests. We need to involve
the community at large by doing crowd reviews. It will allow the
working groups to only focus on the controversial tests.</p>
<p> Once the test got reviewed, we need to run them on the browsers, as many
as possible. Human tests for example are easy to run on all of
them, but it does require a lot of humans. Automatic layout
tests are a lot trickier, especially on mobiles. We focused on
one method during our gathering: screenshot based approach. The basic
idea here is that a screenshot of the page is compared to a
reference. Mozilla developed a technology called ref-tests that
compares Web pages themselves. You write two pages differently that are
supposed the exact same rendering and compare their
screenshots. It avoids a lot of cross-platforms issues one can.
The way Mozilla is doing that is via the mozPaint API in debug
mode. That works well, but only works in Mozilla. You can guess
that other browser vendors have a similar to automatically take
screenshots as well. We wanted to find a way to do this with all
browsers without forcing them or us to write significant amounts
of code. We found a Web site called browsertests.org and we got
in touch with that Sylvain Pasche and, with his help, we started
to make some improvements on his application. It works well on
desktops at least. Once again, we don't think W3C is big enough
to replicate all types of browser environments, so we should
make it easy for people to run the tests in their browser and
report the results back to us. Plenty of testing frameworks have
been done already and we should try to leverage them as much as
possible.</p>
<p> We started to set up a database for receiving the tests and
their results. We'd like to continue the efforts on the
server/database side, as well as continuing to improve Sylvain's
application, allowing more tests methods and formats. Testing
the CSS or HTML5 parser should be allowed for example.</p>
<p> You'll find more information at our <a href="http://omocha.w3.org/">unstable server</a> but keep in mind that:</p>
<ol><li>we're in the very early stages</li>
<li>this server is a temporary one that I managed to steal for a
few days from our system folks. They'll want it back one of
those days and I need to find a more stable home prior to
that event. I'll update the link once this happens but expect it to break if you bookmark it.</li>
<li>Unless I can secure more resources for the project, we won't
go far by ourselves.</li>
</ol>
<p>The server also contains links to more resources on the Web related to
various testing efforts, as well as a more complete of what we
wish the testing framework to accomplish.</p>
<p> For the conclusion, I'd like to thank Mike Smith and Doug
Schepers, and especially Jonathan Watt and Fantasai from the
Mozilla Foundation. They all accepted to argue and code for 8
days around the simple idea of improving the state of testing at
W3C. I hope we're going to be able to take this project off the
ground in the near future. If you're interested in contributing,
got ideas and time, don't hesitate to contact me.</p>
</div>
<div id="more" class="entry-more">
</div>
<p class="postinfo">Filed by <a href="http://www.w3.org/People/LeHegaret/">Philippe Le Hégaret</a> on September 17, 2009 9:51 PM in <a href="http://www.w3.org/QA/archive/technology/css/">CSS</a>, <a href="http://www.w3.org/QA/archive/technology/html/">HTML</a>, <a href="http://www.w3.org/QA/archive/technology/svg/">SVG</a>, <a href="http://www.w3.org/QA/archive/w3cqa_news/tools/">Tools</a><br />
<span class="separator">|</span> <a class="permalink" href="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html">Permalink</a>
| <a href="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html#comments">Comments (4)</a>
| <a href="http://www.w3.org/QA/2009/09/how_do_we_test_a_web_browser.html#trackback">TrackBacks (0)</a>
</p>
<h3 class="comments-header" id="comments">Comments</h3>
<div class="comment" id="comment-183584">
<p class="comment-meta" id="c183584">
<span class="comment-meta-author"><strong>Doug Schepers </strong></span>
<span class="comment-meta-date"><a href="#c183584">#</a> 2009-09-17</span>
</p>
<div class="comment-bulk">
<p>I also want to thank the other participants of the SVG WG here... jwatt and myself were the only ones in the group who could attend the Test Hackathon, but Anthony Grasso, Chris Lilley, Cameron McCormack, and Erik Dahlström have also been actively involved in the process of improving our test framework most recently, and the rest of the SVG WG laid the groundwork in years past.</p>
<p>We know from first hand experience how time consuming and draining building a comprehensive test suite is, and we are confident that with more community participation, and more direct implementer contributions, the process of bringing features to the market (and the hands of developers) will be much faster and smoother.</p>
<p>The more complete the test suite, the easier it is to implement (and implement right), and the better the interoperability will be. This is a high priority for the SVG WG, and we are eager to use what comes out of this effort.</p>
</div>
</div>
<div class="comment" id="comment-183608">
<p class="comment-meta" id="c183608">
<span class="comment-meta-author"><strong>Chris Macdonald </strong></span>
<span class="comment-meta-date"><a href="#c183608">#</a> 2009-09-19</span>
</p>
<div class="comment-bulk">
<p>In the smaller unexpanded text description I think this change should be made to be more easily read. From</p>
<p>"Testing all possible of Web browsers out there is hard and require more effort than one organization can afford by itself. The idea increasing the level of Web browser testing done in W3C is to involve the community at large as much as possible. If we really want an interoperable Web, that's what W3C should move to."</p>
<p>to...</p>
<p>Testing all possible Web browsers out there is hard and requires more effort than one organization can afford by itself. The idea of increasing the level of Web browser testing done in W3C is to involve the community at large as much as possible. If we really want an interoperable Web, that's what W3C should move to.</p>
</div>
</div>
<div class="comment" id="comment-186928">
<p class="comment-meta" id="c186928">
<span class="comment-meta-author"><strong>Emma </strong></span>
<span class="comment-meta-date"><a href="#c186928">#</a> 2010-02-15</span>
</p>
<div class="comment-bulk">
<p>This sounds like a huge project - do you really think it's feasible?</p>
</div>
</div>
<div class="comment" id="comment-222415">
<p class="comment-meta" id="c222415">
<span class="comment-meta-author"><strong>Shanti Bhushan </strong></span>
<span class="comment-meta-date"><a href="#c222415">#</a> 2011-03-24</span>
</p>
<div class="comment-bulk">
<p>how can i contribute in this? please do mail the process once you finalize the execution.
Already i am in to browser testing...
i feel 1st of all basic functionality of the browser should be tested, then we should move for w3c standard testing like HTM5 ,Dom etc. </p>
</div>
</div>
<div class="comments-open" id="comments-open">
<h3 class="comments-open-header">Leave a comment</h3>
<div class="comments-open-moderated">
<p>
Note: this blog is intended to foster <strong>polite
on-topic discussions</strong>. Comments failing these
requirements and spam will not get published. Please,
enter your real name and email address. Every
individual comment is reviewed by the W3C staff.
This may take some time, thank you for your patience.
</p>
<p>
You can use the following HTML markup (a href, b, i,
br/, p, strong, em, ul, ol, li, blockquote, pre)
and/or <a href="http://daringfireball.net/projects/markdown/syntax">Markdown syntax</a>.</p>
</div>
<div id="comments-open-data">
<form method="post" action="http://www.w3.org/QA/sununga/beach.pl" id="comments-form">
<h4>Your comment</h4>
<div id="comments-open-text">
<textarea id="comment-text" name="text" rows="20" cols="100"></textarea><br />
<label for="comment-text">Write your comment text here. Remember, keep the discussion on topic and courteous.</label>
</div>
<h4>About you</h4>
<div id="comment-form-name">
<input type="hidden" name="static" value="1" />
<input type="hidden" name="entry_id" value="6478" />
<input type="hidden" name="__lang" value="en" />
<label for="comment-author">Your Name</label>
<input id="comment-author" name="author" size="30" value="" />
</div>
<div id="comment-form-email">
<label for="comment-email">Your Email Address</label>
<input id="comment-email" name="email" size="30" value="" />
</div>
<div id="comments-open-footer">
<input type="submit" accesskey="s" name="post" id="comment-submit" value="Submit" />
</div>
</form>
</div>
</div>
<p id="gentime">This page was last generated on $Date: 2011/12/15 23:20:55 $</p>
</div><!-- End of "main" DIV. -->
<address>
This blog is written by W3C staff and working group participants,<br />
and maintained by <a href="/People/CMercier/">Coralie Mercier</a>.<br />
Authorized parties may <a href="/QA/new">log in</a> to create a new entry.<br/>
<span id="poweredby">Powered by Movable Type, magpierss and a lot of Web Technology</span>
</address>
<p class="copyright">
<a rel="Copyright" href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a> © 1994-2011
<a href="http://www.w3.org/"><acronym title="World Wide Web Consortium">W3C</acronym></a>®
(<a href="http://www.csail.mit.edu/"><acronym title="Massachusetts Institute of Technology">MIT</acronym></a>,
<a href="http://www.ercim.eu/"><acronym title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>,
<a href="http://www.keio.ac.jp/">Keio</a>),
All Rights Reserved.
W3C <a href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>,
<a href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a>,
<a rel="Copyright" href="http://www.w3.org/Consortium/Legal/copyright-documents">document use</a>
and <a rel="Copyright" href="http://www.w3.org/Consortium/Legal/copyright-software">software licensing</a>
rules apply. Your interactions with this site are in accordance
with our <a href="http://www.w3.org/Consortium/Legal/privacy-statement#Public">public</a> and
<a href="http://www.w3.org/Consortium/Legal/privacy-statement#Members">Member</a> privacy
statements.
</p>
</body>
</html>