2008-03-26 21:00 UTC Tests that are never quite finished
Since we announced the Acid3 test a few short weeks ago, two major browser vendors have been tripping over themselves trying to pass the test, with both Opera and Safari getting very close in the last few hours.
Of course, with a test as complex as this one, I was bound to make some mistakes, which the browser vendors were very quick to tell me about. Sometimes they were mistaken, and the test was actually correct; other times I was wrong, and had to fix the test, and in one case, the spec changed.
Here's a quick overview of the more major changes I made to the test. Luckily, none of the errors were too serious.
- Sub-pixel testing
- It turns out that the original test accidentally required that browsers implement sub-pixel positioning and layout (and in fact the reference rendering got it wrong too, and relied on the same kind of rounding as Firefox does), which is somewhat dubious. I've changed the test to not rely on sub-pixel layout. However, it is very likely that this will be tested in Acid4, if we can get the specs to be clearer on this.
- Surrogate pairs in SVG APIs
- One of the submitted tests assumed that SVG APIs worked on Unicode codepoints, but the SVG spec changed to work on UTF-16 codepoints, like the rest of the DOM API, so the test was changed there. (The test changed a couple of times, because I originally got the fix wrong.)
- The
click()method - The test originally assumed that the
click()method was reentrant, but the specs were vague on this and someone suggested making it fail if calls to it were nested, so I removed this part of the test (the spec hasn't been updated yet). I replaced it with an attribute test (the new second part of subtest 64). - The Performance Test
- I made the loop counter in the performance test (a part of subtest 63) less complicated and shorter, to make it at least plausible that browsers could be fixed to pass that test quickly enough that it wouldn't always feel jerky. At the same time, I updated the test's infrastructure to report more details about pass and fail conditions and how long each subtest takes to run.
- Namespace bug
- Someone noticed that
http://www.w3.org/1998/XML/namespaceshould have beenhttp://www.w3.org/XML/1998/namespacein one of the subtests. - Linktest timeout
- I made the linktest more resilient to slow network conditions. However, the test is still going to give you major issues if you are on a network with multi-second latency, or if the acidtests.org site is being slow.
When we released Acid2, the first browser passed it in about a fortnight. Acid3 is orders of magnitude more complicated. I really didn't expect to see passing browsers this side of August, let alone within a month. I am really impressed by the level of commitment to standards that Opera and the WebKit team (and Apple in particular) are showing here.