Anomalous Anomaly

The Acid 3 web standards-compliance test has been up for a while now, and there’s no release-quality browser in existence yet that fully succeeds at the test.

Let me explain what these results are telling us before I show them. The World Wide Web Consortium (W3C) develops standards for Internet applications. For example, XHTML, HTML, CSS, etc. Acid 3 basically takes a lot of W3C’s newer standards and tests to be sure that the browser supports the features it should and that it behaves as it should when using them. Acid 3 is a collection of 100 different test suites to be sure that these standards are being met. So when I say that browser X gets a score of 55%, this means that browser X passed 55 of the 100 test suites.

Updated August 28, 2008. Decided it was time for an update. Outdated entries are italicized and grayed. New entries are not italicized. It’s a work in progress, and it may be a while before I have all the entries updated. It’s worth noting that Safari 4′s public beta passes Acid 3 without breaking a sweat. Safari 4.0 may well be the first release-quality browser to pass Acid 3!

Updated September 7, 2008. Added Google Chrome to the list.

Updated May 19, 2009. Decided to get up to speed with some of the browsers. Not a lot has changed, though Opera’s latest release puts it in the lead as far as officially-released browsers go. Note that I do still accept submissions/requests. This list may be really out of date now, but I’d like to bring it up to date. Any help is graciously accepted. If you submit a result, please include a summary including the operating system version, browser version, and Acid3 score as well as a screenshot of the result (mainly just as proof for browsers I can’t personally verify the results for).

Updated June 11, 2009. Safari 4.0 was released on June 9, 2009. This makes it the first official release of a browser to have 100% on ACID3.

You can use this table of results as you please, but I would appreciate if people would not just copy/paste it everywhere, because the results will outdate fairly quickly, and I would like this table to get updated more frequently. So please link back to here instead of simply copying this entire table. Thanks!

Note: Italicized lines are tests which were done a long time ago, and no longer represent the current release (beta or otherwise) of the browser.

Beta Browsers
Browser	Version	Operating System	Acid 3 Score
Safari	WebKit Nightly (r43808)	Mac OS X 10.5.7	100% (0.69s)
Opera	9.0 Beta (build 636) “WinGogi”	Windows Vista 32-bit	100% (1.31s)
Firefox	3.1b3	Mac OS X 10.5.7	93%
Seamonkey	2.0a1pre nightly (2008082800)	Mac OS X 10.5.4	85%
Google Chrome	0.2.149.29	Windows Vista 32-bit	79%
Opera	9.50 Beta (build 9864)	Windows XP Service Pack 2	79%
Camino	2.0a1pre nightly (2008082800)	Mac OS X 10.5.4	71%
Firefox	3.0b4 (20081016)	Mac OS X 10.5.2	68%
Firefox	3.0b4 (2008030714)	Windows XP Service Pack 2	68%
Firefox	3.0b4 (2008030714)	Windows Vista (32-bit)	68%
Internet Explorer	8.0.6001.18241 (Beta 2)	Windows Vista (32-bit)	21%
Internet Explorer	8.0.6001.17184 (Beta 1)	Windows Vista (32-bit)	18%
Internet Explorer	8.0.6001.17184 (Beta 1)	Windows XP Service Pack 2	18%

Released Browsers
Browser	Version	Operating System	Acid 3 Score
Safari	4.0 (5530.17)	Mac OS X 10.5.7	100% (0.51s)
Opera	9.64	Mac OS X 10.5.7	85%
Safari	3.1.2 (5525.20.1) *	Mac OS X 10.5.4	75%
Safari	3.1.2 (525.21) *	Windows Vista (32-bit)	75%
Flock	2.0.3	Mac OS X 10.5.7	71%
Firefox	3.0.10	Mac OS X 10.5.7	71%
Firefox	3.0.10	Windows 2000 SP4	71%
Konqueror	4.0.3	Linux	65%
Epiphany	2.22	Ubuntu 8.04 (beta)	59%
Camino	1.6.7	Mac OS X 10.5.7	53%
SeaMonkey	1.1.11	Mac OS X 10.5.4	53%
Epiphany	2.20.3	Gentoo Linux	51%
Konqueror	3.5.9	Gentoo Linux	51%
Opera	9.26	Windows Vista (32-bit)	46%
Safari	3.0.4 (5523.15) *	Mac OS X 10.5.2	39%
Safari	3.0.4 (523.15) *	Windows XP Service Pack 2	39%
Safari	3.0.4 (523.15) *	Windows Vista (32-bit)	39%
Mobile Safari	3.0 (420.1)	iPhone	39%
Internet Explorer	5.50.4807.2300 (SP2)	Windows XP Service Pack 2 (Multiple IE)	14%
Internet Explorer	5.50.4134.0600	Windows ME	14%
Internet Explorer	5.50.4807.2300 (SP2)	Windows 2000 Service Pack 4	13%
Internet Explorer	7.0.5730.13	Windows XP Service Pack 2	12%
Internet Explorer	7.0.6000.16609	Windows Vista (32-bit)	12%
Internet Explorer	6.0	Windows XP Service Pack 2	12%
Internet Explorer	6.0	Windows 2000 Service Pack 4	11%
* I don’t know whether Apple meant to put 5xx or 5xxx. I highly doubt that they intended to use such an inconsistent versioning scheme, but I’m going to cite here whatever they put on the About window. Thanks to Mark Rowe from Apple for explaining this: “The leading digit of the build number signifies the platform version. 45xx.y.z is Tiger (10.4), 55xx.y.z is Leopard (10.5), and 5xx.y.z is Windows.”

I guess it really comes as no surprise that Internet Explorer is currently in last place. But really, how did Internet Explorer 7 and 6 do worse than v5.5?

I’ve been working on a new benchmark for the past several weeks (would have finished it much sooner too, had it not been for college). I think the results are very much worth sharing.

The first thing to know is that the source code to the new benchmark (I’ve named it CheckMark), is one of the CrissCross library’s example programs and is available in the CrissCross source repository. It will be more easily accessible when it goes into the CrissCross v0.7.0 release (probably in a few weeks).

CheckMark has several different checksum/hashing algorithms in it: Adler-32, CRC32, MD4, MD5, SHA-1, SHA-256, SHA-512, and Tiger. Tiger and SHA-512 were written to be especially fast on 64-bit machines (they use 64-bit word sizes), and the rest are optimized for 32-bit use.

The benchmarks I’ve done are fairly simple to do. First, for consistency, it’s best to run in an environment as close to the one I used as possible. I run the Linux 2.6.24 kernel and use GCC 4.1.2. To compile and run CheckMark, you need to first get a copy of the source code using Subversion, and then once the source is downloaded, do ‘make check’ in the main folder. If your machine, compiler, and CPU pass the CrissCross test suite tests, then you can go ahead and run ‘make example’. And then to run CheckMark (assuming the build goes fine), simply ‘examples/CheckMark/checkmark’. CheckMark generates 100 1024-byte strings and times how long it takes to hash all of them 500 times. The output of the program for each hash algorithm is the number of hashes calculated per second. Since each string is 1024 bytes long though, the hashes per second number is equivalent to the number of kilobytes hashed per second.

The machines I tested with CheckMark used the following processors in 32-bit mode: ARM9 (specifically, the one used in my Nintendo DS), Intel Core Duo, Intel Core 2 Duo, Intel Xeon (with a Pentium 4 core, a.k.a. NetBurst), and a PowerPC G4. I also tested the Intel Core 2 Duo in 64-bit mode to show the speed difference in 32 vs 64-bit modes.

My results are pretty intriguing, and they can be found here. Please note that in the graphs, a smaller bar is better, because the processor is accomplishing more work per clock cycle. Also note that since the graphs aren’t based on time but instead based on work per clock cycle, the clock speed of the processors doesn’t affect the graphs. In fact, the clock speed is taken into account when calculating the number of clock cycles per hash. This shows more of the raw performance of the processor and how efficient it truly is. This just goes to show yet again that clock speed is not everything!

Okay then, how are we to read these graphs? What do these results mean? Well, the most obvious thing I see when looking at the graphs is that you probably shouldn’t bother with SHA-512 or Tiger hashes unless you’re running on a 64-bit processor running in 64-bit mode. It’s also plain to see that the Core 2′s 64-bit mode very very easily outperforms its 32-bit mode when doing 64-bit math. This makes perfect sense, as the SHA-512 and Tiger algorithms make heavy use of 64-bit math. In 32-bit mode, a lot of extra work has to be done to complete a single 64-bit math operation.

Another observation I could make about these graphs is that the PowerPC just isn’t doing well at any of them, except SHA-1, where it appears to do well because the NetBurst core takes a huge performance hit. I’m fairly certain that the NetBurst’s speed issues here are due to the fact that the NetBurst core lacks a barrel shifter, the component which makes shift and rotate operations blindingly fast. There are a large number of shifts and rotates that occur in the SHA-1 algorithm, so the performance hit is understandable. Not surprisingly, the barrel shifter has returned in the Core and Core 2 architectures.

I was very impressed with how efficient the ARM9 processor was at doing these hashing algorithms. It was accomplishing so much more work per clock, even though it is a RISC processor, like the PowerPC. The only case where the ARM9 does very badly is when 64-bit math is involved.

I would love to hear your thoughts about this, as well as your own results, if you’d like me to add them to the list. Email me directly at steven@uplinklabs.net and let me know what’s on your mind!

Why would a computer engineering degree be better than a computer science one? Computer science works in the theoretical plane. Stuff that you have to run for months on end on a supercomputer or not even solve for years. Engineering is about making things work with what you have to meet a set of constraints, and deals in milliseconds or microseconds. In game programming, you have a CPU, a GPU, memory that has a certain latency, a disk with a certain seek time, etc. You need to accomplish so much every 60th of a second given those hardware constraints or else the game will suck.

Put another way, computer science is about exploring new ideas and not having to worry about the practical implementation, which is the engineering portion. Prime numbers for example. Computer scientists love to talk about ridiculously large prime numbers and all you can do with such things, like unbreakable encryption and such. Engineering is taking that abstract algorithm and making it work on, say, a 200MHz ARM processor running on your battery-powered MP3 player, where things like battery life and response time matter.

What I’ve found is that computer science majors write really bad code. Not code that doesn’t work, but code that’s slow. They don’t know how to optimize. They can’t tell you the difference between an AMD Athlon XP and an Intel Pentium 4. They can’t explain why the new Core 2 is such a good thing. In their little abstract world of trees and lists and Java, they don’t need to understand the low level hardware. Many of them can’t ever read an x86 disassembly or tell me the first thing about how many registers in a Pentium processor or what the registers are for.

Parallelization is the big thing right now. The Xbox 360, for example, has a 6-way processor. How does one write video games when you can have 6 concurrent pieces of code running at the same time? How does that change your rendering engine? Your game logic? Memory allocation? The Playstation 3 has nine hardware threads. This totally changes the way one writes video games from now on.

Computer Science has a long standing solution to concurrency – the concept of a lock. Which in Linux and Windows are synchronization objects known as locks, mutexes, semaphores, and other names. They’re used in every operating system and just about every shipping Windows and Linux application today. Even calling malloc() is a seralizing operation on the memory heap that causes a lock. Have multiple threads calling malloc() and they basically get to stand in line (a queue data structure) and execute serially. That’s not very parallel. So once again, Computer Science has given us something that doesn’t translate well into real-world performance.

My point of this is that you should NOT go into pure computer science. You’ll rot your head with abstract ideas and end up writing very poor code. Take either computer science with electrical (or computer) engineering electives, what’s known as “CS triple E”, or take computer engineering to get the best of both worlds. If you can’t visualize an algorithm or a piece of code and understand what that touches in the microprocessor, in the memory, on the disk, and what the costs and delays of all the steps are, you’re going to write crappy code.