Anomalous Anomaly

My Mom recently complained to me that her Hewlett-Packard brand PC was booting awfully slowly. She blamed it on my actions, claiming that it had to be the copy of Safari that I installed. She’s horrified to install anything on her computer for fear that it will “screw it up” or “slow it down”. In her case, it’s not a terribly unreasonable fear. If something does go wrong, she doesn’t know what to do to fix it. But blaming Safari for this problem did not make sense in the slightest, so I decided to take a peek under the hood and see why Windows Vista would boot so slowly on such a decent computer (besides the obvious, of course).

It took a while to find the Windows Vista equivalent to Windows XP’s BootVis tool, but apparently they made a new set of utilities for the task, called the Microsoft Windows Performance Toolkit. I won’t go into the usage of it, because you can Google it if you need that information. But I will show you how to interpret the results you get from it, and expose the culprit of this particular performance nightmare.

The first run showed a pretty horrendous sight:

Unfortunately the first results were rather worthless, as they were absolutely littered with disk input/output caused by AVG:

So I uninstalled AVG and ran the test a second time. This time, I got some conclusive results:

There are a few things to note about this graph. First of all, the core system boot sequence is shown to be taking 75 seconds (by “core system” I mean the time it takes between BIOS and loading a login screen). This is horrifying. Consider for a moment that Linux can finish the system boot in less than 20 seconds and reach a fully logged in graphical user interface in an additional 5-15 seconds. So in perspective, a total boot time of roughly 190 seconds (75 system + 115 userspace) is absolutely terrible.

It’s time to figure out the proximate cause. I noticed in the graph above that there is roughly a 20 second period during the system boot where there is minimal CPU usage, very few disk I/O requests, but the disk utilization is maxed out. This tells me that it’s doing a lot of slow requests which stall the system. I decided to look at the disk access graph, which shows where the drive is physically reading/storing data at a point in time.

Criminy. That 20 second stall (highlighted in the second image above) is being caused by thrashing between some files that are physically very far apart on disk. Here’s a zoom on that section:

Well, details are necessary now. We need to know what files it is thrashing between (and why) in order to be able to do anything about it.

Aha! Success! Er, well, maybe not. The files are the “COMPONENTS” and “SOFTWARE” registry hives. This is pretty much bad news because the Windows defragmenter can’t resolve this problem for two reasons:

It’s very strange though, that Windows didn’t simply read the two files in their entirety in two large reads instead of this silly thousands-of-tiny-reads crap. I think this could very well be a bug in Windows Vista. Come on, Microsoft. What were you thinking? A programming 101 student could tell you that a large disk read is faster than lots of smaller ones.

So in order to work around the problem, I first needed to be able to defragment my Mom’s computer’s hard disk completely without running the installed operating system. It’s unfortunate she doesn’t have a Mac, because if she did, we could simply use Target Disk Mode which basically makes the computer behave like an external FireWire drive; all that would be necessary in such a case is hooking it up to another machine as you would with an external hard drive, and then defragment that way. Unfortunately for her, the easiest way to do this was to repartition the drive so that we could install a temporary copy of Windows in a second partition.

The second issue is that we needed to be able to optimize the drive, and not just defragment it. By “optimize”, I mean physically arranging files on disk in such a way that they’re quickly accessed during the system’s boot sequence. This is different from defragmentation, which simply consolidates split files. Thankfully, there’s an awesome open source defragmenter available called JkDefrag, which is capable of optimizing as well as defragmenting. Sweet!

So I installed Windows 7 RC1 onto another partition (created via Disk Management), and then ran JkDefrag with the command-line option ‘-a 7′, which tells JkDefrag to optimize by arranging the files alphabetically based on their path and file name. This should cluster the contents of folders together, making accesses much faster.

So after a couple hours, it finally became bored and finished defragmenting, so I rebooted to re-time Windows Vista’s boot sequence. It was noticeably faster already, but I needed numbers. And I was quite pleased with the results:

Note that there’s no obvious point at which a bottleneck is noticeable. Neither the CPU nor the hard drive are maxed out for too long at any one point. And then there’s the physical layout access graph:

Awesome. JkDefrag did its job and the files are much better clustered. Better yet, the raw numbers are even more impressive. The system boot has been reduced to 45 seconds, which is 60% of the time it took previously. Based on the disk and CPU activity, the machine’s boot is complete after about 90 seconds, which is 47% of the time it took previously. (Keep in mind that xbootmgr continues measuring for 120 seconds after the user logs in, so we can’t just use the last time marker on the boot graph as the time it takes to boot. You have to look at disk activity and CPU utilization, and figuring out the point where the constant activity ceases and the machine begins to settle.)

So now my mom’s machine boots pretty quickly. Now I need to move on to my next project: Operation: Get Mom to Stop Using Internet Explorer.

Well, that was frustrating. I stayed up late last night trying to figure out why my Apache 2.2 server was misbehaving, and it ended up being a Firefox 3 bug. Or more specifically, a bug in the old OpenSSL library that Firefox 3 is statically linked to.

I tried to view our cgit page, and only got this:

And if I refreshed a few times, I got this ‘ssl_error_rx_unexpected_change_cipher’ error:

But strangely, if I used Safari, I got exactly what I should have:

I finally figured out that Firefox was screwing up when using the TLS 1.0 protocol. So all I had to do was edit my Apache 2.2 httpd.conf to have this line:

SSLProtocol -SSLv2 +SSLv3 -TLSv1

I don’t like this solution though. The Mozilla Firefox team should release a build of Firefox that is statically linked to a newer version of the OpenSSL library which doesn’t suffer from this bug. I much prefer TLS because of the added security it provides, so I ended up with a line that opted for security over broken SSL library compatibility:

SSLProtocol -SSLv2 -SSLv3 +TLSv1

Also note that I disable SSL v2 as well bacause it’s widely known to be inherently flawed.

I just installed service pack 1 on a 64-bit Windows Vista machine, and it stupidly took forever and a half to install. It also took multiple reboot cycles, something that wasn’t needed in previous Windows NT operating systems (2000 and XP included). I think Microsoft’s going the wrong direction with Windows service packs, because it is now ridiculously hard to integrate into a customized Windows disc. With Windows XP service packs, it was simple. You’d copy the files to the hard drive, slipstream, and burn the disc. The whole process took about 20 minutes. With Vista, it’s an interminable procedure requiring third party software (vLite). The process takes an hour or more.

Another service pack that I’ve had annoying troubles with is Visual Studio 2008 SP1. It takes more than an hour to install. In fact, it’s still installing on this machine. I wouldn’t be surprised if it’s actually stuck, because the progress bar hasn’t moved in 20 minutes or so.

Why the hell do Visual Studio service packs take forever to install? I assume the problem with 2008 SP1 is similar to the 2005 SP1 issue, which required a very ugly hack to get around it. I mean, it works, but it just takes forever. With Visual Studio 6, you could install a service pack in less than 5 minutes. This process has ballooned to over an hour. Why?!

Another thing that’s bugging me is that Visual Studio 2008 feels slower than molasses. Apparently I’m not alone in noticing this.

Microsoft’s stuff is just seeming less and less appealing. I can get a decent development machine running Mac OS X set up in about an hour or maybe two (including Xcode, Git, development libraries [SDL, etc], CMake, Smultron, etc). On Windows, It’s taken me at least 8 hours so far. I’ve been installing Windows, Visual Studio, service packs and other updates constantly. And I mean constantly. I haven’t let the machine idle once.

Microsoft! Give up on all this stupidity you’ve been adding to Windows, Visual Studio, and Office for Mac! Slim your applications. Make performance and size a priority. To ignore performance will be an absolutely fatal mistake for you. The only teams I think really have it together right now are the Microsoft Office (Windows) team, and the Xbox team. Their stuff installs easily, updates easily, and runs smoothly. Why can’t the rest of Microsoft get this crap right?!

Updated March 2, 2010 Removed unnecessary steps involving Boot Camp and simplified the whole process.

As a software developer, I find it important to ensure that my programs work well on all platforms. Windows, Mac OS X, Linux, whatever. So I decided I should triple-boot my MacBook Pro instead of juggling between a Mac/Windows or Mac/Linux setup.

I’m annoyed that every article I’ve found about setting up a Mac to triple-boot makes it sound like it’s impossible to have greater than four partitions (especially for triple-booting). This simply isn’t the case, and it is hazardous for Linux’s performance in particular because you’re forced to set up a swap file instead of a swap partition and in the case of Gentoo, the /usr/portage mountpoint can’t be separate from the main filesystem (/usr/portage changes frequently and quite randomly, generating awful fragmentation). I’ve had Gentoo installs which had 8 partitions just for Gentoo itself (split up the filesystem to better optimize the usage).

In the past, the problem has been that Intel Macs use the GUID Partition Table (GPT) scheme for representing partition information on disk. Since Windows even now only recognizes the Master Boot Record (MBR) format, Apple came up with a trick that basically makes the disk use a GPT/MBR hybrid. It stores two copies of the partition table. One in GPT format, and one in MBR format. Seems straightforward? It gets tricky.

With MBR, you are limited to 4 ‘primary’ partitions. Once MBR was designed and out the door, people began to realize that 4 partitions just wasn’t enough, and an extension to MBR was developed that would not break compatibility with the existing MBR definition. The extension allowed for one of the primary partitions to be marked as an ‘extended’ partition. Extended partitions sort of store their own partition table which keeps track of the ‘logical’ partitions contained in them.

With the GPT/MBR hybrid, Mac OS X’s Boot Camp software will synchronize the tables once (and once only, which is why we have ‘gptsync’, detailed below). Unfortunately, GPT doesn’t understand extended/logical partitions and gptsync refuses to sync when it detects the MBR has extended partitions defined. So the limitation here is that we cannot have any extended partitions in the MBR if we want GPT/MBR syncing to work.

The solution is pretty simple, really. Windows requires MBR partitioning, but the Linux kernel does not. Simply set up the partition tables with gparted (or with the Mac OS X ‘gpt’ command while booted into an install DVD), run the gptsync tool (which can be run pretty easily if you have rEFIt), install Windows, and finally install Linux. Windows will only see the MBR partitioning scheme, and if your running Linux kernel supports GPT, Linux will see the GPT scheme. Detailed steps are below:

1. Boot a Mac OS X install DVD. Bring up Disk Utility and partition your drive. Use as much space as you want for Mac OS X, but be sure to leave unpartitioned space for other OSes.
2. You can now add partitions for Linux and Windows using the ‘gpt’ command in terminal. A manual page for how to use it is here.
3. I recommend having the partitioning scheme be:
   1. EFI partition (don’t touch this)
   2. Mac OS X partition
   3. Linux /boot partition (about 500MB is fine)
   4. Windows partition

   and then whatever other partitions you want for Linux. At minimum, you need to have a partition for the Linux root, but you should probably go with one for swap, one for /home, and one for /.

4. Once your partitions are set, install Mac OS X to your Mac partition.
5. Install rEFIt. Be sure to enable it, too, by running ‘sudo /efi/refit/enable-always.sh’ in a terminal.
6. Reboot, and run the gptsync tool from the rEFIt menu (it’s the disk icon on the lower left). Make sure the changes it recommends are sane, and then type ‘y’.
7. Your partitions are now ready for your operating systems. Install Linux (Ubuntu, Fedora, whatever), and be sure to specify that the bootloader gets installed to the /boot partition (not to the master boot record, where it would be wiped out by Windows).
8. Once Linux is in, be sure to run ‘gptsync’ again, to be sure the partition table didn’t get clobbered.
9. Install Windows to the 4th partition. This can be tricky because Windows won’t install if the partition isn’t marked ‘active’. Either use a Linux livecd and ‘cfdisk’ to mark it ‘Boot’ or use a Windows Vista or Windows 7 disc’s command prompt to mark the partition ‘active’ with ‘diskpart’.
10. You should now have a Mac+Linux+Windows machine. Great for software development!

I kind of whipped this article up in a hurry. If there are any questions about this, email me.

Have you seen Microsoft’s latest act of desperation aiming to save Windows Vista?

It’s interesting to me that Microsoft has resorted to this strange act. I mean, I knew Microsoft was bleeding money, but I had no idea they were this desperate. You might wonder how this is an act of desperation. Well, Microsoft has taken several non-technically-minded people and asked them to criticize an operating system with technical flaws. As I said back in December, Windows Vista has some glaring flaws that a technically-minded person would pick up on fairly quickly. Unfortunately, all the end users see is the pretty new face, mainly the Aero Glass interface, and think it’s fantastic.

To me, it would have made much more sense to correct the technical flaws, implement the changes in Windows 7 and begin hyping it up. But of course, this isn’t really a new marketing theme, this is truly an underhanded act of desperation.

I don’t know whether to blame the hard drive, Windows 2000, or NTFS-3G. But never in my life do I want to see chkdsk show this again:

It kept scrolling that text for about 10 minutes before finally hitting this BSOD:

So we booted a Windows Vista DVD and ran chkdsk with that to finish the job it couldn’t:

Long story short, we were trying to reformat an external hard drive and ended up putting temporary copies of the data onto the local disk. Unfortunately, the local disk went haywire at that moment, and to make a long story short, most of the files were nicely toasted. The machine needed to be reformatted as well.