Archive for the 'General' Category

Google Summer of Code 2010 – Week One Report

Friday, May 28th, 2010

This week has been slightly heavier on schoolwork than usual (it is very close to the end of the quarter at Central Washington University). So most of my week has been spent working on tcmalloc and trying to figure out why the heap profiler is broken on Mac OS X. You can see the progress on the tcmalloc issue here. There’s a few embarrassing goofs on my part in there, but oh well.

As for Unladen Swallow, I’ve been looking at it in Instruments.

The above screenshot is of two runs of the ‘nbody’ test, which is from the Unladen Swallow test tree. The first run, top, is CPython 2.6.5 (I built my own because I didn’t want any Apple-specific code to interfere). The second run, right beneath it, is Unladen Swallow. ‘nbody’ was the test that had the highest memory usage increase when comparing Unladen Swallow to CPython. Interestingly, this memory usage appears to mostly just be a spike.

I suspect what happened in the above is that Unladen determined that the particular function it was running was considered “hot”, and then proceeded to JIT compile it. The reason it spikes is that compilation is taking quite a bit of memory, but after the end result is obtained, that memory isn’t needed anymore. There’re lots of possibilities for why there’s so much memory usage during that JIT compile. When I look at the objects which are created and still living between the time points I highlighted above, it looks like the overall memory usage increase is from 3.38MB to 5.71MB, an increase of only 2.33MB, which is acceptable. But between those two points, there is a grand total of 78.74MB allocated and freed. At its peak, the memory usage hits 13.71MB. This is where I need to focus my work. Why is it allocating so much for this particular test, considering other tests don’t cause such a large spike? How can I cut this down?

Observant readers familiar with Instruments will notice there’s a memory leak displayed in the screenshot above. This is a memory leak in LLVM 2.7, which has apparently been fixed between release and now, so it’s not particularly relevant.

I’m excited for when school is finally out. I’ll be able to really get going on this project!

Google Summer of Code 2010 – Week One

Monday, May 24th, 2010

And so begins my goal to help resolve Unladen Swallow‘s memory bloat issues. You can look at my proposal here.

Well, the week has gotten off to an interesting start. I’ve been trying to get Google perftools to work, so I can start doing some heap profiling on Unladen Swallow. Unfortunately, it’s not working nicely, even with one of my own projects. My first order of business for the week has been to fix problems I’ve found with perftools: #95, #244, #245, #246.

But I don’t intend to get too sidetracked by this. Perftools will definitely be an essential tool to have, since it allows for programatic, selective profiling. For the moment, I think I’ll just use the tool provided by Apple with Xcode called Instruments. It’ll suffice for the moment, but it doesn’t really have the fine-grained capabilities of perftools. My other tool of choice is Valgrind, which can be used for both finding memory leaks and heap profiling via massif.

Alrighty. Time to get cracking.

Reddit Has A Sense Of Humor

Thursday, May 6th, 2010

Miah suggested that I take a look at the server headers for Reddit.com, because he found something hilarious:

tycho@alcarin ~ $ curl -I http://www.reddit.com/
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Set-Cookie: reddit_first=%7B%22organic_pos%22%3A%201%2C%20%22firsttime%22%3A%20%22first%22%7D; Domain=reddit.com; expires=Thu, 31 Dec 2037 23:59:59 GMT; Path=/
Server: '; DROP TABLE servertypes; --
Date: Thu, 06 May 2010 23:09:17 GMT
Connection: keep-alive

I wonder what prompted them to put that in there…

MinGW Make is Slow

Friday, April 16th, 2010

I was having troubles building Onlink under MinGW, because ‘make’ was so horrendously slow. I wondered if this was possibly because of the GNU Make implicit pre-defined rules. Disabling them helps tremendously, even with a dry run:


tycho@alcarin ~/Development/onlink/uplink/src $ time make -n > /dev/null

real 0m19.124s
user 0m4.506s
sys 0m13.720s


tycho@alcarin ~/Development/onlink/uplink/src $ time make -r -n > /dev/null

real 0m5.432s
user 0m1.731s
sys 0m2.930s

So if your GNU Make is slow under MinGW, try ‘make -r’!

Quad Booting Your Mac

Tuesday, March 2nd, 2010

UPDATE (March 5, 2010): I’ve updated the article to reflect that I now keep my FreeBSD /boot on the ZFS partition.

You may remember my last article on a similar topic. At the time, I thought any more than three operating systems couldn’t be done on a Mac. But surprisingly, I’ve managed to install four operating systems with relative ease.

This time, I’ve got Mac OS X (10.6.2), Windows 7, Ubuntu Linux 9.10, and FreeBSD 8.0 installed.

Historically, there have been a couple problems with doing quad booting. Most operating systems (Linux and FreeBSD namely) don’t do well when booted directly from EFI, because this boot method skips Apple’s BIOS emulation. This prevents using graphics cards with hardware acceleration enabled, among other things. Being a game developer, this means that EFI-booting Linux and FreeBSD is not an option for me. The second problem is that the MBR/GPT hybrid only allows the MBR side to have at most 4 primary partitions and no extended/logical partitions. In the past, this has meant that things booted under BIOS emulation can only “see” the first 4 partitions unless they have built-in support for GPT.

In Linux’s case, it’s simple to get it to “see” past the 4th partition by compiling the kernel with GPT support. Linux’s boot process on a tri-boot or quad-boot mac goes something like this:

  1. rEFIt switches from EFI-boot mode to BIOS emulation and starts GRUB, using the third partition’s boot sector.
  2. GRUB initializes, and since GRUB only has support for the MBR, it can only see the first 4 partitions.
  3. GRUB loads and boots the Linux kernel from the third on-disk partition (which is Linux’s /boot partition).
  4. The Linux kernel initialises, loads its built-in GPT support and can see past the 4th partition.

So Linux has no problem getting past the MBR’s 4-partition limit. Now, how do we deal with FreeBSD? As you can see from the partition table below, FreeBSD’s partitions are past the MBR’s 4-partition limit (illustrated by the gray line).

Partition Table
Index Size GPT Type Comment
1 200MB EFI System (FAT)
2 125GB HFS+ Mac OS X
3 1GB Basic Data Linux /boot
4 125GB Basic Data Windows 7
5 4GB Basic Data FreeBSD swap
6 100GB Basic Data FreeBSD ZFS
7 4GB Linux Swap
8 60GB Basic Data Linux root
9 60GB Basic Data Linux /home

So how do we boot FreeBSD without using a EFI-based boot? We upgrade GRUB to GRUB2. The newer GRUB2 has support for UFS2 (which is a filesystem FreeBSD’s /boot can use) as well as support for GPT. Ubuntu also has a patched GRUB2 which has support for ZFS. It’s certainly possible to compile their spin of GRUB2 on other Linux distributions, but it’s not exactly a cakewalk. I ended up compiling it myself so that I could use Fedora instead of Ubuntu, though.

All that’s needed to get GRUB to boot FreeBSD (in the case of the partition layout above) is to add this to Ubuntu’s /etc/grub.d/40_custom, and then run ‘update-grub’ as root:

menuentry "FreeBSD 8.0" {
	insmod zfs
	set root=(hd0,6)
	freebsd			/@/boot/kernel/kernel
	freebsd_module_elf	/@/boot/kernel/opensolaris.ko
	freebsd_module_elf	/@/boot/kernel/zfs.ko
	freebsd_module		/@/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache
	freebsd_loadenv		/@/boot/device.hints
	set FreeBSD.zfs_load="YES"
	set FreeBSD.vfs.root.mountfrom=zfs:freebsd
	set FreeBSD.vfs.root.mountfrom.options=rw
	set FreeBSD.kern.bootfile=/boot/kernel/kernel
}

And now you have a Mac that quad boots Mac, Linux, Windows, and now FreeBSD. Pretty cool, eh?

One thing that’s important to note is that the partition editor and slice editor used by FreeBSD’s “sysinstall” program, which is often used to install FreeBSD, will destroy your ability to boot Mac OS X, so you need to avoid using it. Manually installing FreeBSD is pretty easy to do, so don’t fret too much. Just use this guide. In my case, I skipped steps 1.3 through 1.5 because I had no intention of using the same partition layout (and their booting method is a pain compared to GRUB2). Before using ‘gpart’ to set up partitions, I first set up my Mac OS X partition using the Mac OS X install disc. Doing so is fairly important, because Mac OS X will behave pretty strangely (or not boot at all) if the EFI System Partition isn’t created in exactly the way Disk Utility creates it. I also deviated from the guide by naming my zpool ‘freebsd’ instead of ‘zroot’.

Note that this guide doesn’t tell you exactly how to do the whole installation, but this is largely because the process is self explanatory. However, it is important to note that in my case, I installed the operating systems in this order: Mac OS X, Linux, FreeBSD, Windows. It’s mandatory to do Mac OS X first, mainly for setting up the first and second partitions. But after getting Mac OS X installed, the rest of the operating systems theoretically can be installed in any order.

If you have questions, please email me and I’ll try to update the article as needed.

Rewriting Onlink

Saturday, August 15th, 2009

Miah and I decided a few weeks back that the code that Onlink is based on is unmaintainable. The code is riddled with some pretty major problems:

  • Poor design decisions
    Every Person is also a Company? Why?!
  • “magic” numbers littered everywhere
    One example: I tried changing the default values for the Date class to be one second past the UNIX epoch (1/1/1970) to allow for use of standard date functions (difftime, mainly). Unfortunately, this didn’t work too well when missions were generated, because other places determined whether the Date had been set or not by comparing the year to the old default (1000). So all the missions actually expired before they could be displayed, since they appeared to be over 50 years old.
  • OpenGL immediate mode
    According to Alastair, immediate mode is deprecated. He’s been harping on me about using immediate mode for graphics code. Generally speaking, he’s right to harp on this. In most cases, immediate mode is much slower than using the logical alternatives: display lists and vertex buffer objects. It’s also a lot harder to code without immediate mode. But we’ll work through this over time.
  • Bizarre problems that can’t be easily or immediately identified
    The current save issues with Windows XP, for instance.
  • Licensing issues
    It’s difficult to get contributions to the project when every developer has to have obtained the Uplink Developer CD [legally]. It just makes sense for this to be an open source project instead.

Long story short, we’re sick of maintaining code that is unwieldy. It’s time for a proper rewrite.

So on July 22, I started working on Codename “Cerberus”. We have decided that keeping the name “Onlink” anymore is inappropriate, since this is no longer just a mod of Uplink. We don’t know what we’ll call it yet, but for the time being, it’s just “Cerberus”.

I basically took the ARC++ code base, ripped out anything ARC-specific, and started coding on it. My first job is to come up with a decent interface. Miah and I asked the community for suggestions, and we saw some pretty interesting ideas that we may integrate.

Miah and I decided that we needed to start off with a few bullet point goals for the project:

  • Fast
    We want to keep slow code out of our project. We don’t want to cut corners and use poor code, even if it saves on development time. We want to focus on the long-run. Maintainable, high-performance code.
  • Customizable
    We’re going to try to make pretty much everything configurable.
  • Intuitive
    Everything should be easy to figure out and learn, especially for people who haven’t even looked at Onlink before. Following in the footsteps of the developers at Valve, we want to make the game teach the player how to play as they play. Hopefully, you shouldn’t need tutorials or manuals to figure out what you’re doing, if we do this properly.

We have more than that, but we don’t want to give away our content concepts that we’re working on.

Also, Miah and I will be discussing open source licenses for Cerberus sometime between August 29th and September 3rd, and we’ll hopefully be able to publish some code at that point.

So within the next couple weeks, we’ll be posting several different screenshots and user interface demos, and hopefully get some comments and suggestions. Subscribe to this blog’s RSS feed to keep tabs on this!

Crazy Backup Solution

Monday, July 13th, 2009

I am thinking about attempting to install FreeBSD on my Mac again. The last time I tried, though, it wiped my GUID Partition Table (GPT) and my Master Boot Record (MBR). So basically, it trashed the system and I had to reformat again. It wasn’t lovely. So this time, I decided to back up important things. My source code is already backed up, thanks to Git. My preferences, bookmarks, etc are all backed up by .Mac (oh, excuse me. “MobileMe“), so all that’s really left is my Applications folder and a few miscellaneous documents.

Normally, when backing up my Applications folder, the process is fairly straightforward:

  • Run AppFresh to get the latest versions of my apps.
  • Run Xslimmer to strip out localizations and PowerPC parts.
  • Hand-pick the applications I want to back up and copy them to a network share.

Unfortunately, this is prone to error, especially in the third step. It’s possible to click too quickly when multi-selecting, and open dozens of applications at once. I don’t much like that. So I hacked together a fairly simple (but quite nifty) solution to deal with backing up my applications.

Let’s start with the problem. The problem is that there are several applications that I shouldn’t back up, because they’re provided by the Mac OS X installer. iTunes, Front Row, etc. In fact, I can’t think of a single application made by Apple that doesn’t either come with the OS or need an installer in order to operate correctly (i.e. Aperture, Logic, Final Cut, iLife, iWork, etc). So these applications need to be excluded while backing up.

So I started with a simple script that filters through the applications I’ve got and finds ones that need to be backed up:

filter.sh

#!/bin/bash

BANLIST="$(cat filter-blacklist)"
WHITELIST="$(cat filter-whitelist)"

# We don't want spaces to muck up the paths.
export IFS=$'\n'

rm -f backup-queue
for a in $(find /Applications -maxdepth 1 -depth 1 -type d | sed 's/\/Applications\///g' | sort -f); do

	# Simple exclusion rule for Apple-provided apps.
	EXCLUDE=$(cat "$a/Contents/Info.plist" 2> /dev/null | grep -A 1 CFBundleIdentifier | grep com.apple)

	# We also run this through a blacklist and a whitelist, just
	# in case there's something that we _did_ want to be banned
	# or vice versa.
	if [ "$EXCLUDE" == "" ]; then
		BANNED=0
		for b in $BANLIST; do
			if [ "$a" == "$b" ]; then
				BANNED=1
				break
			fi
		done
	else
		BANNED=1
		for b in $WHITELIST; do
			if [ "$a" == "$b" ]; then
				BANNED=0
				break
			fi
		done
	fi

	# JUDGEMENT TIME
	if [ $BANNED -eq 0 ]; then
		echo $a OK
		echo $a >> backup-queue
	else
		echo $a BANNED
	fi
done

The script requires two files. A whitelist (applications that should be backed up, but are excluded by the filter) and a blacklist (applications that shouldn’t be backed up, but pass the filter). Here are mine:

filter-blacklist

Adobe Bridge CS4
Adobe Device Central CS4
Adobe Dreamweaver CS4
Adobe Drive CS4
Adobe Extension Manager CS4
Adobe Flash CS4
Adobe Illustrator CS4
Adobe Media Encoder CS4
Adobe Media Player.app
Adobe Photoshop CS4
AppleScript
Microsoft Office 2008
NetBeans.app
Utilities
VirtualBox.app
VMware Fusion.app

 

filter-whitelist

Plasma Pong.app

A bit of explanation on my blacklist and whitelist… Plasma Pong.app is on the whitelist because the author specified that the CFBundleIdentifier is ‘com.apple.plasmapong’, which gets caught by the anti-Apple app filter (a CFBundleIdentifier of ‘com.plasmapong.plasmapong’ would be better). The folders/applications listed on the blacklist are ones which are either provided with the OS (‘AppleScript’, ‘Utilities’), or require an installer to function (‘NetBeans’, ‘VirtualBox’, ‘VMWare Fusion’).

So anyway, the script uses these two files and its filter to figure out what apps should be included or excluded. Let’s look at what the script outputs in my case:

Alcarin:Applications steven$ ./filter.sh
0xED.app OK
Address Book.app BANNED
Adium.app OK
Adobe Bridge CS4 BANNED
Adobe Device Central CS4 BANNED
Adobe Dreamweaver CS4 BANNED
Adobe Extension Manager CS4 BANNED
Adobe Flash CS4 BANNED
Adobe Illustrator CS4 BANNED
Adobe Media Encoder CS4 BANNED
Adobe Media Player.app BANNED
Adobe Photoshop CS4 BANNED
Angband.app OK
Anxiety.app OK
Aperture.app BANNED
AppFresh.app OK
AppleScript BANNED
AppZapper.app OK
Arora.app OK
Audacity.app OK
Automator.app BANNED
BetterZip.app OK
blender 2.48a OK
Braid.app OK
Calculator.app BANNED
Canary.app OK
CandyBar.app OK
Chess.app BANNED
Chmox.app OK
coconutBattery.app OK
coconutIdentityCard.app OK
Coda OK
Colloquy.app OK
CrossOver Games.app OK
CrossOver.app OK
Cyberduck.app OK
DAA Converter.app OK
Darwinia.app OK
Dashboard.app BANNED
Dictionary.app BANNED
Disco.app OK
Dock Library.app OK
DOSBox.app OK
Doukutsu.app OK
DropCopy.app OK
DVD Player.app BANNED
Dwarf Fortress 0.28.181.40d OK
Expose.app BANNED
Firefox.app OK
Flip4Mac OK
Flock.app OK
Font Book.app BANNED
Freeciv.app OK
Front Row.app BANNED
FrostWire.app OK
Geekbench (64-bit).app OK
Geekbench (Rosetta).app OK
Geekbench.app OK
Gimp.app OK
GrandPerspective.app OK
GridWarsOSX OK
Hacker Evolution Untold.app OK
HandBrake.app OK
iCal.app BANNED
iChat.app BANNED
Image Capture.app BANNED
Inkscape.app OK
iPodDisk.app OK
iStumbler.app OK
iSync.app BANNED
iTunes.app BANNED
Jaikoz.app OK
Leopard Cache Cleaner.app OK
LiquidMac.app OK
Little Snitch Configuration.app OK
MacHeist Chat.app OK
MacPorts OK
Mactracker.app OK
Mail.app BANNED
Microsoft Office 2008 BANNED
Multiwinia.app OK
NetBeans OK
NetNewsWire.app OK
Nocturne.app OK
Opera 10 beta.app OK
Opera.app OK
Photo Booth.app BANNED
Picasa.app OK
Picturesque.app OK
Pixelmator.app OK
Plasma Pong.app OK
Preview.app BANNED
Privateer - Ascii Sector OK
Quicksilver.app OK
QuickTime Player.app BANNED
RealPlayer.app OK
Safari.app BANNED
Scribus.app OK
SeaMonkey.app OK
Senuti.app OK
Sequel Pro.app OK
Shimo.app OK
SiteSucker.app OK
SketchFighter 4000 Alpha OK
Skitch.app OK
Skype.app OK
smcFanControl.app OK
Smultron.app OK
Spaces.app BANNED
Speed Download 5 OK
Stickies.app BANNED
SubEthaEdit.app OK
System Preferences.app BANNED
TextEdit.app BANNED
The Unarchiver.app OK
Time Machine.app BANNED
Transmission.app OK
TrueCrypt.app OK
Twitterrific.app OK
Unangband.app OK
Utilities BANNED
Ventrilo.app OK
VLC.app OK
VMware Fusion.app BANNED
Vuze.app OK
WebKit.app OK
World of Goo.app OK
X-Chat Aqua.app OK
Xbench.app OK
Xslimmer.app OK
Yep.app OK
ZAngband.app OK
Alcarin:Applications steven$

Alright, looks good. filter.sh also generates a file called ‘backup-queue’ which contains all the apps which passed the filter. So now, step two is a two-part process:

  • Archive the apps (gzipped tarball works for this)
  • Copy the tarballs to a network share

So I created a couple scripts for this (they can be run in parallel, which is fantastic for a couple reasons which I’ll outline below).

backup.sh

#!/bin/bash

cd /Applications

QUEUE="$(cat backup-queue 2> /dev/null)"

# We don't want spaces to muck up the paths.
export IFS=$'\n'

echo "Running backup queue..."

rm -iv *.tar.gz.lock *.tar.gz

for a in $QUEUE; do

	echo -n "Creating $a.tar.gz... "

	# Create a lock so that the move script won't touch
	# the file until we're done with it here.
	touch "$a.tar.gz.lock"

	# Do the actual work.
	tar -czpf "$a.tar.gz" "$a"

	# Mark this tarball complete.
	rm -f "$a.tar.gz.lock"

	echo "OK"

done

echo "All done."

 

move.sh

#!/bin/bash

DESTINATION="$1"

# The user doesn't know what he's doing, clearly.
if [ "$DESTINATION" == "" ]; then
	echo "Please provide a destination directory."
	exit 1
fi

# I spoke too soon. _NOW_ the user has no idea what they're doing.
if [[ ( ! -d "$DESTINATION" ) || ( ! -w "$DESTINATION" ) ]]; then
	echo "Destination specified is not a writable directory."
	exit 1
fi

# We don't want spaces to muck up the paths.
export IFS=$'\n'

cd /Applications

while true; do

	# Stays zero unless something actually gets moved.
	DIDWORK=0

	QUEUE="$(ls | grep tar.gz$)"
	for a in $QUEUE; do

		# If the lock file for the tarball doesn't exist,
		# we assume that backup.sh has finished its work.
		if [ ! -f "$a.lock" ]; then
			DIDWORK=1
			echo "Moving $a to $DESTINATION/..."

			# Finally move it to the backup storage
			# directory.
			mv "$a" $DESTINATION/ &> /dev/null
		fi
	done

	# We sleep if no work is done because otherwise
	# this infinite loop would waste quite a few CPU
	# cycles.
	if [ $DIDWORK -eq 0 ]; then
		echo "Nothing to do. Sleeping..."
		sleep 2.5s
	fi

done

You might wonder why I didn’t just have it tarball directly to the remote server. The reason is fairly simple. If the network latency is bad enough, it won’t be able to transfer over the network fast enough and the archiving process grinds to a halt while waiting for the network to catch up. And if the bottleneck happens to be in your archiving speed, this doesn’t adversely affect it, either. This parallel method ensures that the maximum amount of work is being done at a time.

8-bit ANTICs, Part One

Sunday, July 5th, 2009

When I was offered an Atari 2600 Junior for perhaps the most awesome reason imaginable, I could hardly believe my luck. I wasn’t able to have one when they were new, and the only time I got to play one was while I was at a grandmother’s house. I still have my NES, but it’s a different sort of thing. One plays an Atari for the experience. Or the nostalgia. As for me, I haven’t played one since I was about four, so it’s a little from column A and a little from column B.

So when a package came for me last week, I was thrilled, knowing what it was. Opening the package was a little less awesome, however. The system was a damaged one, and it was almost certain that all the visible damage was done in transit. Pieces were rattling inside the system and in the box.

100_0609It would suffice to say that the system isn’t exactly working at this moment, but I’ve not lost hope for it. The power light itself did light up for a brief moment before dying out, so I think that this system is savable. It’s sort of out of my scope right now, however, because I don’t know what’s wrong with it. That didn’t stop me from opening it up to take a look though.

Now, I know that things in the mail aren’t typically handled with kid gloves, but there’s one thing I’m particularly curious about, and that is what the hell was done to this package. The system was packed decently, but not superfluously. Yet the corner was, for lack of a better term, shattered. This could be due to the age of the plastic, but could be the sheer force that the package hit.

Case in point: How hard would you as the reader expect the system to be hit if I said peg A and switch B were physically separated?

100_0614Well, they were. I had though the switch was busted, but as it would happen, the peg was set to the “color” mode and the switch was jammed in the “B/W” mode. It’s just happy that the switch itself wasn’t broken.

It doesn’t make me at all happy, though. This isn’t the first time that the post office has manhandled my packages. The prior time was a full desktop computer mailed from southern Texas and showed up with a fractured motherboard, bent GPU, and shattered case. Best part was that this particular one was an insured package, and the post office refused entirely to honour that insurance.

But I digress.

At this point, I am simply wondering what might be broken. Seeing as I have never seen the insides to one of these things while functional, I really don’t know what might have been knocked loose or is now missing. I really am not thrilled with this turnout so far, but am hoping there’s something to be had from it.

I am welcoming any suggestion for repair. I have a few thoughts. In the first image below, it would appear that one piece here that I assume acts as a ground is separated. There was no screw to anchor it down, however, so that might have been incidental. The third and fourth pictures are the obverse and reverse of the main circuit board. Hopefully someone will have some insight.

A person can hope, right?

100_0610100_0611100_0612100_0613

(with apologies and givings of thanks nevertheless to frigginjoe)

The Future Definition of “Cross Platform”

Thursday, June 25th, 2009

Preamble

I would like to spend some time with the community to discuss a new technology. Well, not really “new” per se (it’s been in use since 2007), but it’s certainly something that’s recently gone under the crosshairs of my ire.

I speak of the Cider system, a method to deploy Windows games to the Mac cheaply and quickly.

So let’s have a discussion on this system. As usual, I hope you like text.

What is Cider?

From the TransGaming site: “TransGaming’s Ciderâ„¢ Portability Engine is a proprietary technology that allows PC games to be enabled on Apple’s Intel Macs without the traditionally expensive and arduous need to redevelop a game from the ground-up. Cider acts as a “wrapper” around the PC game dynamically translating PC API calls to the Mac OS X operating system. As such, games can be enabled with Cider in a matter of days to weeks as opposed to the typical man years that traditional development takes.

Why use Cider? (or: what’s the point?)

Cider is often used as a too-little-too-late correction for a major mistake made in early development: the use of Windows-only APIs like .NET, Direct3D, DirectSound, GDI, etc. Programs written to use these APIs can’t run on platforms other than Windows without the use of an emulation layer.

The .NET Framework – Strictly speaking, i don’t find a lot of utility in the .NET framework and tend to use other tools that the framework was meant to simplify. However, it is a best-option in a lot of circumstances for game developers. Portability is probably the biggest thing to this. By “portability”, we speak strictly of Microsoft-only systems; desktop and handheld devices, but the big deal here for the gaming industry is the Xbox portability. Since it’s a gaming platform, it makes it possible to have one’s product be both a PC game and a console game without much additional overhead. Compare this to the Wii method which includes a separate system the size of a 4U server.

DirectX – DirectX is a multitude of utilities from complex graphic rendering to control handling (keyboard, mice, and game control pads) to sound and music playback. Along with the .NET framework, DirectX is used because it works on Window 95 and above, Windows CE (to a limited extent), and most importantly, the Xbox.

Who uses Cider? (or: Dear God, why?)

At this point, game companies, rather exclusively, make use of Cider. You might know a few games: The Sims 3, Spore, City of Heroes and EVE Online to name a few.

Cider’s aim is to make porting a Windows game to Mac a lot easier, eliminating many hours of coding time. For instance, DirectX isn’t supported on Mac OS X, so the graphics API would need to be remapped to use OpenGL. Generally speaking, Mac games from major distributors don’t use Cocoa or Quartz, but rather OpenGL. The reason for this is simple: OpenGL works on pretty much anything. Mac OS X, Linux, FreeBSD, Windows, the iPhone, etc.

By why use DirectX at all if compatibility is an issue? There’s a few reasons. Several years ago, DirectX was a lot better in terms of performance than OpenGL. As a result, several companies spent good sums of money to train their developers to use DirectX Instead of switching to OpenGL, companies generally avoid the extra cost of training and continue to use DirectX. Once a game is done, it’s passed along to a separate team that does the port. Because all the code is in place as far as what a function call can do, it’s basically a codeswap; DirectX calls translated to OpenGL calls. Of course, it’s a lot more complex than this, but this is the general idea to the average layman; consider it like a game with a story written in English being translated to Japanese.

The process really is a long one, however, so a lot of money and hours need to go into it.

So what’s the problem? (or: The Domino Effect)

Know the saying “you get what you pay for”? It doesn’t apply here. Mac gamers pay as much as Windows players, and find that their experience is really diminished. That’s because Cider runs as an emulation layer.

There are various types of emulators. Full hardware emulators (i.e. Bochs, QEMU, etc), virtualizers (i.e. VMware, VirtualBox), and API wrappers (i.e. Wine, Crossover, Cider, etc). Full hardware emulators deal with translating machine code between two completely different architectures. This is perhaps the most complex and painful part of emulation. It causes tremendous overhead unless it’s done very well (i.e. a combination of dynamic recompilation and jitting). In the case of an API wrapper, the overhead shouldn’t be as much. The hard work is already done. The code is already native (x86 or x86_64).

Unfortunately, nobody has yet done an API wrapper particularly well. Current API wrappers such as Wine run much slower than they should. A majority of the Windows APIs can be mapped to POSIX API calls without much effort. Windows threads become pthreads. Windows critical sections become pthread mutexes. DirectX calls become OpenGL calls. But the overhead of an API wrapper such as Cider is much more tremendous than one might expect.

Is there much of a performance impact? Oh yes. Cider doesn’t care if you have more than one processor. Your state-of-the-art graphics card will be chugging along as if it were a 3-year old model. OpenCL, which is coming out in 10.6? Worthless. The Physical Address Extension (PAE) that allows you to run on more than 4GB of RAM? Likely going to be ignored. That’s to name just the hardware limitations.

On top of this, there are software issues. The method of coding to make DirectX fast isn’t the same method as to make OpenGL fast. It’s going to run slower because the original code for the game is totally unmodified. Additionally, and probably the biggest hit to performance, is the translation of the common WinAPI functions, such as page fault handling. Page fault handling is a very basic need; it’s not unusual to see a page fault delta of 20,000 per second on a very active application (i.e. games, which is what this is all about!).

Transgaming, the company that makes Cider and Cedega, have based their products on Wine for Linux. Since Transgaming is making these products with gaming use in mind, we might expect to see that Cedega outperforms Wine. Let’s run some basic micro-benchmarks and see how Cedega and Wine perform compared to native execution on Windows XP. Tests with the largest differences are shown in bold (thanks to Darek Mihocka for the great micro-benchmarking tool!).

Windows XP on a 2.33GHz Intel Core 2 Duo:

test 1 os crit 2 : 237 ms, 79000 ps/instr, 12 MIPS, 184.3 clk
test 2 os try ok : 11 ms, 3666 ps/instr, 272 MIPS, 8.5 clk
test 3 os pg flt : 71 ms, 2366666 ps/instr, 0.422 MIPS, 5528.4 clk
test 4 os virtqr : 14 ms, 466666 ps/instr, 2 MIPS, 1089.1 clk
test 5 os virtpr : 16 ms, 533333 ps/instr, 1 MIPS, 1244.2 clk
test 6 os isbadr : 150 ms, 5000000 ps/instr, 0.200 MIPS, 11665.0 clk
test 7 os isgood : 47 ms, 15666 ps/instr, 63 MIPS, 36.5 clk
test 8 os vxpflt : 71 ms, 2366666 ps/instr, 0.422 MIPS, 5528.4 clk
test 9 CmpExch 2 : 214 ms, 71333 ps/instr, 14 MIPS, 166.4 clk
test 10 GetInpSt : 9 ms, 3000 ps/instr, 333 MIPS, 6.9 clk
test 11 GetQStat : 1186 ms, 395333 ps/instr, 2 MIPS, 922.4 clk
test 12 PeekMsg : 1377 ms, 459000 ps/instr, 2 MIPS, 1071.1 clk
test 13 MemMap 2 : 107 ms, 3566666 ps/instr, 0.280 MIPS, 8332.1 clk
test 14 QueryPrf : 5243 ms, 1747666 ps/instr, 0.572 MIPS, 4078.6 clk

Linux running Wine v1.0.1, on the same 2.33 GHz Intel Core 2 Duo:

test 1 os crit 2 : 295 ms, 98333 ps/instr, 10 MIPS, 229.4 clk (1.24x slower)
test 2 os try ok : 12 ms, 4000 ps/instr, 250 MIPS, 9.3 clk (1.09x slower)
test 3 os pg flt : 163 ms, 5433333 ps/instr, 0.184 MIPS, 12679.3 clk (2.29x slower)
test 4 os virtqr : 44 ms, 1466666 ps/instr, 0.681 MIPS, 3425.8 clk (3.14x slower)
test 5 os virtpr : 91 ms, 3033333 ps/instr, 0.329 MIPS, 7091.1 clk (5.68x slower)
test 6 os isbadr : 156 ms, 5200000 ps/instr, 0.192 MIPS, 12151.0 clk (1.04x slower)
test 7 os isgood : 71 ms, 23666 ps/instr, 42 MIPS, 55.2 clk (1.51x slower)
test 8 os vxpflt : 173 ms, 5766666 ps/instr, 0.173 MIPS, 13485.5 clk (2.43x slower)
test 9 CmpExch 2 : 217 ms, 72333 ps/instr, 13 MIPS, 168.7 clk (2.43x slower)
test 10 GetInpSt : 34990 ms, 11663333 ps/instr, 0.085 MIPS, 27447.0 clk (3887.78x slower)
test 11 GetQStat : 34967 ms, 11655666 ps/instr, 0.085 MIPS, 27447.0 clk (29.48x slower)
test 12 PeekMsg : 42974 ms, 14324666 ps/instr, 0.069 MIPS, 33811.5 clk (31.20x slower)
test 13 MemMap 2 : 2545 ms, 84833333 ps/instr, 0.011 MIPS, 212090.9 clk (23.78x slower)
test 14 QueryPrf : 4723 ms, 1574333 ps/instr, 0.635 MIPS, 3674.0 clk (1.12x faster)

Linux running Cedega 7.3.1 on the same 2.33GHz Core 2 Duo:

test 1 os crit 2 : 286 ms, 95333 ps/instr, 10 MIPS, 222.4 clk (1.21x slower)
test 2 os try ok : 24 ms, 8000 ps/instr, 125 MIPS, 18.6 clk (2.18x slower)
test 3 os pg flt : 1995 ms, 66500000 ps/instr, 0.015 MIPS, 155533.3 clk (28.10x slower)
test 4 os virtqr : 7 ms, 233333 ps/instr, 4 MIPS, 544.4 clk (2.0x faster)
test 5 os virtpr : 96 ms, 3200000 ps/instr, 0.312 MIPS, 7477.5 clk (6.00x slower)
test 6 os isbadr : 1975 ms, 65833333 ps/instr, 0.015 MIPS, 155533.3 clk (13.16x slower)
test 7 os isgood : 102 ms, 34000 ps/instr, 29 MIPS, 79.3 clk (6.00x slower)
test 8 os vxpflt: FAILED!! (games using vectored exception handling would crash)
test 9 CmpExch 2 : 289 ms, 96333 ps/instr, 10 MIPS, 224.7 clk (1.35x slower)
test 10 GetInpSt : 4925 ms, 1641666 ps/instr, 0.609 MIPS, 3830.8 clk (547.22x slower)
test 11 GetQStat : 4796 ms, 1598666 ps/instr, 0.625 MIPS, 3732.8 clk (4.04x slower)
test 12 PeekMsg : 41978 ms, 13992666 ps/instr, 0.071 MIPS, 32859.1 clk (30.49x slower)
test 13 MemMap 2 : 2983 ms, 99433333 ps/instr, 0.010 MIPS, 233300.0 clk (27.88x slower)
test 14 QueryPrf : 4078 ms, 1359333 ps/instr, 0.735 MIPS, 3174.1 clk (1.09x faster)

Oddly enough, the use of Cider is actually worse than using the open-source Wine. The result: a far less enjoyable experience, but probably more enjoyable than, say, running a Windows game in Parallels or VMWare Fusion.

In short, you get what the game developer paid for. Low performance for a low price.

If you’ve play Spore on the Mac platform at all (which I have and I’m throughly unimpressed with the game), then you’re probably thinking that I’m taking things out of proportion. Well no, I’m really not. Games like Spore and the Sims 3 aren’t exactly what we call graphics intensive, but games like EVE Online have the potential for very intensive sequences (namely battles). Here’s the bottom line: you’ll never find games like Bioshock, Crysis or even Portal running on the Mac (more on Portal later).

Should I be concerned? (or: So what’s the problem exactly?)

The use of Cider drops the bottom line. Little additional development is required and the price tag is a lot less than porting to native code.

Should you be concerned? Oh yes. Cider encourages developers to be lazy in the porting process, allowing them to exploit the Mac and Linux gaming markets with underachieving products.

In general, Macs can be good gaming platforms, specifically due to the hardware used in Macs on a given day. Not the best hardware, but certainly among the top. If you’re going to run a Windows game in BootCamp, then you’re going to be pretty well off on high detail modes, even on the laptop models (at least the newer ones). Using a Cider port, however, means that you might be getting intermittent frames, even on low detail modes.

This isn’t really a big deal though if you don’t play games, right? I beg to differ. Games are fairly complex in the emulation layer support they require. The main reason that we’re seeing this behaviour in games is because it’s the one industry that lacks porting the most.

It’s fairly safe to say that constructing a wrapper for other applications would be a lot easier, particularly if the wrapper is not a lot more than Wine. Imagine if other corporations started adopting the Cider method of porting. What could we see?

Imagine Adobe Photoshop or Premiere ported with a Cider wrapper. Or Autodesk Maya, Blender or Bryce. All of these have ports in native code, but what if, in the recessed economy, these companies decide to go the Cider route to cut the bottom line?

This would be a very good way to effectively ruin the Mac platform. While more apps could theoretically be used on Mac OS X, they would all be running in an emulation layer, cutting back on performance. If you use these utilities for a job, there’s a good chance that you would think twice before buying a Mac due to the performance issues.

Apple is trying to make a habit out of optimization. It’s the key focus of Snow Leopard. Unfortunately, no amount of optimized code on the part of Apple will matter if companies start favouring the Cider method outside the gaming industry. Already, gamers that were originally excited to hear of big names like EA coming to the Mac platform are instead feeling disenfranchised rather than empowered. Now imagine if this was happening in the video industry or the graphical arts industry.

It’s been said by some others that Cider could means the death of Mac gaming as we know it, but the potential here is so much more dangerous. It also has lasting implications.

Unintended consequences (or: Piracy Made Easy)

I stated before that I was going to get back to Portal, and now I am. There has been a Cider port of Portal running around for some time now, but it’s not based from Valve.

It is in fact a product of a new fresh mode of piracy. The advent of Cider has actually done more for the piracy movement than anyone else. Generally speaking, pirates understand that they’re already getting more than they are paying for, so certain graphical issues (some things won’t render, namely shaders) aren’t that big of a deal.

However, Portal and Half Life 2 are both critically acclaimed and are on many lists for people as a “must play at least once”. It would thus make sense that they would be targets for piracy. With Cider, that which wasn’t possible without a name brand virtualizer is now possible to run on the Mac, albeit without the quality that you would expect from a natural port.

Pirates don’t care. Free is free and it’s hard to complain when a cracked product is perfectly playable, even if it’s not pristine in graphical quality.

Afterword (or: Digression)

Cider really has a lot going for it in the future, and that’s inescapable. Being able to make any graphically dependent program truly cross platform is really no small feat, and the near-guarantee that a product can work on multiple platforms without significant code changes is a rather tempting reason to use it.

This however really brings to bear the use of the term “cross compatible”. A program that requires a virtualizer certainly does fall under the definition, but something like Cider isn’t so easy to classify. It’s not really even a grey area; it’s more like a splotchy murk. I worry that there might be a point in time that it start affecting the work I do. I rather enjoy the Mac platform, but I’d be damned if I use it to run a dozen wrapped applications instead of a dozen applications with native code.

I mean, the Mac version of Word is bad enough as it is.

Stylish Yet Functional

Friday, May 29th, 2009

I decided my desktop’s current appearance was worth sharing. Enjoy.

My MacBook Pro's desktop