I've been wanting to write this post for a while, but up until now there just hasn't been the time to do it. This post is about how I reverse engineered the Dock with the sole purpose of fixing Spaces in Leopard. There are a couple of reasons why I'm documenting how to do it now. First, 10.5.2 is probably going to be out soon. With some luck, that release will fix Spaces for us, rendering this post along with TripleS moot. Second, just in case 10.5.2 doesn't fix Spaces for us, this post might inspire someone to do the necessary work to make TripleS work on PowerPC based Macs as well. And finally, I've just received feedback from a user on one of the new MacBook Airs, where the current release of TripleS doesn't work. I don't have an Air, so someone who does will need to do the legwork in order to make TripleS work again on the new Airs. At least until 10.5.2.

TripleS works its magic by patching a single function inside the currently running copy of the Dock. Everything rests on knowing the exact address of the function to patch; the rest - while not trivial by no means - is luckily handled by nice libraries like mach_star. To understand how the Dock works, and thus discover the address of said function, we need to get a disassembly of the Dock. This is where we hit the first snag: Apparently, Apple decided it was a good idea to encrypt the Dock binary (which is located in /System/Library/CoreServices/Dock.app/Contents/MacOS/), so just running otool -tV Dock won't do us much good.

However, gdb still works, so we can attach to the Dock and get to its assembly code from memory. To dump the Dock's code, simply run the following commands (you'll need to figure out the current Dock's pid first):

[#] echo "disas 0x1000 0x110000" > disassemble.gdb
[#] gdb -batch -x disassemble.gdb -p dock-pid > Dock.asm

On my MacBook Pro, this takes about half a minute to complete. With the disassembly at hand, we are almost ready to start analyzing the code. But before we can do that, we need a starting point. We already know a bit about what the Dock does to switch [work]spaces. For instance, the function used to switch to a different space is called CGSSetWorkspace. Using gdb, we can set a breakpoint on this function, which will allow us to discover when and from where it is called. In gdb, set a breakpoint on CGSSetWorkspace, and then try to switch spaces:

[#] gdb -p dock-pid
(gdb) br CGSSetWorkspace
(gdb) continue

You'll notice that the switch halts, and that the debugger has stopped on the CGSSetWorkspace function. (Important note: It can be a good idea to start the debugger remotely from an SSH session, as you won't be able to switch spaces or command-tab to different apps once gdb has stopped the Dock. In that case, it is nice to be able to execute the continue command without jumping through about a million hoops to get it done.)

Using the backtrace command (bt for short), we'll be able to determine which places call the CGSSetWorkspace function when we manually initiate a workspace switch. Further, trying to make the Dock activate its infuriating auto-Spaces-switch behaviour, we can also determine how the Spaces switch is initiated automatically. It is then possible to go back to the earlier Dock disassembly we made, and figure out how the different parts fit together.

To make a long story short (well, shorter, at least), the Dock creates a new thread whenever a workspace switch is about to happen. The exact purpose of this thread escapes me at the moment, but suppressing its execution effectively prevents the Dock from switching spaces. For those of you looking at the x86 disassembly of the Dock from 10.5 or 10.5.1, the address where this thread starts is 0x80445.

We know that this thread is created prior to switching workspaces, so there are now (at least) two strategies we can follow in order to discover the new address of this thread in 10.5.2 and on the new MacBook Airs (assuming the mechanism for changing Spaces hasn't changed, which is - of course - always a possibility). The first strategy, which I outlined very briefly above, is to break on CGSSetWorkspace and then backtrack from the place where it is called inside the Dock. One of those places happens to be inside the workspace-switch thread; walking "up" in the disassembly will allow us to discover where the thread begins (typically just after the first ret instruction).

Another way to proceed is to break on pthread_create, which is called at offset 0x799b6 in the 10.5/10.5.1 Dock. Triggering a workspace switch now will allow us to discover the location of the correct pthread_create call. Just a few instructions prior to the offset of this call, the address of the workspace-switch function is pushed on the stack (or put in a register on PowerPC, if my recollection of the PPC ABI is correct). With the new address at hand, TripleS can modified and recompiled to use it. The exact line to modify is TripleS.m:62. If all goes well, TripleS will once again be able to fix Leopard's Spaces. Happy hacking!