I was working on a very bare-bones Cocoa application on friday, and stumbled across a most annoying window server bug. The application I was working on creates windows on demand, but without using NIBs. Instead, I used the code below to create a new window:

[[NSWindow alloc] initWithContentRect:rect styleMask:(NSTitledWindowMask | NSClosableWindowMask | NSMiniaturizableWindowMask | NSResizableWindowMask) backing:NSBackingStoreRetained defer:YES];

Once the window is ordered front and becomes visible, the entire window server hangs. The Mac stops responding to keyboard input, with only the pointer remaining functional. That is, functional in the sense that it continues to move, but you can't actually do anything.

So, what is the cause of this bug? To my luck, I was still able to log in to my computer from the command line. Everything appeared normal - no excessive CPU use, and no particular memory demand. Using the 'sample' utility and gdb, I discovered that my application was hung with the following backtrace:

_displayRectIgnoringOpacity:isVisibleRect:rectIsVisibleRectForView:]
-[NSThemeFrame _recursiveDisplayRectIfNeededIgnoringOpacity...
-[NSView _recursiveDisplayRectIfNeededIgnoringOpacity:...
-[NSView _drawRect:clip:]
-[NSThemeFrame drawRect:]
-[NSFrameView drawRect:]
-[NSThemeFrame drawFrame:]
-[NSThemeFrame _drawFrameInterior:clip:]
-[NSThemeFrame _drawTitleBar:]
Draw3PartImage
-[NSImage _compositeFlipped:atPoint:fromRect:operation:fraction:]
-[NSImage _compositeFlipped:inRect:fromRect:operation:fraction:]
-[NSImage drawInRect:fromRect:operation:fraction:]
-[NSBitmapImageRep _drawFromRect:toRect:operation:alpha:compositing:flipped:ignoreContext:]
CGContextDrawImage
ripc_DrawImage
ripc_RenderImage
ripl_BltImage
ripd_Lock
CGSDeviceLock
_CGSLockWindow
_CGSSynchronizeWindowBackingStore
mach_msg_trap
mach_msg_trap

Notice the CGSSynchronizeWindowBackingStore call at the very end? I did too. It looks like the window server gets deadlocked in some way. If you wait long enough, the window server will come back in brief bursts, allowing you to give input to the application (for instance to quit it). The application can also be killed, but you won't see a response for the next thirty-or-so seconds before the window server collects its senses again.

Going back to my code, after commenting out nearly every part of the very small app, I narrowed it down to the way the window is created. More specifically, the problem is that I was attempting to create a so-called "retained" window, rather than a "buffered" window. Substituting NSBackingStoreRetained for NSBackingStoreBuffered completely solved my problems. (I had no good reason to choose one over the other when I wrote the code, so I picked the backing store constant more or less at random.)

A small example is available for download here. It creates a 128x128 sized window and displays it three seconds after being launched. It exhibits the bug on my MacBook Pro running 10.4.10. To experience the bug, you will need to move the window so that parts of it will be obscured by another application's window(s), or by the Dock's application switcher (Command-tab). You should be able to get out of the app by hitting Command-Q repeatedly, until the window server sees fit to respond. To be on the safe side, save your work before running the app.