Traditional windowing systems rely on a model that requires each pixel on the screen map to the desktop or to exactly one window. This approach is called a "switch model" because the window server repeatedly switches through all of the applications running and gives them each a turn to update their windows. This method is tried and true, but has some serious limitations if you try to do anything fancy like cutouts or transparency.
Consider the problem of updating windows with holes in them using an underlying switch model. Depending on implementation details, you could conceivably determine what would be visible in the window's hole and try to redraw this area, but even then the situation gets awfully messy for even remotely complicated circumstances. For example, if the background window were a movie or animation, you'd almost certainly experience some serious lag--not to mention waste a lot of resources in the process.
Quartz Compositor is a departure from the traditional switch-based approach in that it has applications draw onto an area known as a backing store instead of drawing directly to the frame buffer that gets rendered on the screen. Applications may draw onto their backing store using any supported method available, which facilitates and encourages the use of graphics frameworks such as OpenGL, CoreImage, CoreVideo, and QuickTime. Quartz Compositor composites each window's backing store together to create a single unified display, which it writes to the frame buffer using Quartz 2D. Since it doesn't care how the images in the backing stores get drawn, Quartz Compositor is said to be agnostic with respect to its drawing model.
Because the compositing approach allows pixels to be shared automatically by multiple windows and the desktop, the chore of designing a window with a hole in it now becomes trivial. When combined with support for transparency, its drawing model enables some amazing effects such as round transparent windows, drop shadows, nice rounded corners, the dashboard ripple, and the genie effect.
If you cut out a portion of the image used for the custom view, you can modify Apple's round transparent window example to demonstrate a window with a hole in it .
To say that all of this compositing requires a lot of processing power would be an understatement, but that's where Quartz Extreme fits in. Quartz Extreme is basically the marketing term that describes the hardware acceleration that's gained by moving Quartz Compositor's workload to the Graphics Processing Unit (GPU). Since GPUs are designed to accelerate OpenGL graphics, Quartz Extreme is further realized by treating the contents of each window as a two-dimensional OpenGL texture. This enables the GPU to process intense calculations such as shading, and frees the CPU for other work.
For a nice overview of how Quartz has evolved since the conception of OS X, check out this overview by Ars Technica. Or if you're interested in learning more about specific differences between Quartz Compositor and another windowing system, such as X11, take a look at this Slashdot thread. It provides a nice starting point for a self-guided journey into the world of advanced graphical techniques such as the Porter-Duff compositing, mesh warping, and Affine transforms that Quartz uses to strut its stuff.