Audio Processing with Image Units

Buy PDF

Apple Notes

Published in SOS September 2006
Printer-friendly version

Technique : Apple Notes

With Apple's Worldwide Developer Conference (WWDC) taking place at the beginning of August, the rumour mill has been grinding away in anticipation of the announcements CEO Steve Jobs might make in his opening keynote speech. While Apple have already pre-announced that Leopard, the next major version of Mac OS X (10.5), will be unveiled, there's speculation that the highly anticipated Intel-based successor to the Power Mac will also be shown.

Although Intel only introduced their Core processor in January this year, the processor that now powers Macbook Pros, Macbooks, iMacs and the Mac Mini, this month the company launched the Core 2 processor, which is part of the first family of chips to feature Intel's new Core micro-architecture. Simply put, the micro-architecture of a processor refers to its actual design, and, in addition to simply performing better and consuming less power, the new Core micro-architecture has two features in particular that will be useful to musicians: firstly, it offers support for EMT64 (Intel's 64-bit memory-addressing instructions), meaning that if and when Apple adopt the processor, the resulting Macs will be able to run 64-bit applications, which was previously only possible on G5-based iMacs and Power Macs. Samplers that can address large amounts of RAM are just around the corner...

Secondly, the Core micro-architecture introduces additional SSE-style instructions (termed by some as SSE4), the kind used heavily by audio developers to optimise performance for DSP algorithms, along with a more buzzword-friendly feature called Advanced Digital Media Boost (ADMB). SSE operations previously required two processor clock cycles to execute, and ADMB is best described by Intel's developer-friendly subtitle 'single-cycle SSE', meaning that an SSE instruction is fully executed in a single clock cycle.

Monalisa-AU enables you to process audio using Image Units intended for image processing by Apple's Core Image Technology. Ever wondered what a drum loop processed by a Triangle Tile filter would look and sound like? Now you can find out. Monalisa-AU enables you to process audio using Image Units intended for image processing by Apple's Core Image Technology. Ever wondered what a drum loop processed by a Triangle Tile filter would look and sound like? Now you can find out.

In theory, this means that a Core 2 processor can now execute twice as many SSE instructions in the same amount of time which, for time-critical audio algorithms, will translate into a huge performance benefit. Coupled with the fact that EMT64 already offers more SSE registers (a register is the part of the processor that stores data being worked on by SSE instructions), the Core 2 processor could be a huge leap forward for those working with audio.

Come On See The Noize

One particularly interesting technology Apple introduced in Tiger was Core Image, which makes it easy for applications to handle image processing in the same way that Core Audio makes it easy for applications to support audio. And while Core Audio provides a plug-in framework known as Audio Units, allowing third parties to create additional audio effects and instruments, Core Image offers the Image Units plug-in framework, allowing third parties to create image-processing tools, like the filters in Photoshop, for example.

If you're thinking "this is all well and good but what does it have to do with music and audio?", don't worry: we haven't changed our name to Picture On Picture. Image processing and audio processing actually have quite a bit in common. If you break it down simply, digital audio is represented by the level of a signal at a point in time, while a digital image is represented by the value of a colour at a point in space. Because of this, both audio and image manipulation make use of signal processing techniques such as filtering; so what would happen if you processed audio using an image filter?

While audio processing with image filters might not be something that would naturally occur to you, some Japanese developers have come up with a free Audio Unit called Monalisa-Audio Unit (www.monalisa-au.org) that lets you do exactly that: process audio via an Image Unit. It's currently Power PC-only, and although many of the Image Units produce a distorted mess, some of the results can be quite useful — and at least it would give you a useful way of answering the next person who asks if something you're working on could sound a bit more purple!

Monalisa-Audio Unit is still in its infancy and has some limitations. Currently, buffer sizes of 1024 are recommended, as the plug-in uses fairly long look-ahead times in order to have a big enough block to make the Image Unit's processing produce an audible result. There are also some stability issues to be worked out, but the basic idea is really nice and this will easily become the most curiously destructive Audio Unit in your collection!

Audio Developers On Universal Binaries For those, like myself, who enjoy technical details, Apple have posted an interesting interview on their Developer Connection pages (http://developer.apple.com/audio/audiodevinterviews.html). Here, engineers from Ableton, BIAS, Metric Halo, Native Instruments and Roland discuss the process of creating Universal Binary versions of the various companies' products. There are comments about the use of Apple's developer tools; information on how BIAS were already using Apple's hardware-independent Accelerate framework, so that the company's products offered some SSE-optimised improvements for Intel-based Macs with little extra work; and observations from Metric Halo about how a Power PC version of Spectrafoo ran faster on a Macbook Pro under Rosetta than it did natively on a 1.67MHz Powerbook G4 before they even created a Universal Binary.

Audio Shading

The technology behind Image Units is rather neat. The actual processing routines are written in a dialect of the Open GL Shading Language (which is known as the Core Image kernel language) and are compiled just before they are needed by the system. What this means for the user is that Core Image targets the most suitable hardware on which to run the Image Unit's processing; if you don't have any suitable graphics hardware in your Mac, the processing runs on the regular system processor core (or cores). However, with a graphics card featuring a GPU (Graphics Processing Unit) that supports programmable shaders (and most modern graphics cards do), Core Image will offload the processing to your graphics card.

For those who aren't sure about the term 'shader', this is simply a word used in 3D computer graphics for an algorithm that defines how the surface of an object should be rendered, such as how light should be reflected. However, shaders can implement other types of image processing features for 2D graphics as well, such as those exploited by Core Image, and many people have carried out research as to how shaders on GPUs could be exploited for other types of algorithms, audio being an obvious example.

With the current division between the computer's processor cores or separate DSP chips for audio — either the algorithm runs on the computer's processor or a specific DSP chip — a similar system for audio plug-ins, where the same plug-in was compiled 'just in time' (as they say in the computer world) to take advantage of whatever hardware you had available would be a huge step forward. Sadly, it's not that easy; as in the GPU example, shaders are better suited to some audio algorithms than others.

However, the authors of Monalisa-Audio Unit have also experimented with a couple of Image Units that implement delay and band-pass filter audio algorithms in the Core Image kernel language. Using Monalisa-Audio Unit to run these Image Units, you can now experiment with audio-based algorithms running on a GPU. It's not necessarily groundbreaking computer research, but it is the first time most users will have been able to experiment with such technology so readily. The same authors also hope to develop a reverb algorithm as an Image Unit, which was a goal announced by a company called Bionic FX (www.bionicfx.com) back in 2004, although its web site appeared to have recently expired at the time of writing.

While technology like Monalisa-Audio Unit is still experimental, it shows promise for future methods of audio processing, taking some of the strain off the computer's main CPU, in addition to providing a way of processing audio with tools you would never have considered using before. 

New Mighty Mouse Flies Without Wires Apple's new wireless Mighty Mouse is perfect for laptop users who want Mighty Mouse functionality, but without the cable — and without the need to give up a precious USB port. Apple's new wireless Mighty Mouse is perfect for laptop users who want Mighty Mouse functionality, but without the cable — and without the need to give up a precious USB port. A product that's long been both rumoured and expected from Apple is a wireless version of the company's Mighty Mouse featuring Bluetooth 2.0 — and it was released this month. The original, wired Mighty Mouse is now included as standard with all desktop Macs and was discussed back in October 2005's Apple Notes (www.soundonsound.com/sos/oct05/articles/applenotes.htm). Although it was technically still a single-button mouse, Apple's engineers used sensors to detect where your fingers were on the mouse, to provide multi-button functionality. Many users (myself included) had reservations about the use of sensors, which often triggered the wrong functions if you rested all your fingers on the mouse, but personally I've grown to like the Mighty Mouse, especially the scroll wheel that works both horizontally and vertically, which is great for navigating around the Arrange/Edit/Project windows in music and audio applications (despite the fact that it does need to be cleaned occasionally). The Bluetooth-enabled Mighty Mouse offers all the functionality of the wired version (with the exception of the USB cable, of course), and more. Unlike the wired Mighty Mouse, which used optical tracking to determine movement, the new wireless version offers laser tracking instead, which, according to Apple, offers 20 times the accuracy. The wireless Mighty Mouse requires two AA batteries (which Apple kindly include in the box) and features an advanced power-management system, so that less power is consumed when the mouse isn't in use. The wireless Mighty Mouse requires Mac OS 10.4.6 or later, and there's no mention of Windows or 10.3 compatibility on Apple's web site (the wired version officially supports Windows 2000 or XP, and could be used with 10.3 when it was released). The original, wired Mighty Mouse is still available for £35, while the new wireless version costs £39.