PC Benchmark Tests

A promising new PC system benchmark test has emerged that shows up a previously elusive audio interface problem. PC Notes investigates...

Back in SOS November 2003, I discussed benchmark testing of music PCs, pointing out that the traditional tests used by most mainstream PC magazines don't shed much light on how a particular computer will perform when running music applications.

For several years I've tested music PCs using the Cubase 'Five Towers' test, because its needs are relatively modest, so you can run it successfully on a wide range of PCs and compare their audio performance. However, its processor drain is so low on many modern PCs that while its results are still valid they are beginning to bear little relation to most real-world songs.

However, various more strenuous benchmark songs are now available that run rather more plug-ins and soft synths, and, in most cases, a clutch of audio tracks, bringing them more into line with what the majority of musicians are doing with their computers. Examples include the Nuendo/Cubase SX3 'Thonex' song, the more demanding Nuendo/Cubase SX3 'Fudd' test, with a different mix of plug-ins and soft synths plus loads of audio tracks, and two versions of Scott Reams' Sonar 4 benchmark song. You can download all of these from the ADK Audio web site at www.adkproaudio.com/downloads.cfm.

'DAW Bench'

There's now a new benchmark test around that has already caused quite a stir amongst manufacturers and users alike. 'DAW Bench' (http://dawbench.vze.com) can be downloaded and run in Nuendo/Cubase SX3 on both Mac and PC (although it's hoped that in future it can also be adapted for other sequencers).

Developed by Australian Vin Curigliano of AAVIM technology, it's been in development since August 2005, with the periodic involvement of DAW vendors, some audio manufacturers and various end users. The aim was to produce a test that could be reliably used by all parties as a reference for measuring DAW performance. What actually happened was rather more revealing, as we shall see.

The benchmark started life with the well-known 'Blofeld's Return' demo song that can be found on the Nuendo/Cubase CD-ROMs, which was then modified to keep the audio (hard drive) loading more constant and had 25 30-second audio tracks added to it, each containing a sine-wave tone. Then a Magneto plug-in (initially disabled) was added to each of the audio tracks.

Overall, this 'Blofelds DSP40' test runs 40 stereo audio tracks, 40 4-band EQ plug-ins, 40 VST Dynamics plug-ins, one Multiband Dynamics plug-in and up to 40 Magneto plug-ins. The idea is to run the song and then start enabling the single Magneto plug-in on each track in turn until you hear audio breakup, and then report the number of Magnetos your system can manage.

There are two strengths to this approach, compared with running a static test containing a fixed number of plug-ins and then simply reading the Cubase or Nuendo CPU meter. First, you avoid inaccurate meter readings (particularly at low latencies), but second, you also find out exactly how many plug-ins you can run on a particular system. Moreover, because even the slowest dual-core PC can manage nearly 40 Magneto s at 12ms latency, and you're enabling them one by one, listening for obvious audio breakup rather than relying on your interpretation of a CPU meter reading, you can reliably measure performance increments to about 2.5 percent (one in 40). Faster dual-core PC results become even more accurate because they can run even more Magneto instances. Such incremental accuracy not only makes it easier to quantify the relative performance of different dual-core processor families, but even to spot fairly small differences between different PCs with identical dual-core processors, which may be due to different BIOS settings, RAM speeds, and so on.

ASIO Driver Issues

I've never seen much point in testing how many instances of a plug-in you can run on a PC before it falls over, as I feel this bears little relation to the real world of the musician. However, in the case of 'DAW Bench' the balance of audio tracks, plug-ins and soft synths is a very reasonable representation of many real-world songs. It has also exposed some fascinating facts.

The most contentious of these is that while most PCs will manage to run a certain number of Magneto plug-ins without problems, if you save the test song at this point and then re-load it, it won't run without glitching unless you disable some of them again. The lower the latency, the more pronounced this effect. To give you an idea of the scale of the problem, one system managed 42 Magneto s at 6ms latency, but would only run properly with 29 when the same song was re-loaded.
The Protoplasm TSM synth from designer HG Fortune should excite any Wavestation-loving musician with a PC.

This phenomenon finally confirms anecdotal evidence from various musicians who've reported that a particular song ran perfectly one night, but wouldn't run without serious glitching when they booted up their PC next morning. Tracking down the cause could have been an extremely frustrating exercise, but for one fact — the Save/Re-Open 'droop' didn't happen on any system running a Lynx Two soundcard. The Lynx Two not only managed to run the tests with remarkable consistency between sessions, but was also the only product across a handful of audio interfaces from different manufacturers that managed to run 'DAW Bench' right down to 3ms latency at 44.1kHz (most others refused to run it without severe glitching).

Further tests confirmed that the Lynx Two was immune to the above issues across a selection of PCs running AMD and Intel hardware, whereas both PCI and Firewire interfaces from other manufacturers, including MOTU, Presonus, RME and Terratec, had large variations reported between sessions. This appears to be an ASIO driver issue, and that impression is reinforced by the fact that RME quickly released a beta driver for their popular HDSP-series interface that hugely improves its low-latency performance in this test, on both AMD and Intel systems.

System Results To Date

The test results so far have now been gathered together into graphic form on the DAW Bench web site, and some initial conclusions drawn. Using the Lynx Two as the reference audio interface, for consistency, systems based on Intel's Pentium D940 dual-core and AMD's X2 4200+ processors turned in fairly similar results. After the poor performance of Intel's D800 series, this is good news for Intel enthusiasts and those who need maximum compatibility with audio hardware.

In this test, at least, PCI interfaces also performed better than Firewire ones with AMD systems, but this interface variance was less pronounced with Intel-based PCs. On the other hand, Intel dual-core system results proved to be quite dependent on memory bandwidth — so the most appropriate RAM modules and BIOS timings are important. Results for both AMD and Intel overclocked systems have also been published that don't display a simple proportional increase in audio performance with clock speed, which suggests that other system variables are involved.

The Future

Overall, 'DAW Bench' already has a lot of potential for exposing issues with ASIO drivers, and I suspect that it's already being used by interface manufacturers keen to improve this aspect of their product's performance. No doubt driver revisions will emerge as a result, especially now that this test is available to end-users, who will probably want to investigate how their own interface performs at low latencies, compared to others.

I suspect that the test could also prove to be a great way for both professional and DIY DAW builders to find the optimum BIOS settings for each system they assemble, since with a suitable audio interface you can reliably push any system to the edge, reboot, tweak various aspects of the BIOS, such as memory timings, and try again.

There will always be those who claim that a single test can be biased to favour certain systems (Intel or AMD, and so on), via the choice of particular plug-ins, but if you suspect this it's easy enough to replace Magneto with something else and check for yourself. This benchmark was specifically designed as a torture test for the audio card, its drivers, memory bandwidth and hard drives. However, it doesn't specifically stress the CPU (the component that's often the limiting factor for many musicians), so I'm looking forward to the next benchmark test in the series, which will concentrate more on gradually loading soft synths into a project. I suspect that this test may be just as revealing in different ways.