Symphonova

Conductor, pianist and architect of the Symphonova system, Dr Shelley Katz.

Using a unique combination of virtual musicians and acoustics manipulation, Dr Shelley Katz aims to bring orchestral performances to even the most modest of venues.

The world of classical music has remained more or less serenely unaffected by the technological revolution that has been both nursemaid and concubine to popular music since the 1950s — at least, in the live sphere. Things may be different in the recorded world, where the search for ever higher fidelity has resulted in many significant technical developments but, for the most part, live orchestras still play acoustic instruments that evolve at a geological pace, in large and often old concert halls. Even when the music itself has turned futuristic, the surroundings and manner in which it is played would be familiar to audiences from 100 years ago.

This is not, of course, necessarily a bad thing; indeed, the ‘period instruments’ movement has achieved great artistic and commercial success by deliberately setting its face against the supremacy of the 19th-Century orchestra. But there are serious problems with the status quo.

Large-scale orchestral music is restricted by its very nature to major concert halls, most of which are located in big cities. This means that people who don’t happen to live in cities rarely get the opportunity to enjoy full orchestral performances, which in turn affects career opportunities for musicians and the popularity of orchestral music itself. It has other ramifications, too. Much of the ‘art music’ world relies on subsidies from patrons and governments, and questions can fairly be asked about why all such money is spent on city dwelling populations, leaving people in rural and isolated areas culturally deprived.

One answer to these objections could be to take orchestras to the people, much as leading theatrical companies try to do by touring the provinces — but while finding a theatre big enough to stage a serious play is one thing, finding a suitably sized concert hall with acoustics good enough to show a professional orchestra at its best is quite another. And then there remains that not-so-small matter of the cost. You might be able to make the maths work with a string quartet, or even a chamber orchestra of, say, 40 musicians, but a full-blown symphony orchestra can involve over 100 players, each of whom must not only be paid, but transported, fed and watered. It’s not a new equation and was one of the factors that put paid to the big-band phenomenon of the 1930s and ‘40s.

Bring It Home

On the face of it, the problem is intractable. Even if you could find the money to pay the orchestra, there are too few venues in which they could play. And it is here that technology has finally stepped in with a possible solution. On the surface, this might seem like something out of the pages of Professor Brainstawm or Heath Robinson (think Rube Goldberg for readers in the USA), but Symphonova actually has the potential to solve all of the above problems. It’s a system whereby a conductor can employ gestural control and a handful of real musicians to play an orchestra’s worth of instruments — and it is really quite remarkable. What is more, it can even simulate the sound of a large, world-famous concert hall in the alien and acoustically hostile environment of the sort of small venue most towns can provide.

Dr Katz’s conducting directs both the real musicians, and a Vienna Symphonic Library software orchestra.Symphonova is the brainchild of the Canadian conductor, pianist and inventor Dr Shelley Katz. Juilliard-trained, after graduation Dr Katz studied under several notable conductors, including Leonard Bernstein. After relocating to Germany, he worked as Studienleiter at the Staatstheater Mainz and the Theater der Stadt Koblenz, as solorepetitor at the Deutsche Oper am Rhein, and as conductor of the Duisburg Studio Orchestra. Apparently, he came up with the germ of the Symphonova concept in 1989 while working as Studienleiter and Kappelmeister at the opera house in Koblenz. Despite enjoying a career that has seen him work with many illustrious figures from the world of classical music and in prestigious venues around the world, Dr Katz made time to pursue his fascination with the ultra-high-quality sound reproduction and the gestural control of artificial systems, and earned a PhD through the Tonmeister (audio engineering) program at the University of Surrey. According to his CV, his thesis examines rule-based expression in computer-mediated performances of orchestral excerpts from Romantic opera, while his work on flat-panel loudspeakers has resulted in him holding several patents. All of the above, and quite a bit more, are put to work in the Symphonova concept.

The University of Surrey was home to the Symphonova project until 2014, supported by grants from the UK Engineering and Physical Sciences Research Council and by private and commercial sponsorship. Later, Dr Katz and Dr Abigail Dolan founded Symphonova Ltd and the project has moved from the laboratory stage to real-life performances. Armed with the technology Dr Katz has put together, what amounts to a small chamber-sized orchestra can undertake the works of composers who require a full-sized symphony orchestra, and do so in venues no bigger than a small cinema or town hall — and despite these limitations, they can produce a sound quality that wouldn’t disgrace the leading concert hall of a major city.

Having A Ball

The 15th-Century Herstmonceux castle, the ballroom of which was the venue for the Symphonova performance of Mahler’s Fourth Symphony.My personal encounter with Symphonova is worth recounting because it exemplifies just how far the concept can be taken. In the Autumn of 2017 Dr Katz took what I can’t help thinking of as a semi-robotic orchestra to Herstmonceux Castle in Sussex, to perform Mahler’s Fourth Symphony in the Castle’s ballroom. Now the overseas undergraduate college of Queens University, Canada, Herstmonceux Castle is an imposing 15th Century building which is no stranger to science and technology, having been the home of the Royal Observatory, Greenwich, from the late 1950s to the 1980s. The ballroom, however, is not as grand as it sounds, being a relatively plain, even boxy, rectangular room in the bowels of the castle and not, so far as I was able to tell, particularly notable acoustically. In fact, it really wasn’t somewhere you would want to place an orchestra at all, with a large portion of one end of the room occupied by rows of seats for the evening’s audience. In the hands of Drs Katz and Dolan, however, it was transformed.

It’s when you get to the ‘How does it work?’ stage that the project becomes tricky to describe. The room simulation uses modelling software from the Swedish acoustics company Dirac Research (see box), assisted by Dr Katz’s flat-panel speaker technology, while the orchestra itself is even more extraordinary. Instruments on stage that are not played by actual musicians are driven by transducers, in effect turning them into loudspeakers. These are fed from a computer-based system controlled, much like a real orchestra, by gestures from the conductor. Dr Katz’s inspiration to use as his ‘virtual’ musicians unplayed instruments that remain on stands throughout the performance is a marvellous example of lateral thinking. As he says, each instrument naturally produces the sound it is known for, so why attempt to simulate it from a loudspeaker? Why not use the instrument itself to produce its own characteristic sound? It sounds improbable, but it actually works.

According to Symphonova, the gestural control is a critical component of the system. This isn’t classical karaoke but a real performance from a MIDI score, using the Vienna Symphony Library of sampled sounds, which is controlled by a “new type of conducting wand”. The ‘symphonist’ (as Symphonova call what you would otherwise call the conductor) wears a small electronic board which controls the virtual instruments. “The left arm controls dynamics and the right arm is responsible for timing. Using conventional conducting gestures, the symphonist is able to communicate with the musicians present while simultaneously having full and instantaneous control over the virtual instruments,” is how Dr Katz explains it.

“The gestural control has two primary effects: A, the timing of events and, B, the intensity of events. Its primary purpose is to translate the expressive rhythm (rubato) which is communicated visually (using conducting gestures) to the musicians, into a data stream that communicates the salient information to the industry-standard DAW that I use — the excellent MOTU Digital Performer.”

I wondered how much control the conductor or symphonist actually had. Obviously, he or she can adjust tempo, but presumably amplitude as well? What other factors are under the conductor’s control?

“It’s a little more subtle than that,” Dr Katz explains. “By amplitude, I presume you mean the perceived SPL of loudspeakers. But that would be little more than akin to turning a knob on an amplifier. If, for example, one considers a single note played with musical expression by a skilled violinist, let’s consider how it might be performed to render a highly effective crescendo, from pianissimo to fortissimo. The note may start with little or no vibrato, very light bow pressure on the strings, and the bow placement might be close to, or over, the fingerboard. As the note is sustained in time, the violinist may increase the rate and width of the vibrato, increase the bow pressure, and locate the bow between the bridge and end of the fingerboard where the power of the player is best communicated. As a result of this kind of playing, the entire timbre of the violin will change, in conjunction with the SPL one might measure.

“If, as a result of my gestural control, the only change that took place would be to make the instruments louder, the sound of the orchestra would be much less expressive, and much more stilted. Consequently, the gestural control is also linked to a number of interrelated MIDI and audio automation controls, all of which conspire to create an effect that is closer to the behaviour of a live player.”

The ‘dummy’ instruments are fitted with transducers fed from the VSL setup.

Which brings us to the instruments themselves. It’s not hard to see that a transducer can work with a violin or a cello, but how does it work with, for example, brass instruments? “It is indeed a modified transducer,” explains Dr Katz. “Many people mistakenly believe that a wind instrument requires the flow of air for sound to occur. That’s only true because the air flow is required to make something vibrate, so that a standing wave is established in a tube (a horn, trumpet, clarinet or flute). In the case of brass instruments, the air flow is necessary to make the lips of players separate and come together at a particular frequency. The noise of the lips opening and closing — the Bernoulli principle — causes a standing wave to occur in the instrument. In the case of a clarinet, it’s the reed that vibrates, and in the case of a flute, it’s akin to blowing across the top of a bottle — the air stream flutters up and down across the opening of the mouthpiece.

“To turn these instruments into loudspeakers, all that is needed is to create a standing wave in their respective resonant chambers, using a signal that is native to the instrument.”

Instruments chosen and played by professional musicians tend to be carefully selected and are usually very expensive, so what about the ‘dummy’ instruments in the orchestra? Do they have to be as carefully selected as any in a conventional orchestra?

“Yes,” says Dr Katz. “A high-quality trumpet definitely sounds much better than bathtub-plumbing in the shape of a trumpet. If we had a billionaire with vision on board, I’d build an orchestra with the best possible instruments. But it’s been quite interesting and somewhat astonishing to discover how much the Dirac technology corrects deficiencies in instruments one would rank as merely good-quality.

“The ideal would be to use matching-quality instruments, and it would indeed make a difference. But dynamic equalisation in combination with Dirac goes a long way to correct and improve timbral differences, and create a more unified and convincing sound. I admit that I have not been willing to reach to very good string instruments because I cannot bring myself to cut them open and convert them. In time, we will no doubt find a skilled luthier to work with me on the instruments, and then I will upgrade the quality of the strings to fully professional instruments. But we might need to find that visionary billionaire first.”

Flat Panels

The flat-panel speakers in use at the concert puzzled me, I must admit. They are visually very unobtrusive: during the performance, I sat close to the back of the room, and didn’t really notice their presence at all until I happened to hear a slight hiss from them during the intermission. Were they actually doing anything and, if so, what?

Dr Katz explains: “They were absolutely in use! They were, along with the two at the front of the audience, defining and providing the Virtual Acoustics. And the audio they propagated was carefully configured to emulate the late reflections at the back of the hall we chose for the night [the Vienna Konzert Haus, Grosse Saal].”

Symphonova call their proprietary acoustic treatment the Symphonova Virtual Acoustic System, and claim that it can even create the acoustic illusion of a grand concert hall outdoors, but I was curious to know in more detail what it did, and what the Dirac system Symphonova also uses was doing. There are other questions, too. For example, the sound heard by the audience during a Symphonova concert largely comes from the acoustic instruments being played, so what, I asked, is being processed? Is it just the sound emanating from the loudspeakers, and does that take into account the acoustic sound the orchestra is producing and somehow balance the two for a combined effect?

“Yes, you are quite right — there is a whole lot more going on,” he confirms. “For now, I hope it will suffice to say that I use the Dirac to adjust the Instrumental Loudspeakers individually and in sections; I use the Dirac to adjust the acoustics; and then I use the Dirac to adjust the processes I use, which unify and blend the live sound of the musicians with the audio sound of the instrumental loudspeakers. The magic of the system is in the very natural physical tools that I use. The Instrumental Loudspeakers and the Transverse Wave Resonating Panel loudspeakers (for the virtual acoustics) both integrate naturally with the acoustic instruments of the musicians. Dirac is the wonderful oil that cleans and flows between everything, making sure it all works smoothly. Using the TWRP loudspeakers at the back, the Virtual Acoustic in the auditorium was defined and established. It became an active, live and, thanks to Dirac, fully corrected resonant acoustic environment that responded beautifully to the stimulus of the orchestra ‘on stage’, exactly as the auditorium in a great concert house responds to the stimulus of what comes from the stage.

“The goal of all this technology and effort is to do what every instrument maker has done: create an instrument that will be used by thousands of musicians and go where no orchestra has gone before!”

Super Nova?

So does it actually work? I make no claims to being either a Mahler scholar or an acoustician, but I do know that the late-Romantic composer’s works demand physically big orchestras and that even by tackling his relatively modest Symphony Number Four, Dr Katz wasn’t making life easy for himself. As for how close the sound was to that of the Vienna Konzert Haus, I am unqualified to say. However, I can say that the sound bore little resemblance to what you might have expected from the Castle’s ballroom! It was spacious, airy and natural — and if Dr Katz says that is what it would have sounded like in Vienna, I can see no reason to quibble.

Where Symphonova goes from here is going to be fascinating to see. Far from diminishing the number of jobs for professional musicians, it would, if widely adopted, increase by an order of magnitude the number of venues in which symphonic music could be played, making more work for classically trained musicians. How it will be accepted by the world of classical music is another matter, but Symphonova makes no claims to be a replacement for either major orchestras or their venues. Rather, it’s a supplement for them, a way for people who are denied the full experience to get a great deal closer to it. Perhaps, at last, technology has come to the symphony orchestra.