International Broadcasting Conference

This month Dave Shapton reports on the latest in broadcast video, from the International Broadcasting Conference in Amsterdam, and predicts how these innovations will be used by musicians as promotional tools. He also takes a look at the future of software‑only video editing.

I've just got back from the International Broadcasting Convention (IBC) in Amsterdam. It's a big show and it completely takes over the Dutch city — in fact, I actually saw fights for hotel rooms, the way we had petrol‑queue punch‑ups over here!

There's not much to interest the electronic musician there, apart from the usual armada of aircraft‑carrier sized mixing desks in the audio hall. I used to be awestruck at the capabilities of these things, but now I simply wonder how you can justify charging as much as a BMW five series for an eight‑channel module on a big digital desk. It's not as if each module is different. Chances are that they sit on a control buss that gives each one its own configuration, but basically they're all the same, however big the desk.

I know the argument from the manufacturers of these dinosaurs (note that these are my words and my opinions) is that you need a tool that is appropriate for the job, and if that job is working with world‑class talent then you need a big desk. Fine; but how many businesses can justify that kind of expense these days? Especially since, when you look under the hood of some of these desks, what you find is a PC with maybe a few DSP cards and an RS422 port.

Ten or fifteen years ago, if you had a million pounds or so, you could — with only mild accusations of grandiosity — buy a big desk to sit at the centre of your studio. This would then become your business, and you could make money from it. Now you have to be so rich that firstly, you hardly notice the expense and secondly, you don't mind running a loss‑making extravagance. And some of the companies that make these desks are losing money like nobody's business. Meanwhile, the rest of us go out and buy something that runs on our PCs or Macs that does the job very nicely indeed.

Virtual Studios

Away from the audio hall were some exhibits which, with a bit of imagination, could very easily show us the way technology is going for those of us whose main interest in life is music rather than broadcast video. Virtual studios are television studios whose on‑screen appearance is generated by a computer. The idea has its roots in the way weather forecasters stand in front of maps that aren't really there. The easiest way to do this is to use 'chroma key' (or as the BBC still quaintly calls it 'Colour Separation Overlay'). Chroma key is very easy to understand. Imagine that you have two photographs, one on top of the other. The bottom one is a weather map. The top one is the weather forecaster — the only object in the picture. The rest of the frame is pure blue. It's blue because there isn't much blue in skin tones, with the possible exception of varicose veins.

Now for the chroma key bit. Pick a colour on the top photograph, and make it transparent. If you chose blue, you'll see the weather map through the see‑through bits which means that you'll see the weather forecaster in the foreground, and the weather map in the background. If you chose skin colour you'd see a hole in the top picture the shape of the presenter's head, hands and arms, and you'd see the weather map through the hole. You can also do luminance keying, which is where the transparency is based on brightness rather than colour. It's a similar idea to velocity switching in samplers!

Basic chroma key is all very well, but you've probably seen what happens when it goes wrong. Just about the worst thing that can happen is that the foreground camera moves. Of course, whatever is going on in the background is completely unaware of the camera position, which would lead to some very strange effects. For example, if you zoom in on the weather presenter, all that will happen is that he or she will appear to get bigger. Because of this inflexibility, chroma key has been restricted to static shots and to special effects. To make it more flexible, and to create a true virtual studio, the background layer has to be able to track the position of the camera. And that's not easy (if you're not sure what I'm talking about here, imagine yourself on the set of Doom — but with photo‑realistic backgrounds).

Actually, to say it's not easy is a very big understatement. There are two significant technological hurdles. First, you have to devise a way to locate the position of the camera, and when I say position I don't just mean where it is in the room: I need to know its height, where it's pointing — even its vertical angle. I also need to know about its focus settings. I need to know all this because I have to generate a whole 3D environment, as it would appear in the camera's lens if it were actually there. That's the second difficult bit. It's hard enough generating any kind of photo‑realisitc 3D environment, never mind doing it convincingly in real‑time based on the position and movements of what might actually be a hand‑held camera!

Well, to cut a long story short, all of this is now possible due to clever sensor arrays on cameras and ceilings, and massive processing power. And you can buy it off the shelf for a few hundred thousand pounds. It sounds expensive, but it costs peanuts compared to the expense of both building a full‑size studio and manning it. Expect to see more and more virtual studios on TV from now on — especially when you consider that the same budget that used to fund four or five channels now has to pay for several hundred!

I've just got time to remind you that you really are reading a music technology magazine and not Video Engineering International or something like that. There's more clever video stuff to come, but it's all leading to an idea that could revolutionise the way musicians work with each other. Read on...

Motion Capture

Virtual studios have real presenters in artificially generated environments. You can also have virtual presenters in real — or artificial — environments. Alongside the stunning virtual studio demonstrations at IBC were examples of what is somewhat prosaically called 'motion capture'. That's like calling a Van Gogh a 'picture thing' for all it tells you about the subject. What the phrase 'motion capture' completely fails to tell you is that it's the science of collecting data about the motion of objects and feeding them into a computer so that they can be used as the basis for animating a computer generated model. If you've ever watched a computer game involving two opponents fighting, then the chances are that you've seen the result of motion capture.

The way it's done is to dress someone up in what looks like a black skin‑tight body stocking coverered with ping‑pong balls. The balls, so to speak, are attached to significant parts of the anatomy (don't titter...) which might include the ears, nose, shoulders, elbows, wrists and so on. Using multiple camera angles, the position of the balls is tracked while the 'actor' moves. These moves are mapped onto a generic model of a human being. This androgynous avatar, mirroring the movement of the real human actor, can then be 'clothed' from a wardrobe of 3D garments (and, of course, given a face) and can be made to look like just about anyone.

A demonstration by the Canadian company Kaydara showed how two‑dimensional cartoon characters can be made to interact with the real world in real time. One television channel, for example, plans to have a well known cartoon character take part in live phone‑ins.

For The Musician

Here's where it gets interesting for musicians... Firstly, having a quality video of your band or your songs is becoming an essential promotional tool. As videos get more sophisticated they get more expensive as well. Virtual studios are still way outside the budget of all but the wealthiest organisations at the moment, but I reckon we possibly only need a five to ten‑fold increase in computer power to do this kind of thing on a desktop PC. How long will that take? No idea, except to say that processing power has probably increased a thousand times in the last twelve years. You could argue that we're almost there already with the 3D processing power of the latest wave of graphics cards. Just think: for the price of a powerful PC (which you'll probably already be using as the basis for your studio) and a can of blue paint, you'll be able to video your band playing anywhere in the universe.

Then, after donning your motion capture suit, and possibly your motion capture guitar or saxophone, you'll be able to share your virtual environment with other musicians who might actually not be there, physically. I'm in a band and we struggle to get together because our bassist lives in Sunderland and the drummer lives in Eastbourne. The rest of us live at points in between. When we do meet up, it's brilliant, because there's nothing like working with other musicians (or drummers, for that matter). Just think, with your motion capture suit on, you could make yourself look like other people; Elvis or Bono perhaps. If you are Elvis, you can look like Homer Simpson.

Staying with video, I've seen the future and it's Purple — a video editing package by a company called FAST Multimedia AG, whose feverish imagination has previously named other products 'Silver' and 'Blue'. Purple really is the future because it's a professional video editing package that runs on FireWire‑equipped PCs. Working with the DV format, it's written entirely in 32‑bit code and runs under Windows NT. Rendering, which is what you have to do when the available processing power isn't enough to work in real‑time, takes place in the background, letting you carry on editing. My feeling is that Purple is roughly on a par with Steinberg's Nuendo: it's a successful attempt to do professional‑type work with a software‑only solution. It's got eight tracks of audio, audio punch‑in and decent audio scrub. Mac users might want to compare it with Apple's Final Cut Pro, another software‑based DV editing application. I'd love to see a head‑to‑head contest between these packages. My money would be on Purple...