Celemony Melodyne

Audio Manipulation Software For Mac

Published in SOS November 2001
Bookmark and Share

Reviews : Software: ALL

Few recent software products have stirred as much interest as Melodyne — a program offering unique pitch, time and formant-processing features, which was created not by one of the major music developers but by a maverick Bavarian musician and his software-designer friend.


Derek Johnson & Debbie Poyser

The small team behind Melodyne first came to the attention of the hi-tech music world early this year, when they attended the Los Angeles NAMM show to publicise their high-quality audio processing and manipulation software, which is based on some unusual and rather secret technology. Less than a year on, Melodyne has reached release level.

Capabilities

Melodyne is currently a stand-alone application for Mac (a PC version is coming), and its purpose in life is to analyse audio minutely and then process it such that almost everything about it can be altered at will. It can shift the pitch of monophonic audio without changing duration, change duration or tempo with or without changing pitch, and manipulate formants to alter voice or instrument character or to improve the character of shifted audio. It allows you to create harmonies from one vocal, or indeed construct whole new melodies, as well as making vocals and instruments do things they couldn't normally do. It can force audio of different tempos to run at one tempo, quantise audio timing, correct pitching errors across an entire track or one note at a time, superimpose pitch information from one melody onto another without affecting lyrics, alter amplitude, edit transitions between notes, and even alter vibrato. While it usually requires monophonic audio, it works on drum and percussion loops and will also process strummed guitar. Sounds impressive? It does to us...

Software & Hardware Setup

Installing Melodyne — which is compatible with any Mac OS above 9.0.4, including Mac OS X — is simple. Apple's CarbonLib 1.3.1 extension is required, but if you don't have it, it's a quick download from Apple's web site. Also, Melodyne wouldn't run for us until we enabled the Apple Enet extension.

CELEMONY MELODYNE £680
pros
Enormously flexible and controllable audio processing with many corrective and creative applications.
Renders all aspects of audio independently editable.
Most natural results we've heard.
cons
Not all features promised in pre-release literature have been implemented.
User interface and manual make it harder to learn the program than it should be.
Requires quite a lot of user input — and music-reading ability is a distinct advantage.
summary
Pitch-correction and tempo-matching are just two small aspects of what this creative and inspiring software can do. Melodyne is a tool that vast numbers of musicians, producers, engineers and audio professionals in general could benefit from.

MIDI isn't implemented in this release, so there's no OMS to set up, and Melodyne doesn't even have an ASIO driver folder: all you need do is tell the software the location of any ASIO driver in your system and it can then employ ASIO-compatible hardware's inputs or outputs. Sound Manager is supported, too, for use with the Mac's built-in hardware. The program CD contains a PDF manual — happily, in addition to a printed manual — and two folders of tutorial and demo files.

Recording & Importing Audio

Booting up Melodyne takes a while compared to other music applications, and we noticed a similar sluggishness elsewhere in the program, including when saving files. The first thing you see is the Arrange Window, which is roughly equivalent to the Arrange page of a mainstream sequencer. You can record audio directly into here, or import it. Celemony quote a typical maximum track count of 20 to 30, depending on your Mac. Our G4 managed 20, though on-screen controls began responding rather slowly at this point. We're told that Melodyne isn't currently optimised for song-length tracks, but we managed to work with long sections of music.

If audio is to be recorded (at 16- or 24-bit, with sampling rate depending on your hardware), Melodyne supplies a click, but since it is always at a fixed, rather loud, level it can be annoying. The best plan may be to import a suitable drum part and play along with that. However, anyone who hates playing to a click will be happy to hear that Melodyne allows a tempo to be assigned to freely recorded audio later. This sort of flexibility is at the heart of the program.

Actually recording with Melodyne is inconvenient, at least with this first release, as there's no play-through, so you can't monitor what's going to disk. Our solution involved patching the headphone output of our Digi 001 hardware into an analogue mixer alongside the audio to be recorded, so that the latter could be monitored during overdubbing. A direct out from the audio's mixer channel routed it to a 001 input, extending the signal path unnecessarily. A similar dodge was required with Sound Manager. However, recording was always successful.

Importing audio is rather easier. Whatever the type of audio, not only should it be monophonic, it should also be as dry as possible. Even small amounts of reverb can render audio unusable for certain processes, since reverb essentially provides an aural simulation of polyphony.

Detecting Melody

Melodyne does some preliminary analysis on every new 'melody' (Celemony's term for the contents of a track — which could equally be a speech or percussion recording), and has a stab, which is quite often 100 percent accurate, at determining its tempo. The next step prepares the audio for further manipulation. Choosing Detect Melody from the Define Melody pop-up launches a window offering a number of parameters. Usually, you can leave these alone, as Melodyne functions quite well with the default settings. Exceptions to the rule are speech and percussion: it's best to select their specific Detection options. There are settings for many instrument and vocal types, which define lowest and highest notes detected, although note-range choices can be overridden. (There's a good practical reason for limiting note ranges: if what's being detected suffers from spill, limiting note range should mean that unwanted audio won't confuse the process.) A tempo-detection parameter offers two choices. Present Tempo uses the overall Arrangement's tempo, while Own Tempo derives tempo from the audio being detected. We preferred to use Own Tempo and then match that to the Arrangement tempo, though either option can produce satisfactory results, since there are post-detection tools that allow further manipulation and matching of tempos.

The manual advises that you tell Melodyne what tonality you'll be working in, though we did pretty well without this step. You can choose from the 12 equal-tempered keys, or from a range of non-equal-tempered tone schemes (Pythagorean, mean-tone and pure ratios), or create a custom tuning. The tonality chosen can affect what notes are detected, and also how far the software moves notes when correcting pitches. The fact that the overall pitch reference can be offset, in cents or Hertz values, allows you to account for sound sources not at concert pitch.

Detection should take approximately as long as the audio itself, depending on computer power and, to some extent, the complexity of the audio. Once the audio is processed, it's possible to view the notes detected in the melody superimposed in staff notation over the waveform. This will be your first indication of whether Detection has been successful. Prepare to be impressed: for the most part, it will have been.

If speech or percussion has been Detected, the note overlay represents the rhythm of the audio only. And on the subject of rhythmic material, one consequence of Melodyne's ability to work out the tempos of drum parts, and then allow them to be manipulated, is that it's rather easy to match loops with wildly varying tempos. One tempo is always the main reference for the Arrangement, but there are tools for instantly matching foreign tempos to that reference — and then you can change the tempo of the whole Arrangement. There are methods for tempo-matching in other software, not to mention certain samplers and Roland's VP9000, but it happens here with much less in the way of artifacts, at least to our ears.

  Playback Algorithms  
  How a melody performs in Melodyne is affected by the user's choice of one of five Playback Algorithms:

• Sampler: with this algorithm, the least processor-hungry, audio isn't manipulated using Celemony's Local Sound Synthesis (LSS), and behaves as if it had been traditionally sampled or recorded to tape, with no formant or pitch correction when changing length or tempo.

• Time: best for percussive material, this algorithm ensures that attacks are maintained, with only decays being affected by LSS. Pitch can be changed, but not formants.

• Pitch: The entire sound is subject to LSS. Attack and decay elements aren't differentiated, and it's thus more suited to legato vocals and instruments. There's no formant control: if you try to drag a formant bar in the Editor window, the algorithm switches to Voice.

• Formant: This is similar to Pitch, but allows you to change the formant position of a note, though the changes are more drastic than with the Voice algorithm.

• Voice: This most complex algorithm offers the most natural sound, especially for vocals. As with the Formant algorithm, formant positions are altered during pitch changes, and small shifts from both algorithms produce similar results. For more drastic shifts, though, the formant-position adaptation of the Voice algorithm is designed to be more natural.

 

Editing The Melody Definition

Perhaps it would be too much to expect, but the results of the Detect Melody routine are seldom perfect. Its 80-90 percent accuracy is amazing enough — the melody of what you've sung or played into Melodyne will be clearly identifiable in the note display — but almost because the process works so well you'll immediately expect every nuance and grace note to be represented. If they're not, it's time to get your hands dirty in the Edit Definition window (shown below). This window organises Detected audio on a grid, where you can see individual note relationships against a vertical piano keyboard, with the score-based transcription running across the top. Here, with special tools, you can separate out notes that Melodyne hasn't caught, insert or correct bar lines (which also automatically corrects the audio file's Melodyne 'tempo map', useful for freely recorded audio), and fix mis-detected notes. Besides missed notes, common problems include notes detected correctly but placed in the wrong octave, and coughs and breaths detected as notes.

During editing, it becomes apparent that Melodyne may actually have detected notes that it appears to have missed. In the Edit Definition window, a 'phrasing' graph is superimposed on the note waveforms. Phrasing in this case indicates vibrato, swoops and falls, and also pitch deviations that Melodyne doesn't consider significant enough to count as separate notes. However, it's these deviations that you should look for first when editing a melody: using the Separation tool on or near such a deviation nearly always separates out a note that was missed during detection. It's as if Melodyne wasn't sure, but as soon as you nudge it by clicking on an obvious note, it knows what you mean.

Depending on your perspective, you could find the Definition editing process either annoying and fiddly or quite satisfying, but it is essential, because the accuracy of the Melody Definition has repercussions for what you will be asking Melodyne to do. If you haven't separated all the notes from each other, for example, they won't behave properly if tempo is altered later. The same goes for the placing of bar lines in a performance without a steady tempo.

Celemony know that the detection process doesn't always detect every note perfectly. We suggested that they consider a 'sensitivity' option in the Melody Detection window, to get Melodyne to work harder and pick up fractional pitch/rhythm values more reliably. Celemony's Carsten Gehle responded: "It is a general problem that a 'sensitivity control' would not be a linear parameter here, like 'less or more', but a whole set of parameters. We found that it is faster for the user to correct some separations in the Edit Definition window than to specify a set of detection parameters and still never be sure of getting 100 percent correct note separations. It's often very obvious where note separations are, but only if you know the music and the context. Generally, we would like the user to consider it a necessary step to edit the Melody Definition after detection."

Closing the Edit Melody window causes a Save Melody Definition? box to appear; clicking Save returns you to the Arrange window and assigns your tweaks to the melody — or so you'd think. What actually happens is that no changes appear to have been made, and the melody still appears as originally Detected. This is because you need to do a couple more things to make these changes update, and you're not told of them if you work through the manual tutorial. First, you have to choose Reset to Definition in the Define Melody pop-up. Then you select Tempo from Melody in the Edit Score Time tool's Edit Tempo pop-up if the new tempo is to be used for the whole Arrangement. Alternatively, you use Adapt Time if you want the Arrangement's tempo to override that of the Detected audio. This process speeds up or slows down the audio so it fits its correct number of bars at the new tempo.

It's easy to get confused by the Edit Definition window, especially since the manual fails to explain the software properly. This, in conjuction with a certain lack of streamlining in the interface, makes Melodyne seem more daunting than it really is. It's worth remembering that, whatever you do to the audio in the Arrange or Editor windows, in the Edit Definition window you'll always see the original Detected (and manually corrected) melody. The hierarchy seems to be: audio file, which is always untouched; Melody Definition, which determines how pitch, expression and timing information are interpreted; and a third level in which the melody is manipulated in the Arrange or Editor windows.

  An Interview With Peter Neubaecker, Melodyne's Creator  
  What is your musical and software development background?

"Although I am a guitarist, my main approach to music has always been trying to understand 'what music is all about' in philosophical and mathematical terms. So I came to programming, but I never made programs for sale. Rather, each was a tool for my compositions, or even a composition in itself — for example, transforming planetary movements or prime numbers into audibility."

Did you create Melodyne for your own applications, or did you discover its core ideas and realise that there was a market for them?

"First, there was no intention at all to create software like Melodyne. Rather, there was the question 'What is a sound?', and the idea was to separate the ideal experience of a sound from its appearance in time. In fact, the solution came from the question 'What would be the sound of a stone?', presuming that a shape that has no relation to time might nevertheless represent a sound. The result was a method and algorithm that allowed the handling of the quality, pitch, and time of a sound completely independent of each other. This was four years ago, and after some time playing around with the method, I realised how different it would be to work with audio on the basis of this technique. I didn't look at the market at all; I just thought there should be a piece of software that works like Melodyne."

Who will use the software?

"I see Melodyne as a new concept of handling audio and music. If recorded material is not 'frozen' any more, but becomes as flexible as the musical process itself, you will handle the music in a different way. I think it needs some time to find out what that does to the process of composing and arranging music."

Can you tell us how Melodyne's principles differ from those of other pitch and time manipulation tools?

"The main difference is that other techniques were developed with the aim of 'manipulating pitch and time', whereas in Melodyne pitch, time and timbre are independent entities from the start. This is realised with our Local Sound Synthesis, which implies that you have a consistent local sound at each 'place' in the original sound. You can stop the sound anywhere and hear the timbre of that sound there, with any pitch you like. Imagine you generate a wavetable for any 'place' in the sound. Our algorithm doesn't generate these wavetables, it rather calculates what that table might look like for the values that need to be rendered. It always extracts these waveforms out of the original sound file in real time, and very fast. That's why you can manipulate the pitch and time of a lot of tracks in real time."

Why did you decide to 'go it alone' with Melodyne rather than teaming up with a large software house?

"I am not a businessman, and at first I preferred to sell the technology. I offered it to some major music software companies. They wanted it, but I felt that they did not really get the point of what it should be; it was considered another technique for pitch and time manipulation. That was not my vision. The other point was that I and a few friends have so many more good ideas for new ways of audio processing that we decided to have our own platform where new tools can be published. So, last year, I, my wife, and our friend Carsten [Gehle], who is a great software designer, founded Celemony Software GmbH. The name is derived from CELEstial harMONY, as the basic ideas were found in the Pythagorean and Platonic concepts of 'music of the spheres'."

 

The Editor Window

If the Edit Definition window is for corrective processing, creative activity starts in the Editor window (above), which is something like the individual track editor in a MIDI + Audio sequencer. Pitch information, rhythm, formant response, expression, articulation and velocity are available for manipulation here. You're able to specify extreme values well beyond the bounds of sensible usability — pitch and formant shifts of several thousand cents, for example. It's even possible to drag the tempo right down to 0bpm, at which point the Local Sound Synthesis at the heart of Melodyne resynthesizes indefinitely a single sonic point in time. The result sounds like an infinite version of the note(s) being played at that point, with no artifacts.

'Expression' includes such aspects of a performance as note transition, vibrato and swoops/falls. It's an eerie experience to tweak vibrato as simply as if it were an LFO control: you can literally remove vibrato from a vocal, or crank it up to serious pitch-modulation levels. Problems caused by vibrato changing speed when audio is stretched or time-compressed can be mitigated in Melodyne by editing vibrato depth (you can't change speed) until the result sounds natural — a feature unique to the program, as far as we know. Even Roland's VP9000 advanced vocal processor can't deal with vibrato, while TC's VoicePrism Plus (reviewed in this issue) can add it, but can't remove existing vibrato. Melodyne's ability to tweak a swooped transition between two sung notes is also shared only by the VoicePrism Plus. We weren't able to compare the two, but Melodyne's implementation is very powerful, creating convincing new transitions between original notes in a melody and ones that you've moved to create a new tune.

One obvious question to ask about a program like Melodyne is whether it can do automatic pitch correction, like Antares' Auto-Tune plug-in and hardware processor. The answer is yes, via a simple three-step process: highlight the note or notes you want to correct, choose Note Snap in the Quantise pop-up, then click on any note in the highlighted selection. Highlighted notes snap to the nearest semitone.

Even timing quantisation of audio is possible, with resolution of between a bar and a 32nd note (there are no triplet options yet). You highlight the notes to be quantised with the Move Notes tool active, select the quantise value and click on any note in the highlighted group, whereupon all events move to the nearest quantise position. It works very well. Unfortunately, the background display's quarter-note resolution can't be changed.

Formant offsets, applied at note or melody level, let you make extreme pitch-shifts sound more natural. For example, if you shift a vocal an octave down and it doesn't sound completely natural, you can choose the formant manipulation tool and drag intuitive on-screen bars to alter formant response. Surprisingly little tweaking produces the best results. Such a shifted vocal won't always stand in isolation, but when creating harmonies, for example, it's ideal.

The Arrange Window

The Arrange Window is the place to create multitrack Arrangements and do limited cut, copy and paste editing. Here you can build up a piece from loops, snippets or sections of takes, and re-use existing audio to create new material for harmonies, counter-melodies and so on. One minor irritation is that you can't highlight audio across several tracks.

You can also access a Temporary Play Offsets window in the Arrange page, to quickly create pitch and formant offsets and tempo changes for the whole Arrangement. If you produce a desirable result, it's possible to 'Fixate' (Melodyne's term) the settings to the Arrangement.

One unusual feature in the Edit menu is Paste Notes, which magically applies pitch data extracted from one melody onto another. This seems to work best when source and target have roughly the same rhythm or number of syllables, and could be used for copying harmonies created in Melodyne from one section of a song to another, where the tune is the same but the lyrics are different. We successfully pasted a tune onto a recording of a held synth note, after first 'separating' the synth note into 16th notes in the Edit Definition window. (This process had no audible effect on the note's playback.) Pasting the pitches of another melody onto the note then caused it to play back with those pitches — very clever stuff. A remarkable example of what can be achieved with this function is provided in the audio demos: the melodies of a six-part piece of Bavarian wind music used during the manual tutorial have been imposed on a sloppily recorded vocal version of the same piece.

Balancing tracks in an Arrangement is done with the basic mixer that's also used during recording. It's a flexible enough device, and grows with each new track that you add. Unfortunately, it'll grow right off your screen if you let it, and there are no scroll bars to help — you need to keep dragging it back and forth. Graphically, the mixer is very plain and is one of the more unfinished aspects of Melodyne. There's no track numbering, little labelling, controls are hard to move with the mouse, and the metering is inadequate. VST plug-in support isn't available in this version of Melodyne, so the only effect you can apply is a simple, uneditable reverb.

  Melodyne & MIDI  
  Detected melodies can be exported from Melodyne as MIDI Files. This is an attractive idea, which we first encountered in Emagic's Logic Audio, but the results aren't always satisfactory. One would presume that, once a melody had been tweaked until it both played well and displayed properly in notation, it would be the notation that's exported. Not quite: the export often appears to be based on the original Detected melody, and there often appears to be extra mis-Detected material derived from the performance that hadn't been apparent in Melodyne. There are some clever aspects to this procedure — velocity is successfully translated, for example — but overall it needs work.

There are no other MIDI aspects to Melodyne at the moment. Perhaps Celemony could consider adding the ability to create a new tune for a vocal by playing it in via MIDI, if they develop a MIDI side for the program in the future.

 

In Use

It took us several days to even begin to feel familiar with the way Melodyne's user interface works. Its audio-processing powers are extremely impressive (many of them are unique), and once you grasp the basics you can achieve great results without knowing all there is to know about the software, but it's not the easiest program to learn.

A major source of annoyance is the fact that there's a slightly different set of tools for each window, and each tool has its own buttons and pop-ups, which change when a new tool is selected. So if you're looking for a function you know you've come across somewhere, it's a case of searching through windows, clicking on all their tools and trying to spot the option you're after. Why not leave all the options for a particular window on display, but greyed out if the right tool is not currently selected? Your impulse might be to look up the elusive function in the manual, but you'll be hindered by a lack of consistency in nomenclature: tools have one name in the program, and one, two or three others (some of which are long-winded and confusing) in the manual. And there's no index.

Other things that bothered us were the difficulty of recording audio conveniently, and the lack of undo. This first release of Melodyne also misses several key features that were promised, namely an ASIO pipeline to compatible applications, and VST plug-in support. Most users will probably prefer to record in a dedicated recording application and import audio into Melodyne. Celemony don't see the program trying to rival audio-recording applications anyway. Audio import and export, in WAV, AIFF, SDII or SND formats, is easy, though we once or twice found unexplained gaps inserted at the start of audio exported from Melodyne. Also, the occasional exported file turned out to be stereo white noise. Even more oddly, the same files re-imported just fine into Melodyne.

Missing features to be introduced by the imminent, and free, v1.1 upgrade include ASIO monitoring, an improved click, improved (faster) handling of longer recordings, and multiple undo — all good news. Perhaps Celemony can also look at one or two odd, fleeting bugs, whereby tools or program elements don't appear when they should, or appear in the wrong form. It can also be disconcerting that the zoom levels and song position you've set are reset whenever you leave and then return to the Editor Window.

To put our comments here in context, Melodyne is still an amazing achievement from a small, new company, and if they listen to what users tell them about their experience of the software it will rapidly become more streamlined. In the meantime, no-one should be deterred from trying it just because it requires a bit of head-scratching!

Conclusion

What you can do with this program is amazing. Extreme changes to essential elements of all sorts of audio can be done smoothly and instantly. As always, you should try it for yourself before buying, but Melodyne seems to us to offer unprecedented levels of transparency, with minimal processing artifacts. Loops and snips of audio work together immediately, sopranos become tenors and vice versa (with some work on formants), tunes can be convincingly created from practically any source material. At the end of the review period, just for interest, we randomly threw together 13 tracks of disparate bits of Detected audio we'd been working with — a lead vocal and some 'created' harmonies, strummed guitar, a bass part (modified with Melodyne's help to play a different line), some sampled speech, four thoroughly unrelated drum loops, and a simple piano line — used the Adapt Time function so that they ran at the same speed, and it sounded fantastic. Weird, but fantastic. And armed with our edited Melody Definitions, we could then do pretty much anything with those audio elements.

Melodyne costs more than Auto-Tune, but then it does vastly more than Auto-Tune (which itself is a great tool for its purposes). However, it costs a lot less than either Roland's VP9000 or TC's VoicePrism Plus, which come closest to sharing its unique functionality. Even then, neither unit does everything Melodyne does (although the VoicePrism Plus's modelling of vocal characteristics such as rasp and growl is exclusive to it). This really is unique software, with numerous applications in composition, recording, remixing, even broadcast and post-production (its speech handling has to be heard to be believed). It's not cheap, and its user interface needs polishing, but it's a truly revolutionary creative musical tool, and we don't get so many of those these days.

  Test Spec• Melodyne v1.0.  
  • Apple G4 450MHz Mac with 384Mb RAM running Mac OS 9.0.4.

• Digidesign Digi 001 audio interface.

 

  To hear examples of Melodyne at work, follow these links:  
  MELODYNE & BRASS • In this example, you can hear how you can alter the playback tempo of a phrase without affecting the sound or pitch.

• Once Melodyne has detected the pitches of incoming audio, you can change the key of played phrases with complete freedom. Here you can hear the scale being changed several times, all in real time.

• Here you can hear the results of scrubbing the mouse cursor over a short section of analysed audio.

• Melodyne also offers complete freedom to change the formants of a recording, in real time if you wish. by changing the formants, the trumpet sound changes to resemble that of a trombone and a muted trumpet.

MELODYNE & VOCALS • This shows Melodyne at work on a recording of a medieval vocal piece, changing the pitch in real time.

• Here with each pass of the melody line, the last note is moved higher. The internal phrasing of a note can also be changed: in the penultimate pass, the vibrato is attenuated, in the last one it is exaggerated.

• Finally, this example shows how a performance recorded without reference to a click track or metronome can be quantised to fit a rhythm track, without affecting the perfomance.

 

 information
£679.99 including VAT.
Arbiter
+44 (0)20 8202 1199.
+44 (0)20 8202 7076.
Click here to email
www.celemony.com

SOS Readers Ads
GRAB A BARGAIN

£481,895

of Second-User Gear for sale now — don't miss out!
WIN Great Prizes in SOS Competitions!

 

Home | Search | News | Current Issue | Tablet Mag | Articles | Forum | Subscribe | Shop | Readers Ads

Advertise | Information | Digital Editions | Privacy Policy | Support | Login Help

 

Email: Contact SOS

Telephone: +44 (0)1954 789888

Fax: +44 (0)1954 789895

Registered Office: Media House, Trafalgar Way, Bar Hill, Cambridge, CB23 8SQ, United Kingdom.

Sound On Sound Ltd is registered in England and Wales.

Company number: 3015516 VAT number: GB 638 5307 26

         

All contents copyright © SOS Publications Group and/or its licensors, 1985-2014. All rights reserved.
The contents of this article are subject to worldwide copyright protection and reproduction in whole or part, whether mechanical or electronic, is expressly forbidden without the prior written consent of the Publishers. Great care has been taken to ensure accuracy in the preparation of this article but neither Sound On Sound Limited nor the publishers can be held responsible for its contents. The views expressed are those of the contributors and not necessarily those of the publishers.

Web site designed & maintained by PB Associates | SOS | Relative Media