Musical pitch has always been putty in the hands of Melodyne. Now, revolutionary features in version 4 promise to make tempo and timbre equally fluid.
Can you keep a secret? In the world of music technology, it’s usually not all that difficult. Many of the non–disclosure agreements we sign are about things so dull that no–one would have bothered to disclose them anyway.
Just occasionally, however, a product comes along that is genuinely revolutionary, and when Celemony first showed me a beta of Melodyne 4, it was immediately obvious that they had come up with something a bit special. A couple of months down the line, it’s ready to be unleashed on the world, and if anything, I’m even more impressed. Without wishing to give too much away now, I think it’s fair to say that this is not your average software update.
The first question on some readers’ lips is likely to be “What happened to Melodyne 3?” The last full release of Melodyne to be reviewed in SOS was Melodyne 2, as long ago as December 2009. This was the release that introduced Celemony’s landmark DNA technology, and it’s a mark of how advanced their algorithms are that, more than six years later, Melodyne is still almost unique in its ability to manipulate individual notes within a polyphonic audio recording after the fact.
Melodyne 2 was available in several different editions. The flagship application was the stand–alone Melodyne Studio, a multitrack audio recording and processing environment which could be linked to a DAW either using ReWire or through a special Melodyne Bridge plug–in. However, for some reason, Melodyne Studio never included the polyphonic DNA technology, and relatively few users needed its multitrack capabilities, so most opted instead for the more affordable and DNA–equipped Melodyne Editor package. This comprised a stand–alone Melodyne Singletrack application and a plug–in version which would typically be used as an insert effect within a DAW. Those who didn’t need polyphonic pitch-correction, meanwhile, could avail themselves of the cut–down Melodyne Assistant and Melodyne Essential versions.
Over time, however, Melodyne Studio evolved in a rather different direction from the rest of the range, and the answer to the question “What happened to Melodyne 3?” is that the Melodyne Studio application was the only one to receive an official v3 release. If, like most Melodyne users, you work mainly with Melodyne Singletrack or with the plug–in, you’ll still be using a point release of v2.
So the first thing to be noted about the new versions is that it brings all of the different flavours of Melodyne back into sync. Whether you buy Studio, Editor, Assistant or Essential, you’ll be using Melodyne 4, and the hierarchy of functionality between them should now be much more logical; there’s nothing in Melodyne Editor 4 which isn’t also in Melodyne Studio 4. The only possible fly in this particular ointment concerns backwards compatibility. Projects created in any previous version of Melodyne should open without problem in Melodyne 4. However, the Melodyne Bridge plug–in is no more, and there are a few features which were unique to Melodyne Studio 3 that have been dropped. It’s no longer possible to host VST plug–ins within Melodyne Studio, and nor can it accept MIDI input. Celemony say that these features were not much used, which I can well believe. However, anyone upgrading to Melodyne Studio 4 can keep their licence for v3, so if you do have DAW sessions that used Melodyne Bridge, or Melodyne 3 projects that employed these missing features, you’ll still be able to open them.
Ever since the launch of the very first version of Melodyne, Celemony have been at pains to present it as more than just a tool for correcting wayward pitching. It does, of course, excel at correcting wayward pitching; but it also lets users explore new creative pathways, working with recorded sound in ways that were never previously possible. Celemony’s data apparently indicates that about 60 percent of existing users employ Melodyne primarily as a corrective tool, with the remaining two–fifths of the user base exploring the creative side of the program. And in Melodyne 4, they’ve added features that will be very interesting to both camps.
Some of the biggest improvements are universal, and relate to the user interface. The stand–alone Melodyne Studio 3 application could open multiple audio files and display them in lanes, DAW–style, but you could only edit notes for one track at a time. Melodyne Studio 4 introduces genuine multitrack capability and, for the first time, extends it to the plug–in version.
Audio files can be opened in the stand–alone Melodyne 4 through the Import dialogue, or by drag and drop, and additional files can be brought into a Melodyne project at any stage. Files are analysed as they are added; Melodyne will select the most appropriate algorithm for the material, or you can force everything to be analysed using the algorithm of your choice. An optional Info Pane, akin to the track inspector in Cubase or Logic, displays detailed information about the selected track, and also provides a ‘quick and dirty’ way of auditioning pitch, amplitude and formant changes so that you can check whether the chosen analysis algorithm is the most appropriate to the material.
The plug–in implementation is, of necessity, a little different, but very well thought through. The plug–in window for any instance of Melodyne within a project can display a list of all the other instances down the left–hand side. As before, any audio you want to edit in Melodyne needs to be ‘transferred’ to the plug–in by playing it back in real time within the DAW — but it’s now possible to do this all at once, for every instance of the Melodyne plug–in in your project. Once the audio has been captured, the experience of working in the Melodyne Studio plug–in is in most respects now no different from working in the stand–alone program. In essence, it doesn’t really matter which plug–in window you have open, because each of them basically shows the same multitrack project, allowing you to edit any or all of the tracks at once. The window is fully resizable, and not only are keyboard shortcuts now freely assignable within Melodyne, but they work in the plug–in version as well.
I’m probably not alone in that I rarely used the conventional plug–in version of Melodyne 2. Especially in a large project where multiple parts needed to be processed, it usually seemed quicker and easier simply to render them as audio parts within the DAW and load them into the stand–alone Melodyne Editor, or to work within a DAW, such as Studio One or Sonar, which could host Melodyne as an ARA–format plug–in. Not any more. While ARA still has some advantages, and I’d love to see it offered in more host programs, the Melodyne 4 plug–in is quite simply a huge leap forward in usability, and the Studio version’s ability to offer full multitrack working within a single plug–in window is nothing short of revolutionary.
Once analysed, the stand–alone Melodyne Studio displays each file or track in its own lane, with waveform display, in the multitrack view that forms the top half of the window. You can slide audio back and forth along the timeline, but there’s no editing to speak of, so this view serves mainly as a handy visual reference (and, like most of Melodyne’s panes, can be hidden if you don’t find it useful). As well as a miniature fader with mute, solo and record enable buttons, the lane headers also feature greyscale and colour versions of the characteristic Melodyne ‘blob’ icon. Clicking on either of these icons for a track displays its contents in the main Note Editor window, the difference being that tracks shown in greyscale are for reference only and can’t be edited. So, for example, in a situation where you have several backing vocal parts in a song, you can now display them all together, and select, edit or apply macros to any combination of notes from multiple parts simultaneously; and if it helps to be able to view the notes that the piano or guitar are playing at the same time, you can make them visible without making it possible to edit them accidentally. A particularly neat touch is a feature called ‘spread unison tracks’, which applies a vertical offset between the parts being edited so that overlapping notes are clearly visible. It did occur to me that it might also be a nice visual aid if you could apply different colours to the tracks being edited, but in practice, I usually found it clear what was going on.
Another neat touch is that Celemony have minimised the need to mess around with faders or mute and solo buttons in order to hear what you need to hear for editing purposes, thanks to a single slider at the top right of the Note Editor area. In its leftmost position, this solos all and only the tracks selected for editing within the Note Editor. As you move it rightwards towards the centre position, tracks selected for reference within the Note Editor are brought up; and as you move it further right, the remaining tracks become audible, so that in its rightmost position, all Melodyne tracks are heard at equal levels (unless muted or attenuated in the track header). This is a beautifully simple yet effective piece of interface design, which condenses all the level control you really need into a single slider.
I think it’s fair to say that Melodyne hasn’t always been renowned for its elegant and intuitive interface, so I can’t overstate how much of a difference these developments make in practice. They turn what was a quirky and occasionally frustrating program into something that is a genuine pleasure to use. It’s no exaggeration to say that for existing users, the improvements to the user interface easily justify the upgrade to Melodyne 4 on their own. In fact, I’d go further. If you ever find yourself having to work with multiple backing vocal parts, or close–miked ensemble recordings, these changes fully justify the price of upgrading from Melodyne Editor to the multitrack Studio version of Melodyne.
Upon loading some audio into Melodyne, you’ll notice that there is now a broader selection of analysis algorithms to choose from. The Melodic and Rhythmic options are unchanged, but in the Melodyne editions that support polyphonic analysis, there is now a choice of two options. Polyphonic Decay is similar to the original polyphonic algorithm, whereas Polyphonic Sustain is designed to work with sounds such as legato strings that do not contain clear note attacks. Celemony say that they have improved the sound quality of their algorithms in version 4, and my experience bears out this claim. The polyphonic pitch analysis and correction in Melodyne 2.x was an extraordinary feat of signal processing, but in practice I often found that its side–effects were too obvious to allow me to use the pitch–corrected audio in any sort of exposed or natural–sounding context. Although success can never be guaranteed, the new polyphonic algorithms in Melodyne 4 more often generate results that are sufficiently transparent to work within a mix — and if they don’t do so first time, you now have improved options for fine–tuning them, as explained in the ‘Note Assignment’ box.
Pitch editing within polyphonic material is available only within the Studio and Editor editions of Melodyne. However, another new algorithm is supported within all versions of the program, right down to the lowliest Melodyne Essential 4. On the face of it, this new Universal algorithm might not sound that exciting. It doesn’t offer any pitch analysis or editing at all; yet, as we’ll see, it’s the key to some of the most powerful and innovative features introduced in the new version.
In a nutshell, the Universal algorithm allows the user to perform very high–quality time–stretching and time correction, and within the basic Assistant and Essential editions, this is probably its main attraction. Most of us have access to pretty good time–stretching within our DAW software, but there isn’t actually much choice on offer: behind the scenes, DAWs typically employ the same handful of algorithms, the most widely offered being Zplane’s Elastique. There’s nothing wrong with this, and compared with what was available only a few years ago, Elastique is capable of most impressive results. However, like all time–stretching algorithms, it can be hit or miss, and Melodyne’s Universal mode will make a welcome alternative. My own experience was that it seemed ‘lumpier’ than Elastique on complete mixes and other dense material, tending to alter the rhythmic feel in ways that weren’t always musical, but often gave more natural results on source tracks and solo instrument recordings.
The incorporation of such high–quality time–stretching makes the basic versions of Melodyne look very attractive, but in Melodyne Editor and Studio, it’s only the start of something much more remarkable. In the stand–alone versions of these programs, the Universal mode forms the basis of tempo analysis and manipulation features that sometimes defy belief. (Most of what I’m about to describe is not available in the plug–in versions, as it would ‘fight’ with features of any host DAW.)
An empty Melodyne document contains no tempo information at all. Drop an audio file into it, and whatever analysis mode you apply, Melodyne will create a tempo map, which looks fairly similar to the tempo tracks in DAW software such as Cubase or Pro Tools. Once created, there are two editing modes which can be engaged: in Tempo Assign mode, you seek to improve the way the map represents the music, and in Tempo Edit mode, you change the tempo of the music itself by editing the map. You can also add or edit time–signature changes in a little ribbon above the map. Where Melodyne detects major discontinuities in the song tempo, for instance as a result of long pauses or tempo changes, it will divide the map into regions; these can also be adjusted, added or removed manually. For most editing actions, the start and end points of tempo regions act as boundaries, so that what you do affects only what’s within a region. Where a part of the song has no discernible rhythmic pulse, you can create a ‘free’ tempo region, and Melodyne won’t attempt to edit it.
In both Tempo Assign and Tempo Edit modes, the mouse acts as a context–sensitive ‘smart tool’ with numerous different functions. Depending on what’s selected and where you hover the pointer, it can scale the tempo of everything to the left or right of the selection, apply a ‘flat’ tempo change to all selected beats, make a ‘bell curve’ edit with progressively larger tempo changes towards the middle of the selection, impose a ritardando or accelerando, and so on. Particularly neat is the ‘wave’ tool mode, which simultaneously speeds up the first half of a given selection and slows down the second half, or vice versa, thus changing its feel while keeping its overall duration the same. It’s also possible to force the entire selection to a fixed tempo of your choosing, and if your edits do result in a map that looks like a cross–section through the Alps, there are three further menu options to Smooth Tempo over several bars, several beats, or at a sub–beat level.
It’s possible to manipulate tempo to a certain extent in most DAW software, at least with audio that has a strong percussive element to it. Using Elastic Audio in Pro Tools or Audio Warp in Cubase, for example, you’d analyse the audio to find hit points, engage a warp mode and then quantise the hitpoints to an existing tempo map or use them to generate a new one. However, any resemblance between these typical DAW features and what’s possible in Melodyne 4 is really little more than skin deep.
For one thing, the way in which Melodyne analyses music is, well, musical. It’s capable of deriving useful and accurate tempo maps not only from drum–heavy music that’s already nearly in strict tempo, but from less obviously rhythmical audio recordings, sometimes with considerable freedom of playing. Even more important is that the editing tools it makes available are musical, too. Even if you are working only with MIDI data, I find that fiddling with tempo maps in a DAW is usually a painstaking and difficult process. Unless you’re simply conforming everything to a strict tempo, a great detail of detailed editing and trial and error are necessary to produce results that sound convincingly ‘human’. By contrast, the editing tools in Melodyne 4 just seem to do exactly what you want them to. It’s almost as though you are retrospectively changing the way in which the musicians played the music in the first place! This functionality, teamed up with the very high–quality time–stretching offered by the Universal analysis mode, has the potential to deliver stunningly natural results.
In the single–track Melodyne Editor application, tempo maps can be saved and loaded independently of the audio, which opens up some very interesting possibilities. For example, if you are working with a multitrack project, you could make stem mixes, open the drums or percussion stem in Melodyne Editor, generate a tempo map, and drop into Tempo Edit mode to correct a bit of speeding up on the second chorus, and render the drum stem as a new file. Then, you could open each of the other stems in turn, load the tempo map from the corrected drums, and conform all the other parts to that, quantising any sloppy playing as you go. An hour or two’s work, and you’ll have a stem session that retains all the wanted tempo fluctuations in the original recording, yet with a tighter and more musical lock between the instruments and no annoying speed–up. Better still, if you want to add loops or other audio content from a different session, you can easily conform them to the same tempo map, so that they not only are brought to a fixed tempo that matches the song, but follow its ebb and flow. Alternatively, you can export the tempo map as part of a MIDI file, so it can be transferred to your DAW’s timeline.
That’s all as cool as it sounds, but it’s nothing to what you can do in the multitrack Melodyne Studio. Here, of course, it’s possible to open all the files that make up a multitrack project at once; and when you do so, Melodyne not only figures out the most appropriate algorithm for each of them, but will calculate an overall tempo map that reflects the timing of the project as a whole — usually, it seems, by identifying the tracks that have the strongest rhythmic elements and averaging their timing in some inscrutable way. Tempo edits in a multitrack Melodyne Studio session preserve phase accuracy between tracks, so you can work with multitrack drums or other multi–miked sources without incurring phase problems. Engage the Auto–Stretch button, and any further audio you add to the project will automatically be conformed to the current tempo map.
One of the projects I used as a testbed was a multitrack recording of a folk song where we had dubbed piano, penny whistle, electric guitar and a harmony vocal over a live take of vocals and fingerstyle guitar. The original guitar part was rhythmic, but after a bright start at 95bpm or so, slowed down gradually through the take, ending at 85bpm or so. Apart from the piano, none of the other parts had much rhythmic content, and there were lots of legato slurs in the whistle part.
I wanted to add further rhythmic elements to the track, but the obvious tempo fluctuations made any drum part sit uneasily, so I decided to see whether Melodyne Studio could nudge the entire project towards strict tempo. To that end, I dropped all the files into a new Melodyne document, waited for the analysis to complete, and opened the Tempo Editor. Slightly to my amazement, the analysis had not only correctly focused on the guitar as the main rhythmic element in the multitrack, but had created a tempo map that fitted it like a glove, requiring no editing at all. Thinking this was possibly a bit too good to be true, I then tried forcing the whole thing to a strict 93bpm — and it worked! As long as I used Universal analysis mode for the polyphonic sources, there were no audible time–stretching artifacts that I could detect; and as importantly, the process preserved all the musical nuances of the performance. The legato parts retained their timing relationships with the guitar, and the small pushes and pulls that gave each part its ‘feel’ were retained; yet the large–scale tempo variation in the recording was gone, as though it had never been there in the first place.
Slightly spooked, I undid the automatic tempo adjustment and turned instead to the manual editing tools. It was soon apparent that these have been very cleverly crafted to work with patterns of tempo variation that are typical of musicians playing without a click. With a little practice, it becomes second nature to identify a slump, lump or hump in the tempo curve, select the appropriate section of it and drag to iron it out (or exaggerate it, if you want). Five minutes’ work gave me a tempo map that was nearly flat throughout, yet which retained a hint of the original ritardando in the middle and preserved the wanted slow–down at the very end.
Emboldened by this success, I engaged Auto–Stretch and dropped a drum loop onto a new track. It matched up perfectly. Taking the opportunity to perform a little corrective pitch work on the vocals, I then exported new versions of all the tracks — which, naturally, you can do all at once — and loaded them into Pro Tools for mixing. I honestly think that no–one would ever be able to tell it hadn’t been recorded to a click in the first place. If there’s any other program out there that could have achieved the same thing, let alone with the same speed and elegance, I haven’t come across it.
I must point out that Melodyne won’t deliver the same level of success with every project, and there are one or two wrinkles in the process that can trip you up. For instance, problems can arise if you decide to change the algorithm of a track after having performed tempo edits. In the example described above, I belatedly noticed that the piano was out of tune and decided to re–analyse it using Polyphonic Decay mode to apply some sweetening. When I did so, it came adrift from the modified tempo map; in fact, if I kept Auto–Stretch on, Melodyne trimmed a large section of silence from the start of the piano part, mistakenly assuming it was supposed to begin at bar 1. Even with Auto–Stretch turned off, it ended up in the wrong place, and although you can drag tracks left and right in the waveform overview, they won’t remap themselves to the project tempo at that point — there is no way to force Melodyne to locate the start of the piano at a particular bar number and have it remapped accordingly. There are workarounds, but in general, it pays to be certain that you’ve chosen the right detection algorithms before doing any tempo editing.
There are also occasions where a bit of lateral thinking is required in order to get Melodyne to place the bar lines properly. It spotted that one of my other test recordings was in 4/4 time and mapped its tempo perfectly, but was misled into placing the bar lines a quaver early by the presence of some syncopated accents that fell on eighth–note sub–beats. You can click and drag in the time–signature ribbon to reposition the bar lines, but only in the units of the time signature that Melodyne has chosen. In this case, the time signature had been correctly set to 4/4, and in order to drag the bar lines to the correct positions, I had to temporarily change it to 8/8.
There are a couple of minor but obvious ways in which the tempo functionality could be improved in future versions, too. The Make Tempo Constant menu option is currently all or nothing, and to my mind, would be more useful if it offered a continuously variable degree of correction. And in general, I think working with tempo could be made even faster if Melodyne Studio’s tracks view offered more editing options. The bars and beats ruler that runs along the top of this view doesn’t display time–signature changes, and nor can it be picked up and dragged to move the bar positions as you can with the same ruler in the Assign Tempo pane. And it’s not possible to select only part of an audio file in the tracks view, so if you want to, say, conform or detect the tempo of just one song section, you’ll need to go into the Note Edit view to select that section.
I must emphasise, though, that these are minor issues. In the majority of real–world situations, the new tools for working with tempo in Melodyne 4 are a pleasure to use, and they carry out their work with a musicality and transparency that is, as far as I know, completely unprecedented. In the hands of Melodyne, the pitch of recorded audio has always been fluid; these tools apply the same philosophy to the time domain, and with equally stunning results.
I mentioned earlier that Melodyne users tend to fall into two groups: those who use the program primarily as a corrective tool, and those who employ it to more creative ends. If the most obvious applications of the new tempo functionality lie towards the former end of the spectrum, then the other major innovation in Melodyne 4 definitely redresses the balance. Available only in the full Studio edition, the Sound Editor is a suite of tools that provide fresh ways of altering the tonality of recorded sound.
As is well known, any complex pitched waveform can be broken down into a number of sine waves at different, but related, frequencies. The lowest of these is called the ‘fundamental frequency’ of the note, while the others are described variously as ‘overtones’, ‘partials’ or ‘harmonics’. The timbre or colour of an instrument’s sound is partly due to the relative amplitude of the individual harmonics it generates; for example, the sound of a clarinet is characterised by prominent odd–numbered harmonics.
What Melodyne does when it analyses an audio recording using Melodic or one of the Polyphonic detection algorithms is to identify the fundamental and the harmonics belonging to each note. This information is has always been the basis for its pitch–based tools, but the Sound Editor exploits it in some new and highly original ways.
The Sound Editor has three different editing modes, accessed using buttons at the top labelled Harmonics, EQ and Synth. These are paired with a couple of slider controls which are visible in all modes. The first of these is called Emphasis, and its effects are perhaps easier to hear than to explain. As we have seen, each instrument produces a characteristic pattern of harmonic amplitudes, but these can vary quite a bit from note to note, as the instrument is taken up and down its range, or is played loudly or quietly. When Melodyne analyses a track, it calculates harmonic spectra for each note, but also what Celemony call the ‘mean spectrum’: a spectral ‘profile’ averaged from all the individual notes it has detected. (If multiple tracks are selected for editing, Melodyne averages them all out to calculate a single mean spectrum.)
What the Emphasis slider does is to exaggerate or subdue the ways in which individual notes diverge from the mean spectrum. Consider, for instance, a bass part which has been played rather unevenly, so that some notes stand out and others are buried. Compression might help to iron out brute level differences, but it can’t compensate for the tonal differences between notes that have been twanged hard and others that have been stroked gently. By moving the Emphasis control to the left of its default centre position, however, you can do exactly that, pushing the sound of all the notes towards a hypothetical tonal mid–point and introducing sonic consistency where there was once chaos. Alternatively, by going in the other direction, you can often make dull–sounding or lifeless instruments seem more dynamic and pokey without actually changing their level in the mix.
The other global Sound Editor control has been christened Dynamics, perhaps an unfortunate choice of name for something that isn’t a standard compressor by any means. This parameter does not affect the dynamics of the performance as a whole, or of one note relative to another. Rather, it changes the shape of each note’s ‘internal dynamics’. And unlike any conventional dynamics tool, it acts independently on each note within a polyphonic audio track.
In a typical subtractive synthesizer, you’d control the ‘shape’ of each note by assigning an envelope generator to control both the output level of the VCA, and the cutoff point of a low–pass filter. You could think of Melodyne’s Dynamics slider as allowing you to do the same thing to notes within recorded audio. Move it to the right, and dynamic variation within the sound is exaggerated; move it to the left, and the attack phase of the note is de–emphasised. Thus, ramping up the Dynamics of a piano recording makes it sound increasingly ‘plinky’ and Clavinet–like, while going the other way pushes it towards something more organ–esque. Reducing the Dynamics of a vocal part has a distinctive effect that is both like and unlike the action of a conventional compressor, and will no doubt be all over rap records by the time you read this.
Although they may be applying unfamiliar concepts, neither Emphasis nor Dynamics is mysterious from a practical point of view, and you soon get a feel for what they can do. In both cases, small adjustments (you can hold down the Alt key whilst moving controls to do so in fine increments) can often make a real difference to the ‘mixability’ of tracks without making them sound obviously effected or unnatural. More extreme settings are less predictable, but have plenty of special–effect potential.
As far as the capabilities of Melodyne’s Sound Editor go, though, these two sliders are the tips of the iceberg. To explore its novel sound–shaping powers further, it’s necessary to delve into one of the editing views (or ‘working areas’ as Celemony describe them).
Of these, it’s perhaps the EQ editor that is the closest to any existing processor I’m familiar with. At its heart is a graphic equaliser which operates in semitone bands, labelled according to the notes of the chromatic scale. The averaged frequency spectrum of your track is displayed as a complex curve, and as you play it back, the instantaneous energy within each band is shown by a tiny ball that moves rapidly up and down. Hover the mouse pointer over the editing area, and just as with the tempo map, it becomes a context–sensitive smart tool, allowing you to make subtle or radical changes to the frequency spectrum of your sound. Clicking and dragging above the curve selects a number of bands for editing, whereupon you can apply the same flat or bell–curve cut/boost shapes as in the Tempo Editor.
If you want to abandon your edits and return to the original curve, clicking on the cog–wheel icon will bring up a local menu that includes this option — along with some other, more creative choices. Global copy and paste options let you apply the mean spectrum of one track to another, and it’s also possible to copy the spectrum of a single note or notes within the Note Editor and paste this onto an entire track. The last two options let you clear the mean spectrum entirely, or randomise it, to predictably unpredictable effect.
There are also four ‘macro’ sliders below the EQ curve, which provide instantaneous ways of making particular kinds of edit. The Brilliance control boosts or attenuates the bands towards the top of the frequency spectrum, while Contour offers global control over the unevenness of the curve: move it to the left and frequency discrepancies are ironed out, while positive Contour values exaggerate hills and valleys within the spectrum. Towards its leftmost extreme, it actually begins to invert the original spectrum, so that peaks become troughs and vice versa.
There are other fine–band graphic equalisers that support this kind of editing and EQ ‘matching’, but what distinguishes Melodyne’s is that it is musically aware, a feature exploited in the Tonality slider. This attenuates or boosts individual semitone bands according to whether that band’s musical note is part of the detected scale. Pushing it rightwards emphasises ‘in key’ bands, while going to the left brings up the ‘foreign’ frequencies. The final slider is Comb, which applies a related but different concept, progressively attenuating frequencies depending on how far away their musical note is from the song’s tonic note in the circle of fifths.
The lowest part of each band within the EQ editor is shaded a darker grey, and this area is used to modify the ‘formants’ of the sound. Formants are unpitched resonances that are usually related to physical constants in a sound source, such as the dimensions of a violin body, or the shape of a person’s vocal tract. These resonances are present at fixed frequencies, regardless of the particular note being played, and are part of what makes each individual instrument or voice distinctive.
Melodyne has always offered formant shifting as part of its Note Editor, and this can be a very helpful feature. For instance, if you want to shift a note up by several semitones yet keep the singer’s timbre as consistent as possible, you’d want to transpose only its pitched components, leaving the formants unchanged. Alternatively, you might want to change the apparent character of a voice without shifting its pitch, in which case you’d do the opposite.
What’s new in Melodyne 4 is, in effect, that you can perform formant shifting on a global as well as a per–note basis, and in several different ways. The simplest of these is to turn the Formant dial in the Info Pane mentioned earlier, but if you have the full Melodyne Studio and you want to make more complex and detailed changes, the EQ page of the Sound Editor is the place to turn. The dark grey part of any band or group of bands at the bottom of the EQ editor can be picked up with the mouse and dragged sideways, shifting the contribution of any formants within those bands up or down.
Needless to say, this is pretty deep stuff, and although I worked with Melodyne for a couple of months while writing this review, I don’t feel that I’ve mastered all the possibilities of the formant editing and other EQ features. However, the beauty of Celemony’s implementation is that you don’t really need to understand what’s going on under the hood in order to make use of them. The tools themselves are straightforward to use, and the visual display provides a clear reflection of any edits you’ve made, so you can simply dive in and experiment. I never got to the stage where I could reliably anticipate whether a particular formant edit or Comb setting would improve matters, but I had a lot of fun applying the principles of trial and error!
Whereas the EQ page presents the entire audio spectrum divided up into semitone bands, the bands that are visible in the Harmonics editor represent the pitched spectral content of the audio: its fundamental and overtones. As these are, in frequency terms, discrete rather than continuous, there’s no curve joining them together.
To understand the difference between the EQ and Harmonics pages, let’s imagine a simple piano part that plays a middle C followed by the G a fifth above it. Clicking and dragging the fourth G band in the EQ editor would affect the same frequency in both notes: you’d be attenuating or boosting the first harmonic of the G but the second harmonic of the C, so their respective timbres would change in different ways. By contrast, if you click and drag the second band in the Harmonics view, you’re adjusting the level of the first harmonic in every note, regardless of that note’s pitch, and thus making a timbral change that’s consistent across all the notes.
With that in mind, the bands representing partials within the Harmonics editor can be manipulated with the mouse in similar ways to the bands within the EQ. So, too, can darker grey regions beneath the partials, though since these are pitch–dependent, they are not ‘formants’ as such. To the left of the fundamental, there’s a slider which controls the level of any residual audio belonging to the note that is lower than its fundamental frequency, such as you might get from the mechanics of a piano for example; pulling this down can help to clean the sound up, somewhat like a high–pass filter that operates on a per–note basis.
The Harmonics editor has two sub–pages labelled Lo and Hi, which allow you to vary the changes you’re applying depending on the pitch of the notes. With instruments that have a very wide range, such as the piano, the timbre is naturally quite variable across that range, and you might find that the changes you want to make to the upper octaves are counter–productive when applied to the lower registers too. When Melodyne analyses a file, it automatically determines the mid–point of its musical range and divides notes between the Lo and Hi editors, depending on which side of this point they fall. Changes made in the Hi editor leave low notes unaffected, and vice versa; both can be used in conjunction with the main Harmonics page, so you can combine global and treble– or bass–specific edits.
Like the EQ editor, the Harmonics page features a local menu that allows you to revert to the unedited spectrum, or copy and paste spectra between tracks. This opens up possible applications in both the corrective and the creative sphere. In the former, for example, I found that by copying the harmonic spectrum of a miked acoustic guitar and pasting it into a track recorded from the same guitar using a pickup, I could make the pickup sound a lot more like a guitar: not sufficiently natural that it could be used on its own, but a very viable option alongside the miked track. And the creative possibilities for morphing one instrument into another are, of course, legion.
As well as direct counterparts of the Brilliance and Contour sliders found in the EQ editor, the Harmonics editor presents two further ‘macro’ controls. Odd/even changes the balance between odd– and even–numbered harmonics, progressively favouring the former as you push it to the right and the latter as you go left. In small doses, this can make quite a natural and subtle change to the character of a sound. Finally, as the manual explains, the Harmonics editor’s Comb slider “thins out the harmonic spectrum”, progressively removing harmonics and/or shifting the harmonic spectrum sideways, so that the level of the first harmonic is forced to the level of the second, and so on. It’s as bizarre in reality as it sounds!
And so to the third element of the Sound Editor: the Synth Editor. As we have seen, the action of the Dynamics slider is somewhat like having global control over the attack and decay parameters within a polyphonic synthesizer. The Synth Editor opens up this model in much more depth, and actually gives you synth–like envelope generators to play with. There are three of these, labelled Spectrum, Formant and Amplitude, and each offers six parameters — starting level, attack time, sustain level, sustain time, decay time and final level — set graphically by dragging small arrows around with the mouse. Each acts on a per–note basis, even within polyphonic audio.
Conceptually, at least, the Spectrum and Amplitude envelopes are analogous to the filter and amplifier envelopes within a conventional synthesizer, though in practice, the action of the former in particular is very different. The Formant envelope has no direct counterpart in any synth that I know, though it could perhaps be thought of as the equivalent of altering the spectrum produced by a synth’s oscillators in some way. Together, they make up a set of editing tools that should keep inquisitive sound designers busy for years to come.
They are joined by two additional Resynthesis controls labelled Magnitudes and Phases. As you move the former away from its default zero position, Melodyne starts to fade out the actual sound of your recording, and to replace each note with a resynthesized version built from ‘pure’ sine waves. It is, in other words, performing resynthesis in much the same way that a true additive synthesizer such as Alchemy (formerly marketed by Camel Audio and now included in Apple’s Logic 10.1) can, with the difference that Melodyne can only change the character of existing audio; unlike Alchemy, it can’t abstract a sound and make it playable as a MIDI instrument. The Phases control, meanwhile, adjusts the relative phase of the partials within your sound, and can have quite pronounced effects.
The resynthesis feature was one I was most eager to try out, but in practice, I found it harder than I expected to get interesting results. Simply whacking up the Magnitudes slider to full just makes everything sound like a rather dull synth brass patch, and you usually need to make fairly sweeping adjustments to the envelopes to restore some life and movement to the sound. Considered on their own merits, however, the envelopes offer a good deal of potential. As with many of the other tools in Melodyne’s Sound Editor, when used conservatively, they provide subtle but effective ways to re–shape a sound, sometimes retaining a natural feel and in other cases adding a slightly uncanny, eerie quality. I found that more radical settings often worked better on monophonic material; pianos and guitars tended to degenerate into an unpleasant mess of artifacts, but I could pull the penny whistle solo described earlier into some very strange places indeed!
All in all, the Sound Editor represents a huge extension of Melodyne’s functionality. As with the rest of Melodyne 4, Celemony have clearly put much thought into developing a user interface which makes that functionality accessible. The controls are unfamiliar but never intimidating, and they’re logical, in the sense that they all have clear, audible effects which change smoothly and progressively as you adjust them. So, given the complexity of what it does, it’s impressively easy to use; but nevertheless, I think most users will need to put in quite a bit of work to discover how it will fit into their world. Conceptually, it’s the polar opposite of something like, say, a de–esser, which is designed to meet a very specific, pragmatic need. Melodyne’s Sound Editor opens up a new world of sound shaping, but operates on a very abstract plane. It’s entirely up to the user to figure out what real–world ends all this power can be brought to bear on. In other words, learning to use the Sound Editor is not hard, but working out what to use it for might be more of a challenge.
Remarkable as it is, the Sound Editor also highlights a limitation of Melodyne in general, which is that there’s no support for automation within the stand–alone program, and almost none within the plug–in. The conventional idea of automation isn’t really relevant to Melodyne’s core pitch–processing features, since you can assign different settings to each note in any case. With regard to the Sound Editor, however, the ability to automate global parameters such as Magnitudes, Phases, Dynamics, Contour and so on would really help to exploit the power that’s available. The resynthesis controls cry out to be used for gradual morphing effects, but as things stand, if you want to do this you’ll have to move them in real time during playback and record Melodyne’s audio output in another program. As we’ve seen, the tracks view in the stand–alone Melodyne Studio doesn’t currently offer any real functionality beyond reminding you what your various audio tracks look like. I hope that in a future update, Celemony will consider adding Sound Editor automation curves within this view, as well as incorporating some basic editing features (or, at least, the ability to select just part of a track rather than all of it).
Melodyne 4 has been a long time coming. But, as you can tell from the sheer length of this review, Celemony have wasted none of that time. This is an absolutely huge update, and thank heavens, it’s accompanied by a superb, searchable, cross–linked online manual which is a big improvement over the v2 documentation. Early adopters of Melodyne 2 will remember the stability problems that marred that release, so I’m pleased to report that Celemony appear to have got it right this time around. I tested a late beta and the release versions of Melodyne Studio 4, predominantly on Mac OS but also on Windows, and found it robust. There were a handful of times where it quit unexpectedly after saving and closing a project, but my work was never lost or interrupted; by way of comparison, I’d say it was at least as stable as Pro Tools 11 and 12 on my MacBook Air.
Existing users will find an avalanche of new features that will revolutionise the way they work with the program. The improvements to the user interface alone make the upgrade a no–brainer; add in the extraordinary tempo–manipulation features, true multitrack operation, new analysis algorithms and the limitless possibilities opened up by the Sound Editor, and you have a colossal step forward. And if you previously felt that shelling out the extra for the Studio version wasn’t worthwhile, be prepared to change your mind, because it’s only here that the full sweep of Celemony’s ambition and imagination is apparent. Melodyne Studio 4 is a release that puts a huge gulf between this program and any other pitch–processing software on the market.
What we hear as pure consonant musical intervals are built on simple frequency ratios. For example, the upper note of a pure fifth should have a frequency 3/2 that of the root note, while the frequency ratio between a major third and its root is 5:4, and that of a minor sixth 8:5. Singers, and players of instruments such as the violin family that are capable of free pitching, will often pitch notes so that they form these pure intervals. However, a problem arises with instruments that produce fixed pitches, such as keyboard and fretted string instruments.
Let’s suppose a piano is tuned so that the fundamental frequency of the A below middle C is 440Hz. Taking that as a reference, we tune the other notes to pure ratios of 440: for instance, C sharp will have a frequency of (440 x 5 / 4 =) 550Hz, E will be (440 x 3 / 2 =) 660Hz and the F-natural above that (440 x 8 / 5 =) 704Hz. Our piano has now been tuned to ‘just intonation’ or Pythagorean tuning for the key of A, and will sound beautifully sweet as long as our music remains in that key. It also works pretty well in related keys such as D major. But consider what happens if we want to play something in the key of C-sharp major, taking 550Hz as our tuning reference. The note a major third above it should have a frequency of (550 x 5 / 4 =) 687.5Hz. Yet, on our A piano, that key is already tuned to 704Hz — so unless we can find the world’s quickest piano tuner, our C-sharp major piece is going to sound horribly out of tune.
As a result, instruments that produce fixed pitches are rarely tuned with just intonation, because it ties them to a specific key. Instead, we have the unholy compromise that is equal temperament, where nothing is ever perfectly in tune, but nor is any one key more sour than any other.
Note pitches in equal temperament can be surprisingly different from their ‘pure’ equivalents. For instance, in the key of C, a justly intonated E will be 14 cents flat compared with its counterpart in equal temperament. You don’t have to have a highly trained ear to differentiate justly intonated chords from those played in equal temperament. Despite this, every pitch–correction tool I’m aware of defaults to using equal temperament. It seems ironic that we often use pitch–correction tools to apply equal temperament to sources which don’t need or suit it. Singers who are harmonising an existing melody, for instance, will typically aim to do so in the purest possible way — so it’s quite possible that if we blindly and automatically run their contributions through a pitch–correction plug–in, we might actually be making them more out of tune, not less; and it may also be that the characteristic ‘sound’ of vocal pitch-correction results partly from forcing voices to intervals they wouldn’t naturally use.
Some pitch–correction software, including earlier versions of Melodyne, allow you to create user scales which are microtonal, so if you know your song is in C major and doesn’t modulate away from it, you could in principle create a C major just scale and correct to it. What’s new in Melodyne Studio and Editor 4 is an option called Dynamic Just Tuning, which avoids the compromises of equal temperament without requiring any user input. Instead of correcting all notes to a fixed scale, Dynamic Just Tuning tries to find the sweetest intonation for each chord, given the musical context, and re–pitches the whole chord to minimise the degree by which the most affected note differs from equal temperament. This last point is important given that pitch correction is often used in musical contexts where the corrected material needs to ‘sit’ alongside equal–tempered instruments such as guitars or pianos.
The extent to which Dynamic Just Tuning makes a difference is highly dependent on the source and the context, but if you have the Studio edition, it’s definitely an option worth having when you’re working with vocal harmonies, not to mention strings and so forth. It can also be interesting to apply it to polyphonic instruments such as guitar and pianos to produce subtle effects that wouldn’t be possible in real life.
As Mike Senior pointed out in his review of Melodyne 2, one of the more obvious limitations of the program’s Polyphonic algorithm was its tendency to misinterpret the onset of notes that fall close together, such as grace notes or chords with a slightly ‘spread’ attack. It would sometimes mistakenly treat the noise component that marks the attack phase of a new note as belonging to the sustain phase of an existing note, with very audible results. In Melodyne 4, there are now greatly improved tools for tackling this sort of situation.
When a polyphonic file is analysed, Melodyne detects patterns of ‘energy’ within it and uses them to determine likely ‘starting points’ for notes. In version 2, two global settings governed the thresholds at which a burst of energy was considered significant, and the sensitivity with which they were divided into individual notes. These were, and still are, set by moving ball and bracket icons along a slider. What’s new is that in Note Assignment mode, ‘starting points’ are now visible and editable within the Note Editor. Melodyne places a hollow triangle in the ruler at every point where a note could conceivably begin; of these, the ones that are deemed to be correct according to the current setting of the sensitivity slider are filled in and get a vertical line. Two new modes of the Note Separation tool now let you activate and deactivate these potential starting points individually, and add and remove them as you see fit. In situations where there are a handful of misses in an otherwise correctly analysed file, five minutes’ work with this can make the difference between a recording’s being usable and not.
Celemony tell me that fine–tuning their polyphonic algorithm requires their developers to configure numerous internal parameters to achieve the best possible balance between universal success, sound quality and other desirable ends. The new Decay and Sustain versions of the algorithm represent two different sets of settings, optimised for different functions, but some of these internal workings have now been exposed to end users in the Note and Algorithm sections of the new Info Pane. You can adjust parameters such as the amount of formant compensation to be applied, solo the track, and quickly audition how global pitch or time changes will affect it to check that the most appropriate algorithm is selected. You can also change the behaviour of the Melodic and Polyphonic algorithms by switching their Playback Type setting between Tonal or Complex; in the latter mode you also get a choice of Smooth or Crisp Character settings. In practice, I rarely found that there was much improvement to be gained by changing these, though.
All of these are very worthwhile improvements, but as yet, the Note Assignment mode seems to be one area where Celemony haven’t managed to implement multitrack working, so if for instance you’re working with a multi–miked capture of a guitar or piano, you’ll have to fine–tune algorithm settings and starting points separately for each track.
Melodyne Studio €699; upgrade from previous version €149.
Melodyne Editor €399; upgrade €99.
Melodyne Assistant €249.
Melodyne Essential €99.
Prices include VAT.
Melodyne Studio $699; upgrade from previous versions $149.
Melodyne Editor $399; upgrade $99.
Melodyne Assistant $199.
Melodyne Essential $99.
I’ve prepared these audio files to illustrate some of the features described in my February 2016 review of Celemony’s Melodyne 4. Listen to them here.