Celemony Melodyne 2

Version 2 of Celemony's revolutionary pitch- and tempo-shifting software includes new features, better sequencer integration and an improved interface.

Melodyne attracted a lot of positive press when it was first unveiled, promising as it did the ability to modify audio in the time, pitch and formant domains with minimal processing artifacts. Whole musical phrases could be time-stretched and single notes could be extended seemingly indefinitely. We saw demos of audio recordings where the individual notes were shown as waveforms on a piano-roll-type grid depicting actual pitches and where pitch-bends and vibratos were shown by a 'bendy thread' running through the waveform. Notes could be pitch-corrected, lengthened and dragged to completely new pitches (by more than an octave), while their phrasing and vibrato could be adjusted. You could even make new melodies out of existing material, while rhythmic loops could be tempo-shifted over a wide range and freely played melodies forced to fit an existing tempo. It seemed that you could bend audio just like chewing gum, and the processing artifacts remained amazingly small.

Very positive reviews followed, including our own in November 2001, though the user interface came in for some criticism, as did the manual. Having just read the manual for v2.0, I have to say that it still isn't entirely user-friendly — some procedures seem to be talked about rather than described logically and it's not immediately obvious how the processing works at a file level. However, the package does include some demo session files that are fairly well documented, and it's well worth working through these before you start 'Melodyning' in anger.

Overview & New Features

Melodyne works only with monophonic pitched sounds (or rhythmic unpitched sounds), and these also need to be free from spill and delay-based processing such as echo or reverb for the process to track the pitches properly. The Melodyne environment includes multitrack playback and recording, with an Arrange page, rather like a sequencer, and it also has a simple mixer that now support VST and Audio Units plug-ins in addition to its inbuilt reverb. For most projects where the work has been started in a sequencer, files can be imported into Melodyne (either via an import routine or by dragging them into Melodyne's Arrange page), processed and then saved to be reused back in the original project. Note that the original audio files remain unchanged — processing takes place in real time under guidance of a special Melody Definition file created when the original audio file is analysed by Melodyne.

Since version 1.0 was released a couple of years ago (see our original review for an account of the basic features), various improvements have been added, such as pitch and time quantisation, Undo (at multiple levels), the ability to deal with stereo files, and support for more platforms, including Mac OS X and Windows. It's even possible to make the timing of one piece of audio match that of a reference melody. As well as many further large and small improvements (see www.celemony.com/news/releasenotes.html for a full list) the latest version 2.0 reviewed here adds Melodyne Bridge, enabling Melodyne to be connected to any VST or Audio Units host and controlled via a plug-in insert slot. Though this might suggest that Melodyne can now be used as a VST or AU plug-in — something that would make sense in some applications — it still functions only as a stand-alone application, but Melodyne Bridge enables audio to be recorded into Melodyne and then replayed in context directly from the host application. If only part of a track is recorded into Melodyne and then processed, the host software will play back the original unprocessed track except for the 'Melodyne-ised' section.

Melodyne v2.0 also supports Rewire, making it compatible in this respect with programs such as Reason, Cubase, Digital Performer, Logic and Pro Tools. MIDI Clock sync has also been added along with SMPTE and a significant degree of MIDI functionality. Where an audio file has been processed to detect its pitch, the musical notes can be output as MIDI data in real time, including velocity and pitch-bend information, and it's now also possible to adjust various Melodyne parameters in real time via MIDI, including pitch, formant and so on. This means that using MIDI Controller data, it's possible to control Melodyne via MIDI keyboards and dedicated remote control devices such as Mackie Control, Logic Control, and others.

Melodyne v2.0's track count is limited mainly by the available computing power, and audio of up to 192kHz sample rate and 32-bit resolution is now supported. For any serious multitrack work, a fairly brisk computer is recommended, while less power is needed if you're using Melodyne simply to fix individual tracks or to modify loops. The program can now be run under Mac OS X, Windows SE, ME and XP. Support is also included in v2.0 for Direct X, Sound Manager, Core Audio and ASIO 2, Core MIDI, Rewire 2, VST and Audio Units plug-ins, but it should be noted that Direct X and Sound Manager can only be used for playback, not recording.

Once Melodyne has analysed an audio file, it separates the individual notes and displays them at the detected pitches. Individual Melodies can be opened in the Edit window (above) for detailed adjustments.

Interface

Melodyne's interface is based around three main windows: the Arrangement, the Editor and the Melody Definition window. Work invariably starts in the main Arrange window, which is visually similar to a simplified sequencer arrange page. This shows the audio tracks running from left to right, and audio may be recorded directly into this page, imported from an existing audio file or dragged directly into the Arrangement. A small transport window is used when recording audio directly into Melodyne. Here you can engage a cycle function, set auto punch-in and out points and loop sections of audio. Melodyne also includes a metronome click against which live performances can be played, but in order to get a better feel, it may be easier to simply import an audio file based on a drum groove and work to that, much as you might in a sequencer.

Arrange tracks can be balanced using Melodyne's mixer, which offers most of what you'd expect in a modern sequencer mixer and now supports third-party plug-in processing in VST and AU formats, which is a big improvement over earlier versions. It has level faders, pan pots, an insert point and an effects send buss as well as basic EQ for each channel. The mixer can also support multiple groups, which are dictated by your choice of audio hardware and driver.

Melodyne can now host VST and Audio Units plug-ins, and its mixer has also been improved to offer better visual feedback and labelling.

Detecting Melody

Once you've recorded or imported a clean monophonic audio file melody (AIFF, WAV, SDII and SND files are all valid), Melodyne needs to analyse this for pitch, tempo and note divisions, after which the audio should still play exactly as it was recorded until you start to make changes to it. The process of analysis takes a little while, but this shouldn't be longer than the 'real-time' length of the file unless you have an ancient computer. Once analysis is complete, Melodyne creates an additional Melody Definition file that contains information about the audio, and which must not be deleted until a job is complete. Incidentally, the manual refers to the contents of any track as a Melody, regardless of whether the material is pitched or unpitched.

Analysing the audio file involves choosing Detect Melody from the Define Melody menu, where you can also set up a number of parameters that relate to the type of audio you're working on, including the selection of voice or instrument types, which also sets the range for the instruments being detected to minimise the risk of 'wrong octave' errors. Prior to analysing the Melody, you see the waveform with no pitch information, but once analysis is complete, the individual note waveforms are displaced, much like the notes on a musical stave. Note that when you save a Melodyne Arrangement, the source audio files are not saved along with it, so these must be left in place until the project is finished.

Detect Melody makes an impressive job of extracting the melodic information from a file, but it can occasionally get caught out, so Melodyne includes an Edit Definition window that lets you see the notes on a piano-roll-style grid. Tools are provided that let you split legato notes if Melodyne didn't catch the split point automatically, and you can also insert or correct bar lines. Making these edits updates Melodyne's Melody Definition file so that it can take account of your changes. Depending on the instrument type, Melodyne may also occasionally choose the right note but the wrong octave, so you have the option to fix this manually.

Once you've made changes, you need to use 'Reset to Definition' in the Define Melody menu in order to update what's on the Arrange page to match the newly updated Melody Definition file.The procedure is normally: load your audio into the Arrange page, detect the Melody, check the analysis is correct in the Melody Definition window where you can make revisions if it's not, then move onto the Edit Window to make your own creative changes. Last port of call is back at the Arrange window where your edits should be reflected in the Arrangement ready to be saved, exported or mixed. Complete arrangements can be exported as new sound files, and there's a file selector box that allows you to determine which track or tracks will be exported. If you are working with a multitrack Arrangement, you can opt to save the mix as set up in Melodyne's own mixer, or as a stereo sound file. MIDI files based on the pitch analysis of the various tracks may also be saved using an export routine, complete with bend and velocity information.

The tempo-detection side of the process can be based on the actual tempo of the incoming audio, in which case you choose Own Tempo, or you can use the tempo of the Melodyne project by selecting Present Tempo. The inbuilt tempo-guessing facility is impressively accurate, but if Melodyne does get it wrong and you know what the actual tempo is, you can enter the correct value. Tools are available after doing the analysis to help match the tempo manually so nothing is set in stone until you say so. For pitch detection, you can provide Melodyne with scale information based on a choice of preset scale types, including 12 equal-tempered keys, chromatic, and less common scales such as Pythagorean. Users can also create custom scales and offset the pitch centre for tunings above or below concert pitch.

Notes can also be viewed in more detail in a matrix-like Edit window (opened by clicking on a Melody track) and a Melody may also be presented as a conventional score by clicking the stave icon. If the audio is essentially unpitched, as in the case of percussion, plain speech or any recent Bob Dylan concert, the display shows timing information only. A set of dedicated Edit window tools is available for editing the audio in the pitch, time and formant domains as well as for marking note start points that Melodyne may have misinterpreted, and each of these tools has a subset of further related tools that can be accessed by holding the mouse button over the tool. An inspector bar above the time ruler shows different information depending on which tools is chosen, and many edits can be done using the mouse, though there's often an alternative method where you can type in values directly. There's also a context-sensitive Actions menu next to the toolbox, which changes with the tool type.

Melodyne is pretty clever in the tempo department as it not only allows you to work out loop tempos automatically, it also provides tools for matching different elements that were original recorded at different tempos. The procedure is that one of the parts is used to provide a reference tempo and the other parts forced to match that. It's also possible to change the tempo of a whole multitrack arrangement, much as you might with a MIDI sequence.

The Play Algorithms

Melodyne uses a proprietary algorithm called Local Sound Synthesis or LSS to allow independent pitch and time editing. As the process can be pretty processor-intensive, Melodyne offers a few processing variations based on LSS that may allow some jobs to be done more efficiently. A Plain/Processed Parameter switch determines whether a track plays back with or without processing.

Processing can be Note Based or Time Based, and in Note mode, the attack of the sound always follows the original sound. This works well with plucked or percussive sounds but precludes formant correction. Time mode works better for sounds that have transitions between notes. There's also a choice of Crisp or Smooth algorithms where Crisp gives more presence and detail, while Smooth may be useful where large degrees of time-stretch are employed as it can help hide unmusical characteristics of the sound that may become evident in the stretching process.

Formant Correction, in the modes where this is available, can be set from 0 to 100 percent and is best adjusted by ear. It can be useful in avoiding the Mickey Mouse effect when restructuring a melody where some notes are moved a long way from their original pitch. There's also a Formant Style parameter that seems to automatically adjust the amount of format correction depending on how far you move a note from its original pitch. Again this can be adjusted from 0 to 100 percent and is best judged by ear as it's not always clear from the descriptions exactly what these controls do. Any of the algorithms can be applied to mono or stereo sound files.

Editing Melodies

All the clever manual manipulation of time, pitch, pitch articulation (such as portamento and vibrato) and formant takes place in the Edit window. Here you can change the tempo over a vast range, and slowing a file down to a standstill will result in an infinitely long note, which suggests that some sort of resynthesis is being used to recreate the harmonic structure of the sounds. The ability to increase or decrease the depth of natural vibrato is particularly impressive, and where legato notes change pitch, you can vary the pitch of single notes within the phrase while retaining a natural pitch scoop between them.

Comparisons will obviously be made between Melodyne and Auto-Tune, but they are really very different programs with quite different intended applications. You can use pitch quantisation with Melodyne to achieve Auto-Tune-like results or you can nudge errant notes into place manually, but unless you actually need to go into a file at a very surgical level, I'd suggest that Auto-Tune is a far more appropriate and convenient tool for that job. However, Auto-Tune doesn't enable you to make such drastic and precise changes to the pitch of a file without incurring processing artifacts, it doesn't include formant correction, and of course it can't do any of the clever time-stretching, vibrato or scoop scaling stuff. Melodyne even allows you to time-quantise audio events, providing they are clear enough to detect unambiguously.

Another seriously impressive feature of Melodyne is its ability to paste the melody from one recording onto another, which allow you to do such cunning things as creating a harmony vocal part from the original lead line. Of course the general phrasing of the source and processed piece have to be similar enough, and in isolation, you may notice some of the trickery, but once in a mix, you'd probably get away with using Melodyne-created backing vocals if you were careful.

MIDI

As intimated earlier, melodies can be exported from Melodyne as MIDI files, though the quality of the outcome depends on the type of audio being played in. This feature was present in the original version, but new facilities have since been added. For example, you can now use a MIDI keyboard to apply pitches to your audio in real time or, if the transport is stopped, in step time. This is called Play mode, but a similar mode known as Edit Notes means your audio will move to the MIDI note and then stay there until a new MIDI note is received. Alternatively, you can use your MIDI input to transpose the audio by the relative value of the note you play to middle C, rather as some hardware pitch-shifters do.

In Use

You can help Melodyne's pitch-detection algorithms by specifying the range of the instrument under analysis.Despite the improvements to Melodyne, of which there are many, the manual still makes it harder to use the system than is necessary and it's only after using it for a couple of days that everything clicks and you realise it's not that difficult after all. The long-overdue Undo feature is very welcome, as is ASIO/Core Audio support and the ability to use third-party plug-ins in the mixer, though I can think of few occasions when I'd want to work on a complete project in Melodyne other than when creating multi-part backing vocals. Mostly I'd use it to fix up tracks with intonation problems or to create single harmony vocal parts — I might even use it to manipulate loops, but I can't imagine needing to compose or mix within it on a regular basis. Melodyne Bridge is a great new feature and probably gets as close to making Melodyne behave as a VST plug-in as is practical, even though it still requires more user attention than I'd like.

On an ergonomic level, I think that although you soon get used to the tools and sub-tools method of control, it's perhaps not the best way to have arranged the user interface. Nevertheless, once you have the right tool, the audio manipulations are very easy and the sound quality remains just as startlingly impressive, no matter how many times you hear it. In fact the only area that's less than impressive is the MIDI output from difficult melodies, but this isn't easy to do and some manual intervention will probably always be necessary except in the case of very simple tunes with no legato playing.

Having said how great Melodyne can sound, don't let this imply that you won't have to do some of the work, as editing inflections and tweaking formants is very important in achieving a natural and believable result. If you simple barge in and move things, voices can end up sounding quite unnatural, though for minor corrections as opposed to major creative surgery, the user has to do very little other than nudge pitches or timings. For example, I used Melodyne to tune up an acoustic upright bass part for a song I'm working on, and because the pitch changes were only fractional, the result was completely natural, with no need for me to tweak anything other than pitch.

Conclusion

There's no denying that the audio processing inside Melodyne is still cutting-edge and about as artifact-free (when used with sensitivity) as you could reasonably hope for. If you are the type of musician who likes to create or modify parts by processing, then you couldn't ask for a better tool, and if you're into multi-layered loops, the tempo sync'ing abilities of Melodyne allows you do achieve results that few other programs would even attempt. As a purely corrective tool, Melodyne costs more than Auto-Tune and is less immediate in operation. On the former point, many users will be happy with the more affordable, cut-down Melodyne Cre8, which is limited to eight mono tracks at up to 16-bit, 48kHz. On the latter, I still feel there would be a market for a single-track, pitch-only version to compete directly with Auto-Tune. Melodyne Bridge is a great improvement over importing and exporting whole slabs of audio, but it still doesn't have the immediacy of a processor that was designed from the ground up to be used as a plug-in.

Taken as it is, Melodyne is a relatively costly and sophisticated tool, but if you need the functionality Melodyne offers, it's well worth it and there's no more effective alternative at any price. Most of the criticisms of the earlier versions have been addressed, and once you find your feet with the program, it's actually fairly straightforward to use.