Beta-testing Reason

Reason Notes

Published in SOS October 2005
Bookmark and Share

Technique : Reason Notes

Propellerheads have always had a rather unusual approach to beta-testing, routinely involving thousands of registered users. We talk to the Props about how it's done.

Derek Johnson

Last month's column had a quick rundown of the fixes and new features of Reason's v3.0.3 update. As many of you will know, the update went through Propellerhead's beta-testing process, involving numerous registered users testing fixes to destruction.

I thought it would be interesting to have a look at Propellerhead's development process, by way of a few words with Reason's product manager, Mats Karlöf. Mats joined Propellerhead earlier this year — "This was a dream job for me" — and must have hit the ground running, as v3 was released and discovered to have a few bugs.

Manifesto Destiny
Marco Raaphorst, known to the Reason community as Raapie, is a long-standing creator of patches for our virtual studio. He also contributed to the factory Refill for v3.0. His latest work includes a modest but intricately programmed Combinator patch collection, together with a set of all-new patches for the Subtractor and Malström devices that are these Combis' only sound sources. Combi Manifesto 1 features 22 patches that aim to keep CPU load as low as possible while pushing the device's signal-routing and sound-design possibilities. Combinator controllers have been thoughtfully set up (some even change pattern parameters on Matrix devices), and a new graphic 'backdrop' has been created for each patch. Surf to www.reasonbanks.com, have a listen to the demos and be prepared to fork over a mere £5.50/US$10/Euro 8.25 for the zipped 5.9MB download.
Beta Background

Beta testing comes after a few months of internal testing, which is ongoing but generally begins at the final stages of the product development. Mats: "When we no longer can find any errors, bugs or issues with the software, we start the public beta testing. The purpose of beta testing is to see if problems appear on machines, operating systems and hardware setups that are different from ours, and also to see if other installed software might conflict with Reason."

The problems with v3 came to the surface fairly soon after its release: "Almost everyone in the company reads the Propellerhead forums and boards on a daily basis. We keep track of what users experience, think, feel or wish to see from us. So we were aware about problems that our users had at a very early stage. We started working on the fixes almost immediately after the first couple of reports. Some of the bugs were quite unpleasant, so we were eager to release fixes as soon as possible."

User Friendly

Beta testers are then recruited from the registered user base. "We normally have a lot of interested users who would like to contribute. From these we randomly select a few hundred that we start the testing with. In each stage of testing after that we gradually add more and more users, until the final stages, where we have thousands of people testing the software."

reason notes.s
From left to right, Propellerhead's Mats Karlöf, Loui Westin and Erik Agsjö.
From left to right, Propellerhead's Mats Karlöf, Loui Westin and Erik Agsjö.

With so many people testing the software, it's impossible to keep track of what each tester is doing via normal channels. "We set up a beta-test forum, which is managed by Loui Westin, our beta-test leader, who communicates with the testers and reports bugs to Erik Agsjö, our project manager for the Reason 3 bug-fixes. Every issue or question is posted on the beta forum, as are all updates and new versions of Reason. This has the advantage that all beta testers can follow what has been reported, what is actually a bug, what is intended as a feature, and what has been fixed. It also makes it a lot easier to communicate to a large group of testers." The communication is two-way, despite the numbers of beta testers involved: "The beta-test leader can sometimes try to identify a particular problem and test solutions. It is important that the bug or error is reproducible at least by the tester that first found the bug.

"All bugs and issues are logged as separate cases; each case contains one bug or issue and is always assigned to one individual, who is responsible for fixing the bug or delegating it to someone else. Normally the beta-test leader would open a case and assign it to the project manager, who in turn assigns it to one of the developers for fixing. When the bug is fixed, the case is resolved by the project manager and finally closed by the beta-test leader after he has confirmed the fix. It is important that a developer never confirms and closes his own fix. We try to make sure that all bugs are fixed before we build a new version of Reason. Unfortunately we can mistakenly create a new 'issue' when we are fixing an old bug."

Managing a beta-testing program, especially one involving thousands of testers, seems like a potentially stressful endeavour. Mats comments: "It is not really stressful. One has to be meticulous. It is like being at school and having to go back and do your lesson over and over again until you get it right. The internal system we use makes everyone organised."

A 'final' version of Reason 3 was officially released to all users, only to be followed a few hours later by v3.0.3, due to a newly-introduced bug having been spotted a little too late. How certain are the team that v3.0.3 is 100 percent? Mats is honest: "Unfortunately, we are now certain that v3.0.3 is only 99.9 percent. We must do another bug fix, as we aim for Reason to be rock-solid and flawless. The up-side of all these bug fixes is that we can optimise Reason to be less and less CPU hungry. In the end, we hope users appreciate that they will have a music production system we believe is the best possible."


DAW Tips from SOS

 

Home | Search | News | Current Issue | Tablet Mag | Articles | Forum | Subscribe | Shop | Readers Ads

Advertise | Information | Digital Editions | Privacy Policy | Support | Login Help

 

Email: Contact SOS

Telephone: +44 (0)1954 789888

Fax: +44 (0)1954 789895

Registered Office: Media House, Trafalgar Way, Bar Hill, Cambridge, CB23 8SQ, United Kingdom.

Sound On Sound Ltd is registered in England and Wales.

Company number: 3015516 VAT number: GB 638 5307 26

         

All contents copyright © SOS Publications Group and/or its licensors, 1985-2014. All rights reserved.
The contents of this article are subject to worldwide copyright protection and reproduction in whole or part, whether mechanical or electronic, is expressly forbidden without the prior written consent of the Publishers. Great care has been taken to ensure accuracy in the preparation of this article but neither Sound On Sound Limited nor the publishers can be held responsible for its contents. The views expressed are those of the contributors and not necessarily those of the publishers.

Web site designed & maintained by PB Associates | SOS | Relative Media