Beta-testing Reason

From left to right: Propellerhead's Mats Karlöf, Loui Westin and Erik Agsjö.

Propellerheads have always had a rather unusual approach to beta-testing, routinely involving thousands of registered users. We talk to the Props about how it's done.

Last month's column had a quick rundown of the fixes and new features of Reason 's v3.0.3 update. As many of you will know, the update went through Propellerhead's beta-testing process, involving numerous registered users testing fixes to destruction.

I thought it would be interesting to have a look at Propellerhead's development process, by way of a few words with Reason 's product manager, Mats Karlöf. Mats joined Propellerhead earlier this year — "This was a dream job for me" — and must have hit the ground running, as v3 was released and discovered to have a few bugs.

Beta Background

Beta testing comes after a few months of internal testing, which is ongoing but generally begins at the final stages of the product development. Mats: "When we no longer can find any errors, bugs or issues with the software, we start the public beta testing. The purpose of beta testing is to see if problems appear on machines, operating systems and hardware setups that are different from ours, and also to see if other installed software might conflict with Reason."

The problems with v3 came to the surface fairly soon after its release: "Almost everyone in the company reads the Propellerhead forums and boards on a daily basis. We keep track of what users experience, think, feel or wish to see from us. So we were aware about problems that our users had at a very early stage. We started working on the fixes almost immediately after the first couple of reports. Some of the bugs were quite unpleasant, so we were eager to release fixes as soon as possible."

User Friendly

Beta testers are then recruited from the registered user base. "We normally have a lot of interested users who would like to contribute. From these we randomly select a few hundred that we start the testing with. In each stage of testing after that we gradually add more and more users, until the final stages, where we have thousands of people testing the software."

With so many people testing the software, it's impossible to keep track of what each tester is doing via normal channels. "We set up a beta-test forum, which is managed by Loui Westin, our beta-test leader, who communicates with the testers and reports bugs to Erik Agsjö, our project manager for the Reason 3 bug-fixes. Every issue or question is posted on the beta forum, as are all updates and new versions of Reason. This has the advantage that all beta testers can follow what has been reported, what is actually a bug, what is intended as a feature, and what has been fixed. It also makes it a lot easier to communicate to a large group of testers." The communication is two-way, despite the numbers of beta testers involved: "The beta-test leader can sometimes try to identify a particular problem and test solutions. It is important that the bug or error is reproducible at least by the tester that first found the bug.

"All bugs and issues are logged as separate cases; each case contains one bug or issue and is always assigned to one individual, who is responsible for fixing the bug or delegating it to someone else. Normally the beta-test leader would open a case and assign it to the project manager, who in turn assigns it to one of the developers for fixing. When the bug is fixed, the case is resolved by the project manager and finally closed by the beta-test leader after he has confirmed the fix. It is important that a developer never confirms and closes his own fix. We try to make sure that all bugs are fixed before we build a new version of Reason. Unfortunately we can mistakenly create a new 'issue' when we are fixing an old bug."

Managing a beta-testing program, especially one involving thousands of testers, seems like a potentially stressful endeavour. Mats comments: "It is not really stressful. One has to be meticulous. It is like being at school and having to go back and do your lesson over and over again until you get it right. The internal system we use makes everyone organised."

A 'final' version of Reason 3 was officially released to all users, only to be followed a few hours later by v3.0.3, due to a newly-introduced bug having been spotted a little too late. How certain are the team that v3.0.3 is 100 percent? Mats is honest: "Unfortunately, we are now certain that v3.0.3 is only 99.9 percent. We must do another bug fix, as we aim for Reason to be rock-solid and flawless. The up-side of all these bug fixes is that we can optimise Reason to be less and less CPU hungry. In the end, we hope users appreciate that they will have a music production system we believe is the best possible."