You are here

SOS Product Reviews

For feedback and suggestions about the SOS magazine, app, web site or forums.

SOS Product Reviews

Postby Alvin58 » Thu Sep 17, 2020 1:18 am

One of the main reasons I read SOS is for the reviews, but I have to say they constantly disappoint. I understand SOS needs the advertisers' revenue & does not want to bite the hand that feeds it, but this is at the expense of useful information.

Here's a typical review summary:

Pros

- Really Good Flibbertygibbets
- Smooth Interface
- Fun to use

Cons
- Missing a small insignificant thing

Overall
A really amazing product of its type

When you get review after review like this, they become useless. Yet let's face it, some products are better than others -- but we never learn that from reading SOS reviews.

To fix this, why not give everything a star rating from 1-5? (Assuming you don't give all products a 5). In that way, we can at least quickly compare two similar products.

As it stands now, I don't find the reviews that helpful in making a purchase decision at all -- essentially I would buy everything!
Alvin58
Posts: 3
Joined: Tue Apr 24, 2018 4:05 pm

Re: SOS Product Reviews

Postby zenguitar » Thu Sep 17, 2020 2:34 am

I fully understand where you are coming from. But it might help to give you a little context.

Unlike the vast majority of magazines out there, SOS is from a genuine independent publisher and has been around for well over 3 decades. And throughout that period SOS has used a large number of experienced and highly reputable industry professionals as reviewers.

And from the earliest days SOS has printed honest, warts and all, reviews.

In the past, poor reviews have been published which did lead to manufacturers pulling their adverts for many months. And the reviews editors have always been backed by the editor, publisher, and chairman to go ahead and print bad reviews even if that risks the loss of advertising.

Of course, you are right to be cautious about how far you can trust a review in a magazine. In these forums SOS reviewers have shared stories about receiving equipment for review from another magazine where the equipment in question had never been removed from its original packaging and never used.

And there are numerous examples where the review of a product identified sufficient issues for it to be withdrawn and redeveloped before being put forward again for review. And that was made clear in the published review.

So, bearing that in mind...

If you had a new product that you knew would be scrutinised in great detail by a highly reputable professional with decades of experience and published in a magazine that has a reputation for publishing even if it means losing advertising; would you put it forward for review if it didn't deliver as promised? Of course you wouldn't.

And as a result of that, a theme developing over the last few years is that reviewers report problems they experienced with both hardware and software and how quickly the manufacturers respond with software, firmware updates, and even hardware modifications.

OK, I'm a moderator here so you would expect me to speak up for the reviews. But around a decade ago I was invited to review guitar related music tech equipment and turned the opportunity down because I didn't believe that I was in a position to deliver a review with sufficient value. And I am a trained guitar maker and a very experienced guitar tech.

TL:DR
Manufacturers don't put crap gear up for review in SOS because they know they WILL be called out.
With so much good gear available, why on earth would an SOS reviews editor choose to review a piece of poor hardware/software above good hardware/software?

Andy :beamup:
User avatar
zenguitar
Moderator
Posts: 10610
Joined: Thu Dec 05, 2002 1:00 am
Location: Devon
The more I understand Music, the less good I get at it.

Re: SOS Product Reviews

Postby Alvin58 » Thu Sep 17, 2020 3:05 am

Thanks for your thoughtful reply, it all makes sense.

But let me see if I can be more specific -- the issue becomes this:

- every audio interface is fantastic
- every orchestral software library sounds great
- every reverb module is amazing

Given that, which one should I buy? That's where I would look to SOS for guidance, and that's what I'm not receiving.

Star rankings would help, but what would be even more useful, when they rated a new XYZ, they ended up putting it in a ranking against the previous XYZ's they have reviewed. Then a would-be consumer would gain important information.

Consumer Reports in the US does this -- products are ranked in detail from 1 to 100. So one can be pretty certain that something ranked 95 is better than one ranked 85. And one ranked 95 vs 94 is a personal preference.

I would love to see SOS so something like that too.
Alvin58
Posts: 3
Joined: Tue Apr 24, 2018 4:05 pm

Re: SOS Product Reviews

Postby zenguitar » Thu Sep 17, 2020 4:32 am

Ranking from 1 to 100 is arbitrary. It is loaded with unreferenced and unqualified subjectivity and that renders it meaningless.

every audio interface is fantastic

That's not how I read the reviews. Another way of looking at it is that what the reviews demonstrate is that we a fortunate enough to live in an age where even entry level interfaces surpass anything a pro mixing desk from the '90s or later could achieve. But the reviews go to great lengths to illustrate the strengths and weaknesses each AI offers, and how the hardware integrates with included software.

every orchestral software library sounds great

At a basic level; if your orchestra software library didn't sound great would you honestly offer it up for review in SOS? At a more sophisticated level; it costs a lot of money to hire an orchestra, a suitable studio, recording engineers, producers, and skilled samplists to create a library. Get it wrong and you will lose your house, get it right you will make a living. And you have a clear benchmark; you have to be better than a 10 year old legacy product that was more than good enough to score Hollywood movies in it's day and is now on sale for peanuts.

every reverb module is amazing

My Quadraverb+ is over 30 years old now and still perfectly capable of being amazing. And in hardware or software it is comfortably surpassed, in software often for free.

What I would suggest is that you invest too much attention on playing Top Trumps with the "Pros, Cons & Summary" box at the expense of taking in the detail in the full review.

Put simply, SOS reviews aren't intended to tell you what to buy. They tell you you what gear claims to do, how well that gear achieves what it claims, and (if we are lucky) tells us how a skilled and experienced user might exploit some of the design quirks to discover a valuable bonus.

Star rankings are (at best) a sloppy shorthand. A device that might suit one reviewer's workflow perfectly might throw up loads of problems with another reviewer's workflow. The device is the same, but one reviewer would give it 5* and another 1*, that tells you nothing about the device but a lot about the reviewer. However, you aren't in the market to buy a reviewer.

Of course, it's easy to ask for a comparison between the old XYZ and the new XYZ. The problem lies with the old XYZ being many things to many people. Just think about the classic Yamaha CS80 synth. Many people who really understand synthesis announce that there's nothing special about its topology, and they would be right. What makes it special is how it responds to performance controls.

The bottom line is that, unlike other publications, SOS has no interest whatsoever in telling you what to buy. There really is no BEST AI, no BEST ORCHESTRIAL LIBRAY, no BEST REVERB MODULE. and you can trust the SOS reviewers to give you the pros & cons.

They aren't here to make your decision for you, their job is to give you the information and context to help you make your own informed decision. And they do that very well indeed.

And on my part, I might suggest that the key lesson to take away is to question your starting assumptions much harder, much sooner. :)

Andy :beamup:
User avatar
zenguitar
Moderator
Posts: 10610
Joined: Thu Dec 05, 2002 1:00 am
Location: Devon
The more I understand Music, the less good I get at it.

Re: SOS Product Reviews

Postby resistorman » Thu Sep 17, 2020 4:39 am

Well, Andy beat me to it and expressed all my points far more eloquently than I ever could :D
User avatar
resistorman
Frequent Poster
Posts: 1078
Joined: Sun Nov 22, 2015 1:00 am
Location: Asheville NC

Re: SOS Product Reviews

Postby Arpangel » Thu Sep 17, 2020 7:55 am

The truth is, there’s a lot of good gear out there, and very few bad potatoes, if any really, things have moved on, like cars, there isn’t such a thing as a really bad car anymore, so it becomes difficult to ad anything to a review, apart from a good description, and maybe pointing out potential personal preferences.
User avatar
Arpangel
Jedi Poster
Posts: 5422
Joined: Sat Jul 12, 2003 12:00 am

Re: SOS Product Reviews

Postby CS70 » Thu Sep 17, 2020 8:34 am

Alvin58 wrote:Thanks for your thoughtful reply, it all makes sense.

But let me see if I can be more specific -- the issue becomes this:

- every audio interface is fantastic
- every orchestral software library sounds great
- every reverb module is amazing

Given that, which one should I buy? That's where I would look to SOS for guidance, and that's what I'm not receiving.

Funny, I had he same impression myself at times.. then started actually to think about it. Two things:

1) You are receiving exactly the guidance you seek. Only, it doesn't look like you think it should look. :)

The guidance is: in many cases, hardware quality is so high that differences are irrelevant. Pick what you want and your preferred feature set (for which there's no "best") and it's all about your skill in using it.

The reality of most modern hardware (at least the kind the gets reviewed in SOS and other mags) and much software is that they are amazing - or they would be to anyone used to stuff produced 30 years ago. Due to manufacturing improvements, modularization, component/design reuse and volume of production, we have reached a point where then difference between interfaces and other hardware is basically features (and perhaps longevity/durability), not immediate performance.

Even in consumer context, there's many other fields where you find the same phenomenon: fossil fuel car engines, for example, where any manufacturer worth its name can squeeze unreal amounts of power from low-cost, low-maintenance, powerplants which are durable and small; computers, of course, where the amount of computing power made available to us at low cost is incredible... in 1976 a Cray 500 could execute 160 million instructions per second and it costed more than a villa in Montecarlo. A quad-core i7 processor does about 82 thousand millions (almost 4 orders of magnitude more) and you will be hard pressed to sell it second hand for fifty bucks.

And, oh, microphones.. :D The base level has increased so much, even in the short period I've been interested, that if you have skills you can actually buy a Berry microphone and interface, produce a recording and nobody will be able to tell. 8-9 years ago it still wasn't the case..

So the guidance is clear: from a performance ("quality") point of view, there's a very high likelihood that any audio hardware worth reviewing in a magazine will be pretty good.. and for you as an user you've got to pick the one you like best as feature set, which simply depends on what you want to do. And if something's occasionally - rarely - isn't good, the review will tell it..

These days, for the most part it ain't the equipment.. it's you.

So I still read the reviews, but mostly to figure out what more a certain device does with respect to what I have/know.. because these days that's what it is about in most cases.

2) another factor is that there's much more stuff to review than review space, I guess.. I have no idea how the decision of what ends up in a magazine is made (and I guess different mags have different approaches) but what's certain is that they need to make a choice. And somehow I don't think it's a "first come first served" approach.

If you get a big product from a well known brand, in which readers will be interested, it's much more likely that you want to review that, because it will pull readers. And due to the above, it's very likely that it's good. If there are wrinkles, you will get in touch with the manufacturer and discuss them (and usually these things are reported in reviews).

Now, an aspect I miss from reviews (but it's a bit structural) is how a device perform in the long-term, daily grind as opposite to a short usage period by a reviewer. A bit like "long ownership" reviews for cars. But for the same reason as cars, it's hard to make it work (by the time enough time has passed, the product won't be hot anymore and thus less interesting; equipment is usually returned after a review but doing long-term implies having it for years). Or a "freshen up" of existing reviews - which would be uber cool but require an immense amount of manpower..
User avatar
CS70
Jedi Poster
Posts: 6265
Joined: Mon Nov 26, 2012 1:00 am
Location: Oslo, Norway
Silver Spoon - Check out our latest video and the FB page

Re: SOS Product Reviews

Postby Kwackman » Thu Sep 17, 2020 9:15 am

Alvin58 wrote: - every audio interface is fantastic

If a crap one appears on the market, I'd expect to be told in the review!
The thing is, they probably all are technically fantastic these days!
But....

Alvin58 wrote:Given that, which one should I buy? That's where I would look to SOS for guidance, and that's what I'm not receiving..

Because the one you should buy is the one that fits your needs.
That's a major point of a review.
e.g. if you must have 4 inputs, then a 2 channel interface is useless to you, regardless of how technically brilliant it is. But the review will tell you how many inputs it has.
You might want some monitor controls built into your interface, you might want one that doesn't require an external power unit, you might want more than one headphone out, and guitar/di input, switchable phantom power, MIDI I/O, Optical (ADAT) expansion possibilities etc. etc. So the review will give you this information to help you narrow down which interface suits YOU. If one interface doesn't have MIDI, that might be a deal breaker for someone, but it doesn't make the interface better or worse than another one. It's whatever suits your needs.
User avatar
Kwackman
Frequent Poster
Posts: 1912
Joined: Thu Nov 07, 2002 1:00 am
Location: Belfast
Cubase, guitars.

Re: SOS Product Reviews

Postby The Elf » Thu Sep 17, 2020 9:23 am

I have written reviews for SOS. I can utterly refute this idea that they 'fear to bite the hand that feeds them'. I know because I did write a very rough review for one piece of gear - and from a major company. Other than asking me to make sure that my facts were doubly correct they published it 'warts 'n' all'.

I think you are simply reading into the reviews what you already believe.

As to ranking gear to make choosing simpler... I hate 'desktop' gear, mini keyboards, four-octave keyboards, external power supplies and keyboards without aftertouch. Some of the best selling synths on the planet have all of these (to me) undesirable features. So how would I rank a synth with all of these 'features'? And how might most others rank it?
User avatar
The Elf
Jedi Poster
Posts: 15046
Joined: Tue Aug 14, 2001 12:00 am
Location: Sheffield, UK
An Eagle for an Emperor, A Kestrel for a Knave.

Re: SOS Product Reviews

Postby ef37a » Thu Sep 17, 2020 9:43 am

I generally agree with Andy and the others and I CERTAINLY do not think SOS reviewers 'cow-tow' to companies to protect advertising, but I am not entirely happy with all reviews.

First of all I have been reading audio and electronics magazines for some 60 years. Hi Fi News, Tape Recorder which morphed into Studio Sound, Radio Constructor (LOVED Smiffy!) ,
The Gramaphone (rarely) Wireless World and, of course, Sound on Sound since Feb 2005. (forgot Guitarists! Forgive me Paul!) As computer recording took off I would buy Computer Music, Future Music and others now an again.
Where these publications produced product reviews there was always the same debate (by sloooooow, gentlemanly letter) about 'fairness, independence, to much, too little, bad pictures, in short EVERYTHING that gets complained about now (except the computer used of course!)

My personal beefs (as Doc R well knows!) are,
Reviewers not castigating companies for poor, sometimes virtually non-existent specifications and rubbish 'One Sheet' instruction manuals. I also want to be able to easily access such information as there is and not spend 5 minutes hacking through menus.

The 'excuse' I have been given is, "Most of the population are technically thick and the mnfctrs don't think it worth their time to write decent specs etc. My take on that is...If they can't be bothered (for data that must be in the system anyway) I shall skip over their site and buy elsewhere.

Pleased to see firm still get a mild bllking for not supplying a TB cable. Don't let that slip SOS!

Photos of kit are sometimes poor, too dark to make out connections.
I want to see inside! At least where practical, e.g. compressors, mic pres. Don't TELL me it is nicely built? I will be the judge of that thank you!

Where gear can work on both mac and Windows platforms please report on both.

I could go on but in fact SOS reviews are the best I know of and basically, if it ain't been under the SOS ruler, I don't buy it.

Dave.
ef37a
Jedi Poster
Posts: 12148
Joined: Mon May 29, 2006 12:00 am
Location: northampton uk

Re: SOS Product Reviews

Postby blinddrew » Thu Sep 17, 2020 10:06 am

Alvin58 wrote:Thanks for your thoughtful reply, it all makes sense.

But let me see if I can be more specific -- the issue becomes this:

- every audio interface is fantastic
- every orchestral software library sounds great
- every reverb module is amazing

Given that, which one should I buy? That's where I would look to SOS for guidance, and that's what I'm not receiving.

Star rankings would help, but what would be even more useful, when they rated a new XYZ, they ended up putting it in a ranking against the previous XYZ's they have reviewed. Then a would-be consumer would gain important information.

Consumer Reports in the US does this -- products are ranked in detail from 1 to 100. So one can be pretty certain that something ranked 95 is better than one ranked 85. And one ranked 95 vs 94 is a personal preference.

I would love to see SOS so something like that too.
I completely get where you're coming from here, when I first started reading the magazine a few years ago I was frequently surprised that you'd get 3/4 or even 80% of the way through a review before the reviewer got to how a product really sounded.
It took a while for the penny to drop and for me to realise that, as others have mentioned above (and despite what the marketing and some of the persistent myths around magical bits of gear), basically anything that's good enough to be sent in for review is going to sound great in its price-bracket. Yes, a £5k set of monitors will sound better than a £500 set, but the market is so competitive that choosing between different £500 sets is probably going to come down to what fits on the shelf and what colour you like...
Likewise things like Orchestral libraries, you might get the occasional duff sample (normally picked up in the review), but most of them will all sound fantastic. The biggest differences will be about how you can fit them into your workflow.

But as to helping you make a buying decision, I think all you can expect a magazine to do is to help you put something on your short-list. Or potentially help you take something off your short-list if a review highlights a feature that's lacking, or not well implemented.

There's never been a better time to be buying consumer hardware or software, and I think what you see in the magazine is a reflection of that.
User avatar
blinddrew
Jedi Poster
Posts: 11655
Joined: Sun Jul 05, 2015 12:00 am
Location: York
Ignore the post count, I have no idea what I'm doing...

Re: SOS Product Reviews

Postby Trevor Johnson » Thu Sep 17, 2020 10:07 am

i understand your idea of a ranking system, however it would be doomed to failure, because as newer, better, equipment is released, you would have to retrospectively 'de-rank' older gear, which is clearly not possible. (I realise that not everything newer, even mark II of the same piece of gear is necessarily an improvement).

I find the SOS reviews very nuanced and have helped me buy my comparatively few pieces of equipment, all of which I still use, over the last twenty or so years. Many thanks to all the reviewers past and present!
Trevor Johnson
Regular
Posts: 336
Joined: Sat May 15, 2010 12:00 am

Re: SOS Product Reviews

Postby blinddrew » Thu Sep 17, 2020 10:23 am

The other factor is that the reviewers are not always the same people.
If Bob reviewed Mk1 and really liked its detailed presentation, but Pete reviewed Mk2 and found it overly bright... what then? Reading around the review you can gauge other factors (the room, the material, the other equipment, the personal preference etc) but how could you rank those two products?
User avatar
blinddrew
Jedi Poster
Posts: 11655
Joined: Sun Jul 05, 2015 12:00 am
Location: York
Ignore the post count, I have no idea what I'm doing...

Re: SOS Product Reviews

Postby Fishnish » Thu Sep 17, 2020 10:28 am

I find the SOS reviews to be helpful and when I end up owning/using something, the review usually turns out to have been accurate. Sometimes they've actually contributed to dissuading me from buying a particualr product.

Like most of us I assume, I read watch and listen to a lot of reviews and demos when considering a potential gear purchase. I'd consider the SOS reviews to be one of the more useful tools in my purchaser's toolbox.
User avatar
Fishnish
Regular
Posts: 110
Joined: Fri Dec 24, 2010 1:00 am

Re: SOS Product Reviews

Postby Hugh Robjohns » Thu Sep 17, 2020 10:30 am

Alvin58 wrote:But let me see if I can be more specific -- the issue becomes this:

- every audio interface is fantastic
- every orchestral software library sounds great
- every reverb module is amazing

Given that, which one should I buy? That's where I would look to SOS for guidance, and that's what I'm not receiving.

I wonder if part of the problem is the way you digest and interpret our reviews. Your earlier quoting of the pros/cons summary and mention of a star rating system both suggest you want a very direct, simple guide without needing to think about or interpret a nuanced review. Unfortunately, the world isn't that simple.

Most interfaces are fantastic, and most reverb modules are amazing -- at any given price point. (I don't use or review sample libraries, so I won't comment on them! ;-) ) And even the most basic budget equipment available today is capable of making release-quality recordings if used skillfully.

There really is very little truly crap gear around these days, and what little there is doesn't often make it into the pages of SOS anyway. After all, when there are a limited number of pages, why waste the space on junk when there's already far too much good stuff to talk about?

Of course, there are small and often quite subtle differences between similar products, different features, different optimisations or compromises each of which will appeal to some users more than others -- but mostly they are all perfectly legitimate differences rather than outright good or bad.

So the crux of the matter is that different people have very different requirements and expectations of their equipment. An absolutely essential feature to one person is often completely irrelevant to another -- and those kinds of differences can't be captured in a simple pro/con summary -- and even less so in a star rating or ranking system. What one person would rate reasonably as 3* might equally justifiably get a 5* from another user with different requirements.

Added to which, those kinds of simplistic ratings detract from the actual review which is where all the real information resides. SOS is famous for printing reviews that are routinely two, three or four times longer than any of our competitors -- and they're longer because they contain more detailed information and more useful opinion -- something which we all think matters a lot.

Moreover, all of our reviewers and editors are recording musicians, engineers, or producers, all writing about the gear they are interested in and use every day. That matters too.

I'm guessing you're American, and having talked to a lot of Americans over the years I know some struggle with our 'quaint' British ways, and particularly because our criticisms always seem polite and restrained. Some reading between the lines is usually required to perceive the true strength of opinion... But it is always all in there for those that take the time and trouble to really read and understand what is printed!

And as for 'not biting the hand'... Firstly, while roughly half of the SOS income does indeed come from advertising, the other half comes from newsagent sales and subscriptions from the readers -- and if we didn't have something worthwhile for those readers to enjoy there'd be nowhere for the manufacturers to advertise. So it's actually more important to us to keep the readers happy than the manufacturers. The policy here is that if a review upsets a manufacturer, that's just the way it is: our duty is primarily to the readers.

Obviously we are careful to make sure our facts are correct, and our opinions are valid, and when we have a review which is significantly critical of a product we generally discuss it with the manufacturer before press so they can respond if they want to. Naturally, we try and maintain good relationships with the manufacturers and distributors, but that doesn't mean we don't fall out now and again.

It's also worth noting that we often get to see prototypes or pre-production models specifically so that we can provide constructive feedback -- and often those comments are incorporated into firmware or hardware improvements before the production version is released and subsequently reviewed. I've long since lost count of the number of products I've reviewed or examined that have been improved specifically because of my feedback to the manufacturers. Sometimes that was before I completed a review, and sometimes it was after publishing a critical review! And I've had several review products which have been withdrawn from market after I've found issues and written about them in a critical review!

...what would be even more useful, when they rated a new XYZ, they ended up putting it in a ranking against the previous XYZ's they have reviewed.

I think we generally do that already -- relevant previous-generation products or obvious competitors are usually mentioned and compared as part of the review process. But our view is that a simplistic ranking systems wouldn't be fair to the products or the manufacturers, and wouldn't actually be very helpful to the readers, either.

You can't, for example, say a Neumann KM184 ranks 23 and a Rode TF-5 ranks 35... or 12... it's all utterly meaningless because the 'value' of the mic depends on what you need it to do in each situation... Mics -- like all products -- have a certain unique set of qualities that suite some things better than others. A star rating or ranking list inherently ignores all that. A pros/cons list can highlight the major strengths and weaknesses of a product, but only a full, detailed review can discuss all aspects of a product and place things in contexts that relate to different ways of using it.
User avatar
Hugh Robjohns
Moderator
Posts: 28708
Joined: Fri Jul 25, 2003 12:00 am
Location: Worcestershire, UK
Technical Editor, Sound On Sound

Next