You are here

SOS Please consider making the first n best measurements database

For feedback and suggestions about the SOS magazine, app, web site or forums.

SOS Please consider making the first n best measurements database

Postby TomChimera » Fri Jul 31, 2020 9:53 am

Hello SOS people,
Been reading the magazine for 15 years and this is my first post :)

So in the headphones world, i guess mostly thanks to the Audiophile community we have measurements :)
Also these are challenged and there's a talk about measurements rigs, consistency and discrepancies.
(Great article about this: https://crinacle.com/2020/04/08/graphs- ... surements/)

For example Reference-audio-analyzer website have frequency responses, impulse responses, waterfall plots, step response, impedance at different frequencies, and more and more...

Here an example of the Sennheiser HD 800s:
https://reference-audio-analyzer.pro/en ... s.php#rw32


While in the microphones and studio monitors world we get the company's frequency response, not knowing what was done to it (and many say and show companies edit the results or present in certain ways),
Which signal was used, in which room, how loud, how far from the mic, the preamp, the converter, which loudspeaker was used to test a mic, and which mic was used to test a loudspeaker, etc...
Making comparing microphones and monitors measurements from different companies non reliable.

The whole measurement rig system is part of the result, and the point is not necessarily to get the "truth" but much more importantly is to make consistent measurements using the same rig, thus making the result comparable.

In the SOS article The Yamaha NS10 Story they used Institute of Acoustics paper dedicated for comparable measurements to find out what's special about the NS10.
While in headphones this information is much more abundant and one can make a research at home using measurements database.

Databases are needed in both studio monitors and microphones.

So my request to SOS is to be the pioneer in this, be the first reliable and comparable database, so we will have an online abundant tool. I will pay for it!

Case studies examples.
Lets say you really like the sound of certain old Ribbon mic on double bass which is not available for sale, it smears in a good way the low mid-bass frequencies, you look in the database at it's waterfall plot, (like impulse response, the time domain in certain frequencies) and see how long and which the resonant frequencies. You might try to emulate digitally after recording it with a different mic, you might see which new and available ribbon mic is similar, and generally just become knowledgeable and have the terminology about what was the sound you liked technically.

Another one, you don't have much money, and you see in the measurments the a very cheap mic measures the same as a very expensive one, even though it might not be the same, it is a hint it might be close and work for you for a cheaper price.
Companies wouldn't want it as it exposes their gear for objective comparison without romanticism, and can show that even expensive mic are not perfect and sometime cheaper ones are better.

That's my request for SOS.
Hope it will ignite you curiosity and passion to create it.
Be well
Tom


RAA HD800s measurement example


Image
TomChimera
Poster
Posts: 28
Joined: Fri Jun 19, 2020 5:38 am

Re: SOS Please consider making the first n best measurements database

Postby Hugh Robjohns » Fri Jul 31, 2020 11:22 am

TomChimera wrote:So my request to SOS is to be the pioneer in this, be the first reliable and comparable database, so we will have an online abundant tool.

If only it was a simple and convenient as measuring headphone performance -- and you've already commented on the arguments arising from different ways of measuring headphones!

To measure speakers accurately you need regular access to a large anechoic chamber. To test microphones accurately you need regular access to a not quite so large anechoic chamber. SOS doesn't have those facilities and -- more importantly -- couldn't commercially justify anything like the true budget required to hire those kinds of facilities or the trained staff necessary to perform the testing.

Resolution magazine does publish fairly comprehensive speaker measurements in its reviews:

https://www.resolutionmag.com/reviews/monitoring/

This is thanks to a mutually beneficial arrangement between Resolution magazine and Dr Keith Holland at the Institute of Sound and Vibration Research at the University of Southampton (which does have suitable anechoic chambers and plentiful students able to enhance their education by helping with the testing. There are very few similar facilities in the UK.

Having said that, a lot of our own speaker reviews are performed by Phil Ward who is a very experienced independent professional loudspeaker designer, and he does have the facilities and knowledge required to perform a variety of measurements where relevant to demonstrating any specific points he raises in his reviews.

I can see why you're asking for the database, and I share your interest in such a thing to a degree, but sadly the commercial practicalities of publishing a magazine simply don't allow us the kind of cost and time involved in testing equipment to sufficiently high standards to build such a database.
User avatar
Hugh Robjohns
Moderator
Posts: 28593
Joined: Fri Jul 25, 2003 12:00 am
Location: Worcestershire, UK
Technical Editor, Sound On Sound

Re: SOS Please consider making the first n best measurements database

Postby TomChimera » Fri Jul 31, 2020 4:49 pm

Hugh Robjohns wrote:If only it was a simple and convenient as measuring headphone performance -- and you've already commented on the arguments arising from different ways of measuring headphones!

To measure speakers accurately you need regular access to a large anechoic chamber. To test microphones accurately you need regular access to a not quite so large anechoic chamber. SOS doesn't have those facilities and -- more importantly -- couldn't commercially justify anything like the true budget required to hire those kinds of facilities or the trained staff necessary to perform the testing.

Resolution magazine does publish fairly comprehensive speaker measurements in its reviews:

https://www.resolutionmag.com/reviews/monitoring/

This is thanks to a mutually beneficial arrangement between Resolution magazine and Dr Keith Holland at the Institute of Sound and Vibration Research at the University of Southampton (which does have suitable anechoic chambers and plentiful students able to enhance their education by helping with the testing. There are very few similar facilities in the UK.

Having said that, a lot of our own speaker reviews are performed by Phil Ward who is a very experienced independent professional loudspeaker designer, and he does have the facilities and knowledge required to perform a variety of measurements where relevant to demonstrating any specific points he raises in his reviews.

I can see why you're asking for the database, and I share your interest in such a thing to a degree, but sadly the commercial practicalities of publishing a magazine simply don't allow us the kind of cost and time involved in testing equipment to sufficiently high standards to build such a database.

Hi Hugh! Thanks for explaining this, I understand much better now what is required for such a project.

I hope to ignite more interest in such database, with the hopes maybe people with available facilities, universities, or passionate individuals with budget would like to create it or collaborate.

Thanks :)
TomChimera
Poster
Posts: 28
Joined: Fri Jun 19, 2020 5:38 am

Re: SOS Please consider making the first n best measurements database

Postby Trevor Johnson » Fri Jul 31, 2020 5:32 pm

Audio isn't my field, but I think that sample variation (i.e. variation within the number of units analysed) should be considered when testing equipment.

In photographic lens testing for example, Roger Cicala, who founded Lensrentals in the USA, buys in multiple of new lenses and can test them with state of the art equipment. The results can be very interesting and there is more sample variation than you might expect on lenses costing >£1000! I wonder what the situation is with audio and whether the differences may be significant, or, not?
Trevor Johnson
Regular
Posts: 334
Joined: Sat May 15, 2010 12:00 am

Re: SOS Please consider making the first n best measurements database

Postby blinddrew » Fri Jul 31, 2020 5:52 pm

Isn't this pretty much what companies like Sonarworks are doing?
User avatar
blinddrew
Jedi Poster
Posts: 11569
Joined: Sun Jul 05, 2015 12:00 am
Location: York
Ignore the post count, I have no idea what I'm doing...

Re: SOS Please consider making the first n best measurements database

Postby TomChimera » Fri Jul 31, 2020 9:19 pm

Trevor Johnson wrote:Audio isn't my field, but I think that sample variation (i.e. variation within the number of units analysed) should be considered when testing equipment.

In photographic lens testing for example, Roger Cicala, who founded Lensrentals in the USA, buys in multiple of new lenses and can test them with state of the art equipment. The results can be very interesting and there is more sample variation than you might expect on lenses costing >£1000! I wonder what the situation is with audio and whether the differences may be significant, or, not?

Hi Trevor, that's very interesting about lenses, didn't know that.
Yes there's is a lot of variation as far as I know, and specifically in headphones even expensive ones like Audeze have been shown to have a quite a lot.
You can see in headphones measurements at crinacle.com "graph comparison tool", he have sample variation an that's only for frequency response! Not including many other measurements..

Subjectively speaking, I noticed quite big differences when same model of gear was manufactured in different years..
TomChimera
Poster
Posts: 28
Joined: Fri Jun 19, 2020 5:38 am

Re: SOS Please consider making the first n best measurements database

Postby TomChimera » Fri Jul 31, 2020 9:28 pm

blinddrew wrote:Isn't this pretty much what companies like Sonarworks are doing?

Hi blinddrew,
In the field of headphones Sonarworks calibrate for their model of HRTF, and unless you get individual profile it's an average between sample variation (which is what Trevor talked about)

As for room calibration, they measure frequency response, I believe it can be very practical for studio work, but I guess it doesn't qualify as measurement for scientific data like Hugh explained.. which I just learned :)
TomChimera
Poster
Posts: 28
Joined: Fri Jun 19, 2020 5:38 am

Re: SOS Please consider making the first n best measurements database

Postby CS70 » Fri Jul 31, 2020 9:49 pm

TomChimera wrote:
Trevor Johnson wrote:Audio isn't my field, but I think that sample variation (i.e. variation within the number of units analysed) should be considered when testing equipment.

In photographic lens testing for example, Roger Cicala, who founded Lensrentals in the USA, buys in multiple of new lenses and can test them with state of the art equipment. The results can be very interesting and there is more sample variation than you might expect on lenses costing >£1000! I wonder what the situation is with audio and whether the differences may be significant, or, not?

Hi Trevor, that's very interesting about lenses, didn't know that.

The fun thing is, it goes for the camera bodies as well.

So it may happen that you're lucky and your camera body sort of counteracts the tolerances of the specific lens and you think that lens is wonderful. But with another body it may perform much worse.

For example I had a kit zoom that I got with my first Panny and always thought was horrible - getting really lo-fi results on video.. the third body I paired it with produce a stunning video result!

That's why every lens you buy should be tried with the specific body you intend to use it with. Unless you buy Leica. :D
User avatar
CS70
Jedi Poster
Posts: 6185
Joined: Mon Nov 26, 2012 1:00 am
Location: Oslo, Norway
Silver Spoon - Check out our latest video and the FB page

Re: SOS Please consider making the first n best measurements database

Postby Hugh Robjohns » Fri Jul 31, 2020 10:45 pm

Everything is manufactured within certain tolerances. This is perfectly normal and unavoidable. The better the manufacturer, and the better the product, the tighter (smaller) the tolerances.

And regarding camera lenses/bodies, the professional cameras are usually adjustable to fine-tune each lens to the body to dial out the tolerances.
User avatar
Hugh Robjohns
Moderator
Posts: 28593
Joined: Fri Jul 25, 2003 12:00 am
Location: Worcestershire, UK
Technical Editor, Sound On Sound

Re: SOS Please consider making the first n best measurements database

Postby James Perrett » Sat Aug 01, 2020 12:26 am

Hugh Robjohns wrote:Everything is manufactured within certain tolerances. This is perfectly normal and unavoidable. The better the manufacturer, and the better the product, the tighter (smaller) the tolerances.

In audio a good example of this is the BBC LS3/5a monitor. Yes, anyone can stick a KEF B110 and T27 in a similar size box and make something that looks like an LS3/5a but the real thing will have drivers that are selected to a much tighter specification than standard. I believe that the production speakers had to match the reference within 1dB. That's why a real LS3/5a would have cost more than a similar looking competitor when new.
User avatar
James Perrett
Moderator
Posts: 9778
Joined: Mon Sep 10, 2001 12:00 am
Location: The wilds of Hampshire
JRP Music - Audio Mastering and Restoration. JRP Music Facebook Page

Re: SOS Please consider making the first n best measurements database

Postby CS70 » Sun Aug 02, 2020 2:31 pm

Hugh Robjohns wrote:Everything is manufactured within certain tolerances. This is perfectly normal and unavoidable. The better the manufacturer, and the better the product, the tighter (smaller) the tolerances.

Hence Leica.
User avatar
CS70
Jedi Poster
Posts: 6185
Joined: Mon Nov 26, 2012 1:00 am
Location: Oslo, Norway
Silver Spoon - Check out our latest video and the FB page

Re: SOS Please consider making the first n best measurements database

Postby ef37a » Fri Aug 21, 2020 10:41 am

Great idea! Sadly as High says, simply not practical for a magazine such as SOS today.

I do however think some reviewers should be more, shall we say, assertive?

There is the almost constant complaint when a piece of Thunderbolt gear does not get supplied with a TB cable and I hope that will keep on being said until every manufacturer supplies one. Some will of course site the extra cost but, unlike say XLR cables, DINs or USB cables, most people I suspect are not knee deep in TB cables? I doubt the AI makers would pay a fraction of the price 'we' have to? (I am actually wondering about the legality here? If the kit cannot work out of the box is that contrary to the Sale of Goods act?)

Reviewers could also criticize firm that do not supply, even for download, good documentation more strongly IMHO. I do not expect a 20 page A4 manual with a £200 AI these days but I DO expect a decent .pdf. They only have to write it once!
They should also castigate companies that give only the most loose and spares of specifications. When I come to buy a product my first task is to download all the data I can about it. If I find that difficult to do (some firms lead you through a right maze!) or I find the documents minging, there I stop and look elsewhere.

Another moan! I find it astonishing when I read "Could not test this chatsbyclansenpipe Mk3 on a PC because I do not own one" Why not? I am poor and I have four!

Ok, yes, I am from another era of G J King and Terry Long, and Hi Fi News before it got silly and those people REALLY had a pop at sloppy specifications!

Dave.
ef37a
Jedi Poster
Posts: 12123
Joined: Mon May 29, 2006 12:00 am
Location: northampton uk

Re: SOS Please consider making the first n best measurements database

Postby Hugh Robjohns » Fri Aug 21, 2020 11:53 am

ef37a wrote:When I come to buy a product my first task is to download all the data I can about it.

But, as we've disccussed often before, you're of a dying breed of technically-minded enthusiasts.

From what I see, most purchasers these days base their purchase decisions on whether enough people says it's AWESOME on GS, and abuse them because they can't possibly record any good tracks unless they have one!

Another moan! I find it astonishing when I read "Could not test this chatsbyclansenpipe Mk3 on a PC because I do not own one" Why not? I am poor and I have four!

How many Macs have you got? ;)

...those people REALLY had a pop at sloppy specifications!

Some of still do! At least, we do when it's relevant... but specs generally aren't relevant anymore. Everything has a DC to daylight frequency response, noise and distortion below audibility, and all the rest. There's barely anything to tell them apart.

Equipment these days is chosen on the basis of facilities rather than specs because that's all that distinguishes them now. 30 or 40 years ago the specs did separate the good from the bad, but that era is long gone.

Interestingly, though, I've just reviewed a very high-end mic preamp that will appear in next month's magazine, and the fact that it came with an astonishingly paltry set of specs did ring alarm bells. And rightly so as it turned out! :shocked:

And yet I read comments on other forums from excited potential customers who stated their intentions to buy purely and solely because of the high-profile badge on the front -- intentionally ignoring it's very dubious technical capabilities completely! That's the world we now live in, sadly!
User avatar
Hugh Robjohns
Moderator
Posts: 28593
Joined: Fri Jul 25, 2003 12:00 am
Location: Worcestershire, UK
Technical Editor, Sound On Sound

Re: SOS Please consider making the first n best measurements database

Postby CS70 » Fri Aug 21, 2020 2:13 pm

Hugh Robjohns wrote:Some of still do! At least, we do when it's relevant... but specs generally aren't relevant anymore. Everything has a DC to daylight frequency response, noise and distortion below audibility, and all the rest. There's barely anything to tell them apart.

Equipment these days is chosen on the basis of facilities rather than specs because that's all that distinguishes them now. 30 or 40 years ago the specs did separate the good from the bad, but that era is long gone.

Since we're moaning :D, it is fun how many people constantly attribute their less than perfect results to equipment, as opposite to their own ability of using it..

Perhaps it's just made more visibile by the internet social media and forums, dunno. But to so many, the idea that the issue may not be in the equipment but in their own lack of understanding seems totally alien. Which in turns prevents them from making progress - skills can be learned and understanding acquired, if only one's aware that that's what might be missing! Where buying another interface won't fix any of that..

Interestingly, though, I've just reviewed a very high-end mic preamp that will appear in next month's magazine, and the fact that it came with an astonishingly paltry set of specs did ring alarm bells. And rightly so as it turned out! :shocked:

And yet I read comments on other forums from excited potential customers who stated their intentions to buy purely and solely because of the high-profile badge on the front -- intentionally ignoring it's very dubious technical capabilities completely! That's the world we now live in, sadly!

Ah! Exciting! Looking forward to the review!
User avatar
CS70
Jedi Poster
Posts: 6185
Joined: Mon Nov 26, 2012 1:00 am
Location: Oslo, Norway
Silver Spoon - Check out our latest video and the FB page

Re: SOS Please consider making the first n best measurements database

Postby blinddrew » Fri Aug 21, 2020 2:31 pm

CS70 wrote:Since we're moaning :D, it is fun how many people constantly attribute their less than perfect results to equipment, as opposite to their own ability of using it..

Perhaps it's just made more visibile by the internet social media and forums, dunno. But to so many, the idea that the issue may not be in the equipment but in their own lack of understanding seems totally alien. Which in turns prevents them from making progress - skills can be learned and understanding acquired, if only one's aware that that's what might be missing! Where buying another interface won't fix any of that..
I think there are a couple of basics that you need (interface, DAW, good headphones, half decent mic - probably less than £400 all in) but after that I agree it's all about the user.
I had this proved to me a couple of years ago when I paid for an afternoon of the Elf's time and he massively improved a couple of my tracks using absolutely nothing that I didn't already have. :)
But I don't regret buying some nice monitors! :D
User avatar
blinddrew
Jedi Poster
Posts: 11569
Joined: Sun Jul 05, 2015 12:00 am
Location: York
Ignore the post count, I have no idea what I'm doing...

Next