You are here

SOS Please consider making the first n best measurements database

For feedback and suggestions about the SOS magazine, app, web site or forums.

SOS Please consider making the first n best measurements database

Postby TomChimera » Fri Jul 31, 2020 9:53 am

Hello SOS people,
Been reading the magazine for 15 years and this is my first post :)

So in the headphones world, i guess mostly thanks to the Audiophile community we have measurements :)
Also these are challenged and there's a talk about measurements rigs, consistency and discrepancies.
(Great article about this: https://crinacle.com/2020/04/08/graphs- ... surements/)

For example Reference-audio-analyzer website have frequency responses, impulse responses, waterfall plots, step response, impedance at different frequencies, and more and more...

Here an example of the Sennheiser HD 800s:
https://reference-audio-analyzer.pro/en ... s.php#rw32


While in the microphones and studio monitors world we get the company's frequency response, not knowing what was done to it (and many say and show companies edit the results or present in certain ways),
Which signal was used, in which room, how loud, how far from the mic, the preamp, the converter, which loudspeaker was used to test a mic, and which mic was used to test a loudspeaker, etc...
Making comparing microphones and monitors measurements from different companies non reliable.

The whole measurement rig system is part of the result, and the point is not necessarily to get the "truth" but much more importantly is to make consistent measurements using the same rig, thus making the result comparable.

In the SOS article The Yamaha NS10 Story they used Institute of Acoustics paper dedicated for comparable measurements to find out what's special about the NS10.
While in headphones this information is much more abundant and one can make a research at home using measurements database.

Databases are needed in both studio monitors and microphones.

So my request to SOS is to be the pioneer in this, be the first reliable and comparable database, so we will have an online abundant tool. I will pay for it!

Case studies examples.
Lets say you really like the sound of certain old Ribbon mic on double bass which is not available for sale, it smears in a good way the low mid-bass frequencies, you look in the database at it's waterfall plot, (like impulse response, the time domain in certain frequencies) and see how long and which the resonant frequencies. You might try to emulate digitally after recording it with a different mic, you might see which new and available ribbon mic is similar, and generally just become knowledgeable and have the terminology about what was the sound you liked technically.

Another one, you don't have much money, and you see in the measurments the a very cheap mic measures the same as a very expensive one, even though it might not be the same, it is a hint it might be close and work for you for a cheaper price.
Companies wouldn't want it as it exposes their gear for objective comparison without romanticism, and can show that even expensive mic are not perfect and sometime cheaper ones are better.

That's my request for SOS.
Hope it will ignite you curiosity and passion to create it.
Be well
Tom


RAA HD800s measurement example


Image
TomChimera
Poster
Posts: 12
Joined: Fri Jun 19, 2020 5:38 am

Re: SOS Please consider making the first n best measurements database

Postby Hugh Robjohns » Fri Jul 31, 2020 11:22 am

TomChimera wrote:So my request to SOS is to be the pioneer in this, be the first reliable and comparable database, so we will have an online abundant tool.

If only it was a simple and convenient as measuring headphone performance -- and you've already commented on the arguments arising from different ways of measuring headphones!

To measure speakers accurately you need regular access to a large anechoic chamber. To test microphones accurately you need regular access to a not quite so large anechoic chamber. SOS doesn't have those facilities and -- more importantly -- couldn't commercially justify anything like the true budget required to hire those kinds of facilities or the trained staff necessary to perform the testing.

Resolution magazine does publish fairly comprehensive speaker measurements in its reviews:

https://www.resolutionmag.com/reviews/monitoring/

This is thanks to a mutually beneficial arrangement between Resolution magazine and Dr Keith Holland at the Institute of Sound and Vibration Research at the University of Southampton (which does have suitable anechoic chambers and plentiful students able to enhance their education by helping with the testing. There are very few similar facilities in the UK.

Having said that, a lot of our own speaker reviews are performed by Phil Ward who is a very experienced independent professional loudspeaker designer, and he does have the facilities and knowledge required to perform a variety of measurements where relevant to demonstrating any specific points he raises in his reviews.

I can see why you're asking for the database, and I share your interest in such a thing to a degree, but sadly the commercial practicalities of publishing a magazine simply don't allow us the kind of cost and time involved in testing equipment to sufficiently high standards to build such a database.
User avatar
Hugh Robjohns
Moderator
Posts: 27928
Joined: Fri Jul 25, 2003 12:00 am
Location: Worcestershire, UK
Technical Editor, Sound On Sound

Re: SOS Please consider making the first n best measurements database

Postby TomChimera » Fri Jul 31, 2020 4:49 pm

Hugh Robjohns wrote:If only it was a simple and convenient as measuring headphone performance -- and you've already commented on the arguments arising from different ways of measuring headphones!

To measure speakers accurately you need regular access to a large anechoic chamber. To test microphones accurately you need regular access to a not quite so large anechoic chamber. SOS doesn't have those facilities and -- more importantly -- couldn't commercially justify anything like the true budget required to hire those kinds of facilities or the trained staff necessary to perform the testing.

Resolution magazine does publish fairly comprehensive speaker measurements in its reviews:

https://www.resolutionmag.com/reviews/monitoring/

This is thanks to a mutually beneficial arrangement between Resolution magazine and Dr Keith Holland at the Institute of Sound and Vibration Research at the University of Southampton (which does have suitable anechoic chambers and plentiful students able to enhance their education by helping with the testing. There are very few similar facilities in the UK.

Having said that, a lot of our own speaker reviews are performed by Phil Ward who is a very experienced independent professional loudspeaker designer, and he does have the facilities and knowledge required to perform a variety of measurements where relevant to demonstrating any specific points he raises in his reviews.

I can see why you're asking for the database, and I share your interest in such a thing to a degree, but sadly the commercial practicalities of publishing a magazine simply don't allow us the kind of cost and time involved in testing equipment to sufficiently high standards to build such a database.

Hi Hugh! Thanks for explaining this, I understand much better now what is required for such a project.

I hope to ignite more interest in such database, with the hopes maybe people with available facilities, universities, or passionate individuals with budget would like to create it or collaborate.

Thanks :)
TomChimera
Poster
Posts: 12
Joined: Fri Jun 19, 2020 5:38 am

Re: SOS Please consider making the first n best measurements database

Postby Trevor Johnson » Fri Jul 31, 2020 5:32 pm

Audio isn't my field, but I think that sample variation (i.e. variation within the number of units analysed) should be considered when testing equipment.

In photographic lens testing for example, Roger Cicala, who founded Lensrentals in the USA, buys in multiple of new lenses and can test them with state of the art equipment. The results can be very interesting and there is more sample variation than you might expect on lenses costing >£1000! I wonder what the situation is with audio and whether the differences may be significant, or, not?
Trevor Johnson
Regular
Posts: 309
Joined: Sat May 15, 2010 12:00 am

Re: SOS Please consider making the first n best measurements database

Postby blinddrew » Fri Jul 31, 2020 5:52 pm

Isn't this pretty much what companies like Sonarworks are doing?
User avatar
blinddrew
Jedi Poster
Posts: 10878
Joined: Sun Jul 05, 2015 12:00 am
Location: York
Ignore the post count, I have no idea what I'm doing...

Re: SOS Please consider making the first n best measurements database

Postby TomChimera » Fri Jul 31, 2020 9:19 pm

Trevor Johnson wrote:Audio isn't my field, but I think that sample variation (i.e. variation within the number of units analysed) should be considered when testing equipment.

In photographic lens testing for example, Roger Cicala, who founded Lensrentals in the USA, buys in multiple of new lenses and can test them with state of the art equipment. The results can be very interesting and there is more sample variation than you might expect on lenses costing >£1000! I wonder what the situation is with audio and whether the differences may be significant, or, not?

Hi Trevor, that's very interesting about lenses, didn't know that.
Yes there's is a lot of variation as far as I know, and specifically in headphones even expensive ones like Audeze have been shown to have a quite a lot.
You can see in headphones measurements at crinacle.com "graph comparison tool", he have sample variation an that's only for frequency response! Not including many other measurements..

Subjectively speaking, I noticed quite big differences when same model of gear was manufactured in different years..
TomChimera
Poster
Posts: 12
Joined: Fri Jun 19, 2020 5:38 am

Re: SOS Please consider making the first n best measurements database

Postby TomChimera » Fri Jul 31, 2020 9:28 pm

blinddrew wrote:Isn't this pretty much what companies like Sonarworks are doing?

Hi blinddrew,
In the field of headphones Sonarworks calibrate for their model of HRTF, and unless you get individual profile it's an average between sample variation (which is what Trevor talked about)

As for room calibration, they measure frequency response, I believe it can be very practical for studio work, but I guess it doesn't qualify as measurement for scientific data like Hugh explained.. which I just learned :)
TomChimera
Poster
Posts: 12
Joined: Fri Jun 19, 2020 5:38 am

Re: SOS Please consider making the first n best measurements database

Postby CS70 » Fri Jul 31, 2020 9:49 pm

TomChimera wrote:
Trevor Johnson wrote:Audio isn't my field, but I think that sample variation (i.e. variation within the number of units analysed) should be considered when testing equipment.

In photographic lens testing for example, Roger Cicala, who founded Lensrentals in the USA, buys in multiple of new lenses and can test them with state of the art equipment. The results can be very interesting and there is more sample variation than you might expect on lenses costing >£1000! I wonder what the situation is with audio and whether the differences may be significant, or, not?

Hi Trevor, that's very interesting about lenses, didn't know that.

The fun thing is, it goes for the camera bodies as well.

So it may happen that you're lucky and your camera body sort of counteracts the tolerances of the specific lens and you think that lens is wonderful. But with another body it may perform much worse.

For example I had a kit zoom that I got with my first Panny and always thought was horrible - getting really lo-fi results on video.. the third body I paired it with produce a stunning video result!

That's why every lens you buy should be tried with the specific body you intend to use it with. Unless you buy Leica. :D
User avatar
CS70
Jedi Poster
Posts: 5787
Joined: Mon Nov 26, 2012 1:00 am
Location: Oslo, Norway
Silver Spoon - Check out our latest video and the FB page

Re: SOS Please consider making the first n best measurements database

Postby Hugh Robjohns » Fri Jul 31, 2020 10:45 pm

Everything is manufactured within certain tolerances. This is perfectly normal and unavoidable. The better the manufacturer, and the better the product, the tighter (smaller) the tolerances.

And regarding camera lenses/bodies, the professional cameras are usually adjustable to fine-tune each lens to the body to dial out the tolerances.
User avatar
Hugh Robjohns
Moderator
Posts: 27928
Joined: Fri Jul 25, 2003 12:00 am
Location: Worcestershire, UK
Technical Editor, Sound On Sound

Re: SOS Please consider making the first n best measurements database

Postby James Perrett » Sat Aug 01, 2020 12:26 am

Hugh Robjohns wrote:Everything is manufactured within certain tolerances. This is perfectly normal and unavoidable. The better the manufacturer, and the better the product, the tighter (smaller) the tolerances.

In audio a good example of this is the BBC LS3/5a monitor. Yes, anyone can stick a KEF B110 and T27 in a similar size box and make something that looks like an LS3/5a but the real thing will have drivers that are selected to a much tighter specification than standard. I believe that the production speakers had to match the reference within 1dB. That's why a real LS3/5a would have cost more than a similar looking competitor when new.
User avatar
James Perrett
Moderator
Posts: 9478
Joined: Mon Sep 10, 2001 12:00 am
Location: The wilds of Hampshire
JRP Music - Audio Mastering and Restoration. JRP Music Facebook Page

Re: SOS Please consider making the first n best measurements database

Postby CS70 » Sun Aug 02, 2020 2:31 pm

Hugh Robjohns wrote:Everything is manufactured within certain tolerances. This is perfectly normal and unavoidable. The better the manufacturer, and the better the product, the tighter (smaller) the tolerances.

Hence Leica.
User avatar
CS70
Jedi Poster
Posts: 5787
Joined: Mon Nov 26, 2012 1:00 am
Location: Oslo, Norway
Silver Spoon - Check out our latest video and the FB page