Friday, June 25, 2010

How to Test Tests

A recent news story reports on tests of the cameras of four high end smart phones, giving high marks to the new iPhone. Reading it, it occurred to me that would be nice to have some measure of the reliability of reports of this sort.

In this case, there is a simple way to do it. Each image was evaluated by five independent judges; their conclusions were combined for the final result. It would be straightforward to calculate the standard deviation of the scores they produced and from that how likely it was that the difference between the phone that won and the phone that came in second was due to chance.

4 comments:

Anonymous said...

Or you could use science. There is a point where the area on the sensor and the number of pixels per area reach a point where the image degrades. After all, there are only so many photons out there.

Anonymous said...

Just because the iPhone 4 is better than your droid....

Anonymous said...

A better test would also have involved, say, 5 of each phone. Individual differences between iPhones might well be larger than differences between different phones. That is at least apocryphally true.

Andrew said...

IF you care enough to look at standard deviation numbers you really should simply compare the sample images for yourself. It's a lot more important to pick images that you like than what experts like.