The Anatomy of A Review

[Mister Pig]

I do a bit of writing for Affordable Audio, and have my spent a bit of time thinking about how to properly set up the reviewing process. It is my opinion that there are two main ways to review a component.

One is to install the new component into an existing reference system, and report the results, either good or bad. I can understand the benefits of proceeding in this manner, but it's not my method of choice.

Another choice is to find a complimentary mix of components that result in a system that the review component works well in. In this case, understanding the mix of components in the review system is as important as the description given of the unit under evaluation. Of course, this method is limited by the stable of stereo gear that the reviewer has access to, and his abilities to properly set the system up. This is the my preferred method of writing a review, although it may lead to articles with a consistently positive slant. The key to understanding the review is to carefully look at the system that has been assembled for the piece, and think about how the pieces will interact.

A review should never be the sole basis for a buying decision. I think many of us who have been in the hobby awhile understand this. Although the review will give a perspective buyer an idea if a component should be on their short list of components to audition.

These are my thoughts on the subject, yours may vary. Please feel free to express your thoughts and opinions on this subject.

Regards
Mister Pig

[Superfly]

With synergy between components so important I think you have to do that. I have had speakers that sound horrible on one amp and excellent on another. To review them on the horrible amp would be a great disservice.

[itlldue]

I agree with your methods.

It's along the same lines as Ian Grant discussing his B-283 processor. He clearly states that if you use it in a total top-of-the-line system, there may be no improvement. However, in a more "average" setup, you should hear a difference. That's the only fair way to evaluate it.

[Grant Fidelity]

I think it should be left up to the reviewers judgement on what the product will likely be used with and then trying the gear with a: his/her reference system with notes taken for the review, that provides a good idea of how it sounds/performs to the reviewer on what the reviewer knows the best. b: then try and put it in a system that matches the intended use and similar cost points. c: if available some form of vintage gear to A/B it against, that is the coolest thing since sliced bread and kudos to Affordable for doing this.

I enjoyed learing how both Todd and Stew do their reviews. They both took time to explain their gear and how they planned to conduct the tests. They asked for my suggestions too, which was cool. I just said however you want and that worked out just fine

I agree completely that you shouldn't buy based on a review, you have to find out for yourself, even word of mouth or friend's recommendation cannot predict your own ears. A good dealer will take the time with you.

I will state this again, Affordable$$Audio is the only reviewers that have:
a. contacted us.
b. did not say here's our rate sheet for advertising.

And that is cool. When I get a bad review from these guys, I'll know that the product needs some more work.

Ian

This post has been edited by Grant Fidelity on January 17, 2008 11:42 am

[rroobbcc]

Good topic! I have been thinking a bit about this myself lately, as I have a couple of "shootouts" I want to conduct with some of my own equipment.

The opinion I am beginning to come to is that I think you have to consider the equipment, the room, and the music. Also that a review should be done from the perspective of both the "best-case" as well as the "typical-case" scenarios.

Determining best-case for a component would usually mean using a "reference" system. For most of us "reference" means "the best I have available to me". However, I would append a qualifier onto that to also say "the stuff I am most familiar with". Both of these conditions, but in particular the latter, should also apply to the music selected for the review. I have a CD that I put together that I use for all of my critical listening. This CD contains many genres of music, each picked to help me evaluate different aspects of a system's performance. Most of the tracks I have been using since the late 80's and early 90's. Suffice it to say that I know them well. I recently updated my reference CD and have been listening to it almost exclusively to allow to become familiar with the many nuances of the new tracks.

So for the sake of discussion consider the pair of B&W DM601 S3 I recently picked up, and how I might go about reviewing them (Let's also assume that construction of my listening room is complete, which sadly it is not). I would start out by listening to these speakers in my dedicated listening room on my "reference system". However, before giving the 601s a turn, I would most likely listen to a few tracks from the reference CD to ensure I had a fresh picture of the reference system's character, allowing a better comparision. I may even switch between the 601s and my reference speakers numerous times during the evaluation to help me identify all of the differences (good and bad) or lack there of. Once this "best case" evaluation is over I carry the little B&Ws to the living room for an evaluation of their performance in a more typical environment, and using electronics they are most likely to be paired with. This would again include listening to the system with its regular speakers, as well as swapping back and forth. By this point I should have a good idea of the "best case" as well as "typical case" performance that should be expected from these speakers.

Of course this means a lot more work for the reviewer as well as a need for a lot more equipment to be on hand. And there are still plenty of other variables, such as synergy between components (especially between amps and speakers). And one thing we all need to realize is that no review will be able to tell you exactly how a particular component will sound in YOUR system, in YOUR room, playing YOUR music. Reviews just try to give you an additional perspective, but ultimately you have to let your own ears make the decision.

Rob