I think a new rating sistem is strongly required.Look at audio.de and stereo.de for example They have 2 ratings : 1 for quality (audio.de reviewed speakers from 46 points to 108 points) and a value rating (1 to 5). I really like that a lot! The quality ratting should never not deppend on price and should not have a maxim value. Thus we will be able to compare a 10 year old product with a new one. Let me give an example : let's say A/V recievers we reviewed 3 years ago got quality points from 40 to 110. Now however high definition audio matter a lot so we give every new reciever up to maximum 10 points for this functionality and so now the maximum is 120.
The Great HT Ratings Debate
I admit I’m surprised by the controversy caused by removing the product review ratings. Most who’ve written want the ratings back and accuse HT of running scared from advertisers/manufacturers. I want to set the record straight, elaborate on the reasons we eliminated the ratings, and engage in some meaningful debate about them. I also want to note that while the ratings in their original format will never return, I’m not totally opposed to a revamped system if you readers still think it’s necessary. There are some features coming up that I think will make you miss the ratings less, and I’ll explain those below.
First and foremost, with complete support of HT’s staff I dropped the ratings because I firmly believed that the ratings system results were artificially high, allowing mediocre components to hide behind ratings that looked good numerically. As I noted in the Letters section of the July issue, of the 160+ components reviewed and rated from January 2007 through April of 2008, only a handful received overall ratings below 84 pts, indicating below average. A whopping 75% of the components were rated at 90 pts or greater, and the average overall rating was 91 pts.
For those of you still insisting that advertiser pressure was the reason behind this move, please read the above again. Manufacturers were not complaining about their ratings. At best the ratings showed some distinction between better and best. But in every instance in which a product was rated as truly exceptional that was abundantly clear reading the review.
And I don’t think this was the fault of the reviewers not being tough enough, but faulty architecture in the system and the way the performance descriptors corresponded to the numbers in place. While the ratings scale ostensibly went to 100, it wasn’t a full 100-point scale. It was really a 74-100 scale, which bunched up a lot of scores at 90 points. It was more like having something that looks like a full 5 point scale, but was really a 3-5 point scale, with of the ratings at 4.5 points and above!
I think some of these issues echo something a reader said in lauding the ratings removal- if you have to explain it every other month then something’s wrong. Frankly, it’s hard to come up with a single ratings system that really works without more explanation than necessary, and then explanations to follow explanations.
Also, while the numbers lent an air of objectivity, they were every bit as subjective as the written review. And with different reviewers, the scale was never really the same across the board as hard numbers inherently imply. And if it’s not, then what is the real value in the numbers? Don’t you have to rely on the review text and the reviewers’ personality to discern these nuances anyway?
Another factor is that with HT’s current design (which will be made over with our October issue) the graphics supporting the ratings used a lot of real estate on the page. So much that a 3 page review was limited to as little as 1400 words. That’s not a lot of space to offer meaningful analysis today’s complex products. Eliminating the ratings allowed us an immediate increase in word count over 25%. So, I saw removing the ratings as mothballing a flawed system, and providing more space for analysis and more authoritative reviews. I feel strongly that the strength of this magazine over its competitors is the knowledge and experience of our writers and their ability to convey in-depth opinion and analysis on the printed page. And I thought that removing the ratings would make this more apparent. But some of you clearly don’t agree.
Now, one significant new feature in HT that I want to ping you on debuts with the August issue, hitting newsstands and subscribers right about now. Every issue of HT will feature two pages that delineate our Top Picks in every product category, broken down by Entry Level, Midrange and High End price points. In every issue the products reviewed that make the grade will appear in Top Picks, and starting in October a Top Picks graphic will be on the opening page of the review, showing you at a glance that the product made the grade. We’re being very selective about these Picks, so they’re not just a list of what we’ve reviewed. And in October, a set of highlights will also be on the opening page of each review, telling you in a capsule what’s hot and what’s not about each product.
So, my question is, do you acknowledge what’s written above and agree? Are these new and upcoming features enough to let you say goodbye to the ratings? Or, do you feel we need to come up with a new ratings system? If a new system is desired, how do you make it meaningful and accurate given the problems I’ve brought to light above?
As I said in my first prologue, Home Theater really is your magazine. Let me know what you think.
»
- Login or register to post comments
Comments
Submitted by Pascal on September 18, 2008 - 7:18pm.
I really had given up and thought I had decided not to write again because it seems poitless, but here I am again. I DO like the new look, great job! But 6 of 7 of the products reviewed are TOP PICKS! Might as well give them all 90s scores! I like one of the posts above suggesting reviews with at least a few other components comparing them against eachother. Also, to my amazement, Atlantic Technology speakers, floor model, were not in the top products section but they show up being the Rave Review! How are they not in the Top Products section? Why dont you do a reader survey? See how many readers want ratings and how they would prefer them. Makes sense to me.
Submitted by Kory on September 23, 2008 - 11:53am.
There should be a rating system that focuses on objective points. Value judgement numbers are useless and are generally purely subjective. An executive editor may score an expensive pre-pro very highly on value while an average reader cannot afford 1/4 the price for this component-useless. When talking about inflation, why limit the top scores at all? If last year's model scores a 90 on video processing, this year's model could score 107. Or you could implement individual criteria rankings, perhaps visually expressing them as lines with the newer, better products in front of the older. This would allow people to focus on what is important to them. These are just a few ideas off the top of my head; if the editors thought about it for a while, I feel confident that they could achieve something that curbs misleading information while retaining the useful aspects of a rating system. I also agree that top picks and recommended components are not an ideal substitution.
Submitted by Dan on October 1, 2008 - 10:20am.
Shane, What's that they say about the customer? The customer is always... misinformed? The customer... just needs a little time to get used to it? Mmmm... Maybe, the customer... just needs a little more convincing? No... none of those sound quite right.Haven't your customers spoken? I can appreciate your desire to keep your artistic integrity, but maybe you should just bring the ratings back.
Submitted by John T on October 2, 2008 - 5:35pm.
I could live without the ratings..but in return I'd like "Home Theater" to spend some time reviewing something other than the same 3 things every issues seams to focus on. Receivers, DVD/Blu-Ray players and flat panel TV's. How about things like projector screens, HT seating, HDMI switchers/cables, Sound absorption panels, star ceilings, concessions, universal remotes ETC. The sound and video are of course important to the home theater experience but it's not the only thing that makes it special. You seem to be missing a very large part of the experience that most people will remember long after they forget that my sub can go down to 25hz.
Submitted by Wes on October 6, 2008 - 8:04pm.
Shane,I don't miss the old rating system; but if you feel compelled to restore something, then make it "stars," similar to what you have with Blu-ray ratings that HT reviews. I love the new look of the magazine -- really compliments the great writing, pics, and content. All excellent! Bravo!
Submitted by Dan Carrero on October 7, 2008 - 2:47pm.
I always felt the rating system was wacked.I mean how does a decent inexpensive receiver get the same score as a full blown,top of the line pre-pro and amp?How is it that a sound bar can get an equal rating as a $10k speaker system?The rating system was more for the manufacturers than the readers.It always left me confused but I bet the marketers saw a 91 rating on a good,(not great)component and rubbed their hands in glee.I believe a rating system is necessary... ratings that will inform the readers and leave no doubt that x receiver is a fantastic bargain but in no way is the equal of a Sunfire Grand,a Pioneer 5000 series or a Yamaha Z series receiver.Shane, give us an accurate rating system!The changes you've brought about so far have been on the money.
Submitted by Keith Ottman on October 13, 2008 - 5:45pm.
At first, I was disappointed in your decision to drop your numerical ratings system. It seemed a good way to help quantify a very subjective arena - and sometimes verbage is not the best way to weed out the overall performance quality of a given product. But your solution in ranking the overall favored pick in each catagory satisfies my need for a "ranking system". Almost. It would be VERY interesting to me if, say, a $3,000 tv was able to outperform a $8,000 tv. I know your reviews can't be all things to all people (in every issue!), but I don't believe I would be in the minority to want to hear about about THAT!
Submitted by annai on July 3, 2009 - 11:31pm.
I don't know if this is the right place to post this comment or not. But i wanted to share my views with everyone and everywhere.All of us are somehow or the other suffering from the global recession. We have to think about recoverying this loss.I tried everything and found that real estate is the best option that can beat the global economic melt down.Good luckSingle Property Websites
Site Map / Direct Links| Displays Speakers |  |
HT Systems Sources Electronics | |
Other Gear Software AV Interiors | |
Top Picks
|
|
Community Show Reports | |
Blogs
|
|
Shop Resources Subscriptions |
