Whisky aficionados devote a lot of time and attention to reviews, ratings and awards allotted to whiskies by ‘experts’, both professional and amateur. Designed to separate the good whiskies from the not so good, and the bargains from the rip off’s, these ratings can play a big part in which whiskies are bought, sold, drunk or collected. To new distilleries, they can be critical in establishing a reputation.
These ratings are published in blogs, books, magazines and as competition results. Are they reliable? Are they a good guide as to what you should buy?
Let’s look at an example of numerical ratings (or rankings) and professional reviews (descriptions) of whisky. I will use the ‘Whisky Magazine’ as the source of the data. This well-established UK periodical also has a strong following in the USA and publishes ratings of various whiskies in each edition. Reviewers change over time but presently they are Phoebe Calver and Christopher Coates. Both professionals in the spirit industry with Christopher being Chair of Judges Panel at the prestigious World Whiskies Awards competition.
The Ratings:
These two experts each rated a total of 40 whiskies blind and (presumably) independently in the last two editions of the magazine. The guidelines to the point rating system are provided to the reviewers to try and create some consistency between reviewers. The results on the 40 whiskies show a range from to 7.1 to 9.0. Were they both working with the same interpretation on how to use the scale? Apparently so……
Results Phoebe Christopher
mean 7.93 7.97
median 7.85 8.00
stdev 0.41 0.44
max 9.00 8.70
min 7.20 7.10
Now the distribution of the results are so close between Phoebe and Christopher, it suggests that they may have used some scale shift to bring the two graders into sync with each other. If so, I am OK with it.
More importantly, did Phoebe and Christopher show general agreement on which were the better and lesser whiskies? The good news is they did. What Phoebe liked, Christopher, for the most part also liked. (For the nerds, there was a Pearson’s correlation coefficient of 0.78). Given that, it seems reasonable that we might also like the whiskies that these two experienced tasters ranked high and hence we can use these results, at least as a starting point, in selecting whiskies to check out.
The Pricing:
Was there a relationship between price and ratings? Unfortunately, the pricing ranges were not defined finely enough to reach any real conclusions. However, it is clear that there are some good whiskies at bargain prices and that a high price is not necessarily a guarantee of a high-quality whisky. But we all knew that, right.
The Reviews: Perhaps the most amusing part of all these reviews are the written descriptions of the whiskies tasted. Here are Phoebe’s and Christopher’s descriptions on the taste of the same whisky, that they both rated at 7.9.
Phoebe: Resinous, then smoky peat, almost mechanical. Savory pie crust, hazelnuts, almond oil and cardamom spice
Christopher: Medium-full bodied. Heaps of orchard fruits are joined by subtle iodine and menthol, anise and blueberry. A bittersweet vegetal note of green bell pepper brings up the rear.
Have to laugh, not much agreement here. That is pretty much the case across all the tasting notes. You almost get the feeling the more fanciful the language the better the review will be received?
Truth is that apart from the strongest elements such as peat smoke, brine and sherry for example, everybody is on their own as far as the specifics of what they smell or taste and/or how they describe it based on their own taste buds and memories.
While these results are based on a limited set of ratings in a single magazine, I do feel that they represent what we can expect from whisky ratings and descriptions in general. Hope this was useful.
Comments