RSS

The Great Coffee Taste Test

18 Apr

… was absolutely exhausting. Which is my excuse for only getting round to writing about it now.

The Premise?

Get two coffees, one “good quality”, one less so. Get as many people as possible to try them. Find out if people preferred the high quality or not.

The Venue?

Last week, taking over what currently passes for the ‘Cafe’ on C-floor of the Geography Department, University of Sheffield.

The  partipants?

Simon from Pollards Coffee Company and I, manning the espresso machine, and over 100 willing participants from the (mainly geography) students and staff. Not exactly coffee gourmets, but certainly plenty of addicts (I quote: “I’d be on drip if I could plug it in”)

If only it were that simple!!

This experiment was not a full-proof design anyway, but a few things ‘distracted’ from the scientific precision, shall we say.

cafe

The New C floor "cafe". Ahem.

The venue served us pretty well once Simon had lugged the Fraccino espresso machine up in the lift and plugged everything in. There (as yet) is no furniture in the cafe, so there was very little for our participants to sit on, let alone write out our reply slips on whilst holding two cups of coffee…. but not the end of the world. However, although I had arranged to hold the taste test on Tuesday and Wednesday several weeks ago, it was left until Monday until someone informed us that there was a department open day on the Wednesday, who also wanted to use the space. Great! I thought. More people! Sadly, no, they all appeared at 4pm, just as we were packing up. A few rather lost looking parents-of-prospective-students appeared though, looking for the department cafe… and only found us. We were of course, happy to oblige but methinks for the sake of the geography department in general, it would have been wise to build and furnish the cafe first, and then put the signs up to it…. And I can suggest someone very willing to take over the space and run it as a cafe properly, by the way... hint hint.

But ANYWAY.

In terms of research design, we had tried to keep the experiment as simple as possible. We gave our participants the sort of coffees they asked for,  cappuccinos, americanos, etc. because we wanted to test the average coffee drinker, drinking what they normal would in a coffee shop. We hoped this would give a more accurate picture of how people taste coffee – the differences should still be apparent even with added milk and sugar etc. If they are not, this is still significant because it implies that it would not matter what sort of coffee goes into a cappuccino, if people are just drinking them for the milky flavours. Also, we were likely to get far more participants this way, than if we forced them all to drink espressos. However, it does leave it open to flaws in the consistancy of our drink construction though – we may well have added more milk or foamed one better than the other or screwed up the espresso at times etc etc etc. Not a highly accurate test in this sense!

setup

Simon setting up.

The biggest test for us though, was how to define ‘quality’ in the first place. A very large proportion of my whole thesis revolves around this issue! Simon helped a great deal here by basically using his knowledge of roasting and then absolving us of responsibility for defining quality. He chose three coffees for an espresso blend, that had all rated very highly on the Speciality Coffee Association of American’s cupping scale. Then, he got three more lots of beans from farms very close by to these first three, but had not been rated by the SCAA. Another factor was the price. The SCAA-rated blend would have cost £12 a kilo, the other, £3 a kilo. Was the ‘higher quality’ one really four times better? Pollards people roasted both sets of beans identically and on the same day, so there was as little variation in the roast as possible. All was set!

We both did this test blind – the beans arrived in bags marked A and B, and we had two grinders, also marked similarly. We then marked all the cups before giving them to our guinea-pigs.  Neither of us knew which blend was which as we made the coffees (although Simon worked it out pretty quickly!), so we couldn’t unconsciously make one coffee better than the other and so on. This is a further issue regarding quality. Quality is not just an inherent characteristic of the green bean – it also depends on the roast and the skills of the barista (amongst other things). We could just about controi these variables, so hopefully all we were testing were the difference in quality of the beans themselves. Complicated, though!

barista

Moi as experimental barista and bemused guinea-pig

guineapigs

More caffeinated guinea pigs.

The Results?

Pretty evenly split!!!

This is a fascinating result and is already causing controversy at Pollards and with Simon’s suppliers. I am not going to put the exact figures on here – I am still waiting for some responses anyway, but mainly because we want to put together a proper paper about this for academic and hopefully some trade journals. For simplicity’s sake, there was no significant difference between the number of people preferring Coffee A to Coffee B. There were a very few people who couldn’t tell the difference at all, but not nearly as many of these as I had thought there might be. If anything, there was a very very tiny skew towards a preference for A, but not enough to make definite assertations.

B was the SCAA rated, high quality blend, worth four times the price of A.

I’ll let Simon explain the origins of Coffee B:

“The El Salvador is La Avila Estate, which came 5th in the 2009 Cup of Excellence awards with a score of 89.43. It is a fantastic cup on its own in a filter of cafetiere but is it worth 4 times more than the standard SHG from the farm next door?
The Brazil was a Daterra Special reserve coming in with a score of 85.5 last year. It is the Catuai varietal and again is fantastic on its own in a filter. But again is it worth the extra?
The monsoon was specially prepared for me by a neighbours estate, Ratnagiri, in Chikmagalur. It costs just the same , but, unlike me he is very good at producing exceptional quality green coffee. This does not carrry an SCAA rating but does have the indian coffee boards gold medal for last year…. “

Coffee A came from neighbouring farms in El Salvador, Brazil and India, but did not have these accolades (a fact which could be used to advocate the idea of ‘terroir coffee’ and geographical indicators… but that is a whole other chapter!).

And roughly half our participants (including me, to my surprise) preferred the cheaper, unrated Coffee A.

What does this actually mean?

Because it was such an even split, I can’t conclude that people actually prefer cheaper, supposedly lower quality coffee, because an equal number did prefer the high quality one. Judging by the comments on the day though, no-one really thought that B was worth four times as much as A.  The overriding conclusion, however, is that Quality and Preference are NOT the same thing. In short, and within reasonable parameters, (ie: not mouldy, not stale, not burnt) the quality of the beans is not a real factor in coffee preference. Take away the price (a major factor for consideration amongst students!), the marketing, the certifications and accolades on the beans, and the comfy pulls of coffee shops and their fashionable social spaces, and really, any coffee seems to be useable – and drinkable.

Where does that leave the idea of “quality” amongst coffee producers and retailers then? For the producers, all the highly skilled techniques employed to enhance the quality of the green beans are not necessarily demanded by the average consumer. However, since producers do not deal directly with consumers, if the buyers and importers are still willing to pay the farmers more for what they consider to be high quality, then it is still in the interest of the farmers to keep the quality as high as possible.

For retailers, however, this does appear to give businesses the perfect excuse to buy in cheaper, low quality coffees, and still sell them to consumers for the same price. Why would any coffee shop want to pay £12 a kilo for coffee, when half their customers are quite happy to drink stuff that costs just £3 per kilo? Quality in this case is very much constructed by the retailers: packaging, exoticism of the country of origin, certifications and labels, cafe branding, presentation of the drink (Simon is going to do another test looking at people’s perception of quality between coffees served with latte art and those without) – and also, price. If coffee costs more, the general assumption is that is must be better quality. Are we naive in this view? It is self-perpetuating – it is thought to be high quality because it costs more, and it costs more because people think it is high quality.

I was reprimanded last week for calling the SCAA’s scale, and various certifications ‘meaningless’ in my thesis chapter. As my supervisor rightly points out, the certifications and quality assurances are not in themselves meaningless, but the meanings they actually represent are not necessarily the most obvious – or what the customer believes them to be. In this case, the SCAA scale is not actually meaningless, but instead of the ratings meaning that the El Salvadorean beans are in the top 15% of all the beans tested, it means that a handful of the self-appointed experts at the SCAA liked the taste of them enough to give the coffee a high score – and in doing so, also gave retailers and importers the leave to charge four times as much for those beans than those from a neighbouring farm. However, this is NOT a quality rating; the points simply show an SCAA taste preference. If, like half of our participants, you happen to agree with the SCAA, then that coffee will be high quality to you. If you prefer the other blend, then so be it; personal preference is, after all, personal, and as our test seemed to show, “quality” is entirely subjective as well!

Advertisements
 
2 Comments

Posted by on April 18, 2010 in Uncategorized

 

Tags: , , , , , , ,

2 responses to “The Great Coffee Taste Test

  1. Jeff Jones

    May 5, 2010 at 4:05 pm

    Annabel;
    Thanks for trying this out! It looks fun and interesting.

    I agree with you, that it is very difficult to differentiate coffee qualities, especially distinguishing between good and very good coffees. It seems intriguing to see if people in general can distinguish between “one very low quality coffee, and one very high quality”.

    However,I do wonder if you really did carry out your intent of comparing highly rated coffee with bad coffee. The way you selected the ‘bad’ coffee did not really guarantee it was bad, just not included in the taste test; and as you are aware, there are lots of reasons why a good coffee does not make it into a taste test. And the fact that ‘bad’ coffees came from the same areas as the good coffees probably meant they were the same varieties, same production conditions, same processing conditions, etc. So yes, terroir, but also lots of other similar factors.

    It seems to me that the more interesting test is what you mention in another point; good coffee vs. schlock coffee made with mixtures of robusta, and/or lower grade beans. That would be a very interesting comparison.

    Jeff Jones

     
    • drcoffee

      May 6, 2010 at 12:53 pm

      I agree that could be another, interesting test, but what we were focussing on in here was more the definition of quality in the first place – that set by the SCAA. Given, as you say, the two coffees were from the same region, (trying to minimise the number of variables in the test as well), and both turned out to be equally drinkable, WHY does the SCAA rate one so highly, and not give any rating at all to the others? That rating alone makes one blend four times the price of the other, so does that mean the ungraded stuff is actually four times worse? from this test it didn’t seem to….

      I would love to test some high street schlock coffee from one of the big chains too, but then to make meaningful comparisons, you’d have to find a high rated blend with similar origins which is extremely diffcult especially when the chain coffee companies are so guarded about what goes in to their blends.

       

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

 
%d bloggers like this: