… was absolutely exhausting. Which is my excuse for only getting round to writing about it now.
Get two coffees, one “good quality”, one less so. Get as many people as possible to try them. Find out if people preferred the high quality or not.
Last week, taking over what currently passes for the ‘Cafe’ on C-floor of the Geography Department, University of Sheffield.
Simon from Pollards Coffee Company and I, manning the espresso machine, and over 100 willing participants from the (mainly geography) students and staff. Not exactly coffee gourmets, but certainly plenty of addicts (I quote: “I’d be on drip if I could plug it in”)
If only it were that simple!!
This experiment was not a full-proof design anyway, but a few things ‘distracted’ from the scientific precision, shall we say.
The venue served us pretty well once Simon had lugged the Fraccino espresso machine up in the lift and plugged everything in. There (as yet) is no furniture in the cafe, so there was very little for our participants to sit on, let alone write out our reply slips on whilst holding two cups of coffee…. but not the end of the world. However, although I had arranged to hold the taste test on Tuesday and Wednesday several weeks ago, it was left until Monday until someone informed us that there was a department open day on the Wednesday, who also wanted to use the space. Great! I thought. More people! Sadly, no, they all appeared at 4pm, just as we were packing up. A few rather lost looking parents-of-prospective-students appeared though, looking for the department cafe… and only found us. We were of course, happy to oblige but methinks for the sake of the geography department in general, it would have been wise to build and furnish the cafe first, and then put the signs up to it…. And I can suggest someone very willing to take over the space and run it as a cafe properly, by the way... hint hint.
In terms of research design, we had tried to keep the experiment as simple as possible. We gave our participants the sort of coffees they asked for, cappuccinos, americanos, etc. because we wanted to test the average coffee drinker, drinking what they normal would in a coffee shop. We hoped this would give a more accurate picture of how people taste coffee – the differences should still be apparent even with added milk and sugar etc. If they are not, this is still significant because it implies that it would not matter what sort of coffee goes into a cappuccino, if people are just drinking them for the milky flavours. Also, we were likely to get far more participants this way, than if we forced them all to drink espressos. However, it does leave it open to flaws in the consistancy of our drink construction though – we may well have added more milk or foamed one better than the other or screwed up the espresso at times etc etc etc. Not a highly accurate test in this sense!
The biggest test for us though, was how to define ‘quality’ in the first place. A very large proportion of my whole thesis revolves around this issue! Simon helped a great deal here by basically using his knowledge of roasting and then absolving us of responsibility for defining quality. He chose three coffees for an espresso blend, that had all rated very highly on the Speciality Coffee Association of American’s cupping scale. Then, he got three more lots of beans from farms very close by to these first three, but had not been rated by the SCAA. Another factor was the price. The SCAA-rated blend would have cost £12 a kilo, the other, £3 a kilo. Was the ‘higher quality’ one really four times better? Pollards people roasted both sets of beans identically and on the same day, so there was as little variation in the roast as possible. All was set!
We both did this test blind – the beans arrived in bags marked A and B, and we had two grinders, also marked similarly. We then marked all the cups before giving them to our guinea-pigs. Neither of us knew which blend was which as we made the coffees (although Simon worked it out pretty quickly!), so we couldn’t unconsciously make one coffee better than the other and so on. This is a further issue regarding quality. Quality is not just an inherent characteristic of the green bean – it also depends on the roast and the skills of the barista (amongst other things). We could just about controi these variables, so hopefully all we were testing were the difference in quality of the beans themselves. Complicated, though!
Pretty evenly split!!!
This is a fascinating result and is already causing controversy at Pollards and with Simon’s suppliers. I am not going to put the exact figures on here – I am still waiting for some responses anyway, but mainly because we want to put together a proper paper about this for academic and hopefully some trade journals. For simplicity’s sake, there was no significant difference between the number of people preferring Coffee A to Coffee B. There were a very few people who couldn’t tell the difference at all, but not nearly as many of these as I had thought there might be. If anything, there was a very very tiny skew towards a preference for A, but not enough to make definite assertations.
B was the SCAA rated, high quality blend, worth four times the price of A.
I’ll let Simon explain the origins of Coffee B:
“The El Salvador is La Avila Estate, which came 5th in the 2009 Cup of Excellence awards with a score of 89.43. It is a fantastic cup on its own in a filter of cafetiere but is it worth 4 times more than the standard SHG from the farm next door?
The Brazil was a Daterra Special reserve coming in with a score of 85.5 last year. It is the Catuai varietal and again is fantastic on its own in a filter. But again is it worth the extra?
The monsoon was specially prepared for me by a neighbours estate, Ratnagiri, in Chikmagalur. It costs just the same , but, unlike me he is very good at producing exceptional quality green coffee. This does not carrry an SCAA rating but does have the indian coffee boards gold medal for last year…. “
Coffee A came from neighbouring farms in El Salvador, Brazil and India, but did not have these accolades (a fact which could be used to advocate the idea of ‘terroir coffee’ and geographical indicators… but that is a whole other chapter!).
And roughly half our participants (including me, to my surprise) preferred the cheaper, unrated Coffee A.
What does this actually mean?
Because it was such an even split, I can’t conclude that people actually prefer cheaper, supposedly lower quality coffee, because an equal number did prefer the high quality one. Judging by the comments on the day though, no-one really thought that B was worth four times as much as A. The overriding conclusion, however, is that Quality and Preference are NOT the same thing. In short, and within reasonable parameters, (ie: not mouldy, not stale, not burnt) the quality of the beans is not a real factor in coffee preference. Take away the price (a major factor for consideration amongst students!), the marketing, the certifications and accolades on the beans, and the comfy pulls of coffee shops and their fashionable social spaces, and really, any coffee seems to be useable – and drinkable.
Where does that leave the idea of “quality” amongst coffee producers and retailers then? For the producers, all the highly skilled techniques employed to enhance the quality of the green beans are not necessarily demanded by the average consumer. However, since producers do not deal directly with consumers, if the buyers and importers are still willing to pay the farmers more for what they consider to be high quality, then it is still in the interest of the farmers to keep the quality as high as possible.
For retailers, however, this does appear to give businesses the perfect excuse to buy in cheaper, low quality coffees, and still sell them to consumers for the same price. Why would any coffee shop want to pay £12 a kilo for coffee, when half their customers are quite happy to drink stuff that costs just £3 per kilo? Quality in this case is very much constructed by the retailers: packaging, exoticism of the country of origin, certifications and labels, cafe branding, presentation of the drink (Simon is going to do another test looking at people’s perception of quality between coffees served with latte art and those without) – and also, price. If coffee costs more, the general assumption is that is must be better quality. Are we naive in this view? It is self-perpetuating – it is thought to be high quality because it costs more, and it costs more because people think it is high quality.
I was reprimanded last week for calling the SCAA’s scale, and various certifications ‘meaningless’ in my thesis chapter. As my supervisor rightly points out, the certifications and quality assurances are not in themselves meaningless, but the meanings they actually represent are not necessarily the most obvious – or what the customer believes them to be. In this case, the SCAA scale is not actually meaningless, but instead of the ratings meaning that the El Salvadorean beans are in the top 15% of all the beans tested, it means that a handful of the self-appointed experts at the SCAA liked the taste of them enough to give the coffee a high score – and in doing so, also gave retailers and importers the leave to charge four times as much for those beans than those from a neighbouring farm. However, this is NOT a quality rating; the points simply show an SCAA taste preference. If, like half of our participants, you happen to agree with the SCAA, then that coffee will be high quality to you. If you prefer the other blend, then so be it; personal preference is, after all, personal, and as our test seemed to show, “quality” is entirely subjective as well!