I get very irritated when I see the mainstream media uncritically reporting some agenda pusher’s assertion with the words “Research shows that ...”. I’m not the only one. Frank Furedi, writing in the UK’s Times Higher Education Supplement, sums it up neatly:
Even in otherwise respectable research, too many studies ask about future intentions. On very short term intentions, such as who you will vote for next week, you might get a useful answer. I say might, because research shows (geddit?) that people sometimes lie, some mischievously, some deviously, some to hide indecision, some to give what they think is a socially acceptable answer. And many longer term intention surveys have no more value than my annual New Year resolutions. As economics writer Tim Harford often says, the only truly compelling evidence is based on independent verifiable empirical data of what people have actually done, not what they say they might do.
Decision makers, marketers, designers, politicians, and journalists: turn on your BS detector whenever you hear the words “research shows that ...”
First posted November 26th, 2008
The status of research is not only exploited to prove the obvious, but also to validate the researcher’s political beliefs, lifestyle and prejudice.I remember my university market research class being given two contrasting questionnaires which led you down a logic path to the last question on capital punishment, and a highly skewed result. So the same sample of 20 students gave completely opposite results on two questionnaires asking essentially the same question. It was a clever lesson that I’ve never forgotten. Add to skewed questions the selection of small or biased samples and you can get pretty much any result you want. And some so-called researchers are not above losing some of their sample return sheets to suit their desired outcomes.
Even in otherwise respectable research, too many studies ask about future intentions. On very short term intentions, such as who you will vote for next week, you might get a useful answer. I say might, because research shows (geddit?) that people sometimes lie, some mischievously, some deviously, some to hide indecision, some to give what they think is a socially acceptable answer. And many longer term intention surveys have no more value than my annual New Year resolutions. As economics writer Tim Harford often says, the only truly compelling evidence is based on independent verifiable empirical data of what people have actually done, not what they say they might do.
Decision makers, marketers, designers, politicians, and journalists: turn on your BS detector whenever you hear the words “research shows that ...”
First posted November 26th, 2008