Mad ScienceDeconstructing Bunk Reporting in 5 Easy Steps

British scientists have uncovered the truth behind one of modern culture's greatest mysteries: why little girls play with pink toys. Is it because toy companies flood whole store aisles with the color? Or because well-meaning relatives shower girl babies with pink blankets and clothing? Nope. According to the men in lab coats, it's purely biological.

Apparently, women are hardwired to like pink because our cavewoman foremothers spent their days gathering red leaves and berries amongst the trees while their husbands were out hunting. Later, women needed to notice red-faced babies and blushing boyfriends. And why do men like blue? Because it's the color of the sky.

This evolutionary just-so story takes up three pages of a 2007 issue of Current Biology. To back up the assertion that pink is a universal girly preference worth examining, the authors refer to a 1985 study finding that little girls use more pink and red crayons in their drawings than little boys do.

Dig further, however, and the story completely falls apart. British women do prefer pink, but the author's claim of a “robust, cross-cultural sex difference” turns out to be neither. The scientists compared British natives with Chinese immigrants to Britain, and glossed over the differences. For example: The girliest color in the British results, a purplish-pink, was in fact the Chinese men's favorite.

Nowhere do scientific findings get more mangled than when they're about the differences between men and women. According to the science pages, women aren't just biologically hardwired to prefer pink to blue. We're also predisposed to backstab one another in the workplace, cry in the boardroom, and have both lower iqs and less of a sense of humor than men.

Some misleading stories come from bad science, where the study authors' conclusions aren't supported by their own data. Others are well-conducted studies whose conclusions mutate upon contact with the mainstream media. Newspapers and websites are prone to playing fast and loose with their reports on studies, often neglecting to reveal salient facts about a study's sample group or methodology.

The fact is that science articles aren't designed to be read by non-scientists. College and grad students in the sciences are trained in how to do it: They review papers and discuss them in journal clubs; learn how to question methodologies (Is that sample really big enough? Was that the right test to use?); and learn how to be critical of authors' interpretations (Do the results really mean what they say they mean?). Students also know to look at context for each study, looking up previous papers on the subject, reviewing the authors' previous work, and searching out any evidence of bias that might color a study's findings.

Journalists looking for a quick story, however, do little such research. And in an age where news sites, wire services, and blogs pick up stories with lightning-fast speed, bad research gets around. When London's Sunday Times reported on a 2007 study claiming that men get dumber in the presence of blond women, the paper got the name of the journal wrong, citing the Journal of Experimental Psychology rather than the Journal of Experimental Social Psychology. Nearly every subsequent news article repeated the error because they were content to simply reword the Times' version of the story rather than finding and discussing the study itself.

The Times reported that blond-exposed subjects “mimic the unconscious stereotype of the dumb blonde.” But that's not exactly what the study tested. Rather, subjects—most of them female—fared slightly worse on online trivia quizzes after rating hair color (is she a blond, brunet, or redhead?) on pictures of beauty queens. You could just as easily say that beauty queens make people dumb, or photos of dazzling smiles make people dumb. It seems this study made the news mostly because it could be illustrated with photos of Marilyn Monroe and filled out with dopey quotes from blond models and actresses, as well as blond jokes from the Times itself.

Ben Goldacre, who writes the “Bad Science” column for the UK's Guardian, speculates that science stories come in three varieties: the wacky story, the breakthrough story, and the scare story. Most widely reported studies on gender seem to fall into the wacky category—the supposed innate preference for pink is one of them—and their media strength is that they tend to support existing stereotypes of women, reassuring readers that social stereotypes do, in fact, reflect reality.

We can't put all the blame on mainstream media, of course. Scientists are part of the same culture as the rest of us, and they too have biases that shape their hypotheses and interpretations. The scientific community can also be as fad-driven as popular culture, creating a climate in which many researchers simultaneously geek out over one specific theory while competing ideas get lost or abandoned. So let's learn how to read between the lines of these dubious articles. Next time you see an article reporting that women are happiest when they're picking up their man's dirty socks, try asking these questions:

1 Do the Conclusions Fit a Little Too Well With Cultural Stereotypes?

Science has the capacity to surprise and amaze us, but sometimes it's more satisfying when you can jump up and say, “Yes! I knew it all along!” Which is why articles touting the awesomeness of traditional gender roles are an evergreen subject in the science pages.

A 2007 study from the American Society for Cancer Research journal Cancer Epidemiology Biomarkers and Prevention titled “Physical activity and breast-cancer risk” found fame in such headlines as the BBC's “Housework cuts breast-cancer risk.” That's not to mention the 2006 study on housework and cancer in Canadian women, the 2005 study on housework and cancer in Chinese women, or the 2004 study…you get the idea. [See “Home Is Where the Cardio Is,” Bitch no. 27]

The reality? Being physically active seems to help prevent cancer, and the researchers behind the recent studies have been counting housework as physical activity. Housework, sports, and active jobs all had significant effects in reducing cancer risk, and the authors think the key may be frequent, low-impact exercise.

An author on several of these studies, Christine Friedenreich, told the Calgary Herald that in past studies, researchers counted jobs like construction work as physical activity, but not housework—and it turns out that domestic tasks are, duh, hard work.

This means that many women are getting more exercise than they (or their doctors) had realized. That should be good news for them—but instead, the message imparted by the news reports is, “Get back into the kitchen! That's all the exercise you need!”

It's worth noting that one of the study's sponsors, Cancer Research UK, answered questions about the 2006 study on its website, pointing out that for many of the older women in the study group, housework was their primary form of exercise. The organization went on to address charges of sexism directly, making sure to mention a related 2006 study that found housework cuts the risk of bowel cancer for both men and women, concluding, “There's absolutely no excuse for men to dodge the dusting!”

2 Does the Study Agree With the Headline?

Behind every junk-science headline is a scientific journal article. Sometimes the university or organization that was home to the study sends out a press release to mainstream outlets, hoping for attention; other times, journalists simply scan the abstracts of academic journals for newsworthy fodder. Chances are a story will make the papers if it's got some kind of hook—weird (like the idea that housework has curative properties), controversial (like claims that men are smarter than women), can be illustrated with bikini babes (like the dumb-blond study), etc. Especially for online news outlets, these hooks are valuable because they make good linkbait: the kind of thing they hope you'll forward to friends or post on your blog.

The London Times probably hadn't read the full study titled “Prejudice against women in male-congenial environments: Perceptions of gender-role congruity in leadership” when they summarized it under the headline “Office Queen Bees Hold Back Women's Careers” in a 2006 article. The paper's charge—that “women bosses are significantly more likely than men to discriminate against female employees”—may indeed have surprised the study's authors.

The actual study went something like this: Participants weren't put in a boss's role, but an observer's. They read a purposely vague description of a manager who was being considered for promotion and were asked to imagine how qualified the candidate was, and whether he or she was likely to succeed. The study made a number of interesting points that the Times could easily have reported on—for instance, that female managers were judged to have both very masculine and very feminine traits, possibly in an attempt to reconcile their gender with the traditionally masculine-associated role of leadership.

Both the male and female participants were optimistic about the male manager's success, but not about the woman's (except when she worked in a female-dominated industry). Sounds pretty realistic, right? The researchers thought so too. They write, “Participants' predictions about the [female] candidate's future salary…mirrored the fact that women earn less money in the same position [than] men do in real life.” So where are those “queen bees” that the Times so gleefully name-checked? Exactly.

It's not difficult to track down the science behind the story. Look for the names of the researchers, the journal their work appeared in, and (if you're lucky) the title of the article. Type whatever info you've got into Google Scholar (www.scholar.google.com), and soon you'll be looking at an abstract for the paper. Scientific journals are usually locked behind paywalls, unfortunately, so you may need to call upon a pal at a university for access to the entire study.

3 Can You Spot the Double Standard?

Whether it's lions fathering all the cubs in their pride, or human males getting a pass for cheating on their girlfriends, males sleeping around rarely make the news—it's the natural order, after all—unless the article is happily touting the genetic advantages a male gets from spreading his dna around.

But when female cheetahs were found to do the same by a Zoological Society of London study, the study's words about “promiscuous” felines were quickly outnumbered in Google's index by the phrase, “cheetahs are sluts!”

Study author Dada Gottelli was quoted thus: “Mating with more than one male poses a serious threat to females, increasing the risk of exposure to parasites and diseases. Females also have to travel over large distances to find new mates, making them more vulnerable to predation.” Sounds like a cheetah-specific version of certain sex-ed curricula: Don't sleep around, girls, or you'll catch lots of diseases and the male cheetahs won't respect you in the morning. Male cheetahs, however, aren't “promiscuous”—they're creating a healthier gene pool.

Not too surprising, then, that most of the coverage glossed over the evolutionary benefit of promiscuity for both male and female cheetahs: Multiple cubs by multiple cub daddies increases the likelihood of genetic diversity—a definite positive for a threatened species. Furthermore, the study noted that the rates of infanticide in cheetahs are much lower than in other big-cat populations, likely because male competitors don't know which offspring might be theirs. But why let the facts slow down a good headline?

In a human example of a double-standard story, women were found to be “worse oglers” than men, according to the Sydney Morning Herald summary of a study published in the journal Hormones and Behavior. (The Herald inexplicably illustrated its story with headshots of Sharon Stone and Mr. Bean). What does that even mean, you ask? When researchers showed “sexual stimuli” (read: Internet porn) to heterosexual men and women, they expected women to look more at faces and men to look more at genitals. The newspaper reported that, in fact, “almost the reverse was true.”

Actually, the study says that men looked at women's faces more than women did, and men and women looked with equal frequency at the pictured genitals; women who weren't on oral contraceptives looked slightly more. So where did that headline come from?

The study authors didn't originate the “worse oglers” language; they even warn in the study that they can't say why subjects' gazes lingered where they did, or whether they were turned on as they looked. So it's not fair to say that the study was about “ogling,” a word that suggests that looking is lustful and perhaps inappropriate.

To say that women are “worse” at ogling, we have to believe, first, that ogling is bad, and second, that men do it at some normal, baseline level that women are exceeding. The judgmental language makes it sound like women in the study were indulging a bad habit. Right there in the headline is the double standard: If men ogle, it's normal, but when women do it, they're “worse.”

4 Is There Another Conclusion That Would be Just as Valid?

Sometimes a news story is an accurate representation of the scientists' conclusions, but the scientists' conclusions don't follow their results. Take this 2005 BBC headline: “Men Cleverer Than Women.” The study, at the time of the headline yet to be published in the British Journal of Psychology, claims that as iq scores rise, the gender gap widens, with 5.5 men for every woman scoring at the “genius” level of 155 or higher on iq tests. That's all the evidence the authors (one of whom, Richard Lynn, has published similar studies on racial differences in iq) give to support their claim.

But there is another, equally powerful explanation that's been considered for years before this study came along: iq tests—which don't measure intelligence directly, but try to approximate it—have a wealth of gender, racial, and cultural biases.

In a 2000 survey of sex differences in intelligence called “The Smarter Sex: A Critical Review of Sex Differences in Intelligence,” in the Educational Psychology Review, Diane Halpern and Mary LaMay write that Lynn's approach “rests on the belief that the test of intelligence is really measuring what psychologists mean by intelligence, and that it is doing so in a way that will yield a fair assessment for males and females—two assumptions that may not be justified.”

Statistically, men do outperform women on certain types of questions, but the reverse is also true; test designers use this fact to calibrate iq tests, balancing male-biased with female-biased questions so that men and women average the same scores on the same test. Addressing Lynn's research directly, Halpern and LaMay say, “Using data from tests that are designed to yield no sex differences to argue for a difference is psychometric nonsense.” Either the tests were miscalibrated (and thus biased) or Lynn's results are a fluke: Probably the latter, since other studies (like a 1995 study on a population similar to Lynn's, done by scientists at the Flinders University of South Australia and published in the British Journal of Clinical Psychology) found no difference between males and females.

So why even report the latest study from this obviously biased researcher? Perhaps it's reassuring to believe that sexism isn't sexism, it's science; that the status quo reflects some kind of natural order; and that anyone who claims otherwise is a whiner. Or perhaps Lynn's studies make the news because he's sort of a one-man show of bunk science—after all, this is the same guy who claims that African-Americans have higher iqs than Africans because they have Caucasian genes that make them smarter.

Then there are the stories that point the finger at feminism for a variety of historical incidents and ills. The 2007 Boston Globe story titled “Stone Age Feminism? Females joining hunt may explain Neanderthals' end” is one of these.

The supporting study, authored by archaeologists Steven Kuhn and Mary Stiner, turned on the hypothesis that Neanderthal women participated in hunting alongside Neanderthal men. The dangers of hunting—among them, getting stomped and gored by various beasts—along with the fact that many cavegals' lives were cut short before they could produce baby Neanderthals, meant that the breeding population dwindled and the species died out.

But the evidence for this “stone age feminism” wasn't evidence at all. The hypothesis that the women hunted alongside the men was developed because the study's authors found no clues suggesting that Neanderthal women were, well, homemakers—no grinding stones and bone needles that would signal a traditional division of labor in the species. So is it possible that male and female Neanderthals hunted together successfully, and the species dwindled for some other, totally unrelated reason? And, for that matter, why not hypothesize that coed hunting parties actually contributed to the Neanderthals' longevity? More than 100,000 years of existence is nothing to sneeze at, after all. Why jump to the conclusion that feminism ruins everything? Ah, yes: because it's a story that will sell papers.

5 Is the Study Even Science?

In his “Bad Science” column, Goldacre reminds us that so-called studies may not have studied anything at all. A hair-removal cream company once asked Goldacre to come up with a formula calculating which celebrity had the sexiest walk. “We know what results we want to achieve,” they told him, naming celebrities with shapely legs whose high-ranking walks could move units of their product.

When Goldacre refused, another scientist supplied the company with a formula, thinking it would be used as a joke. The company's press release became an article in the Telegraph, crediting a nonexistent “team of Cambridge mathematicians” and with no mention of the so-called study's actual source.

Lesson learned: If you can't find the source article, it may not actually exist.

In another example of non-science, BBC News studiously reported in late 2007 that humor “comes from testosterone.” The article? Based on a British Medical Journal study recording the casual responses of passers-by to a unicyclist. The article notes that little boys had more “aggressive” responses to the unicyclist (trying to knock him over) and young men made the most jokes—typically an unimaginative variant of “Lost your wheel?”—with elderly men's jokes being less hostile (“Does it crush your bollocks, mate?”). The article also featured a graphic showing the ebb of testosterone in men over time; since young men have the most testosterone and made the greatest number of jokes, the author concludes, testosterone must be the source of humor.

If all you read was the BBC piece, you might think that there's a clutch of professors somewhere in England taking this theory seriously. In fact, the deliberately hilarious study was published in the BMJ's Christmas issue, famous for its joke articles. (A study from the previous year was titled “Surgeons are taller and more handsome than physicians” and used a photo of George Clooney as a control.)

The fact that the BBC didn't pick up on the joke speaks volumes about the mainstream media's unceasing appetite for gendered potshots. How many of us would really be surprised to see a “legitimate” report linking testosterone and humor? Would it look anything like the 2005 report from Stirling and St. Andrews universities in the UK that claims testosterone causes women to be “career-driven” like men? The humbling take-home message from these studies is that traditionally masculine traits still belong to men—even when women share them.

Although there is an element of humor in how wrong the news media can get science, the trend isn't a harmless one. While plenty of smart people question biased headlines of all stripes, casual readers—particularly young ones—are likely to skim the stories and tuck them away in the pocket of their brains where stereotypes are kept. It's bad enough when we see images of women as passive, petty, dumb, or slutty in fiction or advertisements, but stereotypes that come with the lofty stamp of science have the air of being, well, factual.

After all, if women are biologically wired to be weak or catty or dumb or humorless, then there's nothing wrong with writing consistently airheaded female movie and tv characters, dismissing women in positions of power as “bitches” or “too emotional,” or claiming that institutional sexism doesn't exist, women just aren't smart enough to be ceos, grand masters, or surgeons. These studies reassure people that media images reflect reality, that society reflects biology, and that nothing can or should be changed.

Perhaps we should just take solace in one final study, released by the American Psychological Association in 2005 but picked up by very few mainstream sources. The title? “Men and women found more similar than portrayed in popular media.”