Journalist reveals that "chocolate makes you thin" report was fake, designed to test what science media will reprint without checking sources

This spring, the journal International Archives of Medicine published a delicious new study: According to researchers at Germany’s Institute of Diet and Health, people who ate dark chocolate while dieting lost more weight.

The media coverage was instantaneous and jubilant:

“Scientists say eating chocolate can help you lose weight,” read a headline in the Irish Examiner.

“Excellent News: Chocolate Can Help You Lose Weight!” Huffington Post India boasted.

“Dieting? Don’t forget the chocolate,” announced Modern Healthcare.

It was unbelievable news. And reporters shouldn’t have believed it.

It turns out that the Institute of Diet and Health is just a Web site with no institute attached. Johannes Bohannon, health researcher and lead author of the study, is really John Bohannon, a science journalist. And the study, while based on real results of an actual clinical trial, wasn’t aimed at testing the health benefits of chocolate. It was aimed at testing health reporters, to see if they could distinguish a bad science story from a good one.

In many cases, they couldn’t.

Bohannon, who revealed the stunt in an essay for the io9 Web site on Wednesday, was part of a team of gonzo journalists and one doctor who,  in Bohannon’s words, wanted to “demonstrate just how easy it is to turn bad science into the big headlines behind diet fads.” 

“I know people who have gone on diet fads and it has done them no good,” Bohannon said in a phone interview with The Washington Post. “It’s all cloaked in the mantle of science and it’s really troubling. I was ready for taking on the diet industry … for showing how they treat it like lifestyle material rather than real science.”

Bohannon had done similar work before — in 2013 he submitted a fake research paper to more than 300 open-access journals as part of a sting operation for the journal Science. But the idea for this operation came fromPeter Onneken and Diana Lobl, two German reporters working on a documentary about “junk science” in the diet industry. They had recruited a doctor, Gunter Frank, to run a clinical trial, and 15 people to participate in the study. Fifteen people is hardly a reliable sample size, but that was the point.

According to Bohannon, the participants were divided into three groups: one control group, one to follow a low-carb diet and one to follow the same diet while eating 1.5 ounces of dark chocolate daily. They were paid for their time and informed that the study was for a documentary about dieting, and all participants underwent blood tests to ensure no one had diabetes or other illnesses that might endanger them. They dieted, weighed themselves and filled out questionnaires, and when the 21-day study was up, Onneken and a financial analyst friend examined the results to see if there was anything there they could turn into a news story.

The study included 18 different measurements — sleep quality, cholesterol levels, weight and cholesterol, among others — and the “researchers” needed just one of them to be affected by the chocolate dose. It turned out that members in the chocolate group lost 0.1 percent more of their body fat than those on diets alone, so that’s the finding they boasted to reporters.

This, Bohannon says, is problem No. 1 with much of health science. Studies like his are called “underpowered,” meaning that they aren’t designed to distinguish between a real effect and pure luck. A study with thousands of participants being measured for just a few effects is “powerful.” But one like Bohannon’s, with just five people per group being measured according to any of 18 different variables? Any number of factors unrelated to the study could cause one of the variables to fluctuate, allowing researchers to irresponsibly — but not untruthfully — state that eating chocolate while dieting helps you lose more weight. 

A responsible scientist shouldn’t conduct a trial like this, Bohannon said, and a responsible scientific journal shouldn’t publish it. But Bohannon is not a nutrition scientist (he does have a PhD in molecular biology), and the International Archives of Medicine, he says, is not the most responsible journal. Internet Medical Publishing, which publishes the International Archives of Medicine, is listed as a potential “predatory publisher” by open-access critic Jeffrey Beall. According to Bohannon, the journal didn’t peer review his study or even edit it (and the study could have used an edit — “chocolate” is misspelled more than once). 

But according to Bohannon, the main culprit in what he calls the pseudoscience scam of diet fads isn’t scientists or the journals who publish their studies.

“It’s the reporters,” he told The Post. “The reporters and ultimately the editors…. People who are on the health science beat need to treat it like science, and that has to come from the editors. And if you’re reporting on a scientific study, you need to actually look at the paper. You need to talk to a source who has real scientific expertise.”

Bohannon said that those who reported on the study did none of those things. Lots of outlets, including the German newspaper Bild, reported it without ever contacting “Johannes.” Others interviewed the pretend researcher but only asked him perfunctory questions. Few seemed to have read the study or noticed that it makes no mention of its tiny number of subjects. As far as Bohannon is aware, no one contacted an outside researcher to ask their opinion of the results.

“It was fake journalism,” Bohannon said. “It’s like when schoolkids go to museums with those forms they have to fill out, the fill-in-the-blanks. It was like, ‘I have a story with holes, I need to put a quote here, I’m just filling in the blanks.'”

Few major news outlets covered the study, but it got a lot of play on Web sites and in health magazines.

Only one journalist, a reporter from Ohio, asked Bohannon the questions he thought reporters should be asking — ones about the institute and his sample size. Afraid that he was about to lose his cover, Bohannon put the reporter off. Eventually, he stopped calling. As far as Bohannon is aware, the Ohio reporter never published a story.

Bohannon said he didn’t have any ethical qualms about tricking his fellow journalists this way.

“I didn’t lie to reporters, except about my name. And whenever they asked me a scientific question about the study, I gave them a completely honest answer,” he said. “The whole point is that this was as bad as a lot of science that is considered ‘real’ science. It gets reported without people asking the right questions.”

Not everyone agrees.

Source:http://www.washingtonpost.com