Flutterby™! : bad research 2007-09-18 17:14:26.557336+00

bad research

2007-09-18 17:14:26.557336+00 by Dan Lyke 12 comments

It's cranky contrarian day here at Chez Dan, I'm struggling through some PHP and I'm a little cranky (the amount of bad example PHP code out there is truly astounding, I'm shocked at how many potential database and other exploits must be exposed to the world).

So it's without surprise that I read that most peer-reviewed published studies are tainted by sloppy analysis:

These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."

(via) Perhaps what you'd expect, but of course this morning I was reading about Melissa Farley's "research" on the sex industry (press release challenging Farley's work here) and noted that her work is taken seriously in public policy and isn't even peer reviewed. (From the Eyes Open Now? entry at Renegade Evolution)

So all of this feeds into the post-positivist notion that even if reality is "knowable", humans are pretty ill-equipped to figure it out without bias (systematic or otherwise).

My personal take is that academic researchers are driven not just to publish - but to publish in high-rated journals. High rated journals generally don't accept papers that end in "we found no significance in our study". They also don't accept papers based on research that can be done over a long weekend (unless it's really, really earth-shattering). An "good publication" in a "highly rated" journal may be the culmination of several years' work. So in order to replicate the results, you'd have to make a serious time investment. If you come up with different results, then you are likely only to get published if you have something new to add other than "Smith et al was wrong - there is no signifcance". Further, we (academic researchers) are trained from the start to shoot for adding new ideas to our field. That's what fundamentally separates the PhD from the MS.

#Comment Re: made: 2007-09-18 22:45:41.789633+00 by: meuon [edit history]

"the amount of bad example PHP code out there is truly astounding" - why did I get a guilty feeling when I read that? - As I'm moving domains from my old server to my new one, I'm really trying to clean-up and fix code that I now know I should have never written that way. Sigh.. It's just so EASY to make things work in PHP, but that also makes it easy to abuse. my newer stuff has some serious detaint() functions on everything, even things that are supposedly 100% internal data that no-one should be able to get to.

Irrationally held truths may be more harmful than reasoned errors. - Thomas H. Huxley

Meuon, most of what I found was a distinct lack of properly escaped CGI parameters going into databases. mysql_query("UPDATE xyz SET abc='".$_GET["def"]."' WHERE ghi='".$_GET["jkl"]."'", $conn) is not your friend.

Eric, I think that what gets in academic journals is just like what makes the "news". As you say there's a strong emphasis on reporting the unusual, and things that do that while reinforcing the prejudices of the prevailing wisdom are extra likely to make it in. Jon Benet Ramsey isn't indicative of the upbringing or disposition of children in the United States, but her case made the headlines. Why should academic journals be any different, but for focus?

I've learned that you may need to sanity check and detaint things that humans can put into databases that get fed back into the system. mysql_escape() is a good place to start, but not end. In some systems, using & # 3 9 ; (char 39) is a good option for escaping ', and then you find out JavaScript will re-interprete char 39 back into a single quote and...

Want to see new expoits and errors: Let "Subject Matter Experts" who just cut and pasted code from "Experts Exchange" into a CMS, and the JavaScript code they cut and pasted is all url and/or hex encoded and what they can read just unpacks it and executes it. All because he thought the floating post-it note that he could put some text on was cute on his course page. Aargh. Next thing we know, every M$-MSIE System that saw that page has some kind of hidden floating window running who knows what, a keylogger? an ad-ware popper?

So now I check for all kinds of stuff on input, like other people's javascript, or html in fields that should be text only (I love strip_tags()!) or those odd M$-chars that get stuck into inputs that get UTF-8'd into weirder stuff like (this probably will not display here) XXXXXX - which is supposed to be a right quote, really. which I convert that, and rsquo into just normal quotes.. and.. and.. and..

Recently, the "summer of sharks" in Florida had less shark attacks than the average year along the Florida coast. A handful of people received a bacterial infection from tainted veggies and "The Ecoli epidemic facing the nation" gets front page news. Nearly a dozen pets get sick and a handful die from tainted wheat gluten and "The pet food causes renal failure in unknown numbers of pets" becomes the top story.

Media is about sensationalism, and in turn, it bleeds off into other industries. Some less-than-reputable academic journals gain attention and notoriety if they're quoting what's popular in the mass media. Global warming was a recent issue where any journal could make a report that supported what the media wanted to feed to the public and be quoted in major news outlets and on the lips of politicians the following day. On top of the need for notoriety, which means grants and funding as well, outside organizations rarely pay a research firm to come up with an answer, they pay research firms to come up with the answers they want to hear. The question is easy to form when you already know the answer, and even though it's not good scientific method, it is good business. I'm sure we've all read somewhere about how "Windows is the most secure operating system available" by a firm owned or hired by Microsoft.

"Peer reviewed journal" is the easiest term to use which gives a report an air of authenticity. If I report something incorrect and a couple of people in my lab looked at my possibly incorrect data and gained a similar conclusion, it's a "peer reviewed" report. That doesn't mean it's correct, it just means someone using my same potentially flawed data came to the same potentially flawed conclusion. If the "peer reviewed journal" looks at the flawed data and conclusion and decides it looks valid enough to print, it becomes reported in an official publication.

Respectable journals usually will review claims and data and do some of their own fact-checking, source checking and/or research and come up with independent conclusions before publishing any paper that's sent to them. But it seems that not every scientific journal is a reputable institution.

Read "A State of Fear". or just be cognizant of the major theme that the media and many industries ("save the planet" and other kinds) have learned to thrive on, and promote the inducement of "a state of fear" because it's good for business and personal pockets.

"Peer reviewed journal" means quite a bit more than "checked by someone else in my lab". Peer-reviewed journal's have extensive lists of reviewers and every attempt is made to have experts as closely aligned to the article review it. The article is sent for review anonymously (of course, these are small communities and you can likely tell who the submitter was just by their writing style). Unfortunately, if something "seems right" to these other experts, little critical effort is applied. So if you publish something closely aligned with the results of other experts, it'll skate through - compounding any previous errors.

That said, what is a better model? The nature of Logical Postivism is additive. "Proven" hypotheses are added to the Body of Knowledge (Science) by people who are explicitly trained in what are considered the best methods for establishing "proof". People are encouraged to specialize so they can contribute to the body of knowledge in small amounts. As it is, becoming "expert" in an area and "trained" in the methods takes decades.

Eric, yeah, I don't have a better solution than "peer reviewed journal" other than to kick the repute with which we hold "peer reviewed journal" down a few notches. It's not that it's not the best system we've got, we just need to acknowledge that even as the best system we've got it still sucks.

To me the bigger problem is that objectivity has been deprecated. Whether in journalism or in research - my gosh - it appears its not a top priority. Looks like grant writing toward the prejudices of the grantor (gumint) makes you more appreciated. Kinda like global warming advocates.

But science based upon a valid epitemological foundation is self righting like a sail boat in a storm. All we have to do is get rid of the mystics.

I think you are confusing statistical error with mysticism. I get the feeling that you don't think anyone, other than contributers to the Ayn Rand Institute, operate on a basis of reason and understanding. In scientific theory, what you belovedly call "objectivism" is embodied in Logical Postivism. This is the epistemology underlying the hard-sciences and is directly subscribed to by most all physical scientists.

I can tell you that the researchers I personally know who you blanket as "global warming advocates" aren't "advocating" but rather reporting their results. These are people who spend enormous amounts of intellectual and physical effort in trying to really understand what is happening. I can also tell you that they'd love nothing more than to report that global warming isn't happening.

The reason it seems to be a "popular" subject for bodies like the NSF is that it's the general consensus among climate scientists (people who've dedicated their life to understanding how climate works) that there is a lack of understanding there worth directing funds towards. What they are saying is "we don't objectively understand this climate thing but what we have objectively measured is that something screwy is going on and we need to focus our efforts on refining our objective understanding".

I'm not a climate scientist, so I'm not going to argue the details of "global warming" but I can tell you that my friends who spent the Summer on the ice pack in Greenland report that there isn't as much ice there as there was last Summer. I think other objective reports, like the fact that the Northwest Passage just opened (you know - that fabled route west that Columbus was looking for in 1492).