This is not an original idea. I saw it the other day, somewhere on the web. I haven't been able to find it again via Google; maybe it was inside some proprietary comment engine on someone's blog post.
It may well not even have been original with the person whose version I saw the other day. Very little is truly original. Maybe that person copied it; maybe they came up with it independently.
Anyway.
The discussion was about pre-registration of studies. Most people seem to think pre-registration is a good idea, as a way to help fight some of the current problems in science, especially the social sciences, especially psychology, especially (I would argue) social psychology. If you have to pre-register your study, including what you're going to look for, that ought to prevent or reduce all kinds of questionable research practices. If you have to state your hypotheses before you start, you can't HARK later. If you state how many people you're going to have in your sample, you can't apply your own stopping rules, or several other researcher degrees of freedom.
The difficulty seems to be in establishing a register of all these studies. There must be tens or even hundreds of thousands of studies conducted worldwide each year, in the psychology and related departments of thousands of universities worldwide. Just the logistics of creating a central repository for that would be overwhelming. Don't expect the journals to do it, either. Although a few journals have announced their intention to support replication via the concept of a "registered report," this is completely unscalable; it more or less requires the journal to guarantee in advance that the article will be accepted, regardless of the results. And while some journal editors are starting to pay lip service to the idea that it's important to publish null results, the incentive structure of impact factors means that these are likely to remain token efforts, a drop in the bucket alongside the flashy, underpowered Gladwell-bait that nobody will be able to reproduce (but the authors will be way over the line into Tenure County before anyone notices, if they ever do).
And yet, as this person whose contribution I can no longer find (seriously, if you wrote something about this on the Internet in the last 10 days, point me at it and I'll give you full credit) points out, pretty much every PI on any research topic involving human subjects already preregisters their study today via the Institutional Review Board form.
Everyone hates filling out IRB forms. They ask you dozens of questions that have nothing to do with protecting potentially-vulnerable participants and everything to do with covering every imaginable base in case the university gets sued. (Never mind that most such lawsuits would be dismissed as frivolous, or that if you did something really bad, the form wouldn't count for a row of beans.) These forms are so irrelevant to protecting the vulnerable, they even ask you for ridiculous things that are none of the IRB's business, such as your research design and your hypotheses...
Whoa.
You're already forced to register your hypotheses and your design before you can even begin your study. Your institution is going to keep a copy of this for a million years, or at least until the great-great-grandchildren of any of the participants are dead. Probably 80% of the important stuff that a hypothetical universal pre-registration protocol might contain, at some point in an unknown bureaucratic future once the APA and APS and other interested parties have ummmed and ahhhed for a couple of decades, is being recorded right now for almost every study being done in the world, even those cute little quantitative exercises done by Masters' students as course assignments. (Actually, if the APA makes as big a mess of this as they have done with their "Open Data" journal, that 80% will probably turn into something more like 200%, but that's another story.)
To illustrate just how much information PIs are already being asked to provide, I picked a couple of results from a Google search for "IRB form" at random. Cornell's IRB form asks, "Please provide a lay summary of the study, including the purpose, research questions and hypothesis to be evaluated". The University of Minnesota's compact, succinct 18-page form requires PIs to describe "the objective(s) of the proposed research including purpose, research question, hypothesis and relevant background information etc". As a Master's student at the University of East London in the UK, doing a cute little quantitative exercises as a course assignment, I had to fill in a form that asked for "... a statement on the aims and significance of the proposed research, including potential impact to knowledge and understanding in the field (where appropriate, indicate the associated hypothesis which will be tested). This should be a clear justification of the proposed research, why it should proceed and a statement on any anticipated benefits to the community". Helpfully, the form then adds "(Do not exceed 700 words)". 700 words? When did you last read an article with 700 words, or even half that number, of justification of the hypotheses and the research design?
So, here's the challenge to journal editors and IRB chairpersons: this is an approach that would allow you to implement a meaningful, albeit undoubtedly imperfect form of study pre-registration which would have an immediate, overnight, substantial effect on the quality of published research, for a total worldwide cash outlay of approximately zero. All that has to happen is that the editors start to require that authors, instead of mechanically copying and pasting the words "Ethical approval was obtained from the IRB of Pabulum College" from their last 10 articles, actually submit a copy of the form granting that approval for the attention of the reviewers. And of course, the IRBs have to be prepared to give out this data --- but what reason would they have to refuse? Any specific information concerning vulnerable groups could be redacted, along with the PI's mobile phone number; but in most cases, the information on IRB forms is no more sensitive than last week's cafeteria menu. And surely if the IRB forms are as big and complex and bureaucratic as they are, it's because the IRB has a mission, beyond protecting human subjects, of protecting the institution's reputation more widely --- because after all, it would be very embarrassing for a major university if it became associated with the repeated publication of dubious research.
But, as H. L. Mencken said, "There is always an easy solution to every human problem --- neat, plausible, and wrong". So I presume I'm missing something obvious here. Please let me know what that is.
It may well not even have been original with the person whose version I saw the other day. Very little is truly original. Maybe that person copied it; maybe they came up with it independently.
Anyway.
The discussion was about pre-registration of studies. Most people seem to think pre-registration is a good idea, as a way to help fight some of the current problems in science, especially the social sciences, especially psychology, especially (I would argue) social psychology. If you have to pre-register your study, including what you're going to look for, that ought to prevent or reduce all kinds of questionable research practices. If you have to state your hypotheses before you start, you can't HARK later. If you state how many people you're going to have in your sample, you can't apply your own stopping rules, or several other researcher degrees of freedom.
The difficulty seems to be in establishing a register of all these studies. There must be tens or even hundreds of thousands of studies conducted worldwide each year, in the psychology and related departments of thousands of universities worldwide. Just the logistics of creating a central repository for that would be overwhelming. Don't expect the journals to do it, either. Although a few journals have announced their intention to support replication via the concept of a "registered report," this is completely unscalable; it more or less requires the journal to guarantee in advance that the article will be accepted, regardless of the results. And while some journal editors are starting to pay lip service to the idea that it's important to publish null results, the incentive structure of impact factors means that these are likely to remain token efforts, a drop in the bucket alongside the flashy, underpowered Gladwell-bait that nobody will be able to reproduce (but the authors will be way over the line into Tenure County before anyone notices, if they ever do).
And yet, as this person whose contribution I can no longer find (seriously, if you wrote something about this on the Internet in the last 10 days, point me at it and I'll give you full credit) points out, pretty much every PI on any research topic involving human subjects already preregisters their study today via the Institutional Review Board form.
Everyone hates filling out IRB forms. They ask you dozens of questions that have nothing to do with protecting potentially-vulnerable participants and everything to do with covering every imaginable base in case the university gets sued. (Never mind that most such lawsuits would be dismissed as frivolous, or that if you did something really bad, the form wouldn't count for a row of beans.) These forms are so irrelevant to protecting the vulnerable, they even ask you for ridiculous things that are none of the IRB's business, such as your research design and your hypotheses...
Whoa.
You're already forced to register your hypotheses and your design before you can even begin your study. Your institution is going to keep a copy of this for a million years, or at least until the great-great-grandchildren of any of the participants are dead. Probably 80% of the important stuff that a hypothetical universal pre-registration protocol might contain, at some point in an unknown bureaucratic future once the APA and APS and other interested parties have ummmed and ahhhed for a couple of decades, is being recorded right now for almost every study being done in the world, even those cute little quantitative exercises done by Masters' students as course assignments. (Actually, if the APA makes as big a mess of this as they have done with their "Open Data" journal, that 80% will probably turn into something more like 200%, but that's another story.)
To illustrate just how much information PIs are already being asked to provide, I picked a couple of results from a Google search for "IRB form" at random. Cornell's IRB form asks, "Please provide a lay summary of the study, including the purpose, research questions and hypothesis to be evaluated". The University of Minnesota's compact, succinct 18-page form requires PIs to describe "the objective(s) of the proposed research including purpose, research question, hypothesis and relevant background information etc". As a Master's student at the University of East London in the UK, doing a cute little quantitative exercises as a course assignment, I had to fill in a form that asked for "... a statement on the aims and significance of the proposed research, including potential impact to knowledge and understanding in the field (where appropriate, indicate the associated hypothesis which will be tested). This should be a clear justification of the proposed research, why it should proceed and a statement on any anticipated benefits to the community". Helpfully, the form then adds "(Do not exceed 700 words)". 700 words? When did you last read an article with 700 words, or even half that number, of justification of the hypotheses and the research design?
So, here's the challenge to journal editors and IRB chairpersons: this is an approach that would allow you to implement a meaningful, albeit undoubtedly imperfect form of study pre-registration which would have an immediate, overnight, substantial effect on the quality of published research, for a total worldwide cash outlay of approximately zero. All that has to happen is that the editors start to require that authors, instead of mechanically copying and pasting the words "Ethical approval was obtained from the IRB of Pabulum College" from their last 10 articles, actually submit a copy of the form granting that approval for the attention of the reviewers. And of course, the IRBs have to be prepared to give out this data --- but what reason would they have to refuse? Any specific information concerning vulnerable groups could be redacted, along with the PI's mobile phone number; but in most cases, the information on IRB forms is no more sensitive than last week's cafeteria menu. And surely if the IRB forms are as big and complex and bureaucratic as they are, it's because the IRB has a mission, beyond protecting human subjects, of protecting the institution's reputation more widely --- because after all, it would be very embarrassing for a major university if it became associated with the repeated publication of dubious research.
But, as H. L. Mencken said, "There is always an easy solution to every human problem --- neat, plausible, and wrong". So I presume I'm missing something obvious here. Please let me know what that is.
I think this makes sense, depending on how detailed your IRB gets, which really only becomes an issue in longitudinal studies where you might not already know all the specific hypotheses. Another source: Grant proposals. Our lab's funded proposals are on our site (www.teenresearch.org); though people may not like how broad the articulated hypotheses are (Garden of Forking Paths problem).
ReplyDeleteOur IRB documents say that we will measure correlation of genetics with the spread of a non-normal tissue in the esophagus. They don't say whether spread will be measured in centimeters or as a percentage. They say we will measure mutation signatures: they don't say whether we will consider the percentage of mutations in a given signature, or the number of mutations in that signature. When we went to do the analysis we found we were awash in experimenter degrees of freedom that the IRB had been completely silent on.
ReplyDelete