When I used to work in an office, my boss used
to say that any time he had a good idea, he could come to me and in ten
minutes he'd know everything that might go wrong with it. He often went
ahead anyway, and his ideas often worked, but at least he was
forewarned. So in that spirit, here goes.
I'm a little concerned by all the hype around Open Access (OA) journals.
Yes, I know that traditional journal publishers are evil, and make more money and higher gross margins and have bigger car parking spaces than Apple, and I agree that when taxpayers fund research then taxpayers should have access to it. However, I'm not sure that just because all of the above may be true, that our current model of OA journals is necessarily the solution. I have a number of concerns of what may happen as the OA model takes hold and, as everybody tells me is going to happen, becomes dominant. This post is intended to start a discussion on those concerns, if anyone's interested.
1. It's the economy, stupid
One of the strengths of the traditional publishing model is that, to a first approximation and allowing for all kinds of special circumstances, the editor-in-chief of a half-decent journal doesn't have to worry about filling it. Indeed, many journals proudly promote their rejection rate on their home page, next to their average turnaround time. "We reject 90% of submissions; don't waste our time unless you've got a good story to tell", is the message (with, of course, all of the predictable effects on publication bias that this implies). Doubtless the editor-in-chief has some financial targets to meet, perhaps in terms of not blowing the production budget on full-page colour pictures of kittens, but this is not a job whose holder is principally tasked with revenue generation. The money is coming in pretty steadily from sales of packages of journals to institutions around the world (even if some of these institutions are starting to take exception). The editor gets to concentrate on, among other things, maintaining the journal's impact factor --- hopefully using methods that are a little less blatant than this.
With OA journals, funded principally by article processing charges paid by authors, things are likely to be a little different. No matter how dedicated to academic integrity and the highest possible scientific standards the editorial staff want to be, money is right there in the equation every day, especially for an online journal with almost no physical limits to its size. How many articles can we get through the review process this month? Can we upsell the author to the full-colour package? Why do so many people want hardship waivers? (Oh, and I have yet to see any suggestion that OA journals will be less concerned about their impact factor than traditional journals.)
The idea, of course, is that authors will not be reaching into their own pockets to pay the article processing charges. The intention is that the fee of $1,000 or so to publish the results should be budgeted for out of the project's funding. After all, it's only a waffer-thin thousand bucks, the kind of money some projects probably have slopping around at the end anyway if the participants didn't eat all of the M&Ms. But once funding agencies catch on, will they allow grant proposals to include specific line items for OA publishing, when publication in one of the high-prestige traditional journals --- which you promised them, earlier in the proposal, were definitely going to be interested in your groundbreaking project --- is free? And what about the independent researcher with no budget, who may have something interesting to say, but no money? Should such a person have to fund publication from their own pocket?
I'm afraid that the money always, always finds a way to affect things. Someone, somewhere in the process, will be directly incentivised to increase revenue. (In France, where I live, gambling is a state monopoly, which means that whatever arms-length construction they have put together, somewhere there is someone who essentially works for the government and yet has a performance target to sell more scratchcards to the urban poor, even though gambling is officially a social problem.) How does this affect you as the editor-in-chief of an OA journal? Maybe you ask your action editors to tell reviewers to be less picky about certain things. Maybe you suggest to an author that splitting these results into two articles will be to everyone's advantage - after all, the publication fee is coming out of the grant money, and as it stands it is a pretty long paper manuscript for someone to have to wade through at one sitting...
The corollary of this is that the PI presenting an article for publication is a paying customer. Now when I go to make a $1,000 purchase, I'm generally greeted with open arms. I certainly don't expect to have to pass quality control checks before I'm allowed to spend my $1,000. The psychology of the OA model is going to be interesting indeed. (Compare what happened in the UK when public universities started to charge tuition fees; all of a sudden, the idea of a student being given a failing grade became, for many people, a consumer protection issue. "I paid to come here and get a degree, how dare you tell me I can't have one?", ran the argument. Too many unhappy punters, and the Vice-Chancellor is touring the stricter departments to ask them to be a little more, um, flexible in their marking criteria.)
I found a pertinent example shortly before putting this post (which has taken a while to draft) online. Here is a note from Nandita Quaderi, who is "Publishing Director, Open Research" at Scientific Reports, which is part of Nature Publishing Group. Nandita is pleased to announce that henceforth, "a selection of authors submitting a biology manuscript to Scientific Reports will be able to opt-in to a fast-track peer-review service". Needless to say, this service comes "at an additional cost", being provided by a for-profit organisation called Research Square. (An editor of Scientific Reports has resigned over this.) So now, I'm paying to publish, and I'm paying to have my article reviewed. What could possibly go wrong with the objectivity and rigour of the scientific process?
2. Access is not the biggest problem science faces right now
Another issue is that most OA journals do not address the ongoing problems of the peer review system. I would argue that currently, failures of peer review are a bigger threat to science than paywalls. If reviewers are allowing bad science through --- or erroneously recommending rejection of good articles --- then getting free access to the resulting error-filled literature is the least of our problems; and I have yet to see a coherent argument why the OA review process might be inherently any more rigorous than that at traditional journals.
Some online journals, such as The Winnower, have adopted a radical solution to this: anyone can publish an article, without any prior review process, with the idea that people will come along and review it afterwards. This seems attractive at first sight, except that people typically have even less incentive to act as a reviewer once the article is "out there", even if it doesn't yet have the status of a citable article with a DOI (a status which, incidentally, the article's own author decides to award it, at a time of his or her own choosing).
It seems to me that OA journals are to some extent hitching a ride on the back of the traditional journals, which have created (and still sustain) the fundamental mode of operation that we know and love/hate: author sends in MS, editor checks it, editor selects reviewers, reviewers approve or request changes, editor finally accepts or rejects. This system more or less works --- give or take the criticisms of peer review as "broken", which have a lot of merit but which, as I noted above, it seems to me that OA (in and of itself) doesn't do much to address --- because people generally have confidence in it. Not necessarily absolute confidence, but we know how it's meant to work and how to spot when it isn't working. We (like to) believe that the editors do not generally accept (too many) articles from themselves and their buddies (or at least, that they risk getting called out for it if they do), that they select reviewers who are competent in the relevant subfields, that the reviewers do an honest and unbiased job, etc. (Of course, the reviewer who is doing "excellent quality control" with *your* article is an incompetent idiot who has failed to understand even the most basic concepts of *my* article, but that's part of the game.)
So, when something like Collabra, the new OA mega-journal from the University of California, launches, they can put pictures of respected people on the front page where they introduce their editorial board, thus sending a message that the review process will be every bit as rigorous as it is for a traditional journal. Readers are reassured, and authors know they will need to submit work of a high standard. But to me this only works because the majority of people who are being held up as examples of the quality of the journal have good reputations, which have been made within the traditional process. How does this scale? What does the publication process look like in 10 or 20 years time, if the traditional journals have mostly gone and we make our reputations with OA (web-)publishing, blogs, and social media presence? (Yes, impact factor is broken. But where is the dominant, credible alternative that everyone will be prepared to switch to?)
This doesn't mean that Collabra will be full of articles promoting homeopathy after a few months. But over time, the relationship between authors, reviewers, and journals will change, in ways that we can't necessarily predict. That doesn't mean the sky will fall, but it does mean that there will be perverse situations that may or may not be worse than what we have to put up with now.
3. Ham, spam, and all points in between
I also worry that the line between "legitimate" and "spam" OA journals will start to blur. Currently we can all point and laugh at the semi-literate invitations to publish in (or join the Editorial Board of) those pseudo-journals with plausible-sounding names, strange salutation styles in their e-mails, and an editorial address in a Regus suite in San Antonio, from which manuscripts are presumably forwarded to the journal's real staff in Cairo or Mumbai. But these fraudulent (whatever that means...) journals will improve, and it will become hard to tell the "fake" from the "real".
A few weeks ago, I was asked to review an article by an OA journal that was part of a London-based publishing outfit. I genuinely couldn't decide if they were spammers or genuine: the journals mentioned on their web site all seem to exist, and about a third of them are indexed in PubMed. How good or bad is that? I recommended rejection, as the article would have been of little interest to the readers of the journal, according to its own profile. I wonder what the lead author did next (assuming that my recommendation to reject was the editor's verdict as well)? Did he appeal, as a "paying customer", to the editor in chief? Or did he maybe send the article to another OA journal, on the basis that he will eventually find somebody, somewhere, who wants $1,000? (*)
I think, though, that perhaps the bigger risk in the meeting of "legitimate" and "spam" journals is through the trimming of standards at the "legitimate" end. Look at what happened when the Saudis decided to throw some money at education, and suddenly King Abdulaziz University is ranked #7 in the world in mathematics. Uh-huh. Sure. So what happens when that university, or others with rather more money to burn than academic integrity, starts its own OA mega-journal? Exactly what will be the conditions of scientific neutrality under which the editor-in-chief reviews articles by, say, the children of minor members of the Saudi ruling family? Perhaps someone will create an authoritative clearing house to administer a sliding scale of which journals are "real" versus "spam". But who would run such an organisation? The AAAS? ISO? Standard & Poor's? Google? And who would ultimately be responsible for the "legit"/"spam" decisions?
Historically, publisher-led journals seem to have been mostly spam-free; it would be interesting to establish why this was. High barrier to entry in the world of ink and paper? Old-fashioned academic and intellectual integrity, despite the profits? Risk of reputational damage if, say, Springer (cough) or Sage (cough) were to acquire a reputation for publishing garbage? I don't know what the reasons are, but it created the current situation whereby --- whatever the other problems in the system --- a journal that exists in a print edition is generally regarded, at least by default, as having some degree of seriousness. I worry that we will end up in a situation where we don't have a simple way to tell whether we can take a "journal" (in the widest possible sense) seriously or not. In such situations, humans tend to apply some simple heuristics, which scammers have many centuries worth of experience exploiting.
4. A modest (and, as yet, barely sketched out) proposal
Do I have an alternative? Well, when my boss came to me with his ideas, I usually didn't, but in this case I do have a tentative suggestion. What if the funding agencies ran a few journals? After all, these are (generally) the representatives of the taxpayers, who --- as the Open Access movement is right to point out --- pay for the research and ought to have free access to the results. Yet currently, they rely on "the system" to work, and for researchers to muddle their way through that system. In the traditional model, the readers pay, and in theOA model, the authors pay. Both systems have their deficiencies. Supposing we had a parallel model where nobody paid (except a general fund, set up to guarantee neutrality)?
Those of a libertarian bent might argue that the government shouldn't be involved in academic publishing, but the stable door closed on that when we started to take their money to do the research. Some might also argue that an funding agency-sponsored journal might be highly politicised, but then, /a/ why should it be more politicised than the handing out of the money, /b/ the Rind/Lilienfeld saga showed that politicians can pressure "independent" journal publishers into submission too, and /c/ there will always be other outlets; I'm just modestly proposing a "third way". (As a bonus, this would seem to be a good fit with the aims of the pre-registration movement.)
Notes:
1. I'm aware that this is a rather long and at times rambling post. It started life in a frenzied evening of writing just after I got out of hospital after a stay that lasted the better part of three weeks, and that still shows. I should probably have scrapped it and started again, or at least sat down and rearranged the paragraphs, but I wanted to get the ideas out there within a reasonable time frame. I hope some of them are useful.
2. I want to thank Rolf Zwaan for some helpful discussions on an earlier draft of this post. Rolf disagreed with much of what I had written, and I've only made a few changes, so he probably still disagrees with a lot of it. I should point out that my use of the example of Collabra (for whom Rolf is an editor) above is not based on any specific criticism of that journal, but merely as a salient example; Rolf's tweet about his appointment as an editor at Collabra was the spark for my writing of this post.
(*) Update 2016-11-28: I was re-reading this post because reasons, and I noticed this dangling question. I googled the title of the article... sure enough, it was accepted, despite my recommendation to reject.
I'm a little concerned by all the hype around Open Access (OA) journals.
Yes, I know that traditional journal publishers are evil, and make more money and higher gross margins and have bigger car parking spaces than Apple, and I agree that when taxpayers fund research then taxpayers should have access to it. However, I'm not sure that just because all of the above may be true, that our current model of OA journals is necessarily the solution. I have a number of concerns of what may happen as the OA model takes hold and, as everybody tells me is going to happen, becomes dominant. This post is intended to start a discussion on those concerns, if anyone's interested.
1. It's the economy, stupid
One of the strengths of the traditional publishing model is that, to a first approximation and allowing for all kinds of special circumstances, the editor-in-chief of a half-decent journal doesn't have to worry about filling it. Indeed, many journals proudly promote their rejection rate on their home page, next to their average turnaround time. "We reject 90% of submissions; don't waste our time unless you've got a good story to tell", is the message (with, of course, all of the predictable effects on publication bias that this implies). Doubtless the editor-in-chief has some financial targets to meet, perhaps in terms of not blowing the production budget on full-page colour pictures of kittens, but this is not a job whose holder is principally tasked with revenue generation. The money is coming in pretty steadily from sales of packages of journals to institutions around the world (even if some of these institutions are starting to take exception). The editor gets to concentrate on, among other things, maintaining the journal's impact factor --- hopefully using methods that are a little less blatant than this.
With OA journals, funded principally by article processing charges paid by authors, things are likely to be a little different. No matter how dedicated to academic integrity and the highest possible scientific standards the editorial staff want to be, money is right there in the equation every day, especially for an online journal with almost no physical limits to its size. How many articles can we get through the review process this month? Can we upsell the author to the full-colour package? Why do so many people want hardship waivers? (Oh, and I have yet to see any suggestion that OA journals will be less concerned about their impact factor than traditional journals.)
The idea, of course, is that authors will not be reaching into their own pockets to pay the article processing charges. The intention is that the fee of $1,000 or so to publish the results should be budgeted for out of the project's funding. After all, it's only a waffer-thin thousand bucks, the kind of money some projects probably have slopping around at the end anyway if the participants didn't eat all of the M&Ms. But once funding agencies catch on, will they allow grant proposals to include specific line items for OA publishing, when publication in one of the high-prestige traditional journals --- which you promised them, earlier in the proposal, were definitely going to be interested in your groundbreaking project --- is free? And what about the independent researcher with no budget, who may have something interesting to say, but no money? Should such a person have to fund publication from their own pocket?
I'm afraid that the money always, always finds a way to affect things. Someone, somewhere in the process, will be directly incentivised to increase revenue. (In France, where I live, gambling is a state monopoly, which means that whatever arms-length construction they have put together, somewhere there is someone who essentially works for the government and yet has a performance target to sell more scratchcards to the urban poor, even though gambling is officially a social problem.) How does this affect you as the editor-in-chief of an OA journal? Maybe you ask your action editors to tell reviewers to be less picky about certain things. Maybe you suggest to an author that splitting these results into two articles will be to everyone's advantage - after all, the publication fee is coming out of the grant money, and as it stands it is a pretty long paper manuscript for someone to have to wade through at one sitting...
The corollary of this is that the PI presenting an article for publication is a paying customer. Now when I go to make a $1,000 purchase, I'm generally greeted with open arms. I certainly don't expect to have to pass quality control checks before I'm allowed to spend my $1,000. The psychology of the OA model is going to be interesting indeed. (Compare what happened in the UK when public universities started to charge tuition fees; all of a sudden, the idea of a student being given a failing grade became, for many people, a consumer protection issue. "I paid to come here and get a degree, how dare you tell me I can't have one?", ran the argument. Too many unhappy punters, and the Vice-Chancellor is touring the stricter departments to ask them to be a little more, um, flexible in their marking criteria.)
I found a pertinent example shortly before putting this post (which has taken a while to draft) online. Here is a note from Nandita Quaderi, who is "Publishing Director, Open Research" at Scientific Reports, which is part of Nature Publishing Group. Nandita is pleased to announce that henceforth, "a selection of authors submitting a biology manuscript to Scientific Reports will be able to opt-in to a fast-track peer-review service". Needless to say, this service comes "at an additional cost", being provided by a for-profit organisation called Research Square. (An editor of Scientific Reports has resigned over this.) So now, I'm paying to publish, and I'm paying to have my article reviewed. What could possibly go wrong with the objectivity and rigour of the scientific process?
2. Access is not the biggest problem science faces right now
Another issue is that most OA journals do not address the ongoing problems of the peer review system. I would argue that currently, failures of peer review are a bigger threat to science than paywalls. If reviewers are allowing bad science through --- or erroneously recommending rejection of good articles --- then getting free access to the resulting error-filled literature is the least of our problems; and I have yet to see a coherent argument why the OA review process might be inherently any more rigorous than that at traditional journals.
Some online journals, such as The Winnower, have adopted a radical solution to this: anyone can publish an article, without any prior review process, with the idea that people will come along and review it afterwards. This seems attractive at first sight, except that people typically have even less incentive to act as a reviewer once the article is "out there", even if it doesn't yet have the status of a citable article with a DOI (a status which, incidentally, the article's own author decides to award it, at a time of his or her own choosing).
It seems to me that OA journals are to some extent hitching a ride on the back of the traditional journals, which have created (and still sustain) the fundamental mode of operation that we know and love/hate: author sends in MS, editor checks it, editor selects reviewers, reviewers approve or request changes, editor finally accepts or rejects. This system more or less works --- give or take the criticisms of peer review as "broken", which have a lot of merit but which, as I noted above, it seems to me that OA (in and of itself) doesn't do much to address --- because people generally have confidence in it. Not necessarily absolute confidence, but we know how it's meant to work and how to spot when it isn't working. We (like to) believe that the editors do not generally accept (too many) articles from themselves and their buddies (or at least, that they risk getting called out for it if they do), that they select reviewers who are competent in the relevant subfields, that the reviewers do an honest and unbiased job, etc. (Of course, the reviewer who is doing "excellent quality control" with *your* article is an incompetent idiot who has failed to understand even the most basic concepts of *my* article, but that's part of the game.)
So, when something like Collabra, the new OA mega-journal from the University of California, launches, they can put pictures of respected people on the front page where they introduce their editorial board, thus sending a message that the review process will be every bit as rigorous as it is for a traditional journal. Readers are reassured, and authors know they will need to submit work of a high standard. But to me this only works because the majority of people who are being held up as examples of the quality of the journal have good reputations, which have been made within the traditional process. How does this scale? What does the publication process look like in 10 or 20 years time, if the traditional journals have mostly gone and we make our reputations with OA (web-)publishing, blogs, and social media presence? (Yes, impact factor is broken. But where is the dominant, credible alternative that everyone will be prepared to switch to?)
This doesn't mean that Collabra will be full of articles promoting homeopathy after a few months. But over time, the relationship between authors, reviewers, and journals will change, in ways that we can't necessarily predict. That doesn't mean the sky will fall, but it does mean that there will be perverse situations that may or may not be worse than what we have to put up with now.
3. Ham, spam, and all points in between
I also worry that the line between "legitimate" and "spam" OA journals will start to blur. Currently we can all point and laugh at the semi-literate invitations to publish in (or join the Editorial Board of) those pseudo-journals with plausible-sounding names, strange salutation styles in their e-mails, and an editorial address in a Regus suite in San Antonio, from which manuscripts are presumably forwarded to the journal's real staff in Cairo or Mumbai. But these fraudulent (whatever that means...) journals will improve, and it will become hard to tell the "fake" from the "real".
A few weeks ago, I was asked to review an article by an OA journal that was part of a London-based publishing outfit. I genuinely couldn't decide if they were spammers or genuine: the journals mentioned on their web site all seem to exist, and about a third of them are indexed in PubMed. How good or bad is that? I recommended rejection, as the article would have been of little interest to the readers of the journal, according to its own profile. I wonder what the lead author did next (assuming that my recommendation to reject was the editor's verdict as well)? Did he appeal, as a "paying customer", to the editor in chief? Or did he maybe send the article to another OA journal, on the basis that he will eventually find somebody, somewhere, who wants $1,000? (*)
I think, though, that perhaps the bigger risk in the meeting of "legitimate" and "spam" journals is through the trimming of standards at the "legitimate" end. Look at what happened when the Saudis decided to throw some money at education, and suddenly King Abdulaziz University is ranked #7 in the world in mathematics. Uh-huh. Sure. So what happens when that university, or others with rather more money to burn than academic integrity, starts its own OA mega-journal? Exactly what will be the conditions of scientific neutrality under which the editor-in-chief reviews articles by, say, the children of minor members of the Saudi ruling family? Perhaps someone will create an authoritative clearing house to administer a sliding scale of which journals are "real" versus "spam". But who would run such an organisation? The AAAS? ISO? Standard & Poor's? Google? And who would ultimately be responsible for the "legit"/"spam" decisions?
Historically, publisher-led journals seem to have been mostly spam-free; it would be interesting to establish why this was. High barrier to entry in the world of ink and paper? Old-fashioned academic and intellectual integrity, despite the profits? Risk of reputational damage if, say, Springer (cough) or Sage (cough) were to acquire a reputation for publishing garbage? I don't know what the reasons are, but it created the current situation whereby --- whatever the other problems in the system --- a journal that exists in a print edition is generally regarded, at least by default, as having some degree of seriousness. I worry that we will end up in a situation where we don't have a simple way to tell whether we can take a "journal" (in the widest possible sense) seriously or not. In such situations, humans tend to apply some simple heuristics, which scammers have many centuries worth of experience exploiting.
4. A modest (and, as yet, barely sketched out) proposal
Do I have an alternative? Well, when my boss came to me with his ideas, I usually didn't, but in this case I do have a tentative suggestion. What if the funding agencies ran a few journals? After all, these are (generally) the representatives of the taxpayers, who --- as the Open Access movement is right to point out --- pay for the research and ought to have free access to the results. Yet currently, they rely on "the system" to work, and for researchers to muddle their way through that system. In the traditional model, the readers pay, and in theOA model, the authors pay. Both systems have their deficiencies. Supposing we had a parallel model where nobody paid (except a general fund, set up to guarantee neutrality)?
Those of a libertarian bent might argue that the government shouldn't be involved in academic publishing, but the stable door closed on that when we started to take their money to do the research. Some might also argue that an funding agency-sponsored journal might be highly politicised, but then, /a/ why should it be more politicised than the handing out of the money, /b/ the Rind/Lilienfeld saga showed that politicians can pressure "independent" journal publishers into submission too, and /c/ there will always be other outlets; I'm just modestly proposing a "third way". (As a bonus, this would seem to be a good fit with the aims of the pre-registration movement.)
Notes:
1. I'm aware that this is a rather long and at times rambling post. It started life in a frenzied evening of writing just after I got out of hospital after a stay that lasted the better part of three weeks, and that still shows. I should probably have scrapped it and started again, or at least sat down and rearranged the paragraphs, but I wanted to get the ideas out there within a reasonable time frame. I hope some of them are useful.
2. I want to thank Rolf Zwaan for some helpful discussions on an earlier draft of this post. Rolf disagreed with much of what I had written, and I've only made a few changes, so he probably still disagrees with a lot of it. I should point out that my use of the example of Collabra (for whom Rolf is an editor) above is not based on any specific criticism of that journal, but merely as a salient example; Rolf's tweet about his appointment as an editor at Collabra was the spark for my writing of this post.
(*) Update 2016-11-28: I was re-reading this post because reasons, and I noticed this dangling question. I googled the title of the article... sure enough, it was accepted, despite my recommendation to reject.
