29 March 2015

Open Access journals: what's not to like? This, maybe...

When I used to work in an office, my boss used to say that any time he had a good idea, he could come to me and in ten minutes he'd know everything that might go wrong with it.  He often went ahead anyway, and his ideas often worked, but at least he was forewarned.  So in that spirit, here goes.

I'm a little concerned by all the hype around Open Access (OA) journals.

Yes, I know that traditional journal publishers are evil, and make more money and higher gross margins and have bigger car parking spaces than Apple, and I agree that when taxpayers fund research then taxpayers should have access to it.  However, I'm not sure that just because all of the above may be true, that our current model of OA journals is necessarily the solution.  I have a number of concerns of what may happen as the OA model takes hold and, as everybody tells me is going to happen, becomes dominant.  This post is intended to start a discussion on those concerns, if anyone's interested.

1. It's the economy, stupid

One of the strengths of the traditional publishing model is that, to a first approximation and allowing for all kinds of special circumstances, the editor-in-chief of a half-decent journal doesn't have to worry about filling it.  Indeed, many journals proudly promote their rejection rate on their home page, next to their average turnaround time.  "We reject 90% of submissions; don't waste our time unless you've got a good story to tell", is the message (with, of course, all of the predictable effects on publication bias that this implies).  Doubtless the editor-in-chief has some financial targets to meet, perhaps in terms of not blowing the production budget on full-page colour pictures of kittens, but this is not a job whose holder is principally tasked with revenue generation.  The money is coming in pretty steadily from sales of packages of journals to institutions around the world (even if some of these institutions are starting to take exception).  The editor gets to concentrate on, among other things, maintaining the journal's impact factor --- hopefully using methods that are a little less blatant than this.

With OA journals, funded principally by article processing charges paid by authors, things are likely to be a little different.  No matter how dedicated to academic integrity and the highest possible scientific standards the editorial staff want to be, money is right there in the equation every day, especially for an online journal with almost no physical limits to its size.  How many articles can we get through the review process this month?  Can we upsell the author to the full-colour package?  Why do so many people want hardship waivers?  (Oh, and I have yet to see any suggestion that OA journals will be less concerned about their impact factor than traditional journals.)

The idea, of course, is that authors will not be reaching into their own pockets to pay the article processing charges.  The intention is that the fee of $1,000 or so to publish the results should be budgeted for out of the project's funding.  After all, it's only a waffer-thin thousand bucks, the kind of money some projects probably have slopping around at the end anyway if the participants didn't eat all of the M&Ms.  But once funding agencies catch on, will they allow grant proposals to include specific line items for OA publishing, when publication in one of the high-prestige traditional journals --- which you promised them, earlier in the proposal, were definitely going to be interested in your groundbreaking project --- is free?  And what about the independent researcher with no budget, who may have something interesting to say, but no money?  Should such a person have to fund publication from their own pocket?

I'm afraid that the money always, always finds a way to affect things.  Someone, somewhere in the process, will be directly incentivised to increase revenue.  (In France, where I live, gambling is a state monopoly, which means that whatever arms-length construction they have put together, somewhere there is someone who essentially works for the government and yet has a performance target to sell more scratchcards to the urban poor, even though gambling is officially a social problem.)  How does this affect you as the editor-in-chief of an OA journal?  Maybe you ask your action editors to tell reviewers to be less picky about certain things.  Maybe you suggest to an author that splitting these results into two articles will be to everyone's advantage - after all, the publication fee is coming out of the grant money, and as it stands it is a pretty long paper manuscript for someone to have to wade through at one sitting...

The corollary of this is that the PI presenting an article for publication is a paying customer.  Now when I go to make a $1,000 purchase, I'm generally greeted with open arms.  I certainly don't expect to have to pass quality control checks before I'm allowed to spend my $1,000.  The psychology of the OA model is going to be interesting indeed.  (Compare what happened in the UK when public universities started to charge tuition fees; all of a sudden, the idea of a student being given a failing grade became, for many people, a consumer protection issue.  "I paid to come here and get a degree, how dare you tell me I can't have one?", ran the argument.  Too many unhappy punters, and the Vice-Chancellor is touring the stricter departments to ask them to be a little more, um, flexible in their marking criteria.)

I found a pertinent example shortly before putting this post (which has taken a while to draft) online.  Here is a note from Nandita Quaderi, who is "Publishing Director, Open Research" at Scientific Reports, which is part of Nature Publishing Group.  Nandita is pleased to announce that henceforth, "a selection of authors submitting a biology manuscript to Scientific Reports will be able to opt-in to a fast-track peer-review service".  Needless to say, this service comes "at an additional cost", being provided by a for-profit organisation called Research Square.   (An editor of Scientific Reports has resigned over this.)  So now, I'm paying to publish, and I'm paying to have my article reviewed.  What could possibly go wrong with the objectivity and rigour of the scientific process?

2. Access is not the biggest problem science faces right now

Another issue is that most OA journals do not address the ongoing problems of the peer review system.  I would argue that currently, failures of peer review are a bigger threat to science than paywalls.  If reviewers are allowing bad science through --- or erroneously recommending rejection of good articles --- then getting free access to the resulting error-filled literature is the least of our problems; and I have yet to see a coherent argument why the OA review process might be inherently any more rigorous than that at traditional journals.

Some online journals, such as The Winnower, have adopted a radical solution to this: anyone can publish an article, without any prior review process, with the idea that people will come along and review it afterwards.  This seems attractive at first sight, except that people typically have even less incentive to act as a reviewer once the article is "out there", even if it doesn't yet have the status of a citable article with a DOI (a status which, incidentally, the article's own author decides to award it, at a time of his or her own choosing).

It seems to me that OA journals are to some extent hitching a ride on the back of the traditional journals, which have created (and still sustain) the fundamental mode of operation that we know and love/hate: author sends in MS, editor checks it, editor selects reviewers, reviewers approve or request changes, editor finally accepts or rejects.  This system more or less works --- give or take the criticisms of peer review as "broken", which have a lot of merit but which, as I noted above, it seems to me that OA (in and of itself) doesn't do much to address --- because people generally have confidence in it.  Not necessarily absolute confidence, but we know how it's meant to work and how to spot when it isn't working.  We (like to) believe that the editors do not generally accept (too many) articles from themselves and their buddies (or at least, that they risk getting called out for it if they do), that they select reviewers who are competent in the relevant subfields, that the reviewers do an honest and unbiased job, etc.  (Of course, the reviewer who is doing "excellent quality control" with *your* article is an incompetent idiot who has failed to understand even the most basic concepts of *my* article, but that's part of the game.)

So, when something like Collabra, the new OA mega-journal from the University of California, launches, they can put pictures of respected people on the front page where they introduce their editorial board, thus sending a message that the review process will be every bit as rigorous as it is for a traditional journal.  Readers are reassured, and authors know they will need to submit work of a high standard.  But to me this only works because the majority of people who are being held up as examples of the quality of the journal have good reputations, which have been made within the traditional process.  How does this scale?  What does the publication process look like in 10 or 20 years time, if the traditional journals have mostly gone and we make our reputations with OA (web-)publishing, blogs, and social media presence?  (Yes, impact factor is broken. But where is the dominant, credible alternative that everyone will be prepared to switch to?)

This doesn't mean that Collabra will be full of articles promoting homeopathy after a few months.  But over time, the relationship between authors, reviewers, and journals will change, in ways that we can't necessarily predict.  That doesn't mean the sky will fall, but it does mean that there will be perverse situations that may or may not be worse than what we have to put up with now.

3. Ham, spam, and all points in between

I also worry that the line between "legitimate" and "spam" OA journals will start to blur.  Currently we can all point and laugh at the semi-literate invitations to publish in (or join the Editorial Board of) those pseudo-journals with plausible-sounding names, strange salutation styles in their e-mails, and an editorial address in a Regus suite in San Antonio, from which manuscripts are presumably forwarded to the journal's real staff in Cairo or Mumbai.  But these fraudulent (whatever that means...) journals will improve, and it will become hard to tell the "fake" from the "real".
A few weeks ago, I was asked to review an article by an OA journal that was part of a London-based publishing outfit.  I genuinely couldn't decide if they were spammers or genuine: the journals mentioned on their web site all seem to exist, and about a third of them are indexed in PubMed.  How good or bad is that?  I recommended rejection, as the article would have been of little interest to the readers of the journal, according to its own profile.  I wonder what the lead author did next (assuming that my recommendation to reject was the editor's verdict as well)?  Did he appeal, as a "paying customer", to the editor in chief?  Or did he maybe send the article to another OA journal, on the basis that he will eventually find somebody, somewhere, who wants $1,000? (*)

I think, though, that perhaps the bigger risk in the meeting of "legitimate" and "spam" journals is through the trimming of standards at the "legitimate" end. Look at what happened when the Saudis decided to throw some money at education, and suddenly King Abdulaziz University is ranked #7 in the world in mathematics.  Uh-huh.  Sure.  So what happens when that university, or others with rather more money to burn than academic integrity, starts its own OA mega-journal?  Exactly what will be the conditions of scientific neutrality under which the editor-in-chief reviews articles by, say, the children of minor members of the Saudi ruling family?  Perhaps someone will create an authoritative clearing house to administer a sliding scale of which journals are "real" versus "spam".  But who would run such an organisation?  The AAAS?  ISO?  Standard & Poor's?  Google?  And who would ultimately be responsible for the "legit"/"spam" decisions?

Historically, publisher-led journals seem to have been mostly spam-free; it would be interesting to establish why this was. High barrier to entry in the world of ink and paper?  Old-fashioned academic and intellectual integrity, despite the profits?  Risk of reputational damage if, say, Springer (cough) or Sage (cough) were to acquire a reputation for publishing garbage?  I don't know what the reasons are, but it created the current situation whereby --- whatever the other problems in the system --- a journal that exists in a print edition is generally regarded, at least by default, as having some degree of seriousness.  I worry that we will end up in a situation where we don't have a simple way to tell whether we can take a "journal" (in the widest possible sense) seriously or not.  In such situations, humans tend to apply some simple heuristics, which scammers have many centuries worth of experience exploiting.

4. A modest (and, as yet, barely sketched out) proposal

Do I have an alternative?  Well, when my boss came to me with his ideas, I usually didn't, but in this case I do have a tentative suggestion.  What if the funding agencies ran a few journals?  After all, these are (generally) the representatives of the taxpayers, who --- as the Open Access movement is right to point out --- pay for the research and ought to have free access to the results.  Yet currently, they rely on "the system" to work, and for researchers to muddle their way through that system.  In the traditional model, the readers pay, and in theOA model, the authors pay.  Both systems have their deficiencies.  Supposing we had a parallel model where nobody paid (except a general fund, set up to guarantee neutrality)?

Those of a libertarian bent might argue that the government shouldn't be involved in academic publishing, but the stable door closed on that when we started to take their money to do the research.  Some might also argue that an funding agency-sponsored journal might be highly politicised, but then, /a/ why should it be more politicised than the handing out of the money, /b/ the Rind/Lilienfeld saga showed that politicians can pressure "independent" journal publishers into submission too, and /c/ there will always be other outlets; I'm just modestly proposing a "third way".  (As a bonus, this would seem to be a good fit with the aims of the pre-registration movement.)



Notes:

1. I'm aware that this is a rather long and at times rambling post.  It started life in a frenzied evening of writing just after I got out of hospital after a stay that lasted the better part of three weeks, and that still shows.  I should probably have scrapped it and started again, or at least sat down and rearranged the paragraphs, but I wanted to get the ideas out there within a reasonable time frame.  I hope some of them are useful.

2. I want to thank Rolf Zwaan for some helpful discussions on an earlier draft of this post.  Rolf disagreed with much of what I had written, and I've only made a few changes, so he probably still disagrees with a lot of it.  I should point out that my use of the example of Collabra (for whom Rolf is an editor) above is not based on any specific criticism of that journal, but merely as a salient example; Rolf's tweet about his appointment as an editor at Collabra was the spark for my writing of this post.

(*) Update 2016-11-28: I was re-reading this post because reasons, and I noticed this dangling question.  I googled the title of the article... sure enough, it was accepted, despite my recommendation to reject.

10 comments:

  1. Nick - great post, and I share most of these worries.

    One other thing that worries me about the OA model is that it can shift costs away from the developed world (which bought the journal subscriptions) and toward the developing world (whose scientists will have to pay OA fees). Hardship waivers will help, but as you point out, it's not clear why the OA journals will be incentivized to keep providing them.

    ReplyDelete
  2. Thanks, this post is a great conversation starter. I share your worry that the fact that authors pay publishing charges could persuade journals to accept papers a bit too easily. That being said, I have to disagree with you on some other points.
    Sadly, you seem to be one of many who equate having a DOI to being citable. It is not the fact that a document has a DOI that makes it citable; you can, as a publisher, get to assign DOIs to just about anything as long as you promise to preserve it reasonably well. It does make a document a lot easier to reference unambiguously, though. What is citable, however, is really up to the journal etc. in which you are trying to cite the document in question. There seems to be a growing perception in online discussion these days that a document having a DOI means we can safely throw source criticism out the window.
    The fact that the author can decide when a paper is ready and thereby cause it to get a DOI assigned does not mean that everyone has to take it as a seal of approval. You can use the open reviews (or the lack of them) to help you judge whether it is worth citing (or even reading).
    I think the lines between "respectable" and "spam" journals will begin to blur, but that does not have to be a problem. The problem here is that we are all really subscribing to journals for their main services: prestige and seal of approval. But we borrow the prestige of the journal in the hope that our paper published in it will eventually live up to it - which it probably will not (Nine reasons why Impact Factors fail and using them may harm science - see point 1). With respect to seal of approval, we simply have to take the journal's word for it, because we can in most cases not see what the reviewers said about the papers in them. All that is left for journals to provide is subject classification for easier discovery, and even that is changing (see for example the rise of services like Sparrho).
    This is also why the problem of distinguishing the good from the bad journals does not have to be such a big problem. The solution to that is to judge papers on their individual merits instead of which journal they are published in. This is easier said than done, of course, since this would make it irrelevant where a paper is published. This would in turn destroy the traditional publishers' business model, so they will of course fight it.
    My proposal, however, is to make peer reviews open. Of course this also entails problems, as you have also pointed out, because it is hard to attract reviewers. In the end, this could be the value proposition of future journals; the journal that can attract the best and most relevant reviewers wins. If a journal takes care of the review process, the indicator of quality that the reviews provide will be quickly available and you will not have to sit around for years waiting to get cited. So, this provides a qualitative article-level metric that is a lot more useful than wrongly judging a paper by its journal's JIF.

    ReplyDelete
  3. I don't think I implied that something with a DOI is citable. But something that is citable probably ought to have a DOI. I only mentioned this in the context of The Winnower; it's not a major part of my argument.

    There are upsides and downsides to open peer review; it's part of the debate about how the review process is broken. But it seems to me that a journal's choice to use open review or not is likely to be independent of its choice of payment model; that is, OA doesn't fix this problem either.

    ReplyDelete
    Replies
    1. I probably went a bit overboard on the DOI issue as that is really a minor thing in your post. I was referring to "even if it doesn't yet have the status of a citable article with a DOI" which, I guess, one does not have to interpret as DOI = citable.
      There are definitely upsides and downsides to open review. I am trying to propose that it could be used to turn focus to the quality of the paper on its own merits instead of relying on the prestige of the publishing journal.

      Delete
  4. Hi Nick - Interesting post! I'm wondering what you think about traditional journals with open access options. I have a hard time with the ridiculous amounts of money libraries have to pay for us to be able to read our own work, so it seems like any system that helps move us away from that is an improvement. But I wonder if you see any looming issues with that model (i.e., one that shifts costs from libraries to individual researchers/their funders).

    P.S. Sorry about all the tech challenges....

    ReplyDelete
  5. Katie, it seems to me that OA options allow traditional journals to "eat their cake while still remaining in possession of the same cake afterwards"(*). They get all the financial benefits of OA, and the kudos that accrues from fans of that model, without having to change the whole publication model of the journal. If your library subscribes to publisher X's journals for $200,000 and the authors of 20% of the articles published in those journals next year decide to take the OA option, will your library get a $40,000 refund? I suspect not. So all the publisher is foregoing by taking $1,000 for the OA option is however many pay-per-view sales they would have made, which I'm guessing is typically less than one per article.

    I don't worry too much about the amount of money that libraries have to pay. It makes for a big scary bill --- typically more than any of us makes in a year --- to be paid in one shot, but the money has to come from somewhere, and it might well work out more expensive to hand it over in hundreds of waffer-thin slices. If you're at a publicly-funded school then the money has to come from the public purse anyway, whether that's in the form of signing off on the library's budget (which is typically one of the less exciting things for people to talk about in meetings; sometimes there are advantages to being off the radar), or adding a publication fee to the budget of a project. Making it invisible doesn't make things cheaper; in fact, a lot of marketing is all about making it easy to spend money without noticing it adding up (does it feel like $5 when you buy a $5 latte with a contactless card?).

    It seems to me that having the money in the hands of the librarian, rather than scattered around lots of PIs in small amounts, makes the whole process less susceptible to the kind of consumer-type marketing fads that will (in my opinion) inevitably appear with the commercialisation of the publication process. How long before we see "Publish three original research reports and get one free commentary"?



    (*) Any other formulation of this saying tends to be very confusing to non-native speakers of English!

    ReplyDelete
  6. You might also be interested in: http://scitechsociety.blogspot.dk/2014/01/market-capitalism-and-open-access.html

    ReplyDelete
  7. What do you think about the publication model in Machine Learning, especially the new International Conference on Learning Representations (ICLR)?

    ReplyDelete
  8. Al, I have no idea how that works. Maybe you can explain that model to us?

    ReplyDelete
  9. Here's a brief overview of how publications work in Machine Learning:

    -The most important prestigious venues are conference proceedings, journals aren't very prominent (with a few exceptions).

    -All papers are shared online (no paywalls)

    -Double blind. Reviewers bid on papers and reviews are aggregated by area chairs.

    -I believe that most of the administrative fees are covered by conference attendance fees. Attending the conference has no connection with getting published.

    -The best few papers are selected for oral presentation at the conference. One or two papers are selected for the "best paper award". Other papers are presented as posters.

    ICLR is somewhat more radical in that it's single blind, all papers and reviews are posted online (even rejected papers), and any third party can submit non-anonymous reviews.

    Here's are positive and negative reviews of the system:

    http://yann.lecun.com/ex/pamphlets/publishing-models.html

    http://www.cis.jhu.edu/publications/papers_in_database/GEMAN/Ten_Reasons.pdf

    ReplyDelete