14 December 2015

My (current) position on the PACE trial

I have written this post principally for people who have started following me (formally on Twitter, or in some other way) because of my somewhat peripheral involvement on the PACE trial discussions.

First off, while I try to be reasonably politically correct, I don't always get all the details right.  I've tried to be respectful to all involved here.  In particular, someone told me that "CFS/ME" is not always an appropriate label to use.  I hope anyone who thinks that will allow me a pass on that, from my position of ignorance.

I've learned a lot about CFS/ME over the past few weeks.  Some of what I've been told --- but above all, what I've observed --- about how some of the science has been conducted, has disturbed me.  The people whose opinions I tend to trust on most issues, who usually put science ahead of their personal political position, seem to be pretty much unanimous that the PACE trial data need to be released so that disinterested parties can examine them.

But I want to make it clear that I have no specific interest in CFS/ME.  I don't personally know anyone who suffers from it, and it's not something I've really ever thought about much.  I don't especially want to become an advocate for patients, except to the extent that, having had my own health problems in the last couple of years, I wish every sick person a speedy recovery and access to the finest medical treatment they can get.  So I'm not sure I can even call myself an "ally"; allies have to take a non-trivial position, and I don't think my position here is much more than trivial.  If the PACE trial data emerge tomorrow, I will not personally be reanalysing them.  I don't know enough about this kind of study to do so.

What I do care about is the integrity of science.  You can see this, I hope, if you Google some of the stuff I've been doing in psychology.  Science, imperfect though it is, is about the only rational game in town when it comes to solving the problems facing society, and when scientists put their own interests above those of the wider community, it usually doesn't turn out well.

So, on to the PACE trial... I want to say that I can understand a lot of defensiveness on the part of the PACE researchers.  They have heard stories of others being harassed and even receiving death threats.  Maybe some of them have experienced this themselves.  For the purposes of this post (please bear with me!), I'm going to assume --- because I have no evidence to the contrary, and people generally don't make these accusations lightly --- that the stories of CFS/ME researchers being harassed in the past are true; arguably, for the purposes of this discussion, it doesn't make any difference whether they are true or not.  (Of course, in another context, such claims are very important, but let me try to put that aside for now.)  Apart from anything else, given the size of the CFS/ME community, it would be unreasonable not to expect there to be some fairly unpleasant people to have also developed the condition.  We all know people like that, whatever our and their health status.  CFS/ME strikes people from all walks of life, including some saints and some sinners.

Now, with that said, I am unconvinced --- actually, "bewildered" would be a better word --- by the argument that releasing the data would somehow expose the researchers to (further) harassment.  Indeed, it seems to me that withholding the data plays directly into the hands of those who claim that the PACE researchers have "something to hide", and they are presumably the most likely to escalate their anger into harassment.  I actually don't believe that the researchers have anything to hide, in the sense of feeling guilty because they did something bad in their analyses.  I've seen enough cases like this in my working life to know that incompetence --- generally in the form of a misplaced sense of loyalty to a group rather than to the wider truth and public interest --- is always to be preferred as an alternative explanation to malice, first because malice is harder to prove, and second because it just almost always turns out to be the case than incompetence was behind a screw-up.

About the only reason I can sort of imagine for the argument that releasing the data might lead to harassment of the researchers, is if the alternative were for the question to somehow go away.  That's perhaps a reasonable argument with some political issues; for example, there is (I think) a legitimate debate to be had over whether it's helpful to reproduce, say, cartoons that might cause people to get over-excited, when they could just be left to one side.  But that's simply not going to happen here.  People with a chronic, debilitating condition, and no cure in sight, are not going to suddenly forget tomorrow that they have that condition.  So far, none of the replies to people who have asked for the data, and been told it will lead to harassment, have explained the mechanism by which that is supposed to happen.

The researchers' argument also seems to conflate the presence in the CFS/ME activist community of some unpleasant people --- which, again, for the sake of this discussion, I will assume is probably true --- to the idea that "anyone from the CFS/ME activist community who asks about PACE is probably trying to harass us".  This is not good logic.  It's what leads airline passengers to demand that Muslim passengers be thrown off their plane.  It's called the base rate fallacy, and avoiding it is supposed to be what scientists --- particularly, for goodness sake, scientists involved in epidemiology --- are good at.

A further problem with the arguments that a request for the data --- whether it comes from patients with scientific training, or scientists such as Jim Coyne --- is designed to be "vexatious" or to "lack serious purpose" or that its intent is "polemical" (all terms used by King's in their reply to Coyne), is that such arguments are utterly unfalsifiable.  Given the public profile of this matter, essentially anyone who asks for the data is going to have their credentials examined, and unless they meet the unspecified high standards of the researchers, they won't get to see the data.  (Yes, Jim Coyne --- who, full disclosure, is my PhD supervisor --- can be a bit shouty at times.  But this is not kindergarten.  Scientists don't get to withhold data from other scientists just because they don't play nice.  Ask any scientist if science is about robust disagreement and you will get a "Yes", but if that idealism isn't maintained when actual robust disagreement takes place, then we might as well conduct the whole process through everything-is-fine press releases.)

Actually, in their reply to Coyne, King's College did seem to give a hint as to who might be allowed to see the data, in their statement "We would expect any replication of data to be carried out by a trained Health Economist", with an nice piece of innuendo carried over from the preceding sentence that this health economist had better have a lot of free time, because the original analysis took a year to complete.  This suggests that unless you declare your qualifications as an unemployed health economist, you aren't going to be judged worthy to see the data (and if you come up with conclusions after a week, it might well be suggested that you didn't look hard enough). But the idea that it will take a year, or indeed need specialised training in health economics, to determine whether the Fisher's exact tests from the contingency tables were calculated correctly, or whether the results really show that people got better over the course of the study, is absurd.  Apart from anything else, science is about communicating your results in a coherent manner to the rest of the scientific community.  If you submit an article and then claim that its principal conclusions cannot be verified except by a few dozen highly trained specialists with a year's effort, that's an admission right there that your article has failed.  Of course there will be questions of interpretation, over things like what "getting better" means, but nobody should have to accept the researcher's claims that their interpretation is the right one.  There needs to be a debate, so that a consensus, if one is possible, can emerge.  (Who knows?  Maybe the evidence for CBT is overwhelming.  There are plenty of neutral scientists who can reach a fair conclusion about that, but right now, they are being deprived of the opportunity to do so.)

A further point about the failure to share data is that the researchers agreed, when they published in PLoS ONE, to make their data available to anyone who asked for it.  This is a condition of publishing in that journal.  You can't have the cake of "we're transparent, we published in an open access journal" and then eat that cake too with "but you can't see the data".  PLoS ONE must insist that the authors release the data as they agreed to do as a condition of publication, or else retract the article because their conditions of publication have been breached.  See Klaas van Dijk's formal request in this regard.

These data are undoubtedly going to come out at some point anyway.  The UK's Information Commissioner will see to that, even if PLoS ONE doesn't persuade the authors to release the data.  As the risk management specialist Peter Sandman points out, openness and transparency at the earliest possible stage translate into reduced pain and costs further down the line.

I want to end with a small apology.  I wrote a post yesterday on an unrelated topic (OK, it was also critical of some poor science, but the relation with the subject of this post was peripheral).  Two people submitted comments on that post which drew a link with the PACE trial.  After some thought, I decided not to publish those comments, as I wanted to keep discussion on that other post on-topic.  I apologise to the authors of those comments that Blogger.com's moderation system did not let me explain the reasons why they were not published.  I would happily publish those same comments on this post; indeed, I will publish pretty much any reasonable comments on this post.

14 comments:

  1. As a patient with ME, who was once Head of Maths, I'd like to thank you for this balanced contribution to the debate. I am critical of the quality of the statistics on view in the main PACE papers, and by the total reliance on subjective assessments. The results of the objective assessments published so far show no additional improvement due to Cognitive Behaviour Therapy, so when a tiny graph of the last objective assessment, the step test, was published, I asked for the actual plotted values. That was dismissed as vexatious and is currently under appeal.

    It does bother me that, once we become patients, our capacity to analyze and evaluate data can be dismissed. My degree was in Mathematics: my second subject was Experimental Psychology. I am very aware that in academic terms I am a sparrow in comparison with the others who have achieved the label "vexatious", but I can still manage to perform statistical calculations!

    ReplyDelete
  2. Interesting to read how someone from outside the ME/CFS world views the situation. You make a number of reasoned and logical points, in particular that a number of KCL's arguments are utterly unfalsifiable and bewildering. What patients have seen though, and we see it again here with Professor Coyne's request, is that the authors claim vexatiousness, not because it is in any way vexatious, but because we want to see the data for ourselves, and simply put, they don't want anyone to have the freedom to point out with their own data that they are wrong.

    This is clearly not an okay situation. Whereas the ICO may well support the case for the data being released, as they have done for a different request for part of the PACE data recently, QMUL will appeal it as has happened once already, the tribunal may well side with the prominent institution claiming harassment over the individual. You don't even need proof, just the word alone may be enough to swing it.

    Claims of harassment have been, and continue to be, devastatingly effective, both at ensuring data remains hidden and at stealing the patient's voice - and sadly patients have few people helping them as a result. It's pretty hard being bedbound to stick up for yourself when bad science occurs and it affects you directly, as in this case. But this is different to the past; although patient concern underlies all this, the issue at the top of the pile here is one of scientific process, reliability and openness.

    This is why PLOS's action over this is vitally important. They cannot say, for instance, "we'll leave it to the ICO," because passing the buck like that would be akin to saying "we're fine with it, we'll publish anything." And there is no reason why they should come to that decision, as they clearly already have a policy in place on what should happen. ME/CFS is not a special case, it is science and if you make claims you have to be able to back them up and let it all stand up to proper scrutiny.

    I'm waiting, along with a lot of patients, scientists and journalists to see what PLOS do.

    Thanks again for taking the time to write about the situation.

    ReplyDelete
  3. Much like the PACE authors have redefined recovery to allow deterioration of function on some measures, they have redefined harassment to include requests for data as this document shows: https://www.dropbox.com/s/92m09l9tq55pihh/Behind%20the%20Scenes%20-%20Research%20Collaborative.pdf?dl=0

    Under their heading 'Harassment' (of researchers) they list: freedom of information requests, complaints to the GMC and various institutions, parliamentary questions.

    All requests for PACE data have been treated as freedom of information requests which were then rejected as vexatious. I'm not aware of a single request for data that has succeeded. PACE authors cannot credibly claim harassment via freedom of information requests if they have never shared their data with anyone. Their response to Coyne further undermines their credibility.

    There has been far too little discussion about the PACE trial in academics circles. I recommend David Tuller's excellent articles: http://www.virology.ws/mecfs/

    Thank you Nick Brown for a good article.

    ReplyDelete
  4. Many thanks for your well argued blog post.

    There has never been any evidence produced of death threats, despite these 'threats' being used to label a whole patient group as unreasonable. No person with myalgic encephalomyelitis has ever been prosecuted for making such threats. The threats are hearsay.

    Patients requests for freedom of information are being labelled as ' vexatious ' by the PACE trial researchers, and the requests are considered to be harassment (as is any criticism ).

    Denigrating patients is not a professional way for the doctors paid to care for them to behave.

    ReplyDelete
  5. In a lot of ways this blog seems more casually critical of the PACE researchers than many patients would dare to be, but does so with polite and gentle language which indicates that the writer shares (or is sympathetic to) the values of the researchers, and this lets it slip past. I think that a lot of the complaints of 'harassment' of researchers relate to different values and customs between polite society and those who feel a bit more kicked-around.

    eg:

    "I actually don't believe that the researchers have anything to hide, in the sense of feeling guilty because they did something bad in their analyses. I've seen enough cases like this in my working life to know that incompetence --- generally in the form of a misplaced sense of loyalty to a group rather than to the wider truth and public interest --- is always to be preferred as an alternative explanation to malice,"

    If researchers allow a misplaced sense of loyalty to a group to affect how data is analysed, and what claims are made to sick and desperate patients about the efficacy of treatments, then I would describe that as corrupt. And that would be being polite.

    "The researchers' argument also seems to conflate the presence in the CFS/ME activist community of some unpleasant people --- which, again, for the sake of this discussion, I will assume is probably true --- to the idea that "anyone from the CFS/ME activist community who asks about PACE is probably trying to harass us". This is not good logic. It's what leads airline passengers to demand that Muslim passengers be thrown off their plane. It's called the base rate fallacy, and avoiding it is supposed to be what scientists --- particularly, for goodness sake, scientists involved in epidemiology --- are good at."

    It's also called bigotry. After a few years of it, it does start to get frustrating.

    For patients afflicted with a stigmatised health condition, I think it is a mistake to speculate about the actions and motivations of influential researchers. There are serious problems with the PACE trial's design, conduct and presentation of results, yet easily verifiable facts about this have long been breezily dismissed by the PACE trial's defenders, and sometimes still are even now other academics have started promoting patients' concerns for us. I think that the problems around the PACE trial do reflect wider problems within medical research and British society, and that by cautiously building from a solid foundation of criticism patients and their allies might be able to bring about positive changes that benefit far more than just those with CFS. I expect considerable resistance, not least because a lot of those changes will go against the self-interest of researchers who would prefer a comfortable life over "the wider truth and public interest".

    For those who unable/uninterested in putting in the work needed to truly understand the issues around CFS and the PACE trial I think that Nick's approach of simply wanting to see data released so that independent analysis and genuinely open discussion and debate can take place amongst those who are interested is a sensible position.

    ReplyDelete
  6. Thank you for writing about this topic. I think most reasonable people with ME/CFS would, like you, assume that there genuinely will have been some harassment targeted at the PACE authors. As you say, in such a large population of patients - approx. 250,000 in the UK alone - we'll have our fair share of people with personality disorders, just as the cancer population, the MS population and the arthritis population will.

    But, like you, I think most scientists will be wondering whether a “harassment” narrative as a reason for refusing to share the PACE data really stands up to logical examination. Requesting data isn’t harassment. It’s normal in science. The assumption that every data request is being made with malicious intent unless proved otherwise is just extraordinary. That KCL should apply the "vexatious" label to a serious, senior and prominent academic such as Professor Coyne just beggars belief.

    Dr Richard Smith, former editor of the BMJ, has tweeted: "King's College is surely making a mistake in holding back data from a controversial trial. "

    https://twitter.com/Richard56/status/676415734856220672

    Leading methodologist Dr Brian Nosek tweeted: “King’s College data sharing refusal and rationale are antithetical to science”.

    https://twitter.com/BrianNosek/status/675707895892185088?ref_src=twsrc%5Etfw

    Something is broken in science that has let the PACE authors get away with fending off independent analysis of their data for all these years. It's time for that to stop. There are a quarter of a million UK ME/CFS patients who haven’t harassed anyone and would never dream of doing so. We deserve answers about this trial: our health is at stake.

    I really hope that PLOS One will do the right thing and retract the paper if the data are not forthcoming.

    ReplyDelete
  7. Thanks for a terrific blog that exposes with blanket harassment defence brilliantly, and deals with a lot of complex issues in a very readable way (particualry helpful for those of us with cognitive problems).

    ReplyDelete
  8. It's worth adding that whatever the authors fear in terms of damage to their reputations by people with questionable intent, nothing could possibly outweigh the damage they're doing to (a) their own reputations, (b) the reputation of the PACE trial and (c) the reputations of Queen Mary University of London, Kings College London, The Lancet, Psychological Medicine, Lancet Psychiatry, the MRC, and now PLOS One, by withholding their data.

    They should immediately hand over the data for their own sakes as well as everyone else's. This trial was paid for with £5 million of UK taxpayer's money and the PACE authors have destroyed its credibility with these data-withholding antics. Time to let others step in and perform independent analysis.

    ReplyDelete
  9. Your capacity to maintain an open mind on this issue is refreshing. Since this article is about science and the integrity of science, I refer you to documentation of emails and minutes, concerning plans for the contrived promotion of harassment as an issue, between PACE authors and SMC: https://www.dropbox.com/s/92m09l9tq55pihh/Behind%20the%20Scenes%20-%20Research%20Collaborative.pdf?dl=0.

    ReplyDelete
  10. Dear Nick,
    Like you, I came across the PACE trial at a time when I had no direct involvement in the ME/CFS field (I am a retired professor of medicine who does now advise the ME research community). I am pleased to see that you agree that the data should be made public.

    I think you may be unaware of the detail of politics here, however. The situation was made clear to me because I first heard of PACE in the form of a diatribe against patients by one of the authors. I was later 'leant on' by an eminent colleague. The PACE authors did not blunder unsuspecting into hostility. They had generated the hostility in the first place. I have come to see that the hostility is almost entirely justified frustration at very bad science that is persistently marketed as robust by the authors who choose to ignore all criticisms from scientific colleagues. The patients are right to be up in arms. I have never come across anything quite like this before and have never found the need to express opinions like this before. It is not a matter of the ordinary ups and downs of real life in science. It is becoming surreal.

    ReplyDelete
  11. There can be no pass on CFS/ME. It is not about politics but about science. With science it is always about check your premise. ME is not CFS. It is not just about names. PACE trial was on CFS, not ME. It is the mixing switching terms that misinform and then misapply, which ALL so called US so groups contribute to and promote. ALL of these groups work against true ME patients, whom they then pretend to represent.

    There is no such recognized illness as ME/CFS or CFS/ME (no WHO recognition or ICD coding). These are made up terms with different meanings. This is sloppy and unscientific but seems to suit political agendas.

    So as Head of maths, if ME is not CFS, how can we then go on about CFS/ME or ME/CFS. Like saying X is not Y, then talking about X/Y.

    This is what has become surreal.

    ReplyDelete
  12. What is not being much discussed is that the PACE Trial data represents a matter quite literally of life or death for many individuals due to the fact that this trial has largely determined what care and treatment is available to patients in nations where the overwhelming majority of persons rely for health care on highly centralized national health services that are basing their treatment protocols on the conclusions of the PACE Trial.
    In addition, the PACE Trial has provided substantiation internationally for health and disability insurers who wish to limit treatments or compensation they will allow to their policy holders.
    Only individuals rich enough to pay out of pocket for treatments have the option of seeking what is not proscribed either by governments or insurers based on the PACE Trail's (internationally very influential) conclusions.
    Such individuals have the option of getting well, though not without great expense and difficulty. This has been documented by the Times of London in the account of a wealthy young aristocrat who fell ill with mono while in university and then became severely disabled. Eventually she found her way to a private facility that provided biological measures to restore homeostasis to systems afflicted with physical disease. Counterparts without money can only receive CBT or GET; review of PACE Trial data will indicate what expectations patients may have from such treatments. Anecdotal data and the FiNE Trial suggest great expectations would not be in order. One awaits the examination of PACE data for a perhaps more -- or less -- encouraging result.

    ReplyDelete
  13. Like you I have no direct involvement with ME but I have been involved in CBT for many years and consider myself an advocate of science. I have to say I consider the issue of harassment to be totally irrelevant, to send death threats is a criminal offence which should have been reported to the police who have the technology to identify the source. There have been investigated cases in which scientists have made such claims for different reasons and have been found to be lying. If such threats have been made there will be evidence of this.
    However what really sparked my interest was reading about the FINE trail, the sister trail of PACE. I found myself reading about a psychological intervention described as CBT, but it was unlike any CBT I've ever hear of. In fact I found it embarrassing and thought it was potentially damaging.
    Then there is an ethical principle in medical research that it should not be used in way that could be damaging to the patients, this research will no doubt be used in the control of disability benefits and may be very damaging to sufferers. The links of some of the research team to the Department of Work and Pensions, originally undeclared, and the fact that this department provided some funding, the first time it had ever done so for medical research, was very worrying.
    The greater part of the funding which amounted to around £4 million of tax payers money was awarded by the Medical Research Council. They also have policies about the conduct of research paid for by public money which includes a commitment to openness and honesty which again the research team have totally ignored, seemingly with the blessing of the MRC.
    I agree that most problems in research can be attributed to carelessness or incompetence but as this team try with increasing desperation to hide their work, I am far less inclined to believe this of this "study".

    ReplyDelete
  14. There is another recently published article - 4 years after media headlines that climate sceptics are conspiracy theorists.
    https://www.theguardian.com/environment/blog/2012/jul/27/climate-sceptics-conspiracy-theorists

    “The notion that skeptics believed something so silly as the faking of the moon landing is yet another myth essentially concocted by the researchers.” – Lee Jussim

    yet how many hundred of people will ever read the following, vs the hundreds of thousands of people that read Lewandowsky work in the media.


    Can High Moral Purposes Undermine Scientific Integrity? – chapter

    J. Forgas, P. van Lange & L. Jussim, Sydney Symposium on Social Psychology of Morality)

    some extracts (please read whole artcle) :

    The Curious Case of Condemning Climate Skeptics as Conspiracy Theorists (Lewandowsky, Oberauer, & Gignac, 2013)

    Into this mix stepped Lewandowski et al. (2013) with a paper titled, “NASA Faked the Moon Landing – Therefore (Climate) Science is a Hoax” – which strongly implies that people who doubt global warming believe bizarre conspiracy theories. As Lewandowsky et al. (2013, p. 622) put it, “… conspiratorial thinking contributes to the rejection of science.”


    The implication that climate skeptics believe in the faking of the moon landing is another phantom fact. Out of over 1145 respondents, there was a grand total of 10 who believed the moon landing was faked. Among the 134 of participants who “rejected climate science,” only three people (2%) endorsed the moon-landing hoax. The link asserted in the title of the paper did not exist in the sample. Correlations primarily resulted from covariance in levels of agreement among reasonable positions (i.e., people varied in how much they disbelieved hoaxes and in how strongly they accepted science). It would be fair to characterize their results as indicating “the more strongly people disbelieved hoaxes, the more strongly they believed in climate science” – people varied in how strongly they rejected hoaxes and accepted science, but almost no one believed the moon hoax.

    Understanding when people are and are not persuaded by science is an interesting and important area of research. But this curious case highlights the threat to scientific integrity that can stem from high moral missions. The notion that skeptics believed something so silly as the faking of the moon landing is yet another myth essentially concocted by the researchers. No matter how worthy the efforts to advance policy changes to combat human sources of global warming, the goal of “getting it right” is jeopardized when scientists claim their data shows their ideological opponents hold silly beliefs when they, in fact, do not. As such, this constitutes another example of high moral purposes undermining scientific integrity.



    http://www.rci.rutgers.edu/~jussim/CanHighMoralPurposesUnderminescientificIntegrity.docx (doc)

    http://www.rci.rutgers.edu/~jussim/Jussim%20et%20al,%202016,%20High%20Moral%20Purposes.pdf (scan)

    ReplyDelete