An update on #losadagate

Since I've been asked a few times what's happening on "the Losada matter", I thought I'd write it up here so I can point people to it.

The story so far: On July 15, 2013, American Psychologist published our article and Dr. Fredrickson's response [PDF] online.  We had expected to have the last word, in a standard A-B-A sequence, but this was initially denied.  American Psychologist's logic was that the original 2005 article by Fredrickson and Losada [PDF] was A, our article was B, and Dr. Fredrickson's reply to us closed the matter.

We were not very happy about this, to say the least.  Our article addressed two other papers published in other journals (Losada, 1999; Losada & Heaphy, 2004 [PDF]) as well as Fredrickson and Losada's.  Furthermore, we initially had our article summarily rejected because (according to AP) it was a comment, and comments are only accepted for 3 months after the appearance of an article.  (Quite how science is meant to self-correct if any mistakes which are not identified within 90 days become officially true, is an interesting but separate issue.)  Only after we wrote directly to the CEO of the American Psychological Association did we get an offer for our article to be reviewed (which was done in a very thorough and professional manner, I should add).  We expected that Dr. Fredrickson would be invited to reply --- indeed, we encouraged this, on the basis that we would have the last word, as is customary in these discussions.  So when we were later told that we were not going to be offered a final response, we were not very happy.

We considered writing our reply and sending it to a different journal, combined perhaps with some complaining via blogs or the news media.  However, in the end we decided to grit our teeth and embark upon the APA's publications appeal process.  It took a lot of effort (almost all by Harris Friedman), but in the end, we were allowed a 1,000-word reply --- and with some further negotiating, this became 2,000 words.  (I note, in passing, that when AP's position was that Dr. Fredrickson's reply was the end of the A-B-A sequence, she was allowed 5,800 words and 60 references, almost none of which addressed the points made in our article.)

After review, our response to Dr. Fredrickson's reply to our article is now in press at American Psychologist.  You can see the final draft version here [PDF].

But wait --- there's more!

After our article, together with Dr. Fredrickson's reply, appeared in the December 2013 print edition of American Psychologist (which, as part of the benefits of APA membership, has a print run of about 100,000, meaning that my photo is now in the bathroom of most of the psychologists in North America), several people wrote to the editor with comments.  Five of these were selected for publication.  After some further discussion with the editorial team, we obtained the right to respond to these comments, which one would have thought would be automatic... never mind.  We replied to each of the comments, concentrating in particular on the three which critically addressed our article.  Our reply is now also in press, but I don't want to share the full draft in this case because to put it in context requires reading the readers' comments, and I'm not in a position to share them.  (But for those who enjoyed the first article: there are a few zingers in this one to look forward to.)

I'll end on one of the things that didn't make the cut of our reply to those readers' comments.  On p. 821 of the December 2013 issue of American Psychologist, Fredrickson and Losada jointly authored the "withdrawal" of Losada's mathematical model "as invalid".  Yet, if you visit the web site of Losada's consulting company, you will find the model everywhere, right down to the stylised "Lorenz butterfly" which forms the favicon of the site.  Quite how the model can simultaneously be withdrawn as invalid while simultaneously forming the backbone of a training programme that has apparently been delivered to companies such as Apple, Boeing, GE, and AT&T (cf. the "Clients" tab of the Losada Line Consulting site) is, I suppose, just another miracle of mathematical logic to which we mere mortals are not to be allowed access.

PS: Hot off the press: American Behavioral Scientist, which published the Losada and Heaphy (2004) article, has recently issued an Expression of Concern about it.  The last paragraph is particularly intriguing; have you ever heard of the editors of a journal, whether permanent or a guest team for a special issue, taking an incoming article and asking a grad student to polish it up a bit for their target audience?  (I note that one of the guest editors of the special issue in question, Kim Cameron, is a senior professor at the Ross School of Business at the University of Michigan, where Emily Heaphy was studying at the time.)

Study pre-registration? Aren't we all doing that already anyway?

This is not an original idea.  I saw it the other day, somewhere on the web.  I haven't been able to find it again via Google; maybe it was inside some proprietary comment engine on someone's blog post.

It may well not even have been original with the person whose version I saw the other day.  Very little is truly original.  Maybe that person copied it; maybe they came up with it independently.

Anyway.

The discussion was about pre-registration of studies.  Most people seem to think pre-registration is a good idea, as a way to help fight some of the current problems in science, especially the social sciences, especially psychology, especially (I would argue) social psychology.  If you have to pre-register your study, including what you're going to look for, that ought to prevent or reduce all kinds of questionable research practices.  If you have to state your hypotheses before you start, you can't HARK later.  If you state how many people you're going to have in your sample, you can't apply your own stopping rules, or several other researcher degrees of freedom.

The difficulty seems to be in establishing a register of all these studies.  There must be tens or even hundreds of thousands of studies conducted worldwide each year, in the psychology and related departments of thousands of universities worldwide.  Just the logistics of creating a central repository for that would be overwhelming.  Don't expect the journals to do it, either.  Although a few journals have announced their intention to support replication via the concept of a "registered report," this is completely unscalable; it more or less requires the journal to guarantee in advance that the article will be accepted, regardless of the results.  And while some journal editors are starting to pay lip service to the idea that it's important to publish null results, the incentive structure of impact factors means that these are likely to remain token efforts, a drop in the bucket alongside the flashy, underpowered Gladwell-bait that nobody will be able to reproduce (but the authors will be way over the line into Tenure County before anyone notices, if they ever do).

And yet, as this person whose contribution I can no longer find (seriously, if you wrote something about this on the Internet in the last 10 days, point me at it and I'll give you full credit) points out, pretty much every PI on any research topic involving human subjects already preregisters their study today via the Institutional Review Board form.

Everyone hates filling out IRB forms.  They ask you dozens of questions that have nothing to do with protecting potentially-vulnerable participants and everything to do with covering every imaginable base in case the university gets sued.  (Never mind that most such lawsuits would be dismissed as frivolous, or that if you did something really bad, the form wouldn't count for a row of beans.)  These forms are so irrelevant to protecting the vulnerable, they even ask you for ridiculous things that are none of the IRB's business, such as your research design and your hypotheses...


Whoa.

You're already forced to register your hypotheses and your design before you can even begin your study.  Your institution is going to keep a copy of this for a million years, or at least until the great-great-grandchildren of any of the participants are dead.  Probably 80% of the important stuff that a hypothetical universal pre-registration protocol might contain, at some point in an unknown bureaucratic future once the APA and APS and other interested parties have ummmed and ahhhed for a couple of decades, is being recorded right now for almost every study being done in the world, even those cute little quantitative exercises done by Masters' students as course assignments.  (Actually, if the APA makes as big a mess of this as they have done with their "Open Data" journal, that 80% will probably turn into something more like 200%, but that's another story.)

To illustrate just how much information PIs are already being asked to provide, I picked a couple of results from a Google search for "IRB form" at random.  Cornell's IRB form asks, "Please provide a lay summary of the study, including the purpose, research questions and hypothesis to be evaluated".  The University of Minnesota's compact, succinct 18-page form requires PIs to describe "the objective(s) of the proposed research including purpose, research question, hypothesis and relevant background information etc". As a Master's student at the University of East London in the UK, doing a cute little quantitative exercises as a course assignment, I had to fill in a form that asked for "... a statement on the aims and significance of the proposed research, including potential impact to knowledge and understanding in the field (where appropriate, indicate the associated hypothesis which will be tested). This should be a clear justification of the proposed research, why it should proceed and a statement on any anticipated benefits to the community".  Helpfully, the form then adds "(Do not exceed 700 words)".  700 words?  When did you last read an article with 700 words, or even half that number, of justification of the hypotheses and the research design?

So, here's the challenge to journal editors and IRB chairpersons: this is an approach that would allow you to implement a meaningful, albeit undoubtedly imperfect form of study pre-registration which would have an immediate, overnight, substantial effect on the quality of published research, for a total worldwide cash outlay of approximately zero.  All that has to happen is that the editors start to require that authors, instead of mechanically copying and pasting the words "Ethical approval was obtained from the IRB of Pabulum College" from their last 10 articles, actually submit a copy of the form granting that approval for the attention of the reviewers.  And of course, the IRBs have to be prepared to give out this data --- but what reason would they have to refuse?  Any specific information concerning vulnerable groups could be redacted, along with the PI's mobile phone number; but in most cases, the information on IRB forms is no more sensitive than last week's cafeteria menu.  And surely if the IRB forms are as big and complex and bureaucratic as they are, it's because the IRB has a mission, beyond protecting human subjects, of protecting the institution's reputation more widely --- because after all, it would be very embarrassing for a major university if it became associated with the repeated publication of dubious research.

But, as H. L. Mencken said, "There is always an easy solution to every human problem --- neat, plausible, and wrong".  So I presume I'm missing something obvious here.  Please let me know what that is.

Attaining national happiness through chemistry

Apparently, Scandinavia is big in the UK right now.  (Something "foreign" is always big in the UK, it seems.)  And when something is big, the backlash will be along very soon, as exemplified by this Guardian article that I came across last week.  So far, so predictable.  But while I was reading the introductory section of that article, before getting to the dissection of the dark side of each individual Nordic country, this stood out:

[T]he Danes ... claim to be the happiest people in the world, but why no mention of the fact they are second only to Iceland when it comes to consuming anti-depressants?

Hold on a minute.  When those international happiness/wellbeing surveys come out each year, Denmark is always pretty close to the top.  So I checked a couple of surveys of happiness (well-being, etc.).  The United Nations World Happiness Report 2013 ranks Denmark #1.  The OECD Better Life Index (BLI) 2013 rather coyly does not give an immediate overall ranking, but by taking the default option of weighting all the various factors equally, you get a list headed by Australia, Sweden, and Canada, with Denmark in seventh place (still not too shabby).  The OECD report adds helpful commentary ; for example, here you can read that "Denmark, Iceland and Japan feel the most positive in the OECD area, while Turkey, Estonia and Hungary show lower levels of happiness."

So what's with the antidepressants?  Well, it turns out that the OECD has been researching that as well.  Here is a chart listing antidepressant consumption in 2011, in standardised doses per head of population per day, for 23 OECD member states.  Let's see.  Denmark is pretty near the top.  (It's not second, as mentioned in the Guardian article above, because the author of that piece was using the 2010 chart.)  And the other top consumers?  Iceland first (remember, Iceland is among the "most positive [countries] in the OECD area"), followed by Australia, Canada, and (after Denmark) Sweden.  That's right: the top three countries for happiness according to the UN are among the top five consumers of antidepressants in the OECD's survey(*).  And those countries showing "lower levels of happiness"?  Two of the three (Estonia and Hungary) are on the antidepressant list - right near the bottom.  Perhaps they'd be happier if they just took some more pills?

I decided to see if I could apply a little science here.  I wanted to examine more closely the relationship between antidepressant consumption and happiness/wellbeing/etc. So I built a dataset (available on request) with a rank-order number for each of the 23 countries in the antidepressant survey, on each of several measures.  Then I asked my trusty computer to give me the correlation (Spearman's rho) between consumption of antidepressants and each of these measures.  Here are the results:

Measure	Correlation with antidepressant consumption (Spearman's rho)	p
UN World Happiness Report 2013	.590	.003
OECD BLI 2013 - Life Satisfaction	.621	.002
OECD BLI 2013 - Self-reported health	.730	.000
OECD BLI 2013 - Educational attainment	-.077	.727
OECD BLI 2013 - Leisure time	.106	.631
OECD BLI 2013 - Overall (equal weighting)	.653	.001

For the uninitiated: the first three lines, and the last line, show substantial and statistically significant correlations between the item concerned and antidepressant usage. The fourth and fifth lines show no significant correlation. Overall summary: countries with high antidepressant usage also report high happiness/life satisfaction/health.

Note that this is not a case of "everything being correlated with everything else" (as the great Paul Meehl put it).  Only certain measures from the OECD survey are significantly correlated with antidepressant consumption.  (I encourage you to explore the measures that I didn't include.)

Ah, I hear you say, but this is only correlational.  When two variables, A and B, are correlated, there are usually several possible explanations.  A might cause B, B might cause A, or A and B might be caused by C.  So in this case, consuming antidepressants might make people feel happy and healthy; or, being happy and healthy might make people consume antidepressants; or, some other social factor might cause people to consume antidepressants and report that they feel happy and healthy.  I'll let you decide which of those sounds plausible to you.

Now, what does this prove?  Probably not very much; I'm not going to make any truth claims on the basis of some cute numbers.  After all, it's been "shown" that autism correlates better than .99 with sales of organic food.  But here's a thought experiment for you: Imagine what the positive psychology people would be telling us if the results had been the other way around --- that is, if Australia and Denmark and Canada had the lowest levels of antidepressant consumption.  Do you think it's just remotely possible that we might have heard something about that by now?



(*) The OECD has 34 member states, of which some, such as the USA and Switzerland, do not appear in the antidepressant consumption report.  All correlations reported in this post are based on comparisons in rank order among the 23 countries for which antidepressant consumption data are available.

Antidepressant consumption data here [XLS]
UN World Happiness Report 2013 here
OECD Better Life Index 2013 data here


[ Edit 2014-02-20: fixed the figures in the tables, following the discovery of a couple of minor typos in the data. Correlation of UN Happiness Report changed from .595 to .590. Correlation of OECD Better Life Index 2013 - Life Satisfaction changed from .617 to .621. Correlation of OECD Better Life Index 2013 - Self-reported Health changed from .734 to .730. ]

Lessons from a missed plane

For the first time in my life, I missed a plane this week. But it was a chance for a little moment of growth.

I was booked on the 15:40 bus from Stratford (London) to Stansted. Normally this takes 45-50 minutes, so I arrive over two hours before my plane departs at 18:40. I don't mind leaving early as I generally have lots to read, and this way I have a nice margin of safety.

Ha. The A12 was closed due to a potential suicide (try not to read too many of the comments on that article, as the stupid may make your head explode) and every road in and out of East London was pretty much blocked solid. The bus which arrived at 15:55 was actually the scheduled service from 15:10, so the Irish man sitting next to me who'd got on at Liverpool Street was already looking very nervous about his 17:00 departure to Dublin.

As we crawled up Leytonstone High Street, a German passenger approached the driver. He wanted to get off, because he knew that he had no chance of making his flight, so he preferred to get the Tube back into town rather than make a fruitless journey to the Cambridgeshire/Essex border and back. The driver was reluctant ("elf 'n' safety"), and I was half-expecting a shouting match, but to my surprise he took up the German guy's suggesting to call his depot and he was given permission to let people off at the next bus stop.

After this, we carried on, well below walking pace, for what seemed like forever, while 50 kilometres to our north, gates were closing and planes were taking off without us on board. A man aged about 30 was sitting across the corridor from me and asking the driver if he, too could get off.

I asked him when his flight was; he told me the time, which put him in the "unlikely" bracket - towards which my own departure was also starting to slip. I suggested that he stay on the bus anyway, as I intended to do: you never know, I said, the flight could be delayed, and if you get off the bus and later find that the plane was an hour late, you'll kick yourself. He though about it for a moment and seemed convinced. "Driver", he announced, "I'm going to follow this gentleman's suggestion, and maybe I will make it to my plane".

The two of us, seated in the front rows of the bus, chatted to the driver for the rest of the journey; he seemed pleased to have non-hostile customers between him and some of the more nervous-looking people behind him, on what he said was his worst-ever journey in five years as a bus driver. (During the last ten minutes of the trip, he over-ran his maximum 4.5 hours of driving time. I don't think that he would have pulled over and taken his mandatory 45-minute break anyway, but perhaps our input helped. Mark, mate, you did a great job.)

We arrived at Stansted at 18:50, ten minutes after the scheduled departure time of my flight. To my surprise, it was still up on the departure boards, saying "Check-in closed". No other flight with a departure time within half a minute was up there, so I realised that it was most probably delayed. I didn't have a bag to check in, so I headed straight to security, which took far less time than usual. Into the departure area, the flight is still up... "Gate Closed". Yes, but... maybe they're still counting boarding cards. No gate number, though. I asked somebody whose job wasn't to know this sort of thing (you have to pick up a phone, as there is no information desk at Stansted; presumably it would take up valuable selling-you-stuff room), and she guessed a gate number. I ran down there, getting extra directions on the way from people in Ryanair uniforms. Got to the gate... damn! The plane was sitting there, the door had just closed, and nobody was to be seen.

I trudged back the whole length of the terminal - about a kilometre, I reckon - calling Mrs Brown to ask her to get me a ticket for the next day's flight before it filled up. Then we had some fun and games to get back "landside", through some secret doors, and I bought a bus ticket back to London.

I slept all the way. When we arrived, I grabbed my bag and walked to the Tube station, right next to the bus stop. I felt a tap on my elbow; it was the man who had decided to stay on the late bus, who had come back at the same time as me. I said, "I guess you didn't catch your plane, then". He smiled and said "No, but I don't regret having stayed on the bus. I might have caught the plane, and I know I did everything I could. Thank you." And he shook my hand.

It didn't stop missing my flight from costing me over a hundred pounds, but I think the world was a very slightly better place than it otherwise might have been.

Bounded rationality and the Ryanair boarding process

I’m tall. 1.93 metres or 6 feet 4 inches, although being very very old, I've probably started to shrink a bit. Anyway, when I fly, I like to have a decent amount of leg room, which typically means sitting in the exit row.

On most flights, this is a matter of outrageous fortune. Last time I booked a transatlantic flight, Lufthansa allowed me to choose my seat online, but although the flight was 7 months away, the exit row seats had all gone. (On the return trip, I asked the check-in person at the airport very nicely, as an afterthought, if there were any exit row seats available; he looked, smiled, tore up my boarding card, gave me a new one, winked, and said, "Don't tell anybody". I got 6 hours of sleep on a 9-hour flight, which is a record. Presumably somebody got bumped...)

Now, like a lot of people in Europe, the airline I fly with most often is Ryanair. For some reason, people boarding Ryanair flights do not head straight for the exit rows. I'm not sure why this is - maybe they like to be able to put things under the seat in front, or perhaps Ryanair passengers customers don't think that they will want to open the door in an emergency - but I'm not complaining. I usually get a "long leg room" seat, and if I don't, well, it's usually only a 75-minute flight anyway.

I could, of course, more or less guarantee an exit row seat by paying €5 to be first on the plane (or rather, at FKB, first on the bus which takes you to the plane, so you probably aren't first on the plane since you're stuck in the depths of the bus). But that really does feel like an extravagance. (Incidentally, I'm not one of those people who complains about all the "extras" on a Ryanair ticket. I've been paying for my own air fares for thirty years, and I would rather pay one of the nice Mr O'Leary's itemised bills than take a nice, gentle, all-inclusive hammering for three times the price, which is what I had to do until he and the equally nice Mr. Stelios came along.)

You see, the thing I hate most about boarding a Ryanair flight - or any other plane, for that matter - is pointlessly standing for 20 minutes in a queue at the gate. My place on the plane is reserved. If I'm travelling with Mrs. Brown, she's quite happy not to sit next to me while I strike up laboured spontaneous conversations with those around me, so we don't need two seats together. I want to get on last, preferably not even waiting at the foot of the steps of the plane. Fortunately, everyone else seems to be prepared to queue up for me, despite getting almost no benefit out of it.

There's now another good reason to be last. As of two weeks ago, Ryanair have changed the rules about the exit row. You now have to pay €10 to sit there (for that price, you also get the "priority boarding" option). At first, I was tempted to join my fellow passengers in their mutterings of "robbing bar stewards", as they were politely prevented from sitting in the good seats by the long-suffering cabin crew, and I'm certainly not going to be paying €10 extra for a short flight; but then, I just booked a flight to the Canaries, and €10 to avoid four and a half hours of cramp, and/or the urge to fight past people to get to row 16, is well worth it.

This raises an interesting question, however. The exit rows are not in a separate cabin, and €10 is not a lot of money compared to the price of the ticket. That means that Ryanair will sell every single one of the 189 seats on the plane if they can, even if nobody pays the premium for legroom. So what happens when a full planeload of people get on when 18 of the seats are reserved?

I got chatting to a cabin crew member at the bus stop while waiting for the coach up to Stansted. She told me that their instructions are to keep the exit row seats for last. (They don't always know exactly how many passengers will turn up, so it makes sense to protect the value of the €10 upgrade by not giving these seats away until the have to.) So, if you want to have a chance of an exit row seat, wait until everyone else has boarded ("After you" - "No, after you"), and hope the plane is full.

I suppose that there could be some unintended consequences of this slightly perverse incentive to stay sitting down, rather than boarding the plane like a co-operative citizen. I intend to continue to be among the last to board the plane anyway.

AnyDatePicker

Part of my job involves maintaining a Web application which I wrote almost entirely from scratch. I designed the database, wrote pretty much all of the PHP and HTML, and even put my toe in the water of CSS. A colleague now supports part of the application, but v1.0 was more or less entirely my own work.

Unless, that is, you count the date picker. You know what I mean: the little window which opens on a travel site, say, so you can say when you want to leave and when you want to come back. When I was writing the application, I hadn't yet learned Javascript, so I scoured the Internet looking for a date picker that could do times as well as days, have the week start on a Monday, was multilingual, and free software.

Well, 2 out of 4 ain't bad. I found one which did the first two, scraped together just enough Javascript to allow me to send the day name abbreviations (M, T, W, etc) to it in different languages, and ignored the word 'Copyright' because, well, er, we're non-profit and nobody will find out.

That was nearly four years ago. In the meantime, I've found some things I don't like about it, apart from the fact that I would like our application to go open source and then somebody might just notice that I stole the code. It uses Javascript's date objects, so you can't ask people for their date of birth with it unless you're running a school, because Javascript's dates start at 1/1/1970. And it opens a new document window, which often turns into a tab these days, depending on how the user has set up their browser. So we get support calls: "I can't see the calendar" (as most users call it), and when we look, they've got 35 copies, each in its own tab, as a result of frantic clicking.

Anyway, about a month ago I was on a hotel's site and I noticed that the date picker appeared and disappeared in an instant. (It certainly wasn't the first one I've seen like that, but presumably I was in a receptive mood.) It occurred to me that it was running in a DIV rather than a window. I thought this was brilliant and decided to write my own, so that my application could be "pure" (and people wouldn't open 35 tabs trying to specify a date).

Remembering how much of a hassle it was to use the one which I stole (OK, OK, if you break into someone's house and steal their DVD recorder, you don't get the instruction book), I decided to make my date picker as flexible as possible, so that other people could use it without having to touch the code (although you do need to write a couple of Javascript data declarations yourself).

The result is called AnyDatePicker. I don't know what this means, except that it does let you pick any date in any year from 1582 (the start of the Gregorian calendar) through 9999. It also has a lot of ways to allow you to control exactly which dates and times people can choose; for example, it would be pretty good for a system where people book 15-minute appointments and you have different opening hours for Fridays, weekends, and holidays.

Please feel free to check it out.

Podslurping

I learned a new word today: Podslurping.

I've been having fun with StatCounter seeing how many people have been hitting this blog since the Conficker worm made people take the whole business of securing their networks against memory stick worms seriously. (The answer is: about 15000 in the last 7 days.)

One of the sites which referenced my original post was this one at the Postdam Institute for Climate Impact Research in Germany. It notes that with Autorun.inf disabled, Podslurping is made harder.

So what is Podslurping? Well, at its simplest, it consists of plugging a USB storage device (of which an iPod is just one example) into somebody's PC and copying lots of data from its disk, or the network to which the PC is connected. That hardly seems worth giving a name to, but the clever part comes if you automate it. You can write an Autorun.inf file which will start the copy to the USB device as soon as you plug it in, without any need to access the keyboard. All it needs is a reasonable copy program and a few lines of a .BAT file.

So now you literally only need three seconds unsupervised access to the PC on two occasions (one to plug the device in, one to unplug it half an hour later) and you can steal all of the data from it, without having to log in or risk detection by hanging around in the office, leaving a command prompt window open on the screen, etc. If the PC has USB ports on the rear, you don't even have to walk round to the side of the desk where your victim sits; in fact you could probably drop your phone and slip the USB device in while the user is sitting there.

So if you have issues with people potentially stealing data, disabling Autorun might be a useful extra precaution to take.