Should researchers make money from books?

I missed this tweet when it appeared yesterday. By the time someone pointed me to the rather animated thread that followed, it had gone quiet again. (Daniël put out an earlier tweet on the same subject here, but the follow-up thread was a lot shorter.)

Writing a book based on stuff you've learned as an academic and selling it, is like a road worker building a road and asking you for toll.

— Daniël Lakens (@lakens) March 17, 2017

I think it's fair to say that Daniël didn't get a lot of takers for his idea. (I'll let you read the thread/s yourself, as I find that one embedded tweet in a blog post is already annoying enough, especially when reading on a mobile device.) There were discussions about the remuneration system for US academics whereby many of them don't get a salary for three months in the summer (this seems utterly weird to Europeans, but our American colleagues just treat it as part of the landscape), as well as questions about intellectual property.

At one point Daniël revealed that if you want him to come to your university or conference and give some version of his excellent MOOC on statistical inferences (I took it, I passed, I am now quite a bit less ill-informed on several statistical topics) then he will charge you $1,000.  But I think that he would argue—and I certainly wouldn't object—that being compensated for your time to show up somewhere in person is not unreasonable.  (I asked Daniël to comment on this post before I published it.  He pointed out that he does plenty of other teaching for free.  He also adds that the best way to follow his course is to take it online, because there are no "ums" and "erms", and you can schedule your own bathroom breaks.)

Daniël's argument, if I've understood it correctly, is based on the following points:
1. The general principle that knowledge should be freely available.
2. The acknowledgement that in many (most?) cases, the money that paid for either (a) the specific empirical research that led to the findings being discussed in the book, or (b) the basic academic job of the person writing the book, came from taxpayers. I don't think I've ever discussed "Politics with a capital P" with Daniël, not least because he and I don't get to vote in the same country, but I hope he will tolerate me guessing that he is probably somewhere to the left of centre. I find it particularly commendable that he frequently communicates in public the awareness that the money that pays his salary does not fall from the skies every time it rains in Rotterdam(*).
3. A belief that what you have learned, what you have discovered, what you "know", doesn't "belong to you" (cf. point 1).

A quick pause for full disclosure (and a shameless plug) here: I am one of the editors of this forthcoming book (**), for which I will get a royalty of about $7 for every copy sold. Actually, this needn't be a pause --- it can be an interesting case study of Daniël's logic. Some of the knowledge that went into my contribution to that book (which, apart from being the principal assembler and coordinator of the overall manuscript, was as a chapter author and a critical reviewer of many of the chapters) was acquired while I was a *paying* Master's student (my MSc cost me about $12,000 in fees, plus probably another $8,000 in travel and lodging expenses). A little has been acquired in my more recent role as a non-paying, but non-salaried PhD student. Most was just from reading, interacting with people, and some attempts at critical thinking. My situation in science is obviously quite atypical, but any system that attempts to determine what is either legally or morally acceptable has to be able to define and cope with edge cases. On the other hand, my co-editors are (as far as I know) full-time salaried academics. Should different rules apply to the various editors of the same book? Who is going to make and enforce these rules?

Back to Daniël's points: I sort of agree and sort of don't. (People who have read some of my, er, "output" are sometimes surprised to find how ambivalent I am on many issues.) Writing a book is hard work --- it's almost certainly a qualitatively different experience from writing a journal article. (I haven't written a book from scratch. But I know how hard it is to coordinate a big manuscript, and I did translate a full-length book [PDF] --- one which, incidentally, had previously been scanned and put online so that the author couldn't expect to receive much in the way of royalties from its sale afterwards. Also, many books by academics combine accounts of their own (or others') research with their interpretations of what these might mean for society. Some of Roy Baumeister's (sometimes provocative ***) books are a good example of this, I think.

On the other hand, I think that people who defend the idea of deriving income from books need to check that their logic on this point follows their logic when it comes to the publishing of scientific articles. Most of the people who responded to Daniël's tweets (at least, among the names I recognised) are people who strongly defend open access publishing. But it's very easy to denounce the evils of publishing companies when you don't get any revenue yourself from their output. Among the subset of pro-open access people who have also written books, I don't see many of them exhorting people to place PDF copies of those books on Sci-Hub. (Anyone want to bet that we will not hear at some point of a scandal whereby some researcher or other has been taking kickbacks from the article processing charges --- which are typically paid for out of grant money --- at open access journals? I hasten to add that I have no evidence that this is taking place at the moment, but I would be amazed if it didn't happen. People can be very... creative when it comes to spending other people's money.)

One area where I feel less ambivalent is in the writing of "popular" books. If you leverage your government-funded research and/or status as a professor at a publicly-funded university into pop science or self-help books, seminars, and pay-to-listen podcasts, not to mention sponsored keynotes, corporate consulting gigs, and public speaking appearances, then I think that you ought to be making more of a contribution back than just income tax on the royalties and fees, especially since most of the work that got you that speaking gig was probably done by someone else, who may not even believe in the research any more. (I would have included TED talks in this list, but apparently TED speakers don't get paid. I'm sure they all give those talks out of the goodness of their hearts, and not in any way out of a consideration for how they might leverage their appearance into other, more lucrative speaking opportunities.) Naturally, as per the discussion a couple of paragraphs earlier, I don't have a proposed practical solution for this.

In summary, I find this an interesting debate. Perhaps it's destined to remain theoretical for the foreseeable future (apart from anything else, academic norms tend to span the boundaries of legal jurisdictions), but I think it's worth discussing and I'm glad Daniël raised it.



(*) It would be nice if money did arrive that way, because I lived there for four years and it rains a lot, often horizontally.
(**) Yes, that's a great picture on the cover. It's by my brother-in-law, and I believe that he will get a fee for it. Is this nepotism? I just wanted a non-boring cover design (check out what most handbook cover art is like) and I knew that Tony was pretty handy with a brush; this is a picture that he had painted a while ago. Should we have had a call for tenders to avoid possible questions about nepotism? Hmmm.
(***) I linked to Amazon here, but if you don't believe that academics should benefit from writing books about their own or other people's research, you can find a full PDF of the book without too much trouble. ;-)

Cornell salutes America's teenage female combat heroes of WW2

A lot has been written about the four "pizza papers" mentioned by Dr. Brian Wansink in his blog post of 21 November 2016.  But so far, it seems nobody has paid much attention to the fifth article co-authored by "the Turkish grad student" mentioned in that blog.  That's a shame, because it appears to reveal a remarkable bit of American military history.

Here's the article:

Sığırcı, Ö, Rockmore, M., & Wansink, B. (2016). How traumatic violence permanently changes shopping behavior.  Frontiers in Psychology, 7, 1298. http://dx.doi.org/10.3389/fpsyg.2016.01298

Because Frontiers in Psychology is an open access journal, you can read this article here.  However, you might want to look at the annotated copy that I've made available here, where you will also find a couple of other relevant files.

When I first read Sığırcı et al.'s article, I didn't pay it too much attention, partly because Tim van der Zee, Jordan Anaya, and I were busy enough with the pizza papers, and partly because its conclusions seemed plausible, even obvious: Long-retired military veterans who experienced heavy or frequent combat are more cautious, conservative consumers than those who only experienced lighter or infrequent combat.  After all, there's an old military saying: "There are old soldiers, and bold soldiers, but there are no old, bold soldiers".  Maybe a degree of caution helps you survive in war, and also carries over into whatever makes you a careful consumer.  However, after a few weeks of exploring the rest of the output of the Cornell Food and Brand Lab, I decided to revisit this article and see what else I could learn from it.

The usual Cornell comedy stuff

Let's start with the self-plagiarism.  It's actually quite mild, compared to what we've seen; about 350 words of the 500-word method section are copied (absolutely verbatim this time, including the typos; no tweaking of occasional words, as we saw in other cases) from this article:

Wansink, B., Payne, C. R., & van Ittersum, K. (2008).  Profiling the heroic leader: Empirical lessons from combat-decorated veterans of World War II.  The Leadership Quarterly, 19, 547–555.  http://dx.doi.org/10.1016/j.leaqua.2008.07.010

I will spare you the usual yellow-highlighted image here, but you can see the duplicated text in the annotated versions of the files here.

Some other things that regular readers of this blog and Jordan's will be expecting are GRIM inconsistencies (i.e., means or percentages that don't match the sample sizes)  and F statistics that are not consistent with the reported means and standard deviations.  And these readers will not be disappointed.  There are 26 GRIM inconsistencies in Table 1, and seven out of the eight F statistics in Table 2 are inconsistent, as is one in Table 3.  (Actually, this is a little unfair.  The inconsistent F in Table 3 is probably the result of both of the standard deviations on the same line in the table being wrong.)  All of these problems are visible in the annotated copy of the article here.

So we could have included this article in with the four pizza papers; it would have fit right in with the quality of those (and other articles from the same lab going back ten or more years).  But just to blog about it now on that basis would be like publishing a news story saying that the President of the United States was rude and badly informed during a press conference: headline news in 2016, not so much in 2017.  The real fun starts when we look closer at the most basic numbers in the study (indeed, in any study), namely the demographics of the sample.

The unique selling points

Let's start with the ages.  This survey was done in the year 2000.  We don't know exactly when, but let's assume it was January (as we'll see, the results would have been even more extreme if participants had reported their ages in December).  These are veterans of combat operations in World War 2, which ended on September 2, 1945.  Assuming that these veterans were not underage volunteers who lied about their age to join the military, their minimum age on that date must have been about 18 years and 2 months, allowing time for basic training and shipping out to the Pacific theatre of operations.  So the youngest of these veterans would have been born in July 1927 and been 72 years of age in January 2000.

Now look at the standard deviations (SD) for the ages.  They are quite high, especially when there is not much room on the low side of the mean.  In fact, those SDs constrain the pattern of ages quite severely.  James Heathers kindly used his new tool, SPRITE, to build a series of possible distributions of ages matching this constraint (plus a supplementary one that none of the respondents should be older than 105 at the time of the survey, because otherwise it proposes a whole bunch of people aged over 120).  Here's what a typical such distribution looks like:
 
What this means is that, if those means and SDs are correct, 200 of the 235 respondents were between 18 and 18.5 years of age at the end of WW2, having presumably experienced repeated heavy combat only in the last few months of the war.  There were one or two slightly older soldiers, and then a bunch who were 50 at the time (and so were 105 years old in 2000).  Almost none of the soldiers can have been aged 19, or 20, or 21, if these numbers are correct.  That is, if the sample is at all representative of actual US combat veterans from WW2, almost nobody who joined at 21 in 1941, or 18 in 1943, survived until 2000 (whether they were exposed to heavy or light combat).

Even more interesting, however, is just how many of these young men, who fought their way to glory at Anzio, Omaha Beach, and Iwo Jima, must have been... women.  Have a look two lines further down in the demographic data:



Among these veterans, 79.3% who saw heavy combat and 80.3% who saw light combat identified in 2000 as men.  Now I'm going to go out on a limb here for a moment and guess that only a very small proportion of these men came out as transgender since 1945.  With that assumption, the implication here is that 20% of these veterans of heavy combat are, and were at the time, women. (*)

This has to be the historical scoop of the last 70 years.  The role of American women outside the US in World War 2 has up to now been believed to be mostly limited to nursing, well away from the front line.  Woman only officially obtained the right to serve in combat roles in the US Army in March 2016.  Yet here we have evidence to suggest that many women took part in combat of all kinds in World War 2—making up about a fifth of all soldiers who were involved in combat operations.  How come we've never seen this in all those war movies?  Can it just be due to sexism on the part of Hollywood producers?  Why can't the true story of the hundreds of thousands of 18-year-old women whose courageous combat liberated the world from the menace of the Axis powers be told?

[scratching sound of needle being pulled from vinyl record]

OK, fun's over.  Let's be serious for a moment.

First, just to be clear, nothing in the two preceding paragraphs is intended to take a dig at women or trans people.

My purpose in this post has been to show that some of the absurdities in this article (and many, many more, whether from this lab or not) are visible to almost anyone who cares to read it.  You don't need to know a thing about statistics to see that the implication that "20% of the US soldiers who saw combat in World War 2 were women" is absurd.  You don't need to know much more about what distributions look like to see that a mean of 75 and an SD of 9 with a floor of 72 is going to lead to a huge right skew. And you can probably guess that, if a piece of work (whether a scientific article or a restaurant meal or a car) has problems like that visible from the moment you look at it, it may well have a bunch of other problems that you only need some simple tools to uncover.

This article appeared in what claims to be a peer-reviewed journal.  The names of the reviewers and action editor are displayed on the article's web page.  I'm trying to work out exactly how closely any of these people looked at the manuscript before approving it for publication, thus elevating it to the status of "science" so that people can write press releases.  We seriously need to improve the way we go about reviewing.  As it is, though, with some publishers, it seems that things may even be getting worse.



(*) Someone suggested that maybe there was a small percentage of women in the original combat roles, and that they became proportionately more numerous from 1945 to 2000 through having higher survival rates.  I find this fairly unlikely, but since we are arguably conditioning on a collider here, I thought I'd mention it for completeness.

Some instances of apparent duplicate publication from the Cornell Food and Brand Lab

Some concerns have been expressed recently (e.g., here) about a few of the research articles coming from the Cornell Food and Brand Lab.  While reading some past work from the same lab, I noticed some phrases that seemed to recur.  On doing some further comparisons, I found several examples of apparent duplicate publication.  I list five such examples here.

A: Two hundred words for almost any situation

Here are a couple of paragraphs from the same author that have been published at least five times over a 15-year period, with just a few very minor changes of wording each time. I have copied and pasted the relevant text here so that you can see all of the different versions.  (I hope the various publishers will allow this as "fair use".)

1. Wansink, B., & Ray, M. L. (1997).  Developing copy tests that estimate brand usage.  In W. Wells (Ed.), Measuring advertising effectiveness (pp. 359–370). Cambridge, MA: Lexington Books.

From page 361:

These two different measures of usage intent have different relative strengths. With infrequent users of a brand, volume estimates will be skewed toward 0 units (especially over a relatively short period of time). This is partially a drawback of numerical estimates that provide no gradation between 0 and 1 unit. In such cases, volume estimates would provide less variance and less information than an estimate of usage likelihood. As a result, usage likelihood estimates would allow a greater gradation in response and would be more sensitive in detecting any potentially different effects these ads might have on usage.

In contrast, with frequent or heavy users of a brand, a volume estimate is likely to be more accurate than a likelihood estimate. This is because the distribution of these volume estimates is more likely to be normally distributed (Pearl 1981). As a result, a volume estimate of one’s usage intent is likely to provide more variance and more information about the intended usage of heavy users than is a likelihood measure, which would undoubtedly be at or near 1.0 (100 percent probable). Under these circumstances, volume estimates would be a more accurate estimate of a heavy user’s usage volume of a brand.

2. Wansink, B., & Sudman, S. (2002).  Predicting the future of consumer panels.  Journal of Database Marketing, 9, 301–311. http://dx.doi.org/10.1057/palgrave.jdm.3240078

From page 309:

[T]hese two different measures of usage intent have different relative strengths. With infrequent users of a brand, volume estimates will be skewed toward 0 units (especially over a relatively short period of time). This is partially a drawback of numerical estimates that provide no gradation between 0 and 1 unit. In such cases, volume estimates would provide less variance and less information than an estimate of usage likelihood. As a result, usage likelihood estimates would allow a greater gradation in response and would be more sensitive in detecting any potentially different effects these adverts might have on usage.

In contrast, with frequent or heavy users of a brand, a volume estimate is likely to be more accurate than a likelihood estimate. This is because the distribution of these volume estimates is more likely to be normally distributed. As a result, a volume estimate of a person’s usage intent is likely to provide more variance and more information about the intended usage of heavy users than is a likelihood measure, which would undoubtedly be at or near 1.0 (100 per cent probable). Under these circumstances, volume estimates would be a more accurate estimate of a heavy user’s usage volume of a brand.

3. Wansink, B. (2003).  Response to ‘‘Measuring consumer response to food products’’.
Sensory tests that predict consumer acceptance.  Food Quality and Preference, 14, 23–26. http://dx.doi.org/10.1016/S0950-3293(02)00035-6

From page 25:

These two different measures of usage intent have different relative strengths. With infrequent users of a product, frequency estimates will be skewed toward 0 units (especially over a relatively short period of time). This is partially a drawback of numerical estimates that provide no gradation between 0 and 1 unit. In such cases, the frequency estimates provide less variance and less information than an estimate of consumption likelihood. With light users, consumption likelihood estimates will provide greater gradation in response and more sensitivity in detecting any potentially different effects a particular set of sensory qualities would have on consumption.

In contrast, with frequent or heavy users of a product, a frequency estimate is likely to be more accurate than a likelihood estimate. This is because the distribution of these frequency estimates is more likely to be normally distributed. As a result, a frequency estimate of one’s consumption intent is likely to provide more variance and more information about the intended consumption of heavy users than is a likelihood measure, which would undoubtedly be at or near 1.0 (100% probable). With heavy users, frequency estimates would be a more accurate estimate of a heavy user’s future consumption frequency of a product.

4. Bradburn, N. M., Sudman, S., & Wansink, B. (2004).  Asking questions: The definitive guide to questionnaire design—For market research, political polls, and social and health questionnaires (Revised ed.).  San Francisco, CA: Jossey-Bass.

From pages 134–135:

These two different measures of behavioral intent have different relative strengths. With infrequent behaviors, frequency estimates will be skewed toward 0 (especially over a relatively short period of time). This is partially a drawback of numerical estimates that provide no gradation between 0 and 1 unit. In such cases, frequency estimates would provide less variance and less information than an estimate of likelihood. As a result, likelihood estimates would allow a greater gradation in response and would be more sensitive.

In contrast, with frequent behaviors, a frequency estimate will be more accurate than a likelihood estimate. The reason is that frequency estimates are more likely to be normally distributed. As a result, a frequency estimate is likely to provide more variance and more information than is a likelihood measure, which would undoubtedly be at or near 1.0 (100 percent probable). Under these circumstances, frequency estimates more accurately correspond with actual behavior.

5. Wansink, B. (2012).  Measuring food intake in field studies.  In D. B. Allison and M. L. Baskin (Eds.), Handbook of assessment methods for eating behaviors and weight-related problems: Measures, theories, and research (2nd ed., pp. 327–345). Los Angeles, CA: SAGE.

From page 336:

These two different measures of intake intent have different relative strengths. With infrequent users of a food, frequency estimates will be skewed toward 0 units (especially over a relatively short period of time). This is partially a drawback of numerical estimates that provide no gradation between 0 and 1 unit. In such cases, the frequency estimates provide less variance and less information than an estimate of intake likelihood. With light users, intake likelihood estimates will provide greater gradation in response and more sensitivity in detecting any potentially different effects a particular set of sensory qualities would have on intake.

In contrast, with frequent or heavy users of a food, a frequency estimate is likely to be more accurate than a likelihood estimate. This is because the distribution of these frequency estimates is more likely to be normally distributed. As a result, a frequency estimate of one’s intake intent is likely to provide more variance and more information about the intended intake of heavy users than is a likelihood measure, which would undoubtedly be at or near 1.0 (100 percent probable). With heavy users, frequency estimates would be a more accurate estimate of a heavy user’s future intake frequency of a food.

You can check all of these here.  In addition to the draft PDF versions that I have annotated, the first example (from the 1997 book Measuring Advertising Effectiveness) is also available on Google Books here, the fourth example (from the 2004 book Asking Questions...) is available here, and the fifth example (from the 2012 book  Handbook of Assessment Methods for Eating Behaviors...) is available here.  Note that in the case of Asking Questions..., the PDF is an extract from a copy of the entire book that I found when searching on Google; I hope this possible violation of copyright on my own part will be forgiven.

B: Copying and pasting from multiple articles to make a new one

 

Consider this review article from 2015:

Wansink, B. (2015). Change their choice! Changing behavior using the CAN approach and activism research. Psychology & Marketing, 32, 486–500. http://dx.doi.org/10.1002/mar.20794

The image below shows the extent to which the 2015 article appears to consist of duplicated text from other publications.  Everything in yellow, plus three of the four figures (which I couldn't work out how to highlight in the PDF) has been published before, some of it twice; I estimate that this represents about 50% of the article.

Specifically, parts of this article appeared to have been copied without attribution from the following works (listed in approximate descending order of quantity of apparently duplicated text):

Wansink, B. (2011). Activism research: Designing transformative lab and field studies.  In D. G. Mick, S. Pettigrew, C. Pechmann, & J. L. Ozanne (Eds.), Transformative consumer research for personal and collective well-being (pp. 66–88). New York, NY: Routledge.

Wansink, B. (2013). Convenient, attractive, and normative: The CAN approach to making children slim by design. Childhood Obesity, 9, 277-278. http://dx.doi.org/10.1089/chi.2013.9405

Wansink, B. (2015). Slim by design: Moving from Can’t to CAN.  In C. Roberto (Ed.), Behavioral economics and public health (pp. 237–264). New York, NY: Oxford University Press.

Wansink, B. (2010). From mindless eating to mindlessly eating better. Physiology & Behavior, 100, 454–463. http://dx.doi.org/10.1016/j.physbeh.2010.05.003

Wansink, B., Just, D. R., Payne, C. R., & Klinger, M. Z. (2012). Attractive names sustain increased vegetable intake in schools. Preventive Medicine, 55, 330–332. http://dx.doi.org/10.1016/j.ypmed.2012.07.012

Annotated versions of all of these documents are can be found here. A Google Books preview of the 2015 chapter "Slim by Design" is available here to compare with the annotated document, which is a final draft version.

C: An article apparently recycled as the basis of a book chapter, without disclosure

The article:

Wansink, B., van Ittersum, K., & Werle, C. (2009).  How negative experiences shape long-term food preferences: Fifty years from the World War II combat front.  Appetite, 52, 750–752.  http://dx.doi.org/10.1016/j.appet.2009.01.001

The book chapter:

Wansink, B., van Ittersum, K., & Werle, C. (2011).  The lingering impact of negative food experiences: Which World War II veterans won’t eat Chinese food?  In V. R. Preedy, R. R. Watson, & C. R. Martin (Eds.), Handbook of behavior, food and nutrition (Vol. 1, pp. 1705-1714). New York, NY: Springer.

It appears that almost all of the 2009 research article—about 1,400 words—has been duplicated in the 2011 chapter, with only very minor changes and the omission of five sentences, which account for less than 100 words.  No disclosure of this re-use appears in the book chapter.  (In contrast, Chapter 87 in the same book contains, on pages 1357 and 1360, explicit acknowledgements that two passages in that chapter contain material adapted from two other previously published sources; each of these passages corresponds to about 120 words in the original  documents.)

You can examine annotated versions of the article and chapter here (note that the PDF file of the book chapter is an extract from a copy of the entire book that I found on Google).  The book chapter is also available on Google Books here (although three of the ten pages are missing from the preview).

Here is a snapshot of the 2009 article (left) and the 2011 book chapter (right).

D: Two almost-identical book chapters, published more or less simultaneously

There seems to be a very close resemblance between the following two book chapters:

Wansink, B. (2011).  Mindless eating: Environmental contributors to obesity.  In J. Cawley (Ed.), The Oxford handbook of the social science of obesity (pp. 385–414).  New York, NY: Oxford University Press.

Wansink, B. (2012).  Hidden persuaders: Environmental contributors to obesity.  In S. R. Akabas, S. A. Lederman, & B. J. Moore (Eds.), Textbook of obesity: Biological, psychological and cultural influences (pp. 108–122).  Chichester, England: Wiley-Blackwell.

Each chapter is around 7,000 words long.  The paragraph structures are identical.  Most of the sentences are identical, or differ only in trivial details; a typical example is:

(Mindless Eating, p. 388)

While this may appear to describe why many people eat what they are served, it does not explain why they do so or why they may overserve themselves to begin with. Figure 23.1 suggests two reasons that portion size may have a ubiquitous, almost automatic influence on how much we eat: First, portion sizes create our consumption norms; second, we underestimate the calories in large portion sizes.

(Hidden Persuaders, p. 109)

While this may describe why many people eat what they are served, it does not explain why they do so or why they may over-serve themselves to begin with. Figure 6-2 suggests two reasons why portion size may have a ubiquitous, almost automatic influence on how much we eat: First, portion sizes create our consumption norms; second, we underestimate the calories in large portions.

Overall, I estimate that about 85-90% of the text is duplicated, word-for-word, across both chapters.

It seems to be rather unusual to submit the same chapter for publication almost simultaneously to two different books in this way (the books were published less than six months apart, according to their respective pages on Amazon.com).  One occasionally sees a book chapter that is based on an updated version of a previous journal article, but in that case one would expect to find a note making clear that some part of the work had been published before.  I was unable to find any such disclosure in either of these books, whether on the first or last page of the chapters themselves, or in the front matter.  I also contacted the editors of both books, none of whom recalled receiving any indication from the author that any of the text in the chapter was not original and unique to their book.

I found final draft versions of each of these chapters here and here.  Each draft clearly states that it is intended for publication in the respective book in which it finally appeared, which would seem to rule out the possibility that this duplication arose by accident.  Interested readers can compare my annotated versions of these final drafts with each other here.  You can also check these drafts against the published chapters in the Google Books previews here and here (the first has the complete "Mindless Eating" chapter, but four pages of the "Hidden Persuaders" chapter are missing from the second). The degree of similarity is visible in this image, where yellow highlighting indicates text that it identical, word-for-word, between the two drafts ("Mindless Eating" is on the left, "Hidden Persuaders" is on the right).

E: Different studies, same introduction, same summary, different participants, same results

In 2003, when the current director of the Cornell Food and Brand Lab was still at the University of Illinois at Urbana-Champaign [PDF], he published a report of a pair of studies that had a certain number of theoretical aspects in common with another pair of studies that he had previously described in a co-authored article from 2001.  Here are the two references:

Wansink, B., & Seed, S. (2001).  Making brand loyalty programs succeed.  Brand Management, 8, 211–222. http://dx.doi.org/10.1057/palgrave.bm.2540021

Wansink, B. (2003).  Developing a cost-effective brand loyalty program.  Journal of Advertising Research, 43, 301–309. http://dx.doi.org/10.1017/S0021849903030290

I estimate that the introduction and summary sections from the two resulting articles are about 50% identical. You can judge for yourself from this image, in which the 2001 article is on the left, and the 2003 article is on the right.  The introduction is on the first six pages of the 2001 article and the first four pages of the 2003 article.  The summary section is near the end in each case.

Perhaps of greater interest here, though, is a comparison between Table 5 of the 2001 article and Table 2 of the 2003 article, which appear to be almost identical, despite purportedly reporting the results of two completely different studies.

Table 5 in the Wansink and Seed (2001) article apparently represents the results of the second study in that article, in which a questionnaire was administered to "153 members of the Brand Revitalisation Consumer Panel" (p. 216):

As shown in Table 5, this moderate benefit programme captured an average monthly incremental gain of $2.95 from the non-user and $3.10 from the heavy user. For light users, the most cost-effective programme was the one that offered the lowest benefit package. This programme level captured an average monthly incremental gain of $2.00 from the light user. (Wansink & Seed, 2001, p. 218)

Table 2 in the Wansink (2003) article appears to summarise the results of Study 2 of that (i.e., 2003) article, which involved "a nationwide survey of 2,500 adult consumers who had been randomly recruited based on addresses obtained from census records. . . . Of the 2,500 questionnaires, 643 were returned in time to be included in the study".

The second major finding of Study 2 was that, in contrast to the beliefs of the managers, the high reward program appears to be the least cost-effective program across all three segments. Given the simple two-period model noted earlier, Table 2 shows the low reward program is the most cost-effective across all three segments ($1.67), and the moderate reward program is the most cost-effective with the heavy user ($3.10). (Wansink, 2003, p. 307)

(For what it is worth, the average gain for the low reward program across all three segments in the 2003 article appears to be ($0.00 + $2.00 + $2.00 = $4.00) / 3 = $1.33, rather than the $1.67 reported in the text cited above.)


It seems remarkable that two studies that were apparently conducted on separate occasions and with different samples (unless "over 153" from the 2001 article is a very imprecise approximation of 643) should produce almost identical results.  Specifically, of the 45 numbers in each table that are not calculated from the others (i.e., the numbers in the first, third, fourth, and fifth columns, with the first column having two numbers per row), 39 are identical between the two tables (and of the six others, it seems likely that the three numbers corresponding to "Light user/Average Monthly Revenue before Start" were mistyped in the 2001 version of the table, where they are identical to the numbers to their immediate left).  These 39 identical "raw" numbers, plus the 17 out of 18 calculated numbers that are identical in columns 2 and 6, are highlighted in yellow in the two images above.

(*** BEGIN UPDATE 2017-03-12 20:53 UTC ***)
Jordan Anaya points out that the numbers in the third, fourth, and fifth columns of these tables were derived from the two numbers in the first column.  That is, the first number in the first column is multiplied by $3.00 to give the third column; the second number in the first column is multiplied by $3.00 to give the fifth column; and the second number is multiplied by either $1.00, $0.50, or $0.25 to give the value of the coupons used.

So the number of observed values that are exactly the same across these two independent studies is "only" 17 out of 18, identical to one decimal place. (In fact the 18th, "Nonuser/Low reward/After", appears to have been mistyped in the 2001 table, given the dollar amounts on the same line, so it looks like all 18 results were actually identical.)

None of this explains why the calculated value for "Average monthly revenue/After" for "Nonuser/Moderate reward" changes between 2001 and 2003.  See also the footnote under Table 4 from 2001 and Figure 3 from 2003, reproduced about six paragraphs below here.
(*** END UPDATE 2017-03-12 20:53 UTC ***)


The only point at which there appears to be a substantive difference between the two tables—namely, "Average Monthly Revenue after Program Start" and "Dollar Amount of Coupons Used" in the second line ("Nonuser", with a Program Reward Level of "Moderate")—is the explicit focus of discussion at two points in the 2001 article:

As shown in Table 5, this moderate benefit programme captured an average monthly incremental gain of $2.95 from the non-user and $3.10 from the heavy user. (Wansink & Seed, 2001, p. 218)

Across all products, the moderate reward programme was the only programme that motivated non-users to purchase. Perhaps, the high-reward programme required too much involvement on the part of the non-user and the low-reward programme did not offer enough benefit to induce trial. The moderate-reward programme might have struck the right balance. As non-users become more familiar with the product, however, a programme with more benefits might be required to sustain interest. (Wansink & Seed, 2001, p. 220)

In the 2003 article, however, the corresponding numbers for the moderate benefit programme for nonusers are much smaller, so that there is no effect to discuss.

I am at a loss to explain what might have happened here.  The fact that the two tables are not completely identical would seem to rule out the possibility that there was some kind of simple clerical error (e.g., including the wrong file from a folder) in the preparation of the later article.

It is also interesting to compare Table 4 of the 2001 article with Figure 3(a) of the 2003 article, the data for which appear to have been taken directly from the column entitled "Change in Purchases (in units)" in the tables just discussed.  Again, the different presentations of this information suggests that this is not simply a case of adding the wrong table to the second article.

 

(Note here that the value for "Change in Purchases (in units)" for the "Moderate" Program Reward Level for non-users appears to have been miscalculated in both tables (it should read 1.2, not 0.2, if the numbers reported for "Average Monthly Purchases Before/After Program Start (in units)" are correctly reported).  The (apparently) correct figure of 1.2 appears in Table 4 of the 2001 article, but not in Figure 3(a) of the 2003 article.  Also, the "Moderate Reward Program" / "Light User" value in Figure 3a of the 2003 article appears to have become 0.9 instead of 0.8, perhaps due to different rounding in the software being used to make the figure.)

In another twist, I found what appears to be a draft version of the 2003 article here.  In this draft, the table corresponding to Table 2 in the final version of the 2003 article (numbered 4 in the draft) contains the same numbers for "Nonuser" / "Moderate" as Table 5 in the 2001 article, and the effect that was noted in the 2001 article is still reported, as follows:

Based on the survey results of Table 4, the most cost-effective program for
nonusers and heavy users offered average benefits (moderate reward program). This
program level captures an average monthly incremental gain of $2.95 from the nonuser and $3.10 from the heavy user. (Draft, p. 13)

Across all products, the moderate reward program was the only program that motivated nonusers to purchase. Perhaps, the high reward program required too much involvement on the part of the nonuser and the low reward program did not offer enough benefit to induce trial. The moderate reward program might have struck the right balance. As nonusers become more familiar with the product, however, a program with more benefits might be required to sustain interest. (Draft, p. 17)

So apparently, at some stage between drafting and acceptance of the 2003 article, not only was the text describing this effect removed, but the numbers that support its existence were somehow replaced, in what became Table 2, by much smaller numbers that show no effect.

In summary: If the two studies in question are the same, then it would appear that there has been duplicate publication of the same results, which is not normally considered to be a good ethical practice.  This is compounded by the fact that the descriptions of the participants in these two studies diverge considerably; furthermore, there is a difference between the tables across the two articles that results in the disappearance of what appeared, in the 2001 article, to be a very substantial effect (that of the moderate-reward programme for non-users).  On the other hand, if the two studies are not the same, then the exact match between the vast majority of their results represents quite a surprising coincidence.

Readers who wish to explore these two articles further can find annotated versions of them here, as well as an annotated version of the draft of the 2003 article, which shows which parts of that draft did not make it into the final published article.

Conclusion

I have documented here what appear to be multiple varieties of duplicate publication:

• Re-use of the same paragraphs multiple times in multiple publications;
• Assembly of a new article from verbatim or near-verbatim extracts taken from other published work by the same author;
• Apparent recycling of a journal article in a later book chapter, without disclosure;
• Duplicate publication of an entire chapter almost simultaneously in two different books, without disclosure;
• Re-use of the same introduction section for two different empirical articles;
• Apparent republication of the same data, with slightly different conclusions, and different descriptions of the participants.

A different set of problems in an article from the Cornell Food and Brand Lab

[[ Update 2018-09-23 20:30 UTC: Fixed some links that were broken because some original documents had gone missing from sites controlled by the Cornell Food and Brand Lab. ]]
[[ Update 2017-10-19 17:00 UTC: This post now features in a BuzzFeed article here. ]]

(This is the first time I've blogged on the subject of the ongoing kerfuffle around the Cornell Food and Brand Lab, that was started by reactions to this blog post [[2018-09-23: that post is long gone, but an archived version is here]] by the lab's Director, that led to our preprint and subsequent peer-reviewed article showing that their research had a lot of errors and inconsistencies in it, and some more blog posts describing problems at the same lab, and some scrutiny in the media. This story is of another article from the same lab, but with some rather different problems from the others that we've documented so far.)

The article

The American Medical Association has a journal dedicated to all aspects of pediatric medicine, JAMA Pediatrics (formerly known as Archives of Pediatrics & Adolescent Medicine). If you are involved in anything to do with children's health, this is the journal that you really, really want to publish in.  It has an impact factor of 9.5 (which is kind of impressive even if you don't believe in impact factors), making it the "highest-ranking pediatric journal in the world", and it accepts just 9% of the research articles that it receives.

So presumably the editors of JAMA Pediatrics thought that a short article ("research letter") that they accepted for publication in the "Pediatric Forum" section of the journal in 2012, entitled "Can Branding Improve School Lunches?", could result in substantial improvements in nutrition for American schoolchildren. This article, by Brian Wansink and David R. Just at Cornell and Collin R. Payne at New Mexico State, is behind a paywall, but if you know of any alternative ways to get to a published PDF using its DOI (in this case, 10.1001/archpediatrics.2012.999) then you can probably find it.  You can also download a pre-publication draft of the article from here [[2018-09-23: now here]] [PDF].  In fact I encourage you to do that, even if you have access to the published article from the journal, because one of the differences between the two versions is quite important.

The method... a little unclear

Here's how the study worked.  The researchers recruited 208 students at seven elementary schools.  As part of their regular lunch menu, these students were already allowed to take an apple, a cookie, or both, in addition to their main dish.  During the study, the researchers manipulated these "bonus" food items by adding (or not) a sticker of  a cartoon character (presumably to the skin of the apple, or the packaging of the cookie).  The sticker depicted either a well-known, branded cartoon character (specifically, "Elmo", presumably meaning the character from Sesame Street) or an "unknown" character.

The study lasted five days.  On days 1 and 5, there were no stickers on either item.  On days 2, 3, and 4, three of the eight other possible combinations (Elmo/unknown character/no sticker on the apple, Elmo/unknown character/no sticker on the cookie, 3 x 3 = 9, minus the "no stickers on either item" case) were deployed, and the percentage of students taking each item was noted.  The authors do not explain why they only tested three of the possible combinations.  Indeed, it's hard to tell exactly which combinations they tested:

On one day, children were offered a choice between an unbranded apple and a cookie that had a sticker of a familiar popular character (ie, Elmo) on it. On another day, children were offered a choice between an unbranded cookie and an apple that had a sticker of the Elmo icon on it. On another day, their choice was between an unbranded cookie and an apple with a sticker of an unknown character.

Does "unbranded" here mean "with the sticker of the unknown character" or "with no sticker"?  It's completely unclear to me.  The way the sentences are written, particularly the last one, seems to imply that "unbranded" means "no sticker".  But elsewhere in the article, the authors noted that the presence of the unknown character had no effect on the consumption of apples; they also claimed that "this study suggests that the use of branding or appealing branded characters [emphasis added] may benefit healthier foods [sic]" (p. 968), which suggests that the branding/no branding distinction is between Elmo and the unknown cartoon.  As a minimum, this is a confusing mix of terminology, which might lead to inaccuracies when this article is cited.

It's also worth noting the vocabulary that was used to describe the process: "children were offered a choice between [emphasis added]" an apple or a cookie (p. 968).  That exact phrasing is used three times.  To me, this conveys the idea that the children had to choose one or the other (and abandon the unchosen item).  The fact that they could take both an apple and a cookie is only mentioned once, almost in passing, in the Methods section.  (On a tangent, this also seems to me to cast some doubt on the utility of the study, in that nothing was done to try and reduce the number of cookies being taken --- for example, by putting unpleasant stickers on them.  Eating an apple is fine, but it doesn't seem clear how it would counteract the purported deleterious effects of eating a cookie.)

The results... not what they appear to be

The authors set up a web page here [[2018-09-23: this page has been taken down. The last available snapshot at archive.org is here]] to promote the study.  That page claims that "Compared to the pretest, the apple choice nearly doubled when the Elmo sticker was present. There was no effect of the Elmo sticker on the cookie and no effect of the unknown character on the apple."  But an examination of the Results section of the article shows that, using conventional levels of significance, no such effect was demonstrated.  The key statement here is:

The Elmo sticker led children to nearly double their apple choice compared with the pretest control session (Χ²=2.355; P=.06)

There are two problems here.  First, the implicit claim that a p value of .06 is as good as one of .05 is, frankly, cheating in plain sight.  (Of course, we shouldn't be playing the p value game at all, but if we're going to play it, let's at least all play it by the same rules.)  Second, the actual p value associated with a chi-square of 2.355 with one degree of freedom is not .06 but .12.   If you don't want to trust some random online calculator page, here are some tables of critical values for the chi-square distribution from the National Institute of Standards and Technology (NIST); you can see from the first line (df=1) of the table for the upper tail that the value of 2.355 isn't significant at the .10 level, never mind the .05 level.  Or, you can put "1-pchisq(2.355, 1)" into R.  (The degrees of freedom are not reported, but it seems clear that this is just a 2x1 contingency table: number of children taking an apple on day 1 versus day 2.  Even if it's a 2x2 contingency table, including cookies, then there's just one degree of freedom.  Of course, if this chi-square statistic were derived from a larger contingency table, say counting consumption across all five days, the degrees of freedom would be larger, and so the p value would be even higher.)

Now, some readers may be thinking that we can apply the universal statistical get-out-of-jail-free card and do a one-tailed test, which enables us to magically halve our p value because, um, reasons.  Indeed, the authors announced that "All analyses were 1-tailed tests" (p. 968), and they made use of this slightly further on in the results section, when they claimed that t(78)=1.65 has a p value "equal to" .05.  The two-tailed p value for those values is actually .103, so a one-tailed test gives a p of .051, which is not "equal to" .05 (unless you are either using an alpha level of .05499 or think that p values can be rounded down before being compared to .05), but hey-ho, this is what a lot of other people do, so let's pretend it's OK.  But this one-tailed shuffle doesn't work for the chi-square, because the p value reported by statistical software for a chi-square test is already, in effect, a one-tailed value.  The tails of the chi-square distribution are asymmetric; the p value for the upper tail is the one you're usually interested in, and it's what your software will tell you by default.  In any case, but especially with only one degree of freedom, the upper tail is much, much longer than the lower tail; compare the values for 0.95 (lower tail) and 0.05 (upper tail) here.  You can't get out of jail with the one-tailed card when the p value from your chi-square statistic turns out to be a dud.  And you certainly can't just cut your p value in half while shouting "Diminuendo!" (I don't think it's very likely that the .06 emerged in that form from SPSS). In short, that p value should have been reported as .12, not .06; and, whether you play the p value game or not, that is not very convincing evidence of anything.

[[[[[ Update 2017-02-17 14:15 UTC
Carol Nickerson suggests that the design and analysis were not correct for this study.  For example, at the pretest (day 1), the students seem to have had 4 options: plain apple, plain cookie, both, neither.  For the second intervention, they also seem to have had 4 options: Elmo apple, plain cookie, both, neither.  This 4 x 4 design was apparently reduced to a 2 x 2 design with pretest options: no apple, plain apple, and intervention options: no apple, Elmo apple.  Each of the 4 cells of this cross-tabulation should have contained the frequency of the paired observations: (1) no apple, no apple; (2) no apple, Elmo apple; (3) plain apple, no apple; (4) plain apple, Elmo apple, respectively.  This cross-tabulation should have been analyzed with McNemar's test, not the chi square test.
]]]]]

The figure... no, I don't know either

Possibly the strangest thing about this article is the figure that is suppose to illustrate the results.  Here is how the figure looked in the draft article:

That looks pretty reasonable, right?  OK, so it leaves out the days when there were no stickers, but you can see more or less what happened on the days with interventions.  A fairly constant proportion of about 90-92% of the children took a cookie, and between 23-37% of them took an apple.

Now let's see how the results were represented graphically when the article appeared in JAMA Pediatrics:

Whoa.  What's going on here?  Is this even the same study?

Let's start with the leftmost pair of columns ("Unbranded").  The note with the asterisk (*) tells us that these columns represent the baseline percentage of children taking an apple (about 22%, I reckon) and the baseline percentage of children taking a cookie (about 92%).  This presumably shows the results from Day 1 that were missing from the figure in the draft.  (Apples are now the darker column and cookies are the lighter column.)

The dagger (†) on the titles of the other three pairs of columns sends us to a different note, which describes the bars as representing the "percentage of change in selection from baseline".  This needs to be unpacked carefully.  What it seems to mean is that if 22% of children took an apple on Day 1, and 37% of children took an apple on Day 2 (let's assume that the columns are in time order, so "Branded Apples" is Day 2), then we should calculate (37%−22%)/22% which gives around 0.68 or 68% (or maybe a bit more; there is quite a big rounding error factor here since we are obliged to get all of these numbers from visual inspection of the figures, in the absence of any actual numerical results). So the meaning of the height of the bar for apples in "Branded Apples" is that the percentage of children taking an apple increased by 68%.  But it makes absolutely no sense to plot this chart in this way.  The label on the Y axis (%) means something completely different between the pair of columns labelled with an asterisk and the pairs labelled with a dagger; they both happen to be numbers that are expressed as percentages, but those numbers mean completely different things.  And the next two pairs of columns have the same problem.  To see how meaningless this is, consider what would happen if just 0.5% of children took an apple on Day 1 and 2% took an apple on Day 2.  The bar for apples in the second pair ("Branded Apples") would be at 300%, suggesting to the casual reader that the intervention had had a huge effect, even though almost no children actually took an apple on either day. As published, this figure is almost meaningless and arguably deceptive --- but it looks like something spectacular has occurred.  Just the kind of thing that might impress a policymaker in a hurry, for example.

Let's ignore for a moment all of the other obvious problems with this study (including the fact that consumption of cookies didn't decline at all, so that one of the results of the study was a net increase in the calorie consumption of the students who otherwise would not have eaten an apple).  For now, I just want to know how this figure came to be published.  We know that the figure looked much more reasonable in a late version of the draft of the article (the PDF that you can download from the link I gave earlier is dated 2012-05-29, and the article was published online on 2012-08-20, which suggests that the review process wasn't especially long).  I can't help wondering at what point in the submit-revise-accept-typeset process this new figure was added.  I find it very strange that the reviewers at a journal with the reputation, impact factor, and rejection rate of JAMA Pediatrics did not apparently challenge it.

The participants... who exactly?

The article concludes with this sentence: "Just as attractive names have been shown to increase the selection of healthier foods in school lunchrooms, brands and cartoon characters can do the same with preliterate [emphasis added] children" (p. 968).  But we were told in the Methods section that the participants were elementary school students aged 8–11 (let's not worry for now about whether Elmo would be the best choice of cartoon character to appeal to children in this age range).  I have tried, and failed, to imagine how a team of three researchers who are writing up the results of a study in seven elementary schools, during which they identified 208 children aged 8-11 and obtained consent from their parents, could manage to somehow type the word "preliterate" when writing up the results after the study.  The reader could be forgiven for thinking that there might be something about the literacy levels of kids in the third through sixth grades in the state of New York that we should know about.

But it seems that the lead author of the article may have been a little confused about the setting of the study as well.  In an article entitled "Convenient, Attractive, and Normative: The CAN Approach to Making Children Slim by Design" (published in Childhood Obesity in 2013), Dr. Wansink wrote (p. 278): "Even putting an Elmo sticker on apples led 70% more daycare kids [emphasis added] to take and eat an apple instead of a cookie", with a reference to the article I've been discussing here.  Not only have the 8-11 year olds now become "daycare kids", but it is also being claimed that the extra apple was taken instead of a cookie, a claim not supported by the JAMA Pediatrics article; furthermore, the clear implication of "take and eat" is that all of the children ate at least some of their extra apple, whereas the JAMA Pediatrics article claimed only that "The majority of children [emphasis added] who selected a food ate at least a portion of the food" (p. 968).

These claims were repeated in Dr. Wansink's article, "Change Their Choice! Changing Behavior Using the CAN Approach and Activism Research" (published in Psychology & Marketing in 2015): "Even putting an Elmo sticker on apples led to 46% more daycare children taking and eating an apple instead of a cookie" (p.  489).  The efficacy of the intervention seems to have declined somewhat over time, though, as the claimed increase in the number of children taking an apple has dropped from 70% to 46%.  (It's not clear from the JAMA Pediatrics article what the correct figure ought to be, since no percentages were reported at all.)

Conclusion

Dr. Wansink wrote a rather brave blog post recently in which he apologized for the "pizza papers" incident and promised to reform the research practices in his lab.  However, it seems that the problems with the research output from the Cornell Food and Brand Lab go much further than just that set of four articles about the pizza buffet.  In the first paragraph of this post I linked to a couple of blog posts in which my colleague, Jordan Anaya, noted similar issues to those in the "pizza papers" in seven other separate articles from the same lab dating back as far as 2005, and here I have presented another article, published in a top-rated medical journal, that seems to have several different problems.  Dr. Wansink would seem to have a long road ahead of him to rebuild the credibility of his lab.

Acknowledgement

Although I had previously glanced briefly at the draft version of the "Elmo" article while looking at the public policy-related output of the Cornell Food and Brand Lab, I want to thank Eric Robinson for bringing the published version to my attention, along with the problems with the figure, the inconsistent p value, and the "preliterate" question.  All errors of analysis in this post are mine alone.

On [the] edge

This post is mainly addressed to all of my online friends, acquaintainces, contacts, etc etc.  Whether we interact via Twitter, Facebook, Skype, e-mail, or some combination of these.  (Everyone says I have to be on WhatsApp and Instagram too, but I already waste way too much time online as it is.)  Of course, people I've never heard of are welcome to read and comment on this too.

These are strange times.  Since about 24 June 2016 I have had this constant strange feeling of unease.  It's faint, but real.  And since 9 November 2016, it has become a bit less faint.

I don't think I've ever had any problems with mental health.  That is, if I complete a measure of depression, I don't think I've never been at a point in my life where I scored above 0 on any of the items, and even then I only would have scored 1 on a couple.  About once every three years I go through a little phase where I feel strangely lethargic for a couple of days (after controlling for hangovers), but that's about it.

I just looked at the Beck Depression Inventory and today I scored 6 out of 63.  Probably my highest ever, but I didn't score more than 1 on any item, and a score of 1 to 10 is classed as "These ups and downs are considered normal".  So apparently I'm fine at that level.  This sums up more like how I feel:

My "work", such as it is (I don't have a job that involves leaving the house and going to an office with a boss where I have to do stupid shit), involves quite a lot of being critical of other people's work.  I try to do this in as civilised a way as possible.  I prefer to write my critiques of scholarly work in the form of manuscripts that are at least intended for publication in journals (except when something really pisses me off and I dash off a blog post about it, which I usually regret shortly afterwards when it turns out I didn't do my due diligence).  When you work this way, you need a valve to release the pressure, because it's very, very slow and tedious to work your way through a series of articles about how people consume pizza with almost innumerable statistical errors in them (shameless pimping of our new preprint there).  For me, that valve is mostly Twitter, and sometimes Facebook.  But that brings me back face to face with... well, the causes of "that feeling".  80% of the tweets in my feed, every second Facebook post seems to be about what the whole world (or at least, my blinkered, woolly-liberal(*) section of it) is talking about.

I'm starting to think that this feeling of unease may be affecting my interactions.  People who used to be up for stupid, nerdy banter about stuff that doesn't matter seem to be a little bit more sensitive.  Stuff doesn't get discussed that probably ought to.  Or, perhaps worse, stuff that shouldn't be discussed does come up.  I've witnessed people whose fundamental views on a particular question differ by about one hair's width from each other having fights --- well, not quite fights, but exchanges of snarkiness --- over utterly trivial details.  People seem to be a little bit on edge.  I find myself wondering if I ought to drop that bit of banter into a tweet when the only people who will read it are people I've been happily bantering with for a couple of years.

I have been wondering whether I'm alone in experiencing this "gnawing feeling" in the form of (what I presume is) low-level stress.  Today, as I wondered whether to publish this draft (which I've been working on occasionally for a few days now, not that it shows from the quality of the writing), I saw that my occasional co-conspirator James Heathers --- for whom the words "irrepressibly upbeat" are normally a mere pastiche of an understatement --- seems to have been having something similar going on.  So maybe it's not just me.

And I'm lucky.  I'm white and male and all of the other things that place me above the midpoint of luck and privilege on every scale ever.  Just after the US election result, I saw a tweet from a Black person that basically said, "Hey, liberal white folks. That feeling in your stomach right now? Welcome to our world, every day of our lives".  So I'm conscious that this is probably just me having a whine about how I don't feel as good as I think I'm entitled to feel.

Currently I don't have many few ideas for cheering myself up.  Silly, over-the-top prog-rock wigouts work a bit, for a few moments.  My slow acquisition of the documents I need to apply for Irish nationality provided a couple of moments of light relief last Friday, as one certificate arrived in the post and I got e-mail confirmation that another was on its way.  But these are small consolations.

Anyway, back to the first paragraph (all the professional writers seem to have learned at writing school that you have to finish with a quirky point that ties back into your first quirky point).  To my online friends, acquaintances, etc: If I am being "differently annoying" right now --- i.e., not in the normal "Nick, we get it, just shut up now" way :-) --- then I apologise, but things are, well, not normal.

PS: Normally I allow comments on my posts, but it doesn't feel right in this case. That seems to fit in with my theme here.  Heh.


(*) I don't think I'm very political.  I mean, yes, I don't like racism, and I think that multinationals probably ought to pay more tax, and the state in some countries should probably help poor people more, but I do find a lot of "progressive" ideas to be just sloganising.  I think that there are real biological differences between the sexes, and I don't think want it to be impossible to start a business because you might make a lot of money from it.  I just wish the view was better from on top of this pile of fences.

In which science actually self-corrects, for once

Amid all the stories of bad behaviour by researchers confronted with demonstrations of errors and other problems with their work --- I'm sure many readers have their own favourite examples of this --- I thought I'd start the year off with a story of somebody doing the right thing.

You may be familiar with our (that's James Heathers and me) GRIM article, in which we demonstrated a technique for detecting certain kinds of reporting errors in journal articles, and showed that there are a lot of errors out there.  The preprint was even picked up by The Economist.  GRIM has caused a very small stir in skeptical science circles (although nothing compared to Michèle Nuijten's statcheck and Chris Hartgerink's subsequent bulk deployment of it with reporting on PubPeer, a project that has been immortalised under the name of PubCrawler).  Some people have started using the GRIM technique to check on manuscripts that they are reviewing, or to look at older published articles.  Even the classic 1959 empirical demonstration of cognitive dissonance by Festinger and Carlsmith succumbed.

Round about the time that we were finalising the GRIM article for publication, I came across Mueller and Dweck's 1998 article [PDF] in JPSP, entitled "Praise For Intelligence Can Undermine Children's Motivation and Performance".  I'm quite skeptical of the whole "mindset" area, for a variety of reasons that don't matter here, but I was especially interested in this article because of the tables of results on page 38, where there are no less than 50 means and standard deviations, all with sample sizes small enough to permit GRIM testing.

This looked like a goldmine.  Unlike statcheck, GRIM cannot be automated (given the current state of artificial intelligence), so running one or two checks typically requires reading and understanding the Method section of an article, then extracting the sample sizes and conditions from the description ("Fifty-nine participants were recruited, but three did not complete all measures and were excluded from the analyses" is often what you get instead of "N=56"; if anyone reading this works in an AI lab, I'd be interested to know if you have software that can understand that), and then matching those numbers to the reported means in the Results section.  So the opportunity to GRIM-check 50 numbers for the price of reading one article looked like good value for my time.

So I did the GRIM checks, taking into account that some of the measures reported by Mueller and Dweck had two items which effectively doubles the sample size, and found... 17 inconsistencies in the means, out of 50.  Wow.  I rechecked - still 17.  And a couple of the standard deviations didn't seem to  be possible, either.  (I have some code to do some basic SD consistency checks, but the real expert here is Jordan Anaya aka OmnesRes, who has taken the idea of GRIM and done some smart things with it).

What to do?  I got James to have a look, and he found the same problems as me.  We decided to contact Dr. Carol Dweck, the senior and corresponding author on the article.  Would she want to talk to us? Would she even remember what happened back in 1998?

To our slight surprise (given some other recent experiences we have had... more to come on that, but probably not any time soon), Dr. Dweck wrote back to us within 24 hours, saying that she was going to look into the matter.  And within less than four weeks, we had an answer, in the form of a 16-page PDF document in which Dr. Dweck and her co-author, Dr. Claudia Mueller, had brought Dr. David Yeager to help them.  They had gone through the entire article, line by line, and answered every one of our points.

For several of the inconsistencies that we had raised, there was a conclusive explanation.  In some cases this was due to some degree of unclear or omitted reporting in the article, some of which the reader (me) ought perhaps have caught, others not.  (To our amazement, two of the study datasets were still available after all this time, as they are being used by a teacher at Columbia.)  A few other problems had no obvious explanation and were recorded as probable typos or transcription errors, which is a little unsatisfying but perhaps not unreasonable after 18 years.  And in one other case, outside the table with 17 apparent inconsistencies, I had highlighted a mean that was (rather obviously) not wrong; getting a long sequence of precise measurements right is hard for everybody.

So for once --- actually, perhaps this happens more often than we might think, and the skeptical "literature" also suffers from publication bias? --- the process worked as advertised.  We found some apparent inconsistencies and wrote a polite note to the authors; they investigated and identified all of the problems (and were very gracious about us calling out the non-problems, too).  With Dr. Dweck's consent, I have written this story up as an example of how science can still do things right.  I'm still skeptical about mindset as a construct, but at least I feel confident that the main people researching it are dedicated to doing the most careful reporting of their science that they can.

You can find the full report here (look in Files/Output).

Here's to a collegial, collaborative, self-correcting 2017!