16 March 2017

Cornell salutes America's teenage female combat heroes of WW2

A lot has been written about the four "pizza papers" mentioned by Dr. Brian Wansink in his blog post of 21 November 2016.  But so far, it seems nobody has paid much attention to the fifth article co-authored by "the Turkish grad student" mentioned in that blog.  That's a shame, because it appears to reveal a remarkable bit of American military history.

Here's the article:

Sığırcı, Ö, Rockmore, M., & Wansink, B. (2016). How traumatic violence permanently changes shopping behavior.  Frontiers in Psychology, 7, 1298. http://dx.doi.org/10.3389/fpsyg.2016.01298

Because Frontiers in Psychology is an open access journal, you can read this article here.  However, you might want to look at the annotated copy that I've made available here, where you will also find a couple of other relevant files.

When I first read Sığırcı et al.'s article, I didn't pay it too much attention, partly because Tim van der Zee, Jordan Anaya, and I were busy enough with the pizza papers, and partly because its conclusions seemed plausible, even obvious: Long-retired military veterans who experienced heavy or frequent combat are more cautious, conservative consumers than those who only experienced lighter or infrequent combat.  After all, there's an old military saying: "There are old soldiers, and bold soldiers, but there are no old, bold soldiers".  Maybe a degree of caution helps you survive in war, and also carries over into whatever makes you a careful consumer.  However, after a few weeks of exploring the rest of the output of the Cornell Food and Brand Lab, I decided to revisit this article and see what else I could learn from it.

The usual Cornell comedy stuff

Let's start with the self-plagiarism.  It's actually quite mild, compared to what we've seen; about 350 words of the 500-word method section are copied (absolutely verbatim this time, including the typos; no tweaking of occasional words, as we saw in other cases) from this article:

Wansink, B., Payne, C. R., & van Ittersum, K. (2008).  Profiling the heroic leader: Empirical lessons from combat-decorated veterans of World War II.  The Leadership Quarterly, 19, 547–555.  http://dx.doi.org/10.1016/j.leaqua.2008.07.010

I will spare you the usual yellow-highlighted image here, but you can see the duplicated text in the annotated versions of the files here.

Some other things that regular readers of this blog and Jordan's will be expecting are GRIM inconsistencies (i.e., means or percentages that don't match the sample sizes)  and F statistics that are not consistent with the reported means and standard deviations.  And these readers will not be disappointed.  There are 26 GRIM inconsistencies in Table 1, and seven out of the eight F statistics in Table 2 are inconsistent, as is one in Table 3.  (Actually, this is a little unfair.  The inconsistent F in Table 3 is probably the result of both of the standard deviations on the same line in the table being wrong.)  All of these problems are visible in the annotated copy of the article here.

So we could have included this article in with the four pizza papers; it would have fit right in with the quality of those (and other articles from the same lab going back ten or more years).  But just to blog about it now on that basis would be like publishing a news story saying that the President of the United States was rude and badly informed during a press conference: headline news in 2016, not so much in 2017.  The real fun starts when we look closer at the most basic numbers in the study (indeed, in any study), namely the demographics of the sample.

The unique selling points

Let's start with the ages.  This survey was done in the year 2000.  We don't know exactly when, but let's assume it was January (as we'll see, the results would have been even more extreme if participants had reported their ages in December).  These are veterans of combat operations in World War 2, which ended on September 2, 1945.  Assuming that these veterans were not underage volunteers who lied about their age to join the military, their minimum age on that date must have been about 18 years and 2 months, allowing time for basic training and shipping out to the Pacific theatre of operations.  So the youngest of these veterans would have been born in July 1927 and been 72 years of age in January 2000.

Now look at the standard deviations (SD) for the ages.  They are quite high, especially when there is not much room on the low side of the mean.  In fact, those SDs constrain the pattern of ages quite severely.  James Heathers kindly used his new tool, SPRITE, to build a series of possible distributions of ages matching this constraint (plus a supplementary one that none of the respondents should be older than 105 at the time of the survey, because otherwise it proposes a whole bunch of people aged over 120).  Here's what a typical such distribution looks like:
What this means is that, if those means and SDs are correct, 200 of the 235 respondents were between 18 and 18.5 years of age at the end of WW2, having presumably experienced repeated heavy combat only in the last few months of the war.  There were one or two slightly older soldiers, and then a bunch who were 50 at the time (and so were 105 years old in 2000).  Almost none of the soldiers can have been aged 19, or 20, or 21, if these numbers are correct.  That is, if the sample is at all representative of actual US combat veterans from WW2, almost nobody who joined at 21 in 1941, or 18 in 1943, survived until 2000 (whether they were exposed to heavy or light combat).

Even more interesting, however, is just how many of these young men, who fought their way to glory at Anzio, Omaha Beach, and Iwo Jima, must have been... women.  Have a look two lines further down in the demographic data:

Among these veterans, 79.3% who saw heavy combat and 80.3% who saw light combat identified in 2000 as men.  Now I'm going to go out on a limb here for a moment and guess that only a very small proportion of these men came out as transgender since 1945.  With that assumption, the implication here is that 20% of these veterans of heavy combat are, and were at the time, women. (*)

This has to be the historical scoop of the last 70 years.  The role of American women outside the US in World War 2 has up to now been believed to be mostly limited to nursing, well away from the front line.  Woman only officially obtained the right to serve in combat roles in the US Army in March 2016.  Yet here we have evidence to suggest that many women took part in combat of all kinds in World War 2—making up about a fifth of all soldiers who were involved in combat operations.  How come we've never seen this in all those war movies?  Can it just be due to sexism on the part of Hollywood producers?  Why can't the true story of the hundreds of thousands of 18-year-old women whose courageous combat liberated the world from the menace of the Axis powers be told?

[scratching sound of needle being pulled from vinyl record]

OK, fun's over.  Let's be serious for a moment.

First, just to be clear, nothing in the two preceding paragraphs is intended to take a dig at women or trans people.

My purpose in this post has been to show that some of the absurdities in this article (and many, many more, whether from this lab or not) are visible to almost anyone who cares to read it.  You don't need to know a thing about statistics to see that the implication that "20% of the US soldiers who saw combat in World War 2 were women" is absurd.  You don't need to know much more about what distributions look like to see that a mean of 75 and an SD of 9 with a floor of 72 is going to lead to a huge right skew. And you can probably guess that, if a piece of work (whether a scientific article or a restaurant meal or a car) has problems like that visible from the moment you look at it, it may well have a bunch of other problems that you only need some simple tools to uncover.

This article appeared in what claims to be a peer-reviewed journal.  The names of the reviewers and action editor are displayed on the article's web page.  I'm trying to work out exactly how closely any of these people looked at the manuscript before approving it for publication, thus elevating it to the status of "science" so that people can write press releases.  We seriously need to improve the way we go about reviewing.  As it is, though, with some publishers, it seems that things may even be getting worse.

(*) Someone suggested that maybe there was a small percentage of women in the original combat roles, and that they became proportionately more numerous from 1945 to 2000 through having higher survival rates.  I find this fairly unlikely, but since we are arguably conditioning on a collider here, I thought I'd mention it for completeness.


  1. The veteran database has been sliced many times, it seems. I haven't looked up all of the papers, but in the attached paper they explicitly state that a subgroup of the sample were under 17 at enrollment.

    Another curiosity of the data is that some widows filled the surveys for their late husbands. Any chance they might have used these data, resulting in the 20% women ratio?

    (Another option would be that their sample included non-US veterans; people who fought for Russia, Yugoslavia, some other eastern European countries, or even some volunteers for Britain. Women did fight under these flags in WW2. It seems unlikely to be represented in their sample, though)

  2. @Niv:

    In the article to which you linked, the authors stated that "we discarded the responses of ... 239 [respondents] who were age 17 or younger in 1945 since they would not have completed high school before the end of World War II". I think it's reasonable to assume that anyone who had not completed high school before the end of WW2 did not see heavy combat during that war. Similarly, in that linked article, the authors state that they *removed* the surveys filled in by the widows.

    That suggests to me that we can probably assume that when they decided go on a fishing expedition --- oops, sorry, a "deep data dive" --- a few years later, they will have been aware that some of the questionnaires might not have been filled in by actual combat veterans from WW2. Perhaps they could have clarified that for us in their method section, if they had *actually taken the time to write one* instead of copying it verbatim from a previous article.

    More generally, nobody should have to read another article to find out whether the reported demographics are correct or not. If the article says that their sample consisted of US combat veterans from WW2, it is not up to the reader to think, "Oh, well, if this were any other lab I'd believe what they wrote, but because this is the Cornell Food and Brand Lab and we know everything they produce is surrounded by a huge cloud of uncertainty, I'll just assume that what they meant was X and Y and Z", as if they deserve the benefit of the doubt for trying, like if an undergraduate brought their 6-year-old kid brother to class who insisted on doing all the assignments.

    1. I totally agree with your final point, of course. No one is supposed to dig that info up.

      I just wanted to highlight where any assumptions we may make might be biased. So, for instance, I believe that boys as young as 15 did participate in heavy fighting. That was the situation in other armies.

      Also, there was another reply to that tweet that mentioned that many women did participate in fighting, especially at Pearl Harbor.

  3. Our study utilizes data from the 2000 University of Illinois Veteran Survey...
    A random national sample of 7,500 World War II veterans was asked to complete a questionnaire about their experiences before, during, and after the war. ...
    In all, a total of 1123 surveys (25.6%) from World War II veterans were received in a timely enough manner to be included in the study.

    Here is another description of the 2000 Veteran's Survey:
    In the year 2000, two waves of surveys were used to gather information on U.S. veterans. Wave 1 collected a random sample of 500 veteran respondents born before 1928. Wave 2 collected a random sample of 250 veteran respondents.

    Another version has 931 participants in the study.

    If you can find evidence that this ever-growing survey ever actually occurred, you are doing better than me.

    1. In the 2012 account of the Survey, 750 veterans were targeted -- including later wars, so those data are a superset of the present 7500 targets. 467 provided full responses, of whom 62.59% were combat veterans (Table 1), i.e. 229.3.
      Of those combat veterans, 99.62% (228.4) were male, with a std.dev. of 6.19%. The same proportion were married.

  4. Regarding the possibility that the "female" respondants were widows, the paper says that

    "Of the 7,500 questionnaires that were initially mailed, 3188 were undeliverable (due to death), including 72 that were returned by the late veteran’s spouse."

    Interestingly, 19.7% of 120 = 23.6 and 20.7% of 235 = 48.645, total = 72.245

    So the number of female "soldiers" does add up to the number of widow's responses (to the nearest integer anyway).

  5. @Neuroskeptic: I think that a reasonable interpretation of "72 that were returned by the late veteran’s spouse" is that the form came back with a nice cover letter saying "Thank you for sending this form; unfortunately, my husband(*) recently passed away". Even if the spouse filled out the form, she(*) must have indicated that it was her(*) doing so (unless they did something like testing the DNA from where the envelope was licked for Y chromosomes), at which point the data should have been discarded as not being the veteran's own description of his(*) experiences.

    (*) Simplifying assumptions: Male combatant, female spouse.

  6. According to a 2009 account of the project, only 5000 questionnaires were sent out:
    To investigate this, a random selection of 5000 veterans born before 1928 were obtained from census data. In the year 2000, each veteran was sent a survey, a cover letter, and a business reply return envelope (see Wansink, Payne, & van Ittersum, 2008).
    2376 surveys were deliverable, and 493 responses were received.

  7. @Smut Clyde: https://www.youtube.com/watch?v=KF6SNxNIV08

  8. 2009:
    To investigate this, a random selection of 5000 veterans born before 1928 were obtained from census data.
    To solicit respondents, a random sample of [750] veteran addresses was obtained from census data.

    Do US census results really provide the name, address and age of every war veteran?

  9. You might want to add Wansink & Wansink (2013) to the comparison, as it includes the same description of the Survey (7500 surveys sent out to only WWII veterans; 3188 undeliverable, 1123 responses, 25.6% or 43% response rate).

    Relevant responses are divided into No Combat (N = 203), Light combat (132) and Heavy combat (210), 545 in total. This is a relatively small disagreement with the Frontiers paper, which ignores the No Combat group, somehow leaving 355 -- “120 of them had experienced a light combat and 235 of them had experienced heavy combat”.

    It does not fit so well with the 2008 paper, which only looked at the "Heavy and frequent" group, where N=526.

    This could be explained by omitting different individuals from different studies depending on what relevant subsidiary questions have been answered, but it does leave open any number of forking pathways.

  10. I was also puzzled by the reference to Army service. My father fought as a Marine all the way from Guadalcanal [as an 18-year-old] through the end of the war in the South Pacific. Until late in the war there were no or few Army combatants there.
    Also an uncle got his mother to lie on his papers and he got in at age 15 to the Seabees in early 1944. He died in the early 90s, but would have only been 74 in 2000.
    Anyway, that paper is a mess.