19 April 2020

In psychology everything mediates everything


In the past couple of years I have reviewed half a dozen manuscripts with abstracts that go something like this:

<Construct X> is known to be associated with higher levels of well-being and healthy psychological functioning, as indexed by <Construct Y>. However, to date, no study has investigated the role of <Construct M> in this association. The present study bridges this gap by testing a mediation path model in a sample of undergraduates (N = 100). As predicted, M fully mediated the positive association between X and Y.  These results suggest that X predicts higher levels of M, which subsequently predicts higher levels of Y. These results provide new insight that may advance a coherent theoretical framework on the pathways by which M enhances psychological well-being.

There is typically a description of how the 100 participants completed measures of constructs X, M, and Y, with a table of correlations that might look like this:

    X       Y
Y .24*
M .52***  .32**

* p < .05; ** p < .01; *** p < .001.

Then we get to the mediation analysis. More often than not this is done using the PROCESS macro in SPSS, but it can also be done “by hand” using ordinary least-squares regressions. Here are the steps required (cf. Baron & Kenny, 1986):

  1. Show that X is a significant predictor of Y. You don’t actually need to do the regression for this, as the standardized regression coefficient and its associated p value will be identical to the correlation coefficient between X and Y, but sometimes the manuscript will show the SPSS output to prove that the authors conducted this regression anyway. (In the last manuscript that I reviewed, the authors managed to obtain a standardized regression coefficient that was different to the zero-order correlation, which did not enhance my confidence in the rest of their analyses.) Here the p value will be .016.
  2. Show that X is a significant predictor of M. Again, no regression is required for this, as it’s just the correlation coefficient. The p value in this example is about 3E−8.
  3. Regress Y on both X and M. If you get a significant regression coefficient for M then you have at least a “partial” mediation effect. If, in addition, the regression coefficient for X is non-significant then you have “full" mediation. Here, this produces the following standardized coefficients:
    • M: β = 0.268, p = .018
    • X: β = 0.101, p = .368
Ta-da! In this example, we have complete mediation: The p value for the mediator, M, is significant and the p value for X isn’t. We conclude that Construct M fully mediates the relation between Construct X and Construct Y. We can write it up and celebrate contribution to understanding the mechanisms that lead to well-being. Surely the end of mental distress is only one more grant away.

The problem is this: Absolutely any other variable that you might put in place of M, and which is correlated in the same way with X and Y, will also show exactly the same mediation effect. And there is no shortage of things you can measure—in psychology, at least—that are correlated at around .5 and .3 with two other variables, correlated at around .2, that you might have measured. Let’s say that X is some aspect of socioeconomic status and Y is subjective well-being. You can easily come up with any number of ideas for M: gratitude, optimism, self-esteem, all of the Big Five personality traits (if you reverse-score neuroticism as emotional stability), etc., without even needing to resort to Lykken and Meehl’s “crud factor” (“in psychology and sociology everything correlates with everything”; Meehl, 1990, p. 204).

I wrote some R code to demonstrate the example that I gave above, which you can find here. You will see that I performed the calculations in two ways. The first was to generate (with a bit of trial and error) some random data with the correct correlations. (This produces a bit of rounding error, so the p value for the beta for M in the regression is reported at .019, not .018.) The second—my preferred method, since you can generally use this starting with the table of descriptives that appears in an article—is to start with the correlations and perform the regression calculations from there. (A surprising number of people do not seem to know that you can generally determine the standardized coefficients of multiple regression models just from the correlation table. The standard errors—and, hence, the p values—can then be derived from the sample size. If you have the standard deviations as well, you can get the unstandardized coefficients. Add the means and you can calculate the intercepts too. You don't need the raw data for any of this.)

If your initial choice for variable X is more strongly correlated with Y than M is, then you can usually just swap M and Y around, because there is very often nothing to say that whatever X is measuring occurs “before” whatever M is measuring, or vice versa—especially if you just hauled your undergraduates in and gave them measures of X, M, and Y to complete. The reason why you want your mediator, M, to be more strongly correlated than X with the outcome (Y), is a little-known phenomenon of two-variable regression that I like to think of as a sort of “Matthew effect”. Feel free to skip the next paragraph in which I explain this in tedious detail.

When the two predictors are moderately strongly correlated with each other (.52, in our case), then although their zero-order correlations with the outcome variable might be quite close together (.32 and .24 here), their standardized regression coefficients will diverge by quite a bit more than their respective correlation coefficients. Here, M’s correlation of .32 led to a beta of 0.268, which is a 16% reduction, but X’s correlation of .24 was reduced by 58% to a beta of 0.101. If the correlation between M and X had been a little higher (eg, .60 instead of .52), the beta for M would actually have been larger (0.275) while that for X would have been even smaller (0.075). At some point along the M–X correlation continuum (around .75), the beta for M would be exactly equal to the correlation coefficient of M with Y, and the beta for X would be zero. Continuing even further, we would hit “negative suppression” territory, with M’s standardized regression coefficient being greater than the original correlation coefficient of .32, and X’s being negative. Many people seem to have a rather naïve view of multiple regression in which when you add a new predictor, the betas for all of the predictors are reduced in some roughly equal proportion, but the reality is nothing like that. You can explore what happens with just two predictors (with more, things get even wilder) using a Shiny app that I wrote here.

So it’s possible to build an almost infinite number of mediation studies, all of which will appear to tell us something about the mediation of the relation between two psychological variables by a third, although almost all of them are just illustrating a known phenomenon of multiple regression. Again, everything is determined by the three correlations between the variables, plus the sample size if you care about statistical significance. (Alert readers will have noticed that whether or not mediation is “full” or “partial” will depend to a large extent on the sample size; with enough participants even the residual effect of X on Y will be large enough that its p value doesn’t drop below .05. But of course, alter readers will also know that these days statistical significance doesn’t anything, right?)

Now, am I saying that all of the mediation articles that I get to review are based on an atheoretical “throw some numbers at the wall and see what sticks” approach that I may have implied here is what sometimes happens? Well, no... but I’m not saying that none of them are, either. I have heard first-hand from grad students, in several cases, what happens when they have a bunch of variables and no obvious result: Their supervisor suggests that they write them up as a mediation analysis.

I don’t think that preregistration will necessarily help all that much here, because it is quite predictable from previous knowledge that X, M, and Y will have the pattern of correlations needed to produce an apparent mediation effect. I’m going to suggest that the only solution is to refrain from doing this kind of mediation analysis altogether in the absence of (a) much better theoretical justification than we currently see, and (b) some kind of constraint on the order in which changes in X, M, and Y occur. Without a demonstration that the causal arrows are running from X to M and M to Y (MacKinnon & Pirlott, 2014), and not vice versa, we have no way of knowing whether we are dealing with mediation or confounding, especially since in many cases the constructs X, M, and Y may themselves be caused by multiple other factors, and so ad infinitum. (cf. Arah, 2008). In the absence of experimental manipulation, causality is hard to demonstrate, especially in psychology. 


References
Arah, O. A. (2008). The role of causal reasoning in understanding Simpson's paradox, Lord's paradox, and the suppression effect: covariate selection in the analysis of observational studies. Emerging Themes in Epidemiology, 5, 5. https://doi.org/10.1186/1742-7622-5-5
Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182. https://doi.org/10.1037/0022-3514.51.6.1173
MacKinnon, D. P., & Pirlott, A. G. (2014). Statistical approaches for enhancing causal interpretation of the M to Y relation in mediation analysis. Personality and Social Psychology Review19(1), 30–43. https://doi.org/10.1177/1088868314542878
Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66(1), 195–244. https://doi.org/10.2466/pr0.1990.66.1.195

(Thanks to Julia Rohrer for her helpful comments on an earlier draft of this post. If the whole thing is garbage, it's probably because I didn't take incorporate all of her thoughts.)


1 comment:

  1. Hi Nick,
    Thank you for this blog post, which highlights one of the most important issues in mediation testing. I have to admit that I also have been guilty of publishing several mediation analyses based on cross-sectional self-report data. I guess that researchers are just not aware of this problem unless they dive deeper into this topic.
    I just wanted to add some notes:
    Researchers who use PROCESS actually shouldn't write about complete vs. partial mediation. PROCESS estimates the indirect effect as the product of the a and b path and this indirect effect is either signifcant or not. Thus, there is no place for the concept of partial mediation in the framework by Hayes and others. Also, there can be an indirect effect in addition to, or in the absence of, a direct and a total effect. So, the significance of the single paths actually doesn't matter much, that is, significance of the single paths are not a prerequisite for finding a mediation effect (but, of course, as you write it is very likely to find one if all three variables are correlated with each other).
    Unfortunately, it seems that most users of PROCESS haven't read Hayes' book, so they still use the outdated thinking when writing about their analyses (https://open.lnu.se/index.php/metapsychology/article/view/870)...

    ReplyDelete