On or about July 18, Jean Burns sent a message to jcs-online, reading in part:
I think one of the most exciting findings in parapsychology has been the development by an experimental test by Ed May and co-workers (May et al., 1995) which can determine whether psi results have been produced by psychokinesis (PK) or extrasensory perception (ESP).
The above analysis has been applied to large number of experiments using random number generators, in which correlations with operator intention demonstrated the presence of psi (May et al., 1995). However, the analysis of May et al. showed that PK was *not* present.
[...] May, E.C., Utts, J.M., and Spottiswoode, S.J.P. (1995). Decision Augmentation Theory, Journal of Scientific Exploration, 9(4), 453-488.
Presumably the strong statement that May et. al. "showed that PK was *not* present" is driven primarily by the 8.6-sigma refutation quoted in the abstract. May, responding to a post of mine, wrote:
York suggested that I quote an 8-sigma effect based upon a meta analysis of all the RNG data. Wrong. In the abstract of our JP paper we attribute an 8.6 sigma favor of the influence model to and analysis of a large number of individual button presses of PEAR data.
I apologize for my misinterpretation of the statements made by May et.al.; it is very clear from the relevant paragraph (p. 467 of the JSE reference given above) that the 8.6-sigma result comes from examining the data generated by one specific operator at PEAR, at two different sequence lengths.

However, I feel compelled to point out that the figures given in that very same paragraph of the JSE article also show that the 8.6-sigma figure is, to say the least, suspect, and the use of that value without qualification or caveat in the abstract of the article, and in subsequent discussions, is distinctly questionable.

In that paragraph, the authors demonstrate the calculation of the 8.6 figure from the observed Z^2 data in the two subsets used. It is derived by using the observed effect size in the short-sequence data as a prediction for the effect in the long-sequence data. However, they also point out that if one performs the calculation in the other direction, using the long-sequence observation to predict an effect for the short-sequence data, the t-score is only 2.398, rather than 8.643. Since there is no obvious quality that makes one dataset the "prediction" and the other the "test", it is decidedly not clear why one should prefer one value over the other, and the fact that they are in such stark contrast (an overwhelming powerful refutation vs a moderately convincing one) suggests that there is something worrisome about such a strong dependence on an essentially arbitrary choice of analysis method.

I will therefore spend a couple of paragraphs showing the application of a standard, symmetrical test to the data used by May et.al., restricting myself to the data values actually published in the paper lest I be accused of generating a red herring by throwing some extra data into the pot. Fortunately, the raw data used by May et.al. to derive the T-scores above are also given on page 467 of JSE 9/4. The short-sequence data involve 5918 trials at 200 bits per trial; the Stouffer Z for the presence of an anomalous effect is 3.37, the observed Z^2 value is 1.063 +/- 0.019. The long-sequence data comprises 597 trials of 10^5 bits each; the Stouffer Z is 2.45, the observed Z^2 is 1.002 +/- 0.050.

Given these figures, the most natural evaluation for consistency with a hypothesis would seem to be to treat both observations as empirical measurements of a model parameter, and construct a T- or Z- test against the null hypothesis that the parameter has the same value. The first line of May, Utts, and Spottiswoode's Table 1 (p. 461) gives the expected value of Z^2 in terms of model parameters. For the PK or "micro-AP" model, E[Z^2] = 1 + epsilon^2 *n; for the DAT model, it is simply the sum mu(z)^2 + sigma(z)^2. The fact that the latter is a sum of two unknown parameters of the model is essentially irrelevant, since the same sum is measured in both cases.

Calculating from the observed figures listed for Z^2 (we're back on p. 467 again), one finds that for n=200 data epsilon^2*n = 0.063+/-0.019; since n=200, epsilon^2 = (3.15+/-0.95)x10^(-4). (I am keeping more significant digits than I am entitled to in the intermediate figures, to try to avoid too much accumulation of roundoff error.) For the long-sequence data, one finds epsilon^2 *n = 0.002+/-0.05; since n=10^5, this observation gives epsilon^2 = (2+/-50)x10^(-8). Any hand calculator with a square root key will allow the reader to verify that this gives a T score of 3.3 against the micro-AP hypothesis. Both observations are treated symmetrically in this calculation, rather than one being used to predict the other.

For the DAT model, the unknown parameter is simply equal to the observed Z^2, and so the T test is simply the comparison of 1.063+/-0.019 against 1.002 +/- 0.050, yielding T=1.14.

I am not presenting degrees of freedom for any of the T scores, as they have so many that they are for all practical purposes equivalent to Zs.

The two-sample T-test is a straightforward technique that I have found in several standard references on statistics. The predict-and-compare process used by May et.al. to get T=8.6 (or T=2.4, depending on which direction you choose), on the other hand, is not one that I have ever before seen applied to such a simple evaluation. I therefore contend that the data examined on p. 467 of the JSE reference above amount, by standard and well-understood statistical tests, to a T=3.3 result, not to a T=8.6 result, and that people should stop talking about an eight-sigma refutation of PK models for observed anomalies.

York Dobyns

RetroPsychoKinesis Project Home