dRo& .-

Randomicity, P r e d i c t a b i l i t y , and Mathemat ical I n f e r e n c e S t r a t e g i e s 5 1

i n ESP Feedback T r a i n i n g S i t u a t i o n s

Char les T. T a r t

Psychology Department

U n i v e r s i t y o f C a l i f o r n i a - 5%

Davis, CA 95616 7- 6 7- Y,

I have long b e l i e v e d t h a t t h e most p r e s s i n g problem i n parapsychology

i s how t o g e t s t r o n g and r e l i a b l e m a n i f e s t a t i o n s of p s i i n t h e l a b , s o we

can p r o f i t a b l y g e t on t o t h e impor tan t q u e s t i o n s of what p s i i s and how

i t works. A s a r e s u l t , some y e a r s ago I r a t h e r i n n o c e n t l y s e t o u t t o de-

t e rmine i f p r o v i d i n g immediate feedback of r e s u l t s t o p e r c i p i e n t s would

a l l o w them t o a t l e a s t s t a b i l i z e , i f not a c t u a l l y s t r e n g t h e n , t h e i r ESP

a b i l i t i e s . L i t t l e d i d I r e a l i z e what a h o r n e t s ' n e s t o f c o n t r o v e r s y I

would s t i r up! The i n i t i a l r e s e a r c h , pub l i shed i n my Parapsychology

Foundat ion monograph ( T a r t , 1975) , and more wide ly i n my L e a r n i n g -- t o Use

E x t r a s e n s o r y P e r c e p t i o n book ( T a r t , 1976a) , has s t o o d up r a t h e r w e l l t o

l e n g t h y q u e s t i o n i n g s and a t t a c k s from O'Brien (1976), S t a n f o r d (1977) ,

Gardner (1977) , b u t today I f i n d myself under f i r e from G a t l i n ' s guns. I

s h a l l t r y t o a d d r e s s myself t o some u s e f u l q u e s t i o n s t h a t a r i s e when im-

media te feedback of r e s u l t s i s provided t o p e r c i p i e n t s .

When the v a s t m a j o r i t y o f ESP s t u d i e s were done w i t h o u t immediate feed- !

back t o t h e p e r c i p i e n t s , t h e q u e s t i o n o f p o s s i b l e b i a s e s i n t a r g e t sequences ,

s i g n i f i c a n t d e p a r t u r e s from e q u i p r o b a b i l i t y and s e r i a l independence, a r t i -

f a c t u a l l y i n f l a t i n g t h e r e s u l t s was r a t h e r e a s i l y handled. Unless t h e g l o b a l

b i a s e s o f t h e p e r c i p i e n t s j u s t happened t o match t h e g l o b a l b i a s e s o f t h e

t a r g e t sequence, a m a t t e r t h a t cou ld be checked by c o n t r o l match ings , such

b i a s e s were n o t impor tan t u n l e s s o f v e r y l a r g e magnitude. With t h e i n c r e a s -

i n g use o f immediate feedback abou t t a r g e t i d e n t i t y , i n a n e f f o r t t o s t a b i l i z e

T a r t -2

and improve ESP performance ( T a r t , 1966 ; 1977e) , q u e s t i o n s abou t p o s s i b l e

e f f e c t s o f b i a s e s i n t h e t a r g e t sequences a r e more i m p o r t a n t , f o r we can con-

c e i v e of a p e r c i p i e n t g r a d u a l l y l e a r n i n g what t h e b i a s e s of the t a r g e t se -

quences a r e and then a l t e r i n g h i s own response s t r a t e g i e s t o t a k e advantage

o f them, t h u s c r e a t i n g a r t i f a c t u a l " h i t s " which might t e l l us something

abou t mathemat ica l i n f e r e n c e s t r a t e g i e s , , b u t l i t t l e o r n o t h i n g about ESP.

I n t h e b r i e f t ime a l l o t t e d t o me I s h a l l t r y t o o u t l i n e some new per-

s p e c t i v e s on t h i s i s s u e t h a t I have developed, such a s f i n d i n g t h a t t h e

s t a n d a r d Chi-square measures of b i a s . a r e n o t v e r y u s e f u l measures of the

p r e d i c t a b i l i t y of a b i a s e d sequence, and p r e s e n t a b r i e f d e s c r i p t i o n of the

r e s u l t s o f a powerful mathemat ical i n f e r e n c e p r e d i c t o r program developed by

Eugene Dronek and m e t h a t a t t a c k s t h e problem of p r e d i c t i o n of b i a s e d sequences

d i r e c t l y . U n f o r t u n a t e l y , whi le i t i s u s e f u l t h a t D r . G a t l i n r a i s e d t h i s

i s s u e here today, he r own s o l u t i o n t o i t i s g r a v e l y flawed i n a v a r i e t y o f T' Wtl , be q,t u4, /ee<G& &&- &d'/ bi $ey,.zj-&IC f f ics I ways, and t h e p a r t i c u l a r c o n c l u s i o n s she ha reached a r e i n v a l i d . These f l a w s

?=i

i n c l u d e such t h i n g s a s a p e r s i s t e n t f a i l u r e t o unders tand t h e - d i f f e r e n c e be-

tween v a l i d p r e d i c t i o n and t r i v i a l p o s t d i c t i o n , t h e c l a s s i c a l e r r o r of equa t -

i n g c o r r e l a t i o n w i t h c a u s a t i o n , c o n f u s i n g p o t e n t i a l and p r o o f , c l a i m i n g en-

hanced s e n s i t i v i t y f o r her s t a t i s t i c a l p rocedures when they p robab ly l a c k

. v a l i d i t y , making a t l e a s t one major c l a i m f o r which she p r e s e n t s no s u p p o r t i n g

d a t a a t a l l , and i n t e r p r e t i n g i n v a l i d s t a t i s t i c a l a b s t r a c t i o n s i n ways which !

would have r e v e a l e d themselves a s o b v i o u s l y f a l s e i f s h e had looked a t t h e

raw d a t a they were based on. I s h a l l d e t a i l t h e s e problems below, a s they

a p p l y t o D r . ~ a t l i n ' s a n a l y s e s of my feedback t r a i n i n g d a t a . D r . P r a t t w i l l

comment on D r . G a t l i n ' s a n a l y s i s of t h e M a r t i n - S t r i b i c d a t a .

G a t l i n ' s Main A s s e r t i o n s :

A s a number o f c o l l e a g u e s have remarked t o me t h a t they have had d i f f i -

c u l t y f o l l o w i n g t h e w r i t t e n v e r s i o n o f D r . ~ a t l i n ' s paper ( G a t l i n , 1 9 7 8 ~ ) ~

T a r t -3

l e t me b r i e f l y summarize h e r main a s s e r t i o n s . She a s s e r t s t h a t :

( 1 ) . Given two n u m e r i c a l s e q u e n c e s t h a t n o t o n l y a r e e a c h b i a s e d , b u t

have ma tch ing p a t t e r n s o f b i a s e s , when you compare t h e s e s e q u e n c e s w i t h e a c h

o t h e r you w i l l g e t a h i g h e r number o f i d e n t i c a l numbers i n t h e same p o s i t i o n s ,

h i t s , t h a n you would e x p e c t i f you m i s t a k e n l y assume t h a t t h e s e q u e n c e s a r e

random and unb iased .

( 2 ) . Both some t a r g e t and many r e s p o n s e s e q u e n c e s i n my f i r s t f e e d b a c k

T r a i n i n g S t u d y show s i g n i f i c a n t d e g r e e s o f b i a s , a t t h e s i n g l e t , d o u b l e t , and

t r i p l e t l e v e l s .

(3) . Because o f immedia te f e e d b a c k a b o u t t h e i d e n t i t y o f e a c h t a r g e t ,

d a t a is a v a i l a b l e t o p e r c i p i e n t s from which t h e y migh t c a l c u l a t e c h a r a c t e r i s -

t i c s o f p o s s i b l e b i a s e s i n t h e i r i n d i v i d u a l t a r g e t s equences .

(4). The human mind has " f a n t a s t i c " ( h e r term) c a p a b i l i t i e s , presumably

u n c o n s c i o u s , f o r p a t t e r n r e c o g n i t i o n i n n u m e r i c a l s e q u e n c e s t h a t a l l o w u s e

o f t h e i n f o r m a t i o n o b t a i n e d t h r o u g h feedback : a s D r . G a t l i n p u t s i t , ' I . . . e x t r e m e l y s u b t l e b i a s e s a t h i g h n - t u p l e l e v e l s i n f i n i t e s e q u e n c e s c a n be

u t i l i z e d by t h e human mind."

( 5 ) . The p e r c i p i e n t s i n my s t u d y n o t o n l y had t h e p o t e n t i a l t o u t i l i z e

s u c h b i a s p a t t e r n s , t h e y did u s e them t o o b t a i n mos t o r a l l o f t h e h i t s c o r e s

above chance e x p e c t a t i o n . T h e r e f o r e , D r . G a t l i n asserts t h a t :

( 6 ) . A l l o f t h e above-chance s c o r i n g i n my f i r s t T r a i n i n g S t u d y c a n be !

e x p l a i n e d by p e r c i p i e n t s f i g u r i n g o u t and u t i l i z i n g t h e t a r g e t s equence b i a s e s

w i t h a n unconsc ious m a t h e m a t i c a l i n f e r e n c e s t r a t e g y , s o t h e r e i s no need t o

p o s t u l a t e ESP a s a n e x p l a n a t i o n o f t h e d a t a .

The re a r e o t h e r m i s c a l l a n e o u s a s s e r t i o n s i n D r . ~ a t l i n ' s p a p e r , b u t I

b e l i e v e I have a d e q u a t e l y o u t l i n e d he r main argument h e r e .

What Are t h e B i a s e s i n t h e T a r g e t Sequences?

Al though I am p r o b a b l y more f a m i l i a r w i t h D r . ~ a t l i n ' s D-measures o f b i a s

T a r t -4

t h a n most of you, I s t i l l f i n d them d i f f i c u l t t o f o l l o w , s o i n o r d e r t o look

a t p o t e n t i a l b i a s e s * i n t h e t a r g e t sequences of my f i r s t T r a i n i n g S tudy I

s h a l l pres 'ent them i n more f a m i l i a r Chi-square n e a s u r e s , t o which G a t l i n t s

** D-measures a r e s o s t r o n g l y r e l a t e d t h a t f o r p r a c t i c a l purposes t h e y a r e

e q u i v a l e n t , i n s p i t e o f he r s t r e s s on t h e i r uniqueness . Table 1 p r e s e n t s

b o t h Chi s q u a r e measures I have computed .and t h e few D-measures D r . G a t l i n

p r e s e n t e d i n her paper ( s h e p r e s e n t e d o n l y those r e a c h i n g s t a t i s t i c a l s i g n i -

f i c a n c e ) f o r t h e s i n g l e t ' and d o u b l e t l e v e l s . ,~. -,

, , , -, I_ I I I I > I I Y W - . . _ ... . - - - - - - - - - - - - - - - - - - -

I n s e r t Table 1 about he re __ .,..

Two n o t e s on t h e v a l u e s i n Table 1 should be made. My Chi-squares (and

o t h e r c a l c u l a t i o n s ) w i l l be c l o s e t o b u t sometimes n o t e q u i v a l e n t t o any c a l -

c u l a t e d from D r . ~ a t l i n ' s a n a l y s e s , a s she t r e a t e d t h e d a t a I provided her

i n a s l i g h t l y less a c c u r a t e manner by f i l l i n g i n t a r g e t d a t a a s s o c i a t e d w i t h

Passes (no response) by t h e p e r c i p i e n t s w i t h a response d i g i t from her computer

*I s h a l l use t h e term "bias" i n a g e n e r a l way i n t h i s paper t o d e s c r i b e even

t h e s l i g h t e s t d e v i a t i o n from a n e q u i p r o b a b i l i t y and s e r i a l independence model,

w i t h t h e q u e s t i o n o f whether such b i a s i s o n l y a random f l u c t u a t i o n o r is

r s t a t i s t i c a l l y o r p r a c t i c a l l y s i g n i f i c a n t handled s e p a r a t e l y .

**I c o r r e l a t e d f o r s e t s o f s i n g l e t and d o u b l e t D-measures and Chi-squares

andy)-chi-squares (Davis & Akers, 1974), c a l c u l a t e d by D r . G a t l i n , working

from a computer and p r i n t o u t s h e p rov ided me w h i l e s h e was s t i l l honor ing her

commitment t o p r o v i d e me w i t h c o p i e s o f a l l a n a l y s e s she c a r r i e d o u t on my

d a t a , a n d found t h e c o r r e l a t i o n s t o range from .91 t o 1.00, f o r an average

1 c o r r e l a t i o n o f .97.

T a r t -5

,'s pseudorrandom number g e n e r a t o r program, whereas I d e l e t e d t h e s e t r i a l s , s i n c e

t h e p e r c i p i e n t s d i d n o t r e c e i v e feedback when t h e y passed. Second, t h e Chi-

s q u a r e v a l u e s I have c a l c u l a t e d f o r t h e s i n g l e t l e v e l i n Table 1 a r e based

on a model of e q u i p r o b a b i l i t y o f a l l s i n g l e t s (p = . l o ) , b u t s i n c e t h e r e i s

some s i n g l e t b i a s , t h e d o u b l e t l e v e l Chi-square c a l c u l a t i o n s a r e c o r r e c t e d

f o r s i n g l e t b i a s by b e i n g based on m a r g i n a l t o t a l s , r a t h e r t h a n t h e o r e t i c a l

v a l u e s . Without t h i s c o r r e c t i o n , s i g n i f i c a n t Chi-square v a l u e s a t t h e doub-

l e t l e v e l migh t be o n l y r e f l e c t i o n s o f s i n g l e t b i a s , r a t h e r t h a n v a l i d l y i n d i -

c a t i n g a h i g h e r o r d e r b i a s . I n s o f a r a s I unders tand D r . G a t l i n ' s D;(T) mea-

s u r e i t a l s o c a l c u l a t e s d o u b l e t b i a s independent o f s i n g l e t b i a s .

The f a c t shown i n Tab le 1, t h a t seven o f D r . ~ a t l i n ' s d o u b l e t l e v e l

b i a s measures a r e s i g n i f i c a n t , when o n l y t h r e e o f mine a r e , i s a n i n t e r e s t -

i n g d i s c r e p a n c y . D r . G a t l i n c l a i m s g r e a t e r " s e n s i t i v i t y " f o r her D-measures

t h a n f o r c o n v e n t i o n a l Chi-square measures . Whether t h i s c l a i m o f s e n s i t i v i t y

o f h e r D-measures i s a c t u a l l y v a l i d , o r j u s t r e p r e s e n t s a n a r b i t r a r y lower ing

o f s t a n d a r d s f o r s i g n i f i c a n c e i s a p o i n t I w i l l l e a v e f o r t h e more s t a t i s t i -

c a l l y e r u d i t e t o work o u t , b u t what we shou ld n o t e h e r e i s t h a t t h e " s i g n i f i c a n t "

d e p a r t u r e s from t h e model t h a t D r . G a t l i n c l a ims t o have d e t e c t e d w i t h he r

D-measures a r e even t i n i e r t h a n t h o s e d e t e c t e d w i t h t h e Chi-square and, a s

w e s h a l l see l a t e r , t i n y d e p a r t u r e s from e q u i p r o b a b i l i t y and s e r i a l independence

!

* ~ r . G a t l i n d e s c r i b e s u s i n g a random number g e n e r a t o r program i n he r paper ,

b u t t h e computer a t UC Berke ley where s h e c a r r i e d o u t he r a n a l y s e s u s e s a

pseudo-random g e n e r a t o r w i t h a n a lgorhythm, a s p r a c t i c a l l y a l l computer ran-

dom g e n e r a t o r programs do. I t i s p r o b a b l y s a t i s f a c t o r i l y random f o r s h o r t

sequences .

T a r t -6

may n o t be p r a c t i c a l l y u s e f u l f o r making h i t s w i t h a m a t h e m a t i c a l i n f e r e n c e

s t r a t e g y .

As t h e Ch i - squa re measures i n T a b l e 1 show, two t a r g e t s e q u e n c e s were

s i g n i f i c a n t l y b i a s e d a t t h e s i n g l e t l e v e l and t h r e e a t t h e d o u b l e t l e v e l .

TWO o+ J" J-5, P 1 ~ - i x ~ ( ~ l , erst b i w - - -C - Why migh t t h i s have o c c u r r e d ?

I n r e p o r t i n g on t h i s b i a s i n . e a r l i e r p u b l i c a t i o n s ( T a r t , 1977a; 1977b) ,

I p o i n t e d o u t t h a t p r i o r t o c o l l e c t i n g d a t a i n t h e f i r s t s t u d y , my c o l l e a g u e s

and I were aware o f t h e many s t u d i e s which showed t h a t s u b j e c t s ' d e s i r e s t o

a l t e r t h e o u t p u t c o u l d s i g n i f i c a n t l y ' a f f e c t e l e c t r o n i c random number genera-

t o r s (RNGS). A l though we wanted o u r p e r c i p i e n t s t o b e o n l y p e r c i p i e n t s and

n o t a g e n t s , i .e . , t o u s e ESP b u t n o t PK, t h e y n e v e r t h e l e s s c o u l d s c o r e w e l l

b y u n c o n s c i o u s l y PKing o u r e l e c t r o n i c RNG t o make i t s o u t p u t f i t t h e i r r e s p o n s e

p r e f e r e n c e s . Our i n s t r u c t i o n s t o t h e p e r c i p i e n t s t o t r y a v a r i e t y o f s t ra-

t e g i e s may have f u r t h e r enhanced t h e PK p o s s i b i l i t y . For t h i s r e a s o n , we

made a d e c i s i o n b e f o r e s t a r t i n g t h e e x p e r i m e n t t o l e t t h e s a t i s f a c t o r i n e s s

o f o u r RNG rest on two samples o f 1 ,000 t a r g e t s e a c h , t a k e n b e f o r e we i n t r o -

duced our p e r c i p i e n t s t o t h e equipment and a f t e r t h e l a s t p e r c i p i e n t had

f i n i s h e d t h e s t u d y . These checks showed s a t i s f a c t o r y r a n d o m i c i t y a t t h e s i n g -

l e t and d o u b l e t l e v e l s . D r . G a t l i n h e r s e l f r e p o r t s ( G a t l i n , 1978b) t h a t t h e

e n t i r e t a r g e t s e q u e n c e f o r t h e T r a i n i n g S t u d y (5 ,000 t r i a l s ) shows s a t i s f a c -

! t o r y r a n d o m i c i t y a t t h e s i n g l e t and d o u b l e t l e v e l s . The f i n d i n g o f s i g n i f i -

c a n t non-randomici ty i n some i n d i v i d u a l t a r g e t s e q u e n c e s is t h u s n o t unexpec ted ,

a l t h o u g h t h e c u r r e n t i s s u e s would n o t have a r i s e n i f a l l t h e t a r g e t s equences

had shown r a n d o m i c i t y by t h e s t a n d a r d Chi-square tests. I n r e t r o s p e c t , t h e

a p p e a r a n c e o f b i a s i n some s e q u e n c e s h a s been a n a d v a n t a g e , s e r v i n g a s a s t i m -

u l u s t o c l a r i f y some i m p o r t a n t i s s u e s .

An o b s e r v a t i o n s u p p o r t i n g t h e i d e a o f PK a s t h e r e s p o n s i b l e f a c t o r f o r

t h e s i n g l e t b i a s e s i s t h a t t h e h i g h number i n t h e two s i g n i f i c a n t s equences

T a r t -7

i s n o t t h e same, a f i n d i n g s e n s i b l e i n l i g h t o f p s y c h o l o g i c a l number p r e f e r -

e n c e s which would p robab ly d i f f e r between i n d i v i d u a l s , b u t which would n o t

seem l i k e l y t o a r i s e f o r e l e c t r o n i c r e a s o n s .

My s u g g e s t i o n o f PK a s a p o s s i b l e e x p l a n a t i o n f o r l a c k o f randomic i ty

i n two o f t h e t e n t a r g e t sequences i s concerned main ly w i t h t h e s i n g l e t l e v e l

o f b i a s , a s t h e r e i s a more p r o s a i c e x p l a n a t i o n f o r t h e t h r e e t a r g e t sequences

which show s i g n i f i c a n t d o u b l e t b i a s . As e x p l a i n e d e l s e w h e r e ( T a r t , 1977b),

t h e RNG was b u i l t w i t h a pushbut ton s w i t c h t o a c t i v a t e i t f o r e a c h t r i a l .

Th i s pushbut ton was n o t o f t h e type t h a t makes a d i s c e r n i b l e "snap" o r " c l i c k "

when i t i s d e p r e s s e d , b u t a cheaper type i n which r e s i s t a n c e t o b e i n g pushed

s t e a d i l y i n c r e a s e s w i t h b u t t o n t r a v e l . I n i n t e r v i e w i n g some e x p e r i m e n t e r s

a f t e r t h e s i g n i f i c a n t d o u b l e t b i a s was found, t h e y p o i n t e d o u t t h a t sometimes

they would push t h e pushbut ton t o o b t a i n t h e n e x t t a r g e t number, n o t i c e t h a t

t h e number on t h e e l e c t r o n i c d i s p l a y had n o t changed, and then assume t h a t

they had n o t pushed t h e b u t t o n hard enough t o make c o n t a c t and a c t i v a t e t h e

RNG, s o t h e y would push i t again! T h i s would produce a g r e a t d e f i c i e n c y o f

XX d o u b l e t s ( l , l s , 2 , 2 s , e t c . ) , and , indeed , t h i s l a c k of XX d o u b l e t s is the

major c o n t r i b u t i o n t o t h e s i g n i f i c a n t d o u b l e t r e s u l t s . I f t h e t e n XX d o u b l e t

c e l l s a r e l e f t o u t o f t h e Chi-square c a l c u l a t i o n s , two o f the t h r e e s i g n i f i -

c a n t d o u b l e t t e s t s f a l l t o i n s i g n i f i c a n c e and t h e t h i r d one is j u s t s i g n i f i -

c a n t a t t h e .05 l e v e l . r ; ~ 1 1 ~

\fl I n summary, we have two o f t e n t a r g e t sequences which a r e s i g n i f i c a n t l y

be &If C ~ U I C L P vkiaG* , A ~ i l t h o ~ ~ h &> b i a s e d a t t h e s i n g l e t l e v e l , one t h a t i s s i g n i f i c a n t l y b i a s e d a t the d o u b l e t

A l e v e l independent of t h e XX d o u b l e t l a c k , and two t h a t a r e s i g n i f i c a n t l y b i a s e d

a t t h e d o u b l e t l e v e l due t o t h e exper imente r e r r o r which s y s t e m a t i c a l l y de-

p l e t e d XX d o u b l e t s from t h e t a r g e t sequences . The p o s s i b i l i t y o f i n f l a t e d

s c o r i n g through matching o f t h i s l a c k o f XX d o u b l e t s w i t h s i m i l a r b i a s e s on

t h e p e r c i p i e n t s ' p a r t has a l r e a d y been d i s c u s s e d i n t h e l i t e r a t u r e ( T a r t , 1977a;

T a r t -8

1977b; 1 9 7 7 f ) , and shown t o be t r i v i a l , g i v e n t h e v e r y h igh l e v e l of s c o r i n g

i n t h e s tudy .

T r i p l e t Leve l B i a s e s ?

What a r e consp icuous by t h e i r absence i n Table 1 a r e any t r i . p l e t l e v e l

t e s t s , a l t h o u g h D r . G a t l i n r e p o r t s h e r D; (T) measures and makes a number o f

i n t e r p r e t i v e s t a t e m e n t s a b o u t t h e i r s i g n i f i c a n c e . Unless he r D; measure has

v a s t l y d i f f e r e n t p r o p e r t i e s t h a n a t r i p l e t l e v e l Chi-square , however, i t seems , f i g u r e s

c e r t a i n t h a t her are i n v a l i d . I n computing any s t a t i s t i c , we need a c e r -

t a i n minimum sample s i z e i n o r d e r t o assume i t i s r e a s o n a b l y r e p r e s e n t a t i v e

of t h e p o p u l a t i o n i t is drawn from. For Chi-square t e s t s , t h i s i s u s u a l l y

e x p r e s s e d a s t h e r u l e t h a t t h e e x p e c t e d v a l u e i n e a c h c e l l o f t h e m a t r i x

must be f i v e o r g r e a t e r .

I n do ing he r t r i p l e t l e v e l t e s t s , D r . G a t l i n i s s p r e a d i n g a mere 500

d a t a p o i n t s over 1 , 0 0 0 c e l l s , f o r an e x p e c t e d v a l u e of o n l y one-hal f i n e a c h ,

v i o l a t i n g t h e minimum e x p e c t a t i o n r u l e by a f a c t o r of 10. Simple c a l c u l a t i o n

w i l l show t h a t v i o l a t i o n o f t h i s r u l e l e a d s t o g r o s s l y i n f l a t e d Chi-square

v a l u e s . For example, i f e v e r y t r i p l e t i n t h i s sample o f 500 were e x a c t l y

e q u i p r o b a b l e , t h a t i s t h a t t h e y were d i s t r i b u t e d over 500 s e p a r a t e c e l l s

i n t h e t r i p l e t t a b l e , c a l c u l a t i o n would g i v e a s u p e r - s i g n i f i c a n t Chi-square ,

c o r r e s p o n d i n g t o a CR o f -221 D r . G a t l i n e r r s i n even p r e s e n t i n g such in-

! v a l i d measures , much less i n t e r p r e t i n g what s h e t h i n k s t h e y show abou t t h e

p e r c i p i e n t s ' r e s p o n s e p a t t e r n s , o r i n t e r p r e t i n g them a s e v i d e n c e f o r t h e

e x i s t e n c e of h i g h e r o r d e r b i a s p a t t e r n s i n t h e t a r g e t sequences . Indeed , D r .

G a t l i n must n o t have b o t h e r e d t o compare he r c o n c l u s i o n s abou t t h e s e i n v a l i d

D; a b s t r a c t i o n s w i t h t h e a c t u a l d i s t r i b u t i o n of p a t t e r n s i n t h e d a t a a v a i l -

a b l e t o h e r , o r s h e c o u l d n o t have made t h e i n t e r p r e t i v e s t a t e m e n t s s h e made

a b o u t them. Much o f he r c l a i m abou t t h e a l l e g e d s u p e r i o r s e n s i t i v i t y o f he r

T a r t -9

D-measures then , r e s t s on i n v a l i d a n a l y s i s p rocedures . I s h a l l d r o p a l l

f u r t h e r r e f e r e n c e t o D r . ~ a t l i n ' s t r i p l e t d a t a .

D r . G a t l i n r e p o r t s t h a t t h e s c o r i n g r a t e o f t h e p e r c i p i e n t s i s s i g n i -

f i c a n t l y c o r r e l a t e d w i t h t h e d e g r e e o f d e p a r t u r e from a n e q u i p r o b a b i l i t y

and s e r i a l independence model. Th i s i s t r u e . U n f o r t u n a t e l y , s h e goes on t o

make t h e c l a s s i c a l s t u d e n t e r r o r of e q u a t i n g c o r r e l a t i o n w i t h c a u s a t i o n when

she s t a t e s "The s c o r i n g r a t e i s s i g n i f i c a n t l y p o s i t i v e l y c o r r e l a t e d w i t h

1 1 D l (T) and D2 (T) . . . . . . C l e a r l y t h e p a t t e r n i n g i n t h e t a r g e t i s b e i n g

used b~ (my i t a l i c s ) t h e s u b j e c t s t o i n f l a t e t h e i r s c o r e s . " ( G a t l i n , 1978c, - P 1 -

I a g a i n s u s p e c t o c c a s i o n a l PK by t h e p e r c i p i e n t s a s t h e cause of t h i s

c o r r e l a t i o n , w i t h t h e more s u c c e s s f u l p e r c i p i e n t s o c c a s i o n a l l y (unconsc ious ly )

t r y i n g a PK s t r a t e g y i n a d d i t i o n t o t h e i r ESP s t r a t e g i e s . Whatever t h e

c a u s e o f t h i s c o r r e l a t i o n , t h e c o n s i d e r a t i o n s I s h a l l now d i s c u s s e s t a b l i s h

t h a t t h e e x i s t e n c e o f s i g n i f i c a n t (and c o n s i s t e n t ) b i a s p a t t e r n s i n a t a r g e t

sequence e s t a b l i s h e s o n l y a p o t e n t i a l f o r u s i n g a mathemat ica l e s t i m a t o r s t r a -

t e g y , a p o t e n t i a l t h a t may n o t b e p r a c t i c a l l y u s e f u l . I t i s impor tan t t o

n o t e t h a t t h i s p o t e n t i a l e x i s t e d i n o n l y two o f t h e t e n t a r g e t sequences ( f o r

P3 and P5) of t h e f i r s t T r a i n i n g S t u d y , and a s imple and t r a d i t i o n a l way t o

handle i t would be t o j u s t d e l e t e t h e d a t a from those two p e r c i p i e n t s . T h i s

! would s t i l l l e a v e t h e o v e r a l l r e s u l t s enormously s i g n i f i c a n t (495 h i t s i n

4 ,000 t r i a l s , CR = 5.01, p<6x10'7, 2 - t a i l e d ) , b u t i t i s more u s e f u l and r e -

v e a l i n g t o examine t h e n a t u r e o f t h i s p o t e n t i a l , and t h e n c o n s i d e r t h e more

p e r t i n e n t q u e s t i o n s o f j u s t how much s c o r i n g can be g o t t e n from e f f i c i e n t ap-

p l i c a t i o n of s u c h p o t e n t i a l , and whether o r n o t t h e r e i s ev idence t h a t s u c h

a p o t e n t i a l was a c t u a l l y u t i l i z e d by t h e p e r c i p i e n t s .

Chi-Square Bias T e s t s Are Poor P r e d i c t o r Measures:

I f we t h i n k t h a t a p e r c i p i e n t migh t t a k e advantage o f b i a s e s he d i s c o v e r s

T a r t -10

th rough feedback i n a t a r g e t s o u r c e , i t i s p r o b a b l y common t o assume t h a t t h e

magni tude of t h e Chi-square measures ( o r D r . G a t l i n ' s D-measures) of d e p a r t u r e

from a n e q u i p r o b a b i l i t y and s e r i a l independence model i s a measure of how

p r e d i c t a b l e t h e sequence i s . T h i s i s i n c o r r e c t , a s t h e f o l l o w i n g examples w i l l

show.

Suppose a p e r c i p i e n t works i n a n exper iment i n v o l v i n g g u e s s i n g t h e num-

b e r s one t o t e n , i n a n exper iment f i x e d a t 200 t r i a l s . We s h a l l d e l i b e r a t e l y

use a b i a s e d t a r g e t s o u r c e , such t h a t o u t p u t s 1, 2, 3, 4 , and 5 a l l have a

p r o b a b i l i t y o f .15, w h i l e t h e o u t p u t s 6 , 7, 8 , 9, and 10 have a p r o b a b i l i t y

o f .05 , i n s t e a d o f a l l t a r g e t s b e i n g e q u i p r o b a b l e . The observed d i s t r i b u t i o n

of t a r g e t s i n t h e f i r s t h a l f of our exper iment (100 t r i a l s ) would l o o k l i k e

t h i s , d e l i b e r a t e l y g i v i n g i t a p e r f e c t r e f l e c t i o n o f t h e t a r g e t b i a s p a t t e r n

f o r s i m p l i c i t y o f i l l u s t r a t i o n

TARGETS 1 2 3 4 5 6 7 8 9

FREQUENCY 1 5 1 5 15 15 15 5 5 5 5 5

The s t a n d a r d Chi-square test f o r e q u a l f r equency o f o b s e r v e d - t a r g e t s

would t e l l us t h a t t h i s sample of 100 i s n o t from a random s o u r c e , a s we g e t

a Chi-square o f 25.00 w i t h 10 d e g r e e s o f freedom, p < . 0 1 , o n e - t a i l e d .

Suppose o u r p e r c i p i e n t t a k e s t h e f i r s t 100 t r i a l s t o c a t c h on t o t h i s

s i n g l e t b i a s p a t t e r n , s o t h a t w h i l e he has o n l y s c o r e d t h e 1 0 h i t s expec ted

under o u r assumed e q u i p r o b a b i l i t y model i n t h e f i r s t 100 t r i a l s , he w i l l u s e f

a mathemat ica l i n f e r e n c e s t r a t e g y , based on h i s new knowledge, f o r t h e remain-

i n g 100 t r i a l s o f t h e e x p e r i m e n t , H i s b e s t s t r a t e g y i s t o guess a 1, 2, 3,

4 , o r 5 on e v e r y t r i a l . I t does n o t m a t t e r whether he p i c k s one o f t h e h igh

f i v e and a lways g u e s s e s i t o r randomly a l t e r n a t e s among t h e h igh f i v e : we

would e x p e c t him t o s c o r e a b o u t 15 h i t s i n t h e second 1 0 0 . t r i a l s . I f we

a s s e s s t h e s i g n i f i c a n c e o f t h i s second-ha l f s c o r e under o u r a s sumpt ion of

e q u i p r o b a b i l i t y , w e compute a CR of 1.67, p < .05, o n e - t a i l e d . For t h e whole

exper iment o f 200 t r i a l s , we now have ( l W 1 5 ) =25 h i t s w i t h a n a s s o c i a t e d CR

T a r t -11

o f 1 .18, which, w h i l e n o t r e a c h i n g s t a t i s t i c a l s i g n i f i c a n c e , migh t s u g g e s t

t o a n exper imente r t h a t something was happening.

I f t h e exper iment was l o n g e r t h a n 200 t r i a l s t o t a l and t h e b i a s p a t t e r n

and mathemat ica l i n f e r e n c e s t r a t e g y were c o n s i s t e n t , t h e p e r c i p i e n t cou ld

o b v i o u s l y a t t a i n c o n v e n t i o n a l l e v e l s of s i g n i f i c a n c e a s he went f u r t h e r .

For 300 t r i a l s , f o r example, we would have (10+15+15)=40 h i t s , f o r a CR of

1 .92, p C . 0 5 , o n e - t a i l e d . We s h a l l s t a y w i t h a 200 t r i a l exper iment f o r

now, however, t o i l l u s t r a t e c e r t a i n p o i n t s .

Now c o n s i d e r a t a r g e t s o u r c e w i t h a q u i t e d i f f e r e n t s o r t o f b i a s , where

we obse rve t h e f o l l o w i n g d i s t r i b u t i o n of t a r g e t s i n t h e f i r s t 100 t r i a l s :

TARGETS 1 2 3 4 5 6 7 8 9

FREQUENCY 24 10 8 8 8 8 8 8 8

This g e n e r a t o r is h i g h l y b i a s e d toward p roduc ing o n e s , w i t h no o t h e r

l a r g e b i a s e s . The Chi-square t e s t f o r e q u i p r o b a b i l i t y o f s i n g l e t s f o r t h i s

d i s t r i b u t i o n g i v e s a v a l u e o f 22.40, p < . 0 2 , o n e - t a i l e d . I f we m i s t a k e n l y

assumed t h a t t h e magnitude o f t h e Chi-square v a l u e s r e f l e c t e d t h e degree o f

p r e d i c t a b i l i t y of t h i s and t h e p r e v i o u s t a r g e t s o u r c e s f o r a ma themat ica l in-

f e r e n c e s t r a t e g y , we would t h i n k t h i s second t a r g e t s o u r c e was e q u a l l y o r

s l i g h t l y l e s s p r e d i c t a b l e t h a n t h e s o u r c e i n t h e p r e v i o u s example. We shou ld

be q u i t e wrong.

t Again assume t h a t i t t a k e s o u r p e r c i p i e n t t h e f i r s t 100 t r i a l s t o c a t c h

on t o t h e b i a s , s o he s c o r e s o n l y 1 0 h i t s i n t h e f i r s t 100. Now he f o l l o w s

t h e o p t i m a l s t r a t e g y o f c a l l i n g a one f o r e v e r y one o f t h e remain ing 100

t r i a l s , and s c o r e s a b o u t 24 h i t s . For t h e second 100 t r i a l s a l o n e , t h i s

g i v e s a CR of 4 - 6 7 , P < o n e - t a i l e d . For t h e whole exper iment o f 200

t r i a l s , we have (10+24)=34 h i t s , w i t h a CR of 3.30, <.0005, o n e - t a i l e d .

For e q u a l Chi-square v a l u e s i n t e s t s o f b i a s , two sequences may d i f f e r

enormously i n u s e f u l n e s s f o r a m a t h e m a t i c a l i n f e r e n c e s t r a t e g y . F u r t h e r ,

T a r t -12

t h e r e w i l l be f a r , f a r more p o s s i b l e b i a s p a t t e r n s of l e s s u s e f u l n e s s f o r a

ma themat ica l i n f e r e n c e s t r a t e g y t h a n t h e r e w i l l be h i g h l y u s e f u l ones f o r a

g i v e n Chi-square v a l u e . There a r e many, many ways t o r e a r r a n g e t h e b i a s p a t -

t e r n i n our f i r s t example w i t h o u t g i v i n g i t t h e s i n g l e number peak b i a s p a t t e r n

o f o u r second example t h a t i s s o u s e f u l i n a ma themat ica l i n f e r e n c e s t r a t e g y .

Consider a t h i r d example where t h e f o l l o w i n g f r e q u e n c e s o f t a r g e t s a r e

obse rved i n t h e f i r s t 100 t r i a l s :

TARGETS 1 2 3 4 5 6 7 8 9 I FREQUENCY 1 8 10 9 9 9 9 9 9 9

The s t a n d a r d Chi-square test o f e q u i p r o b a b i l i t y t e l l s us t h a t t h i s is

n o t a b i a s e d sequence , f o r Chi-square i s o n l y e q u a l t o 7.20, which would occur - more than h a l f t h e t ime by chance a l o n e . Yet i f o u r t a r g e t s o u r c e i s r e a l l y

b i a s e d toward 18% ones i n t h i s way, and o u r p e r c i p i e n t d e c i d e s t o c a l l a l l

ones i n t h e second 100 t r i a l s of t h e exper iment , he cou ld make 18 h i t s t h e r e ,

f o r a second-hal f CR o f 2.67, p< .01 , o n e - t a i l e d , a n d a t o t a l o f (10+18)=28

h i t s f o r t h e whole exper iment , CR = 1.89, p< .05 , o n e - t a i l e d . It i s e s p e c i a l l y

i n t e r e s t i n g t o n o t e t h a t t h e e n t i r e sequence o f 200 t a r g e t s f o r t h i s p e r c i -

p i e n t , w i t h t h e 18% b i a s c o n t i n u i n g th rough t h e second 100 t r i a l s , s t i l l does

n o t show any s i g n i f i c a n t b i a s : Chi-square i s 14.40, p ) . 10 , o n e - t a i l e d .

We may conc lude t h e f o l l o w i n g f o r s t a n d a r d Chi-square t e s t s o f b i a s .

C For l o n g t o i n f i n i t e l e n g t h e x p e r i m e n t s ,

(1) Lack o f s i g n i f i c a n t Chi-square v a l u e s i n b i a s t e s t s p robab ly i n d i -

c a t e s l a c k o f s i g n i f i c a n t p r e d i c t a b i l i t y by a mathemat ica l i n f e r e n c e s t r a t e g y ;

and

(2 ) The p r e s e n c e o f s i g n i f i c a n t Chi-squares i n b i a s t e s t s i n d i c a t e s - some

d e g r e e o f p r e d i c t a b i l i t y by a mathemat ica l i n f e r e n c e s t r a t e g y , b u t t h e magni-

t u d e o f Chi-square does n o t i n d i c a t e t h e d e g r e e o f p r e d i c t a b i l i t y .

For s h o r t t o moderate l e n g t h e x p e r i m e n t s o f t h e t y p e f r e q u e n t l y c a r r i e d o u t ,

T a r t -13

however,

( 3 ) A s i g n i f i c a n t Chi-square i n d i c a t i o n o f , b i a s does - n o t n e c e s s a r i l y

i n d i c a t e t 'hat a s i g n i f i c a n t o v e r a l l s c o r e c a n be o b t a i n e d th rough a mathema-

t i c a l i n f e r e n c e s t r a t e g y ;

(4) The magni tu re o f t h e o b t a i n e d Chi-square is a v e r y poor measure

of t h e magni tude of r e s u l t s t h a t c a n be o b t a i n e d w i t h a ma themat ica l i n f e r e n c e

s t r a t e g y ; and

(5) A mathemat ica l i n f e r e n c e s t r a t e g y may produce s i g n i f i c a n t r e s u l t s

from a b i a s e d s o u r c e which does n o t appear t o be s i g n i f i c a n t l y b i a s e d by

Chi-square e v a l u a t i o n .

The shor tcomings of s t a n d a r d Chi-square measures o f b i a s i n r e a l i s t i c

l e n g t h exper iments i l l u s t r a t e why more d i r e c t measures o f p r e d i c t a b i l i t y by

ma t h e m a t i c a l i n f e r e n c e s t r a t e g i e s need t o be developed.

The f i n d i n g s o f s i g n i f i c a n t s i n g l e t and d o u b l e t b i a s i n a few of t h e

t a r g e t sequences used i n t h e f i r s t T r a i n i n g S tudy , t h e n , i n d i c a t e a p o t e n t i a l

f o r some k ind of ma themat ica l e s t i m a t i o n s t r a t e g y b e i n g employed, b u t do n o t

t e l l us e i t h e r ( a ) how much i t cou ld u s e f u l l y c o n t r i b u t e t o a r t i f a c t u a l l y

c r e a t i n g s i g n i f i c a n t r e s u l t s ; o r (b ) whe the r i t was a c t u a l l y employed; o r

( c ) how much such a s t r a t e g y c o u l d do compared w i t h t h e a c t u a l s c o r e s o b t a i n e d

by t h e p e r c i p i e n t s .

L e t us now c o n s i d e r t h e i m p o r t a n t d i f f e r e n c e between p r e d i c t i o n and pos t -

d i c t i o n .

Randomici ty , P r e d i c t i o n , and Pos t d i c t i o n :

P robab ly t h e most d i s h e a r t e n i n g a s p e c t o f s e v e r a l y e a r s o f exchanges

w i t h D r . G a t l i n i s h e r p e r s i s t e n t f a i l u r e t o comprehend t h e enormous d i f f e r -

e n c e between e d i c t i o n and p o s t d i c t i o n . I s h a l l q u o t e t h r e e pa ragraphs o f

a l e t t e r o f mine p u b l i s h e d e a r l i e r t h i s y e a r i n t h e J a n u a r y i s s u e of t h e

J o u r n a l o f t h e American S o c i e t y for P s y c h i c a l Research, p o i n t i n g o u t t h i s

problem.

T a r t -14

"Two meanings a r e g e n e r a l l y a s s o c i a t e d w i t h t h e concep t of r andomic i ty .

The f i r s t i s t h a t no p a t t e r n i n g s o r dependencies o f any s o r t c a n be found

i n a sequence o f random numer ica l d a t a . The second i s t h a t r andomic i ty

means a l a c k o f p r e d i c t a b i l i t y of a numerical sequence: t h a t i s , g iven

a sample o f t h e sequence, one cannot p r e d i c t subsequen t numbers i n t h e se-

quence w i t h g r e a t e r than chance s u c c e s s .

While t h e second meaning a s s o c i a t e d w i t h t h e concep t o f r andomic i ty

i s impor tan t f o r b o t h p s y c h o l o g i c a l and p a r a p s y c h o l o g i c a l r e s e a r c h , t h e

f i r s t i s f a l s e . Mathemat ica l ly , one c a n t ake any sequence o f numbers of any

f i n i t e l e n g t h , even i f t h e y have been g e n e r a t e d by a t r u l y random p r o c e s s ,

and f i n d a n a l g o r i t h m which would d e t e r m i n i s t i c a l l y g e n e r a t e t h a t e x a c t

sequence o f numbers. Th i s seems t o imply t h a t t h e sequence o f numbers was

n o t random, b u t r e s u l t e d d e t e r m i n i s t r i c a l l y from t h a t a l g o r i t h m , and t h u s

had a p a t t e r n t o i t t h a t cou ld be d e t e c t e d and made use o f . However, t h e

a l g o r i t h m s o determined w i l l not s u c c e s s f u l l y p r e d i c t f u r t h e r numbers ga thered

from t h e same random s o u r c e a t a l e v e l beyond chance expec tancy . To p u t i t

a n o t h e r way, we c a n always f i n d some kind o f p a t t e r n i n r e t r o s p e c t , a pro-

c e s s a k i n t o t h e p s y c h o l o g i c a l p r o c e s s o f r a t i o n a l i z a t i o n o r p r o j e c t i o n ,

b u t t h a t does n o t mean t h a t t h e sequence was a c t u a l l y genera ted i n t h a t

f a s h i o n o r t h a t i t i s p r e d i c t a b l e . "

t More t e c h n i c a l d i s c u s s i o n s o f t h e s e p o i n t s can be found i n C h a i t i n , 1975, and

Gardner, 19 . My le t ter con t inued :

"Thus t h e q u e s t i o n of whether D r . G a t l i n ' s post-hoc a n a l y s e s can f i n d

any kind o f p a t t e r n ( i n t h e s e n s e o f d e p a r t u r e s from p e q u a l i n g e x a c t l y one-

t e n t h ) i n my t a r g e t d a t a i s n o t r e a l l y t h e r e l e v a n t q u e s t i o n : such p a t t e r n s

can be found, t o v a r y i n g d e g r e e s , i n t h e d a t a o f any and e v e r y p s y c h o l o g i c a l

and p a r a p s y c h o l o g i c a l exper iment . The r e l e v a n t q u e s t i o n i s whether such

p a t t e r n i n g s , s e q u e n t i a l dependenc ies , o r b i a s e s e x i s t i n t h e t a r g e t d a t a

T a r t -15

t o a deg ree s t r o n g enough t o have a l lowed p e r c i p i e n t s i n t h e T ra in ing Study

t o f i g u r e o u t t h e s e b i a s e s a s they went a l o n g (not p o s t hoc) , and make use - of them t o boos t t h e i r s c o r e s t o a l e v e l h igh enough t o make unnecessary

t h e occur rence o f ESP a s a n exp lana t ion ." (Ta r t , 1978, p 8 2 ) .

D r . G a t l i n c la imed i n her JASPR l e t t e r ( G a t l i n , 1978a) t o which I was

responding t h a t he r "monotone" ( s i n g l e t ) s t r a t e g y s co red s i g n i f i c a n t l y w i th

e i g h t o f t h e t e n t a r g e t sequences i n my s tudy . A s I po in t ed o u t i n my r e p l y ,

t h i s s t r a t e g y a p p a r e n t l y c o n s i s t e d o f p o s t hoc coun t i ng of t h e f requency

of observed s i n g l e t s i n the e n t i r e sequence and t hen p r e t e n d i n g you had

c a l l e d t h a t h i g h e s t s i n g l e t f o r your e v e r y response! I n t h e r e a l world

p e r c i p i e n t s do no t have a l l t h i s d a t a u n t i l a f t e r t h e i r c a l l s a r e made, s o

D r . G a t l i n ' s p o s t d i c t i v e procedure i s q u i t e spu r ious . I gave an example

o f performing a G a t l i n monotone p o s t d i c t i v e s t r a t e g y on 25 random numbers

t aken from a random number t a b l e : I s co red s i x spu r ious h i t s , f o r a binomial

p r o b a b i l i t y of .03. D r . G a t l i n is s t r o n g on h igher o rde r b i a s e s : u s ing

a d o u b l e t monotone p o s t d i c t i v e s t r a t e g y o f t he same type , I s co red 10 h i t s

i n 25 t r i a l s , e t c . The h igher t h e l e v e l of t h i s p o s t d i c t i v e s t r a t e g y , the

h igher your s c o r e i s f o r a sequence w i t h any b i a s i n i t , o r even on a random

sequence.

Given t h e t o t a l f a l l a c i o u s n e s s o f any kind of p o s t d i c t i v e s t r a t e g y ,

I found i t hard t o b e l i e v e t h a t D r . G a t l i n would con t i nue t o use i t a f t e r

i t was po in ted o u t , b u t s h e has. I n he r l a t e s t p u b l i c a t i o n (Ga t l i n , 1978b)

i t i s now g iven t he impress ive sounding t i t l e of a 'Vaximal Markov-3 s t r a t e g y . "

To quote D r . ~ a t l i n ' s c u r r e n t paper , "It i s i n s t r u c t i v e t o c a l c u l a t e how

h igh t h e s u b j e c t s cou ld s c o r e i f t h e i r e s t i m a t e s were 100% accu ra t e . I f

w e count t he t r i p l e t f r e q u e n c i e s i n e ach i n d i v i d u a l t a r g e t sequence and use

t h e s e a s a b a s i s f o r a s imple guess ing s t r a t e g y , which we w i l l c a l l a maximal

Markov-3 s t r a t e g y symbolized a s MM3, where in t he s u b j e c t guesses the symbol

T a r t -16

mos t l i k e l y t o o c c u r , g i v e n t h e two p r e c e e d i n g symbols i n t h e t a r g e t , t h e

Z-scores range from abou t 1 7 t o 19 which i s s u b s t a n t i a l l y h i g h e r t h a n any

obse rved i n t h e exper iment ." ( G a t l i n , 1978c, p.13).

I n he r ~ e c e n t l e t t e r t o t h e J o u r n a l of t h e American S o c i e t y for P s y c h i c a l

Resea rch ( G a t l i n , 1978b), D r . G a t l i n i n s i s t s t h a t s h e does know t h e d i f f e r -

ence between p r e d i c t i o n and p o s t d i c t i o n , y e t s h e a g a i n g i v e s a p o s t d i c t i v e

s t r a t e g y a s a n example o f h e r knowledge! Now perhaps I ' m o ld - fash ioned and

c o n s e r v a t i v e , b u t t h e d i c t i o n a r y d e f i n i t i o n o f t h e v e r b " p r e d i c t " i s "TO

t e l l o r d e c l a r e beforehand . . .", b e i n g d e r i v e d from L a t i n r o o t s meaning

t o s p e a k a b o u t something b e f o r e i t happens. D r . G a t l i n j u s t g i v e s a n o t h e r

example o f what p e r c i p i e n t s might have done, g i v e n he r l a t e r knowledge, b u t

t h i s i s h a r d l y p r e d i c t i n g . Her examples remind me o f t h e newspaper columns

o f s t o c k marke t a n a l y s t s who always b r i l l i a n t l y e x p l a i n why t h e market be-

haved t h e way i t d i d l a s t week. These a n a l y s t s seldom make any money on

t h e marke t .

D r . G a t l i n f u r t h e r a r g u e s i n t h i s l e t t e r of r e s p o n s e t h a t I misunder-

s t o o k he r monotone g u e s s i n g s t r a t e g y , and d e c l a r e s t h a t ". . . i n e i g h t o u t

o f t h e 1 0 sequences t h e p r o b a b i l i t y (Dr. ~ a t l i n ' s i t a l i c s ) o f s c o r i n g s i g -

n i f i c a n t l y i s 10% t o 40% . . ." ( ~ a t l i n , 1978b, p. 296). What t h i s means

i n terms of a c t u a l d a t a i s t h a t if a p e r c i p i e n t had happened t o guess t h e

one c o r r e c t o u t o f t e n p o s s i b l e monotone s t r a t e g i e s r i g h t a t t h e s t a r t o f

h i s o r h e r r e s p o n s e s t h e y c o u l d have s c o r e d a t t h e (.05 l e v e l , 2 - t a i l e d .

As ide from t h e f a c t t h a t t h e r e a r e many more ways o f g u e s s i n g wrong w i t h

t h i s s t r a t e g y t h a n g u e s s i n g r i g h t w i t h i t , a s i m p l e i n s p e c t i o n o f t h e d a t a

would have r e v e a l e d t h a t no p e r c i p i e n t used s u c h a monotone s t r a t e g y !

F u r t h e r i n s p e c t i o n of t h e d a t a would have shown t h a t even i f t h e y had used

i t t o maximal advan tage , t h e i r t o t a l h i t s s c o r e s would have been enormously

less s i g n i f i c a n t t h a n t h e y a c t u a l l y were! If I had bought many s h a r e s of

T a r t -17

a c e r t a i n s t o c k l a s t week I would i n d e e d have been r i c h t h i s week, b u t . . . . As l o n g as we are on t h e u n p l e a s a n t s u b j e c t o f m e a n i n g l e s s s t a t i s t i c a l

p rocedures , , I s h o u l d comment o n t h e s o - c a l l e d " c o n t r o l " a n a l y s e s r e p o r t e d by

D r . G a t l i n t h a t presumably s u p p o r t h e r ma in t h e s i s . Fo r t h e s e a n a l y s e s s h e

matched a computer pseudo-random number g e n e r a t o r o u t p u t a g a i n s t t h e t a r g e t

s e q u e n c e s and t h e n a r t i f i c i a l l y c r e a t e d a number o f h i t s e q u a l t o t h o s e t h e

a c t u a l p e r c i p i e n t s made on e a c h t a r g e t s equence by l o o k i n g a t t h e t a r g e t l i s t

a t random i n t e r v a l s and s i m p l y c h a n g i n g t h e pseudo-random o u t p u t o f t h e com-

p u t e r g e n e r a t o r t o ma tch t h e t a r g e t and t h u s make a h i t . She r e p o r t s t h a t

t h e s e r e s p o n s e s e q u e n c e s showed none o f t h e s i g n i f i c a n t D-measures t h a t t h e

a c t u a l p e r c i p i e n t r e s p o n s e s e q u e n c e s d i d .

Her p r o c e d u r e amounts t o t a k i n g a v e r y s m a l l sample from a s l i g h t l y

b i a s e d sequence . A s m a l l s ample , o f c o u r s e , i s u n l i k e l y t o have a d e t e c t a b l e

b i a s i n i t s i m p l y b y t h e l a r g e r e d u c t i o n o f N. Then m i x i n g t h i s s m a l l sample

w i t h s e v e r a l t i m e s as many pseudo-random numbers d i l u t e s any b i a s even f u r t h e r .

It i s no wonder no b i a s e s were found. I n d e e d , t h e pseudo-random g e n e r a t o r

u s e d was p r o b a b l y o f t h e same t y p e t h a t D r . G a t l i n s t a n d a r d i z e d h e r D-measures

on i n t h e f i r s t p l a c e . I c a n n o t u n d e r s t a n d what meaning t h i s s o - c a l l e d con-

t r o l a n a l y s i s has .

Magni tude o f B i a s V e r s u s P a t t e r n o f B i a s :

I g n o r i n g f o r t h e moment t h e t r i p l e t and o t h e r f a l l a c i e s i n D r . G a t l i n ' s

a n a l y s e s , s u p p o s i n g we assumed t h a t h e r a n a l y s e s a t l eas t d e m o n s t r a t e d t h e

p o s s i b i l i t y t h a t a m a t h e m a t i c a l i n f e r e n c e s t r a t e g y m i g h t have been used b y

a t l e a s t some p e r c i p i e n t s . I f we l o o k a t t h e d a t a a v a i l a b l e t o D r . G a t l i n

t o see i f t h e y a c t u a l l y s u p p o r t t h i s p o s s i b i l i t y , we s h a l l s e e e v i d e n c e t h a t

t h e p e r c i p i e n t s d i d not u s e s u c h a s t r a t e g y .

To u s e a m a t h e m a t i c a l i n f e r e n c e s t r a t e g y , a p e r c i p i e n t s h o u l d p a t t e r n

h i s o r he r r e s p o n s e s t r a t e g i e s a s c l o s e t o h i s e s t i m a t e o f t a r g e t b i a s p a t t e r n s

T a r t -18

a s p o s s i b l e . We would t h e n e x p e c t t o s e e a correspondence between t h e most

f r e q u e n t b i a s p a t t e r n s i n t h e t a r g e t sequence, t h e t h i n g s t h a t would be most

u s e f u l f o r a n i n f e r e n c e s t r a t e g y , and t h e p e r c i p i e n t s ' r e sponse p a t t e r n s .

D r . G a t l i n c l a i m s t h a t such b i a s p a t t e r n matchings e x i s t , a s i n t h e opening

s t a t e m e n t of her d i s c u s s i o n s e c t i o n when s h e s t a t e s , "The matching p a t t e r n s

(my i t a l i c s ) demonstra ted i n t h e t a r g e t and guess sequences o f t h e s e two

independent sets o f ESP d a t a i n d i c a t e t h a t e x t r e m e l y s u b t l e b i a s a t h i g h

n - tup le l e v e l s i n f i n i t e sequences can be u t i l i z e d by the human mind."

( G a t l i n , 1978c, pp. 15-16). T h i s c l a i m , however, has no e m p i r i c a l s u p p o r t

a t a l l p r e s e n t e d f o r i t i n D r . G a t l i n ' s paper : she h a s examined t h e magnitude

o f b i a s e s , b u t p r e s e n t e d no d a t a a t a l l on whether t h e s p e c i f i c p a t t e r n s of

b i a s i n t a r g e t and p e r c i p i e n t d a t a a c t u a l l y match. I f a t a r g e t sequence is

h i g h l y b i a s e d toward t h r e e s f o l l o w i n g f i v e s , f o r example, and a p e r c i p i e n t

i s h i g h l y b i a s e d toward responding w i t h s i x e s a f t e r a t a r g e t has been f i v e ,

t h e s e h i g h magni tudes o f b i a s w i l l n o t be a t a l l u s e f u l f o r s c o r i n g , a s t h e

p a t t e r n s do n o t match.

One would have expec ted t h a t D r . G a t l i n would have i n s p e c t e d t h e a c t u a l

b i a s p a t t e r n s i n t h e t a r g e t and p e r c i p i e n t d a t a t o see i f they d i d match.

S i n c e she has e i t h e r n o t done s o o r chosen n o t t o p r e s e n t such d a t a , I c a r r i e d

o u t t h i s a n a l y s i s .

T I f a t a r g e t sequence had a h igh s i n g l e t l e v e l b i a s f o r e i g h t s , e . g . ,

we would e x p e c t t o s e e t h e p e r c i p i e n t showing most of h i s above-chance h i t s

on e i g h t s , r a t h e r t h a n o t h e r t a r g e t s . I f a d o u b l e t l e v e l mathemat ica l e s t ima-

t i o n s t r a t e g y was a l s o u s e f u l , t h e n we shou ld s e e many o f t h e above-chance

h i t s on t h e second term o f t h e d o u b l e t : i f n i n e s fo l lowed f i v e s v e r y f r e q u e n t l y ,

f o r example, we shou ld have many h i t s on n i n e s , a s w e l l a s on t h e e i g h t s ( i n

t h i s example) t h a t we have a l r e a d y d e f i n e d a s u s e f u l f o r a s i n g l e t e s t i m a t i o n

s t r a t e g y . We would n o t e x p e c t more than a chance number o f h i t s on t a r g e t s

T a r t -19

t h a t were n o t t h e h i g h ones i n a n e s t i m a t o r s t r a t e g y . How d o t h e d a t a a c t u a l l y

l o o k ?

Table ' 2 shows t h e t a r g e t numbers which had t h e h i g h s i n g l e t and h i g h

d o u b l e t o c c u r r e n c e s f o r t h e t e n p e r c i p i e n t s and t h e i r r e s p e c t i v e t a r g e t

s e q u e n c e s . The v a s t m a j o r i t y o f t h e s e d o n o t , o f c o u r s e , r e p r e s e n t s t a t i s -

t i c a l l y s i g n i f i c a n t b i a s e s . As c a n be s e e n , o n l y one p e r c i p i e n t had h i s h i g h e s t

s i n g l e t b i a s toward t h e same t a r g e t t h a t was h i g h i n t h e t a r g e t s e q u e n c e , and

no p e r c i p i e n t had h i s h i g h e s t d o u b l e t b i a s i d e n t i c a l t o t h e h i g h e s t d o u b l e t

b i a s i n h i s t a r g e t sequence . T h i s r e s u l t i s n o t s u p p o r t i v e o f Dr. G a t l i n ' s

c l a i m s . I n d e e d , i n s p e c t i o n o f t h e v a r i e t y o f h igh s i n g l e t s and d o u b l e t s

m i g h t l e a d us t o wonder how t h e same e l e c t r o n i c RNG c o u l d p roduce s u c h d i f -

f e r e n t p a t t e r n s f o r d i f f e r e n t p e r c i p i e n t s i f i t were as b i a s e d a s D r . G a t l i n

imp 1 ies .

I n s e r t Tab le 2 a b o u t h e r e

The d a t a i n T a b l e 2 do n o t c o m p l e t e l y d i s p r o v e t h a t t h e p e r c i p i e n t s migh t

have used a m a t h e m a t i c a l i n f e r e n c e s t r a t e g y , however, f o r t h e i r i n c o r r e c t h i g h

r e s p o n s e b i a s e s m i g h t have r e p r e s e n t e d p e r s o n a l i d i o s y n c r a c i e s , y e t t h e y

migh t have s t i l l p i c k e d up enough b i a s toward t h e t a r g e t s equence h i g h s t o

a r t i f a c t u a l l y p roduce t h e e x t r a - c h a n c e h i t s which made t h e i r s c o r e s s i g n i f i - s c a n t . A more d i r e c t test o f t h e h y p o t h e s i s i s shown i n T a b l e 3 . F o r s i m p l i -

c i t y , o n l y t h e f i v e p e r c i p i e n t s who s c o r e d s i g n i f i c a n t l y above chance a r e I

shown. The second column is t h e CR o f t h e i r obse rved h i t s , g i v e n t h e e q u i -

p r o b a b i l i t y model. To compute t h e CRs i n t h e t h i r d column, a l l t h e t r i a l s

(and, of c o u r s e , h i t s ) on t h e m o s t f r e q u e n t l y o c c u r r i n g s i n g l e t t a r g e t were

d e l e t e d --- f rom t h e t o t a l t r i a l s : i f t h e p e r c i p i e n t s were o n l y u s i n g a s i n g l e t

e s t i m a t o r s t r a t e g y , t h i s s h o u l d r e d u c e t h e i r s c o r e s t o chance . Obv ious ly i t

T a r t -20

d o e s no t . They were s c o r i n g q u i t e s i g n i f i c a n t l y on many o t h e r t a r g e t s t h a n

t h e one a s i n g l e t e s t i m a t o r s t r a t e g y would g i v e them a n advantage on. To

t e s t a n i n f e r e n c e h y p o t h e s i s e v e n f u r t h e r , a l l h i t s on t h e r e l e v a n t t a r g e t

p r e d i c t e d by t h e most f r e q u e n t t a r g e t d o u b l e t were a l s o d e l e t e d , y e t t h e C R s

i n t h e f o u r t h column show t h e p e r c i p i e n t s s t i l l c o n t i n u e t o s c o r e v e r y s i g -

n i f i c a n t l y .

I n s e r t Table 3 a b o u t h e r e

Thus s t r a i g h t f o r w a r d i n s p e c t i o n o f d a t a on t h e r e l e v a n t b i a s e s t i m a t e s

s t r o n g l y s u g g e s t s t h a t t h e p e r c i p i e n t s were not u s i n g a u s e f u l e s t i m a t o r

s t r a t e g y , c o n t r a r y t o D r . ~ a t l i n ' s c l a i m s .

G a t l i n ' s Opt imal E s t i m a t i o n Windows:

I f i n d D r . G a t l i n ' s s e c t i o n on o p t i m a l e s t i m a t i o n windows r a t h e r d i f f i -

c u l t t o f o l l o w . Presumably s h e is c l a i m i n g t h a t i f s i g n i f i c a n t and c o n s i s t e n t

b i a s p a t t e r n s e x i s t i n t h e t a r g e t sequence, a p e r c i p i e n t must d e t e c t what

t h e s e p a t t e r n s a r e e a r l y enough i n t h e exper iment t o be a b l e t o e f f e c t i v e l y

u s e them t o i n f l a t e h i s o r h e r s c o r i n g . A t l e a s t t h i s i s my r e a d i n g of what

would make s e n s e . The p rocedure D r . G a t l i n f o l l o w s t o presumably demons t ra te

t h a t t h i s was p o s s i b l e i s u n c l e a r t o m e , however. A t one p o i n t i t seems t o

i n v o l v e t h e c a l c u l a t i o n o f 300 c o r r e l a t i o n c o e f f i c i e n t s a s s h e g e t s h e r rnet r

v a l u e s f o r D;, D;, and D; f o r t e n i n c r e m e n t a l sample l e n g t h s over t e n p e r c i -

p i e n t s , and s u c h a l a r g e number o f c o r r e l a t i o n s seem bound t o y i e l d some s i g -

n i f i c a n t v a l u e s by chance a l o n e . The f i n a l outcome o f a l l t h i s i s even more

p u z z l i n g , however.

I f a p e r c i p i e n t a c t u a l l y f i g u r e s o u t a s i g n i f i c a n t b i a s p a t t e r n i n a

t a r g e t sequence and e x p l o i t s i t , what w e b a s i c a l l y e x p e c t t o s e e i s s c o r i n g

n e a r chance l e v e l f o r a w h i l e , b u t i n c r e a s i n g markedly and s t e a d i l y once t h e

b i a s p a t t e r n i s g rasped . Tha t i s , w e would e x p e c t a " l e a r n i n g " c u r v e , a

T a r t -21

s t e a d y i n c r e a s e i n s c o r i n g w i t h f u r t h e r t r i a l s . This e x p e c t a t i o n o f l e a r n i n g

is e s p e c i a l l y a p p l i c a b l e f o r h i g h e r o r d e r b i a s e s , where i t migh t t a k e some

t ime and a few p a r t i a l l y s u c c e s s f u l s t a r t s f o r t h e p e r c i p i e n t t o g e t i t r i g h t .

As you know, my pr imary i n t e r e s t i n t h i s d a t a was i n l o o k i n g f o r l e a r n i n g ,

and w h i l e I r e p o r t e d some e v i d e n c e f o r i t ( T a r t , 1975, 1976a), i t was h a r d l y

a s p e r v a s i v e a s we would e x p e c t from t h e , a p p l i c a t i o n o f s u c c e s s f u l mathemati-

c a l i n f e r e n c e s t r a t e g i e s by t h e p e r c i p i e n t s !

Consider t h e d a t a p r e s e n t e d by D r . G a t l i n i n he r Table 2 on o p t i m a l

es t i m a t i o n windows. For conven ience , I have p r e s e n t e d them g r a p h i c a l l y i n

F i g u r e Her procedure seems t o be a m a t t e r of c o r r e l a t i n g samples o f t h e

t o t a l t a r g e t d i s t r i b u t i o n f o r a p e r c i p i e n t w i t h t h e whole t a r g e t d i s t r i b u t i o n

o f t h a t p e r c i p i e n t . S i n c e t h e sample s i z e s s t a r t a s o n l y a s m a l l (5%) sample

b u t g e t i n c r e a s i n g l y b i g g e r a s t h e y inc rement , we would e x p e c t t h e s e c o r r e l a -

t i o n s t o g e t i n c r e a s i n g l y b i g g e r : a f t e r a l l , w e a r e c o r r e l a t i n g i n c r e a s i n g l y

more adequa te samples o f a d i s t r i b u t i o n w i t h i t s e l f . Yet D r . ~ a t l i n ' s own

a n a l y s e s show t h a t t h e s i n g l e t samples g e t b e t t e r f o r a w h i l e and t h e n g e t

worse: a s i d e from t h e mathemat ica l oddness o f t h i s , which makes m e s u s p e c t

some e r r o r , how i s t h i s supposed t o be h e l p f u l t o a p e r c i p i e n t ? The d o u b l e t

l e v e l c o r r e l a t i o n s s t a y r e l a t i v e l y c o n s t a n t a s soon as t h e sample s i z e in -

c r e a s e s above i t s i n i t i a l v e r y s m a l l s i z e , and t h i s i s p r o b a b l y main ly a r e -

f l e c t i o n o f t h e c o n s i s t e n c y of t h e e x p e r i m e n t e r e r r o r t h a t l e d t o t h e system- t

a t i c d e p l e t i o n o f XX d o u b l e t s , a n e r r o r I have a l r e a d y shown ( T a r t , 19772)

t o r e q u i r e o n l y a t r i v i a l c o r r e c t i o n o f t h e r e s u l t s . Perhaps someone e l s e

c a n f i g u r e o u t e x a c t l y what D r . G a t l i n d i d t o o b t a i n t h e s e r e s u l t s and what

s i g n i f i c a n c e , i f any, t h e y migh t have.

- - - - - - - - - - - - - - - - - - I n s e r t F i g u r e a b o u t h e r e

T a r t -22

Applying a Powerful I n f e r e n c e S t r a t e g y :

From t h e a n a l y s e s I have p r e s e n t e d , I have shown t h a t w h i l e some p o t e n t i a l

may e x i s t ' i n a s m a l l m i n o r i t y o f t h e t a r g e t sequences by which some s o r t o f

ma themat ica l i n f e r e n c e s t r a t e g y migh t have c o n t r i b u t e d t o a t l e a s t some p a r t

o f t h e obse rved r e s u l t s , D r . ~ a t l i n ' s c l a i m s a b o u t t h i s a r e i n v a l i d because

o f r a t h e r b a s i c p r o c e d u r a l and s t a t i s t i c a l f l aws . I n d e e d , i f s h e had in-

s p e c t e d t h e raw d a t a s h e based he r a n a l y s e s on, s h e would have s e e n t h a t t h e y

c o n t r a d i c t e d he r a s s e r t i o n s . As I ment ioned a t t h e beg inn ing o f t h i s paper ,

however, t h e g e n e r a l q u e s t i o n o f t h e e x t e n t t o which mathemat ica l i n f e r e n c e

s t r a t e g i e s c o u l d a f f e c t r e s u l t s i n feedback s t u d i e s i s a n i m p o r t a n t q u e s t i o n ,

s o l e t me b r i e f l y a d d r e s s i t d i r e c t l y by d e s c r i b i n g t h e r e s u l t s o f a power-

f u l i n f e r e n c e s t r a t e g y a p p l i e d t o t h e d a t a o f t h e f i r s t T r a i n i n g Study. Th i s

w i l l be f a m i l i a r t o some o f you a s I b r i e f l y touched on i t i n my P r e s i d e n t i a l

a d d r e s s ( T a r t , 1977b) l a s t y e a r .

The h y p o t h e s i s t h a t p e r c i p i e n t s c a n i n f l a t e t h e i r s c o r e s by a mathemati-

c a l i n f e r e n c e s t r a t e g y a s a r e s u l t o f f i g u r i n g o u t t a r g e t b i a s e s needs t o be

c a s t i n a s p e c i f i c and t e s t a b l e form t o be s c i e n t i f i c a l l y u s e f u l . F o r t u n a t e l y ,

ma themat ica l i n f e r e n c e l e n d s i t s e l f t o p r e c i s e d e f i n i t i o n . Eugene Dronek, a

c o l l e a g u e i n t h e Computer S c i e n c e s Department o f t h e U n i v e r s i t y o f C a l i f o r n i a

a t Berke ley , and I a r e now s u b m i t t i n g f o r p u b l i c a t i o n t h e r e s u l t s o f a v e r y

powerful ma themat ica l i n f e r e n c e s t r a t e g y t h a t we c a l l t h e P r o b a b l i s t i c Pre-

d i c t o r Program (PPP). I am p r i m a r i l y r e s p o n s i b l e f o r t h e b a s i c s t r a t e g y , and

Dronek i s p r i m a r i l y r e s p o n s i b l e f o r i t s p r a c t i c a l implementa t ion on t h e computer.

We s e t o u r s e l v e s t h e t a s k o f d e v i s i n g a compute r -ass i s t ed i n f e r e n t i a l

c a l l i n g s t r a t e g y t h a t would have enormously more power t h a n we c o u l d r e a s o n a b l y

a t t r i b u t e t o human p e r c i p i e n t s . We gave o u r program powers s u c h a s a n a b s o l u t e l y

p e r f e c t memory f o r a l l p r e v i o u s t a r g e t s t o d a t e , a l l p r e v i o u s t a r g e t d o u b l e t s ,

e t c . , up t o a l l p r e v i o u s t a r g e t s e x t u p l e t s , a s w e l l a s p e r f e c t l y a c c u r a t e and

T a r t -23

w e l l n i g h i n s t a n t a n e o u s ( i n terms o f human t i m e ) computing c a p a c i t y t o a s s e s s

p o s s i b l e b i a s e s .

To get a n overview o f what t h e PPP d o e s , assume t h a t t h e 1 0 1 s t t r i a l

i s coming up. To make i t s c a l l , our PPP i n f e r e n c e program l o o k s a t a l l

hundred p r e v i o u s t a r g e t s which have come up on p r e v i o u s t r i a l s . It has

a l r e a d y s o r t e d them i n t o a s i n g l e t f i l e , , a d o u b l e t f i l e , and s o on th rough

a s e x t u p l e t f i l e . I t l o o k s a t t h e s i n g l e t f i l e , a s k s what has been t h e most

f r e q u e n t s i n g l e t t o d a t e , and, g i v e n 100 t r i a l s , what i s t h e e x a c t b inomia l

p r o b a b i l i t y t h a t a s i n g l e t s h o u l d have appeared w i t h s u c h an obse rved £ r e -

quency compared t o t h e n u l l h y p o t h e s i s t h a t a l l s i n g l e t s have a n e q u a l pro-

b a b i l i t y o f one- ten th? Th i s e x a c t b inomia l p r o b a b i l i t y i s computed and s t o r e d .

The program t h e n a s k s i f t h e r e i s r e l e v a n t i n f o r m a t i o n i n i t s d o u b l e t f i l e :

t h a t i s , s a y t h e 1 0 0 t h t a r g e t was a 7 . Does t h e d o u b l e t f i l e have any i n f o r -

ma t ion on what 7s have been fol lowed by i n t h e p r e v i o u s 100 t r i a l s ? I f n o t ,

i t w i l l g u e s s on t h e b a s i s o f t h e most improbable (compared t o t h e n u l l hypo-

t h e s i s ) t a r g e t t o d a t e i n t h e s i n g l e t f i l e , b u t i f t h e d o u b l e t f i l e does have

r e l e v a n t i n f o r m a t i o n , i t w i l l a g a i n compute t h e e x a c t b inomia l p r o b a b i l i t y

o f t h a t many o r more d o u b l e t s having o c c u r r e d i n t h e 100 t r i a l s t o d a t e , com-

pa red t o t h e n u l l h y p o t h e s i s o f e q u a l p r o b a b i l i t y f o r a l l p o s s i b l e d o u b l e t s .

Th i s b inomia l p r o b a b i l i t y w i l l t h e n b e compared t o t h e b i n o m i a l p r o b a b i l i t y

o f t h e h i g h e s t s i n g l e t t o d a t e : i f t h e h i g h e s t d o u b l e t t o d a t e i s less pro- f

b a b l e , i.e., r e p r e s e n t s more of a d e p a r t u r e from t h e model o f s e q u e n t i a l i n -

dependence t h a n t h e h i g h e s t s i n g l e t t o d a t e r e p r e s e n t s a s a d e p a r t u r e from

t h e e q u i p r o b a b i l i t y model, t h e p r o g r a m w i l l u s e t h a t d o u b l e t i n f o r m a t i o n a s

t h e b a s i s of i t s g u e s s i n g s t r a t e g y . S i m i l a r l y , i f t h e r e i s a r e l e v a n t t r i p l e t ,

q u a d r u p l e t , q u i n t u p l e t , o r s e x t u p l e t , t h e most r a d i c a l d e p a r t u r e from t h e

model o f e q u a l p r o b a b i l i t y and s e q u e n t i a l independence w i l l be used a s a b a s i s

f o r t h e g u e s s i n g s t r a t e g y . On t h e 102nd t r i a l , a l l computa t ions w i l l be re-done

T a r t -24

b e c a u s e t h e r e i s now a d a t a b a s e o f 1 0 1 t r i a l s i n s t e a d o f 100 , e t c . , s o t h e

program c o n s t a n t l y u p d a t e s i t s e l f i n o r d e r t o g e t t h e maximum i n f o r m a t i o n

from a l l t'he m a t e r i a l t o d a t e . Because o f t h i s u p d a t i n g , i t i s q u i t e s e n s i -

t i v e t o l o c a l l y s h i f t i n g b i a s e s , as w e l l a s g e n e r a l b i a s e s .

Note t h a t o u r PPP program i s n o t based o n a n y a s s u m p t i o n s a b o u t r e p r e -

s e n t a t i v e samples o r t h e l i k e : i t d e a l s w i t h what h a s a c t u a l l y been o b s e r v e d

t o d a t e and a lways tries t o c a p i t a l i z e o n t h e s e o b s e r v e d f r e q u e n c i e s .

F i g u r e i s a compar i son o f wha t o u r i n f e r e n t i a l s t r a t e g y program, w i t h

a l l i t s a d v a n t a g e s , c a n d o on t h e t a r g e t s e q u e n c e s , compared t o t h e s c o r e s

o f t h e a c t u a l p e r c i p i e n t s o f t h e f i r s t T r a i n i n g S tudy . As you c a n see, t h e

PPP manages t o r e a c h s t a t i s t i c a l s i g n i f i c a n c e o n o n l y two o f t h e t e n t a r g e t

s e q u e n c e s , and i t i s g e n e r a l l y s c o r i n g w e l l be low t h e a c t u a l p e r c i p i e n t s '

s c o r e s . I n two c a s e s o f p e r c i p i e n t s who d i d n o t show i n d i v i d u a l l y s i g n i f i -

c a n t ESP s c o r e s , t h e i n f e r e n t i a l s t r a t e g y program d i d b e t t e r , a l t h o u g h i t

d i d n o t r e a c h s t a t i s t i c a l s i g n i f i c a n c e . I n d e e d , t h e PPP s c o r e d a t chance

- - - - - - - - - - - - - - - - - - ,

I n s e r t F i g u r e a b o u t h e r e

(CR = .15) on t h e t a r g e t s e q u e n c e o f t h e mos t s u c c e s s f u l p e r c i p i e n t (P3 ) ,

f o r t h e b i a s e s i n t h e t a r g e t d i s t r i b u t i o n w e r e n o t u s e f u l f o r p r e d i c t i o n ,

e v e n i f t h e y were s t a t i s t i c a l l y s i g n i f i c a n t . I n g e n e r a l , t h e PPP c o u l d g e t

o n l y a b o u t 30% as many h i t s as t h e a c t u a l p e r c i p i e n t s g o t o v e r t h e whole

s t u d y , and e v e n i f we a d j u s t e d t h e p e r c i p i e n t s ' s c o r e s downward a c c o r d i n g l y

o n a n h y p o t h e s i s t h a t t h e y a r o s e f rom b o t h ESP and a m a t h e m a t i c a l i n f e r e n c e

s t r a t e g y , t h e amount o f ESP i n t h e e x p e r i m e n t was s t i l l enormous.

D r . G a t l i n p u t s much e m p h a s i s o n h i g h e r b i a s e s i n s p e c u l a t i n g t h a t a

m a t h e m a t i c a l i n f e r e n c e s t r a t e g y was used by t h e p e r c i p i e n t s . I mus t n o t e

t h a t w e compared v a r i o u s l e v e l s o f s e n s i t i v i t y o f t h e PPP t o h i g h e r ' . l e v e l

T a r t -25

b i a s e s , r a n g i n g f rom a l l o w i n g i t t o o p e r a t e o n l y a t t h e s i n g l e t l e v e l a l l

t h e way up t o t h e s e x t u p l e t l e v e l . Adding l e v e l s above t h e s i n g l e t was o f

v e r y l i t t l e h e l p t o t h e PPP, as p r a c t i c a l l y a l l o f i t s s i g n i f i c a n t p e r f o r -

mances came from i t s u s e of s i n g l e t l e v e l b i a s e s : t h e r e s i m p l y were n o t

h i g h e r l e v e l b i a s e s , u p t o t h e s e x t u p l e t l e v e l , t h a t were o f p r a c t i c a l u s e

f o r a m a t h e m a t i c a l e s t i m a t o r s t r a t e g y . T h i s is p e r f e c t l y i n a c c o r d w i t h

t h e k i n d o f o p e r a t i o n we would e x p e c t from a n e l e c t r o n i c r o u l e t t e whee l t y p e

o f RNG, o f c o u r s e .

E v i d e n c e t h a t P e r c i p i e n t s Did Not Use a n I n f e r e n c e S t r a t e g y :

I c o u l d c o n c l u d e a t t h i s p o i n t t h a t less t h a n o n e - t h i r d maximum o f t h e

s c o r i n g c o u l d be a t t r i b u t e d t o t h e b e s t m a t h e m a t i c a l i n f e r e n c e s t r a t e g y we

have b e e n a b l e t o d e v i s e , and a l l o w t h a t c l a i m t o s t a n d u n t i l someone e m p i r i -

c a l l y d e m o n s t r a t e s t h a t some o t h e r p r e d i c t o r program a p p l i e d t o t h i s d a t a

d o e s b e t t e r . I stress e m p i r i c a l l y d e m o n s t r a t e s , f o r I t h i n k c o n c e r n w i t h

D r . G a t l i n ' s r e p e a t e d c l a i m o f f a n t a s t i c comput ing and p a t t e r n r e c o g n i z i n g

a b i l i t i e s o f t h e human mind, s u p p o r t e d by i n v a l i d a n a l y s e s , is a w a s t e o f

o u r t ime as i t s t a n d s . L e t t h e n e x t c l a i m a n t f o r a p r e d i c t o r s t r a t e g y

d e m o n s t r a t e t h a t i t p r e d i c t s .

I t h i n k I can go much f u r t h e r i n my c o n c l u s i o n , however, and c l a i m

t h a t t h e d a t a s t r o n g l y s u g g e s t t h a t e i t h e r no m a t h e m a t i c a l i n f e r e n c e s t r a t e g y

was used t o any s i g n i f i c a n t e x t e n t t h e p e r c i p i e n t s , o r , i f one was u s e d , --- f

i t was a c o n s i d e r a b l y less p o w e r f u l one t h a n Dronek and I have d e v i s e d , and

s o would l e a v e e v e n more t h a n 70% o f t h e h i t t i n g i n t h e f i r s t T r a i n i n g S t u d y

a t t r i b u t a b l e t o ESP. T h i s c o n c l u s i o n has a l r e a d y been s u g g e s t e d by t h e a n a l y s e s

o f t h e p a t t e r n s o f t a r g e t and r e s p o n s e b i a s I p r e s e n t e d e a r l i e r , which showed

t h a t t h e p e r c i p i e n t s ' s t r o n g r e s p o n s e b i a s e s were a l m o s t a lways d i f f e r e n t

from t h e s l i g h t b i a s e s i n t h e t a r g e t s e q u e n c e s . F u r t h e r s t u d i e s o f t h e in -

t e r n a l p a t t e r n i n g o f t h e d a t a g i v e even s t r o n g e r s u p p o r t t o t h i s c o n c l u s i o n .

T a r t -26

I n my P r e s i d e n t i a l Address l a s t yea r ( T a r t , 1977b) and e l sewhere ( T a r t ,

1977c) , I r e p o r t e d on t h e d i s c o v e r y o f a s t r o n g , n e g a t i v e r e l a t i o n s h i p be-

tween t h e magni tude of ESP h i t t i n g on t h e r e a l t i m e t a r g e t and t h e magni tude

o f h i t t i n g on t h e immediate f u t u r e ( + l ) t a r g e t , a f i n d i n g i n d e p e n d e n t l y s i g -

n i f i c a n t i n two s e p a r a t e s t u d i e s . I proposed a t h e o r y o f t r a n s t e m p o r a l i n h i -

b i t i o n , a n i n f o r m a t i o n p r o c e s s i n g mechanFsm f o r ESP, t o accoun t f o r t h i s re-

l a t i o n s h i p , and I a l s o p o i n t e d o u t t h a t t h e f i n d i n g was q u i t e r o b u s t : i f

a l l t h e t a r g e t sequences showing s i g n i f i c a n t s i n g l e t b i a s ( t h r e e , i n two

s t u d i e s ) were d e l e t e d from t h e computa t ions , t h e r e l a t i o n s h i p was s t i l l

s t r o n g l y and s i g n i f i c a n t l y p r e s e n t . F i g u r e shows a t y p i c a l example of

t h e t empora l d i s p l a c e m e n t p a t t e r n o f h i t t i n g and m i s s i n g w i t h a t a l e n t e d

p e r c i p i e n t , w i t h s t r o n g r e a l time h i t t i n g , s t r o n g m i s s i n g on t h e +1 f u t u r e

t a r g e t and on t h e -1 and -2 p a s t t a r g e t s (due a t l e a s t p a r t l y t o r e s p o n s e

b i a s e s ) , and g e n e r a l p o s i t i v e and n e g a t i v e , l a r g e l y chance, f l u c t u a t i o n s f o r

o t h e r t i m e d i s p l a c e m e n t r e g i s t e r s .

As a c o n t r o l on a p o s s i b l e mathemat ica l e s t i m a t i o n s t r a t e g y a r t i f a c -

t u a l l y c r e a t i n g t h e n e g a t i v e r e l a t i o n s h i p between r e a l t ime and +1 h i t t i n g ,

I I r a n t h e same s o r t s o f a n a l y s e s f o r temporal d i s p l a c e m e n t s on t h e r e s p o n s e s

o f t h e PPP t o t h e t a r g e t sequences . F i g u r e is a temporal d i s p l a c e m e n t

a n a l y s i s on t h e same t a r g e t sequence a s t h a t used i n F i g u r e , and i t i s

t y p i c a l o f t h e PPP r e s u l t s . I c o u l d t a b u l a t e v a r i o u s pa ramete r s o f t h e s e

a n a l y s e s s t a t i s t i c a l l y and show enormous d i f f e r e n c e s , b u t t h e two f i g u r e s

convey t h e e s s e n t i a l p o i n t : a ma themat ica l e s t i m a t i o n s t r a t e g y l i k e t h e PPP

g i v e s i n t e r n a l p a t t e r n i n g t o t h e d a t a t h a t i s o b v i o u s l y n o t h i n g l i k e t h a t

shown by a c t u a l p e r c i p i e n t s . The d i f f e r e n c e s a r e d i s c u s s e d i n more d e t a i l

T a r t -27

e l sewhere ( T a r t , 1977b).

- - - - - - - - - - - - - - - - - - -- 5- I n s e r t F igure about here

SL toe R I

Any proponent of a mathemat ical e s t i m a t i o n theory , then , has a cha l l enge

here : what k ind o f p r e d i c t i v e s t r a t e g y can t hey d e v i s e t h a t can be empi r i -

c a l l y demonstra ted t o bo th produce t h e enormous number o f h i t s t h e pe r c i -

p i e n t s showed and t h e same i n t e r n a l p a t t e r n i n g s ?

Conclusions :

Time does no t a l l ow me t o adequa t e ly review the v a r i o u s p o i n t s touched

on i n t h i s paper , nor do I wish t o dwel l f u r t h e r on t h e inadequac ies o f D r .

G a t l i n ' s arguments and ana ly se s . I s h a l l c l o s e by j u s t ment ioning t h e main

p o s i t i v e c o n t r i b u t i o n s and conc lu s ions t h a t have come o u t o f t h i s d i s c u s s i o n .

F i r s t , t h e p o t e n t i a l importance o f i n f l a t i n g s c o r e s through some k ind

of mathemat ica l i n f e r e n c e s t r a t e g y i n ESP s t u d i e s employing immediate feed-

back of r e s u l t s has heen underscored.

Second, t h i s d i s c u s s i o n has emphasized t h a t t h e p r e d i c t a b i l i t y o f a

t a r g e t sequence i s our pr imary concern i n t h i s m a t t e r , no t d e p a r t u r e s from

randomness per se.

Thi rd , t h e inadequacy of t h e s t a n d a r d measure f o r b i a s , t h e Chi-square

test (and D r . G a t l i n ' s D-measures) f o r t e l l i n g us how p r e d i c t a b l e a t a r g e t ?

sequence i s by mathemat ica l i n f e r e n c e s t r a t e g i e s has been demonstra ted empiri-

c a l l y : I am s u r e some o f you who a r e more ma thema t i ca l l y i n c l i n e d than I

could demonstra te i t i n e l e g a n t mathemat ica l form.

Fourth , t h e need t o d e a l d i r e c t l y and e m p i r i c a l l y w i t h t he q u e s t i o n

o f p r e d i c t a b i l i t y of t a r g e t sequences , r a t h e r t han i n f e r e n t i a l l y , has been

underscored. The c o n t r i b u t i o n of Dronek and myself t h a t has been submi t ted

f o r p u b l i c a t i o n i s a powerful s t a r t i n d i r e c t l y d e a l i n g w i t h t h i s ques t i on .

T a r t - 28

F i f t h , a s t o t h e s p e c i f i c q u e s t i o n o f whe the r t h e p e r c i p i e n t s i n my

f i r s t T r a i n i n g S t u d y a c t u a l l y i n f l a t e d t h e i r o u t s t a n d i n g h i t t i n g s c o r e s b y

a m a t h e m a t i c a l i n f e r e n c e s t r a t e g y , i n a d d i t i o n t o o r i n s t e a d o f u s i n g ESP,

s e v e r a l a n a l y s e s s k e t c h e d h e r e have d e m o n s t r a t e d t h a t t h e y c o u l d n o t have

r done anywhere n e a r l y a s w e l l as t h e y d i d w i t h a known e s t i m a t o r s t r a t e g y ; - t w k b ~ ,

t h e e v i d e n c e s t r o n g l y s u g g e s t s t h e y d i d use a known e s t i m a t o r s t r a t e g y

t o any s i g n i f i c a n t e x t e n t .

The r e s u l t s o f t h i s f i r s t T r a i n i n g S t u d y have now been t h o r o u g h l y ques-

t i o n e d on a wide v a r i e t y o f grounds i n t h e l i t e r a t u r e ( G a t l i n , 1978a; 1978b;

Gardner , 1977; O ' B r i e n , 1976; S t a n f o r d , 1977) , and I b e l i e v e t h e r e s u l t s

i n d i c a t i n g t h e p r e s e n c e o f v e r y h i g h l e v e l s o f ESP i n t h e d a t a have w i t h -

s t o o d t h i s q u e s t i o n i n g e x t r e m e l y w e l l (see T a r t , 1976b; 1977d; 1977f ; 1978a)

making them some o f t h e b e s t d a t a i n con tempora ry p a r a p s y c h o l o g i c a l e x p e r i -

m e n t a t i o n . They s u g g e s t , among o t h e r t h i n g s , t h a t ser ia l s e l e c t i o n p r o c e d u r e s

c a n f i n d v e r y h i g h s c o r i n g p e r c i p i e n t s , t h a t immediate f eedback c a n a t l e a s t

s u s t a i n i f n o t i n c r e a s e ESP f u n c t i o n i n g , and t h a t we now have a g l i m p s e o f

a b a s i c i n f o r m a t i o n p r o c e s s i n g s t r a t e g y f o r ESP, t r a n s t e m p o r a l i n h i b i t i o n ,

which i n t u r n l e a d s t o a more s e n s i t i v e test f o r t h e p r e s e n c e o f ESP ( T a r t ,

1977b) . I would s u g g e s t t h a t t h e r e i s more p r o f i t i n t r y i n g t o r e p l i c a t e 4 5 h r q c fw*[

and expand t h e s e f i n d i n g s i n new e x p e r i m e n t a t i o n t h a n i n &

Tart -29

References

Cha i t in , G., Randomness and mathemat ica l proof. S c i e n t i f i c American, 1975,

'232, 47-52.

Davis, J., & Akers, C., Randomization and t e s t s f o r randomness.

J o u r n a l o f Parapsycholoqy, 1974, 38, %id 393-407.

Ga t l i n , L., Comments on t h e c r i t i c a l exchange betweeen Drs. S tan fo rd and

Tart . Jou rna l o f t h e A m e r i c a n Soc ie t y f o r P s y c h i c a l Research, 1978,

72, 77-81. (a )

G a t l i n , L., Dr. G a t l i n t s r e p l y t o Dr. Tart . J o u r n a l o f t h e American So-

c i e t y f o r P s y c h i c a l Research, 1978, 72, 294-296. ( b )

G a t l i n , L., A new measure o f b i a s i n f i n i t e sequences w i t h a p p l i c a t i o n s t o

ESP data. Paper, Parapsycho log ica l Assoc ia t ion , S t . Lou is , 1978. ( c )

Gardner, fl.,

Gardner, Me, ESP a t random. New York Review o f Books, 1977, August 14.

OtBr ien, D., Review of T a r t ' s " A p p l i c a t i o n o f .i?S Learn ing Theory t o ESP

Performance", Journa l o f Parapsycholoqy, 1976, 40, 76-81.

Stanford, R e , The a p p l i c a t i o n o f l e a r n i n g t heo ry t o ESP performance: a rev iew

o f Dr. C. T. T a r t ' s monograph. Jou rna l o f t h e American Soc ie ty f o r Psy-

c h i c a l Researbh, 1977, 71, 55-80.

! Tar t , C., Card guessing t e s t s : l e a r n i n g paradigm o r e x t i n c t i o n paradigm?

Jou rna l o f t he American Soc ie t y f o r Psych i ca l Research, 1966, 60, 46-55.

Tar t , C., The A p p l i c a t i o n o f Learn inq Theory t o ESP Performance. New York:

Parapsychology Foundation, 1975.

Tart -30

T a r t , C., L e a r n i n q t o Use E x t r a s e n s o r y - I J e r c e p t i o n . Chicago: U n i v e r s i t y o f

Ch icago P r e s s , 1976. ( a )

T a r t , C., Rqply t o O tBr i en . J o u r n a l o f P a r a p s y c h o l o q y , 1976, 40 , 240-246. ( b )

T a r t , C., Improv ing s r e a l - t i m e ESP by s u p p r e s s i n g t h e f u t u r e : t r a n s - t e m p o r a l

i n h i b i t i o n . P a p e r , I n s i t t u t e o f E l e c t r i c a l and E l e c t r o n i c E n g i n e e r s ,

New York, 1977. ( a )

T a r t , C., Space , time, and mind. P r e s i d e n t i a l Adres s , P a r a p s y c h o c l o g i c a l

A s s o c i a t i o n , Washington , D.C., 1977. ( b )

T a r t , C., P s i : S c i e n t i f i c S t u d i e s o f t h e P s y c h i c Realm. New York: Du t ton ,

1977. ( c )

T a r t , C., Toward h u m a n i s t i c e x p e r i m e n t a t i o n i n p a r a p s y c h o l o g y : A r e p l y t o

t o D r . S t a n f o r d . J o u r n a l o f t h e American S o c i e t y f o r P s y c h i c a l Resea rch ,

1977, 71 , 81-102. ( d )

T a r t , C., Toward c o n s c i o u s c o n t r o l o f p s i t h r o u g h immed ia t e f e e d b a c k t r a i n -

i n g : some c o n s i d e r a t i o n s o f i n t e r n a l p r o c e s s e s . J o u r n a l of t h e American

S o c i e t y f o r P s y c h i c a l R e s e a r c h , 1977, 71 , 375-408. ( e )

T a r t , C., P s i and s c i e n c e . New York Review o f Books, 1977, O c t o b e r 13. ( f )

T a r t , C., D r . T a r t ' s r e p l y t o D r . G a t l i n . J o u r n a l o f t h e American S o c i e t y

f o r P s y c h i c a l R e s e a r c h , 1978, 72 , 81-07.

Table 1

Bias Measures o f Target Sequences

i n t he F i r s t Training Study

S ing l e t Gat l in , Doublet Gat l in

Perc ip ien t

P5

P3

P4

* i nd i ca t e s P c .05 , l - t a i l ed .

T a b l e 2

High F r e q u e n c i e s o f T a r g e t G e n e r a t o r

V e r s u s P e r c i p i e n t s

Percipient Target Response Target Response

P5 7* 7* 895 7,5*

P3 5* 2* 995* g92*

P4 9 7* 599 7,9*

P2 9 5 5,9* 299

* indicates that averall distribution departed from the equiprobable or serial independence

model with P < .05, 1 -tailed.

+ ties for highest rank were broken randomly

Table 3 . .

Extreme Test of Mathematical Predictive

Strategy: Deleting Hits from Best

Predictors as - Postdicted, Singlet

and Doublet Levels

Deleting Deleting Hits on Hits on

Original Highest Highest Hits Singlet: Singlet&Doublet

Percipient C R* CR CR

* These CRs may differ slightly from published data due to

Gatlin1s practice of substituting random responses for

Pass data.

EXPECTATION

P< .01, ONE-TAILED

1 2 3 4 5 6 7 8 9 1 0 TARGETS . c E - T i -

EXPECTATION

x:,,, = 22.40 I

P < .O2, ON E-TAI LED

1 2 3 4 ' 5 6 7 8 9 10 TARGETS

------ ---------- EXPECT.ATION 1 0 9 9 9 9 9 9 9 9

x : ~ ~ , = 7.20

NS

2 3 4 5 6 7 8 9 10 TARGETS

G A T L I N ' S ' O P T I M A L E S T I M A T I O N W I N D O W S

ESTIMATION WINDOW LENGTH?

I 1 I I

- -

- - DOUBLETS

-

- -

- - \

- -

-

-

- -

I I I

B i a s Measures of Target Sequences

i n t h e F i r s t Tra in ing s tudy '

S i n g l e t Ga t l in Doublet Ga t l in

P e r c i p i e n t ,x2 X' ~ ; ( f )

* i n d i c a t e s P c.05, 1- ta i led .

P e r c i p i e n t

r 5

P3

F4

P2

High Biases of Target Generator

versus P e r c i p i e n t s

+ S i n g l e t s

Target Response

7* 7*

Response

7,5*

9,2*

7,9*

299

5,3*

899

2 ,8*

498

4,6*

5,6*

* i n d i c a t e s t h a t o v e r a l l d i s t r i b u t i o n departed

from t h e equiprobable o r s e r i a l independence

model with P < ,05, 1 - t a i l e d ,

+ t i e s f o r h ighes t rank were broken randomly

Extreme Test of Mathematical Predictive

Strategy: Deleting Hits from Best

Predictors as - Postdicted, Singlet

and Doublet Levels

Deleting Deleting Hits on Hits on

Original Highest Highest Hits Singlet : Singlet&Doublet

Percipient C R* CR CR

* These CRs may differ slightly from published data due to Gatlint s practice of substituting random responses for

Pass data.