International Conference on Artificial Reality and TelexistenceEurographics Symposium on Virtual Environments (2017)T. Huang (Editor)

Poster

Can Face Swapping Technology FacilitateMental Imagery Training?

Haruka Matsumura1, Hironori Watanabe2, Tai Chih Chen2, Takafumi Taketomi1, Yasuhide Yoshitake2,Alexandor Plopski1, Christian Sandor1 and Hirokazu Kato1

1Nara Institute of Science and Technology, Japan2National Institute of Fitness and Sports in KANOYA, Japan

AbstractIn this research, we conducted a preliminary study to investigate the effectiveness of face swapping technology for mentalimagery training. To confirm its effectiveness, we used transcranial magnetic stimulation for measuring motor evoked potential(MEP) as brain excitability during mental imagery training. In the experiment, we used three motions: wrist dorsiflexion asan easy-to-perform motion, and pen spinning and baoding balls rotating motions as difficult motions. In each target motion,we compared MEPs when watching own motion video, another person’s motion video, and another person’s motion videowith the face swapped with own face. The results showed that there was a difference between MEPs in difficult motion videoobservations. Watching another person’s motion video with face swapping showed higher MEP than simply watching anotherperson’s video.

CCS Concepts•Computing methodologies → Mixed / augmented reality; Image processing;

1. Introduction

Mental imagery training is a motor learning method that utilizesimagination in order to learn a particular motion. For example, insports training, athletes usually observe an expert’s motion to im-prove their performance during training. In the neurophysiologyresearch field, it is known that brain activity can be observed froma person watching a video of an athlete performing a motion, evenif the said person is not performing the actual motion [RC04]. Forthis reason, mental imagery training has been known to facilitatemotor performance. In addition, it has been reported that observingone’s own video produces stronger brain activity than observinganother person’s motion video [KBRF10]. From these physiolog-ical backgrounds, we make the hypothesis that "it becomes easyto imagine one’s own motion by observing another person’s mo-tion video but with the face swapped with one’s own face, andas such, face swapping facilitates athlete’s motor learning." Onthe other hand, in the computer vision research field, face swap-ping technologies have been proposed to generate a natural faceswapping result [BKD∗08, DSJ∗11]. Recently, this technology hasbeen widely used in the mixed reality application domain such asSNOW [SNO].

In this study, we investigate an application of face swapping ina mental imagery learning scenario. In order to confirm the effec-tiveness of face swapping, we conducted a preliminary experiment.In the experiment, we measured motor evoked potential (MEP) asa brain activity in the motor cortex by using transcranial magnetic

stimulation (TMS) method while watching videos. Differences ofMEPs are measured between own video, another person’s video,and another person’s video with face swapping. In addition, wealso investigated a method for quantitatively evaluating the qualityof face swapping results. To the best of our knowledge, no researchhas been done yet on evaluating the quality of face swapping resultsquantitatively.

2. Face Swapping Algorithm

The purpose of the preliminary study is to confirm the effectivenessof face swapping for mental imagery training. Therefore, we use asimple configuration; a person sits in front of a camera, and thenhe/she performs a finger or arm motion. For this reason, in theexperiment, the face swapped video is generated using a simplealgorithm. The face swapping algorithm consists of an initializationprocess and a sequential process.

Initialization Process: In the initialization process, face part loca-tion differences dddi between a source image (trainee’s face image)Isrc and a target video (expert’s face image) Idst are calculated asfollows.

dddi =(AAA f it pppi +bbb f it

)−qqqi (1)

where AAA f it and bbb f it are affine transformation parameters to alignfaces in Isrc and Idst . pppi and qqqi are facial landmark locations in Isrcand Idst , respectively.

c© 2017 The Author(s)Eurographics Proceedings c© 2017 The Eurographics Association.

H. Matsumura et al. / Can Face Swapping Technology Facilitate Mental Imagery Training?

(a) wrist dorsiflexion (b) pen spinning (c) baoding balls rotating

Figure 1: Observed motions.

Sequential Process: In this process, facial landmark detection[KS14], facial landmark location compensation, and image warp-ing and composition are repeatedly executed. In the facial land-mark compensation process, detected landmark locations are com-pensated using dddi as follows.

q̄qq′i = sss jdddi + q̄qqi (2)

sss j is a scaling factor, which is calculated as follows.

sss j =

[w′j/w j 0

0 h′j/h j

](3)

where w j and h j are bounding box width and height of a face j, andsuperscript represents the target video. After this compensationprocess, the source image Isrc is warped into the target video Idst .In order to warp the source image, first, a face area is divided intosmall regions using Delaunay triangulation for face landmarks. Isrcis then warped into Idst using affine transformation calculated fromthe corresponded regions. Finally, warped regions are compositedusing poisson image blending [PGB03].

3. Experiment

We conducted a user study to confirm the effectiveness of faceswapping for mental imagery training. In the experiment, we mea-sured MEPs using TMS while imagining a target motion by ob-serving a target motion video. We used three motions as shown inFig. 1. Wrist dorsiflexion motion can be performed by all partici-pants. Therefore, we measured MEPs for own video, another per-son’s video and face swapped video. On the other hand, pen spin-ning and baoding ball rotating motions cannot be performed by allparticipants. Therefore, we used another person’s video and faceswapped video only. The participant’s own video, another person’svideo, and face swapped video were randomly shown to partici-pants. It should be noted that we used mirrored videos to reducethe effect of mental rotation. The number of participants for eachmeasurement are 6 for wrist dorsiflexion, 3 for pen spinning, and4 for baoding balls rotating. MEP was obtained as a peak-to-peakvalue from averaged waveform of 15 trials. It should be noted thatwe also measured a normal MEP in relaxed state. The normal MEPis used as a control data, and then obtained MEPs are normalized bythe normal MEP. Participants tried to imagine each target motionwithout actual motion by themselves.

Fig. 2 shows results of average MEPs in each motion. The re-sults confirmed that average MEPs of face swapped video werehigher than that of another person’s motion video. Specifically,

0.0

50.0

100.0

150.0

200.0

250.0

300.0

wrist dorsiflexion pen spinning baoding balls rotating

Per

senta

ge

of

ME

P p

eak-t

o-p

eak v

alue

norm

aliz

ed b

y c

ontr

ol

val

ue

[%]

own motion another person's motion face swapping motion

�16.9%

�3.7%

�8.3%

�4.1%

Figure 2: Average MEPs for each video observation.

there were large differences between another person’s video ob-servation and face swapped video observation in pen spinning andbaoding balls rotating cases. From these results, it can be consid-ered that the face is important to get good imagination in difficultmotions. In addition, according to the interview, we could con-firm that the relationship between MEP values and user’s sensation.Thus, we believe that TMS can be used to evaluate face swappingalgorithms quantitatively.

4. Conclusion

In this research, we conducted a preliminary user study to confirmthe effectiveness of face swapping in mental imagery training. Wemeasured MEP using TMS while watching videos from recordedmotions. From the results of the experiment, we consider that faceswapping is effective to get higher MEP. However, some partici-pants feel unnatural sensation in the face swapped video, and theirMEPs did not increase from another person’s video. In the future,we are planning to confirm the effectiveness of face swapping withmore participants. In addition, we are planning to develop a faceswapping algorithm that can generate self-recognizable image.

Acknowledgements

This work was supported by the MIC/SCOPE #162107006.

References[BKD∗08] BITOUK D., KUMAR N., DHILLON S., BELHUMEUR P.,

NAYAR S. K.: Face swapping: automatically replacing faces in pho-tographs. ACM Transactions on Graphics 27, 3 (2008), 39. 1

[DSJ∗11] DALE K., SUNKAVALLI K., JOHNSON M. K., VLASIC D.,MATUSIK W., PFISTER H.: Video face replacement. ACM Transactionson Graphics 30, 6 (2011), 130. 1

[KBRF10] KEYES H., BRADY N., REILLY R. B., FOXE J. J.: My faceor yours? event-related potential correlates of self-face processing. Brainand cognition 72, 2 (2010), 244–254. 1

[KS14] KAZEMI V., SULLIVAN J.: One millisecond face alignment withan ensemble of regression trees. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition (2014), pp. 1867–1874. 2

[PGB03] PÉREZ P., GANGNET M., BLAKE A.: Poisson image editing.ACM Transactions on Graphics 22, 3 (2003), 313–318. 2

[RC04] RIZZOLATTI G., CRAIGHERO L.: The mirror-neuron system.Annu. Rev. Neurosci. 27 (2004), 169–192. 1

[SNO] SNOW. http://snow.me/. last checked: 2017-09-19. 1

c© 2017 The Author(s)Eurographics Proceedings c© 2017 The Eurographics Association.