PopObject: A Robotic Screen for EmbodyingVideo-Mediated Object Presentations

Kana Kushida(&) and Hideyuki Nakanishi

Department of Adaptive Machine Systems, Osaka University, 2-1 Yamadaoka,Suita, Osaka 565-0871, Japan

{kana.kushida,nakanishi}@ams.eng.osaka-u.ac.jp

Abstract. Some studies have been conducted on 2.5D display surfaces, whichdisplays a two-dimensional video with a curved or deformable display surface.In this study, we developed a telepresence system, which protrudes a specificpart of a remote video by a 2.5D display surface. The system has a stretchprojection screen and a push-out mechanism. The screen is pushed out frombehind and expresses protrusion. This protrusion is aiming to express the depthinformation of the remote video. We expected that it enhances the remoteconversation partner’s presence. We supposed a conversation such as showingand explaining an object in videoconferencing, and conducted experiments inorder to confirm an effect of the developed system as a telepresence system. Theprotrusion was in synchronization with the movement of the objects on theprojected video. The results of the experiment suggested that the protrusion onthe screen surface provided by this system strengthens the presence of the objectand remote person.

Keywords: Telepresence system � Social telepresenceVideo-mediated communication � Elastic display

1 Introduction

Telepresence is a technology that allows a person to feel as if they were present at aplace other than their true location [1–5]. One of the popular applications of telep-resence is a video conferencing system. However, most of them are designed for flatsurfaces where the perception of depth is lost. This decreases the presence of a remoteperson [6].

Some studies have tried to express a spatial three-dimensional effect of a two-dimensional video with a curved or deformable display surface. For example, “Live-mask” is a surrogate system with a face-shaped screen [7], and “Interactive SpatialCopy Wall” is a system, which represents a three-dimensional shape of a remote personwith hundreds of movable pipes [8]. Furthermore, shape change is increasingly used inphysical user interfaces, both as input and output [9–12].

We focused on the idea of representing a remote person with a flexible and de-formable surface. We applied the elastic displays for a telepresence system as an outputinterface, which represents depth information about the video. The elastic displays areproposed for a new-generation input interface [13–16]. It is attained by a stretch cloth

© Springer Nature Switzerland AG 2018H. Egi et al. (Eds.): CollabTech 2018, LNCS 11000, pp. 200–212, 2018.https://doi.org/10.1007/978-3-319-98743-9_16

and offers new ways to interact with multi-dimensional data by using the deformationof the surface [17]. By applying an elastic display, we can build the deformable screen,which has a smooth surface and can express various shapes.

Additionally, we attempted to verify the effectiveness of this simplified 3D surfacefor a telepresence system. Then, we conducted experiments in order to confirm aneffect of the developed system as a telepresence system.

2 System

The Snapshot of our deformable screen is shown in Fig. 1, and the constitution of thedeveloped system is shown in Fig. 2. This system makes the object on the projectedvideo appear protruded with its deformable surface.

There was a projection screen, which was made of a stretch cloth and was flexiblydeformed by pushing. The extruder was set behind the screen. If the object on theprojected video moved forward, the extruder also moved forward and pushed thescreen to the front. The screen was deformed along a shape of the extruder. Thisdeformation of the screen adds the perception of the depth and a spatial three-dimensional effect to the video. A shape of the extruder imitates that of the object.Figure 3 shows some examples of various objects and the extruders which correspondto each objects.

Linear positioning tables that were set behind the screen moved to the extruder. Thelocation of the extruder was synchronized with that of the object on the projected video.

Fig. 1. Snapshot of our deformable screen

PopObject: A Robotic Screen for Embodying Video-Mediated Object Presentations 201

3 Experiment 1

At first, the experiment 1 was done as a preliminary experiment. We estimated a qualityof the developed system.

3.1 Conditions

In the experiment 1, we compared the following two conditions:

• Flat screen condition: The video of the remote explainer was projected on the flatscreen (Fig. 4(a)).

• Deformable screen condition: The video of the remote explainer was projected onthe screen and the screen was deformed along with the video (Fig. 4(b)).

Extruder

Flexible screendeformed by extruder and expresses three-dimensional effect

Projectorprojects remote videoon flexible screen

Linear motion table device

Fig. 2. Mechanism of our deformable screen

(a) Ball (b) Stuffed animal

Fig. 3. The object (left) and the extruder corresponding to the object (right)

202 K. Kushida and H. Nakanishi

3.2 Setup

In this paper, we focus on the effect of the deformable screen on social telepresence,and we developed the one way system. Figure 5 shows the setup of the experiment.A vertical display monitor situated along one side of a desk provides a nearly identicalimage of the remote side.

3.3 Task

We contrived a situation where the remote person explains while showing a ball or astuffed animal as the task. In the ball task, the experimenter held out the ball and had asimple conversation with the subject. In the stuffed animal task, the experimenter heldout the stuffed animal. In all conditions the presentation began and ended with agreeting. To conduct a controlled experiment, we offered the same conversation time.

3.4 Questionnaire

We conducted a questionnaire after the experiment. The questionnaire included severalstatements and asked the extent to which the statements matched the impression that

(a)Flat screen condition

(b) Deformable screen condition

Fig. 4. Conditions of the experiment 1

PopObject: A Robotic Screen for Embodying Video-Mediated Object Presentations 203

the participant had. The questionnaire asked five questions shown in Fig. 6. Thesubjects answered the questionnaires after they experienced both conditions.

Q1, Q2 and Q3 check the quality of the presentation. Q5 correspond to presence ofthe object. Q4 correspond to presence of the remote person. All the statements wererated on a 7-point Likert scale where 1 = strongly disagree, 4 = neutral, and7 = strongly agree.

3.5 Results and Discussion

We compared two conditions by the within-subjects experiment. Eight subjects con-sisting of seven males and one females participated with the ball task. Also Eightsubjects consisting of six males and two females with the stuffed animal task. Theparticipants were undergraduate students whose ages ranged from 18 to 24 years. They

(a) Local Side

(b) Remote Side

120

130

55

90

6370

Subject

120

70

Explainer

Object

Web cameraand kinect

Fig. 5. Setup of the experiment 1

204 K. Kushida and H. Nakanishi

1 2 3 4 5 6 7

Flat screen condition

Deformable screen condition

*

**

** p<.01* p<.05† p<.1

†

Q4. I felt as if I were viewing the stuffed animal in the same room.

Q1. The video was sufficiently clear.

Q2. The audio was sufficiently clear.

Q3. The presentation was intelligible.

Q5. I felt as if I were viewing the explainer in the same room.

Bal

l St

uffe

d an

imal

Q4. I felt as if I were viewing the stuffed animal in the same room.

Q1. The video was sufficiently clear.

Q2. The audio was sufficiently clear.

Q3. The presentation was intelligible.

Q5. I felt as if I were viewing the explainer in the same room.

Fig. 6. Results of the experiment 1

PopObject: A Robotic Screen for Embodying Video-Mediated Object Presentations 205

were recruited via a part-time job search site, and paid about 20 dollars. We tookcounter balance to not produce an influence by the order of conditions. The result of theexperiment is shown in Fig. 6, where each box represents the mean scores of thequestionnaire, and each bar represents the standard error of the mean. We found nosignificant difference in Q1, Q2 and Q3. We found a significant difference in Q5 (balltask: p < .05, stuffed animal task: p < .01). We also found a significant difference inQ4 (stuffed animal task: p < .1).

The subjects marked higher scores in the deformable screen condition than the flatscreen condition in Q5, the questions regarding presence of the object. Since the sametendency was shown in both the ball task and the stuffed animal task, it was found thatthe effect of the deformation screen was obtained regardless of the complexity of theshape of the extruder.

Additionally, The Q5 shows that presence of the remote person was also stronger inthe deformable screen condition than other conditions. It suggested that the developedsystem enhances the social telepresence.

4 Experiment 2

We conducted experiments 2 in order to confirm an effect of the developed system as atelepresence system. We expected that a spatial three-dimensional feeling expressed bythe deformable screen could enhance the presence of the remote person and object.

We made the following hypotheses.

• Hypothesis 1: a spatial three-dimensional feeling expressed by the deformablescreen enhance the presence of the object.

• Hypothesis 2: a spatial three-dimensional feeling expressed by the deformablescreen enhance the presence of the remote person.

4.1 Conditions

In the experiment, we compared the flat screen condition, deformable screen conditionand Liquid crystal display condition.

• Liquid crystal display condition: The video of the remote explainer was exhibitedon the liquid crystal display monitor like general videoconferencing systems (Fig. 7(c)).

The liquid crystal display condition was added in order to compare our system withthe general videoconferencing systems. Liquid crystal displays have an advantage ofvideo quality over projection screens. We expected that the deformable screen iseffective to express the presence in despite of lower quality of the video.

206 K. Kushida and H. Nakanishi

4.2 Setup

We developed the one way system as with the experiment 1. Figure 9 shows the setupof the experiment. A vertical display monitor situated along one side of a desk providesa nearly identical image of the remote side.

4.3 Task

We contrived a situation where the remote person explains an animal while showing astuffed animal as the task. In all conditions, the experimenter held out the stuffedanimal and had a simple conversation with the subject. To conduct a controlledexperiment, we offered the same conversation time.

In the deformable screen condition, the screen protruded along to the experi-menter’s arm and the stuffed animal with movement of the experimenter (Fig. 8).

(a)Flat screen condition

(b) Deformable screen condition

(c)Liquid crystal display condition

Deformation

Fig. 7. Conditions of the experiment

PopObject: A Robotic Screen for Embodying Video-Mediated Object Presentations 207

(a) Local side

(b) Remote side

120

120

55

42

5674

Subject

120

74

Explainer

Object

Web cameraand kinect

Fig. 9. Setup of the experiment 2 (length unit: centimeters)

(a) Remote video (b) Extruder

Area of the protrusion

Fig. 8. The remote video and the extruder of the experiment

208 K. Kushida and H. Nakanishi

4.4 Questionnaire

We conducted a questionnaire after the experiment. The questionnaire included severalstatements and asked the extent to which the statements matched the impression thatthe participant had. The subjects answered the questionnaires after they experienced allthree conditions. The questionnaire asked eight questions shown in Fig. 10.

Q1, Q2, Q3 and Q4 check the quality of the presentation. Q5 and Q6 correspond topresence of the object. Q7 and Q8 correspond to presence of the remote person. All thestatements were rated on a 7-point Likert scale where 1 = strongly disagree,4 = neutral, and 7 = strongly agree.

4.5 Results and Discussion

To clarify the hypotheses we compared three conditions by the within-subjectsexperiment. Fourteen subjects consisting of eight males and six females participated.The participants were undergraduate students whose ages ranged from 18 to 24 years.They were recruited via a part-time job search site, and paid about 20 dollars. We tookcounter balance to not produce an influence by the order of conditions. The result of theexperiment is shown in Fig. 10, where each box represents the mean scores of thequestionnaire, and each bar represents the standard error of the mean. The figurecompares the conditions by a one-way ANOVA, followed by Bonferroni correction.

We found no significant difference in Q1, Q2, Q3, Q4 and Q8. We found asignificant difference in Q5 (F(2, 26) = 12.82, p < .01) and Q6 (F(2, 26) = 13.26,p < .01). Multiple comparisons showed that presence of the object was significantlystronger in the deformable screen condition than the flat screen condition and liquidcrystal display condition (p < .05). We also found a significant difference in Q7 (F(2,26) = 8.000, p < .01). Multiple comparisons showed that presence of the remoteperson was significantly stronger with the deformable screen condition than the flatscreen condition and liquid crystal display condition (p < .05).

Contrary to expectation, we found no significant differences in Q1, the questionregarding quality of the video, because eight subjects did not perceive a difference inthe display method. Six subjects recognized the difference between the flat screencondition and the liquid crystal display condition. However, the subjects marked higherscores in the deformable screen condition than the liquid crystal display condition inQ5 and Q6, the questions regarding presence of the object. It suggested that thedeveloped system has a potential compared to the general videoconferencing systemsin despite of lower quality of the video.

Additionally, the Q7 shows that presence of the remote person was also stronger inthe deformable screen condition than other conditions. According to the results of theinterviews, we found that there are individual differences in the perception of depthinformation in the deformable screen condition. One subject believed that there was athree-dimensional feeling on the entire screen in despite of the fact that we deformedonly part of the screen. On the other hand, two subjects did not perceive a deformationof the screen. Eleven subjects recognized that only the stuffed animals protruded andthe experimenter does not. During the task, these subjects paid attention to theexperimenter’s face. The three-dimensional feeling of the experimenter’s arms were

PopObject: A Robotic Screen for Embodying Video-Mediated Object Presentations 209

1 2 3 4 5 6 7

Flat screen condition

Deformable screen condition

Liquid crystal display condition

* p<.05

Q7. I felt as if the experimenter were facing me.

Q5. I felt as if I were able to touch the stuffed animal.

Q6. I felt as if I were viewing the explainer in the same room.

Q8. I felt as if I were viewing the stuffed animal in the same room.

* *

Q1. The video was sufficiently clear.

Q2. The audio was sufficiently clear.

Q3. The presentation was intelligible.

Q4. The shape of the stuffed animal was easy to understand.

*

*

* *

Fig. 10. Results of the experiment 2

210 K. Kushida and H. Nakanishi

frequently ignored. Another factor was caused by the weak point of this deformablescreen that cannot express a detail of the complex shapes. The deformable screen couldnot express the shape of the remote person’s arms clearly. However, many of thesubjects felt presence of the remote person although they did not percept the three-dimensional feeling of the remote person. In this case, the presence of the objectenhanced feeling to share the object with remote person, and it enhanced presence ofthe remote person. Future studies are needed in order to compare the deformable screenand an object sharing system.

5 Conclusion

In this study, we developed a telepresence system which has a stretch projection screenand a push-out mechanism. The screen deforms in synchronization with the move ofthe objects on the projected video. This deformation highlights a specific part of aremote video by a protrusion on the display surface.

This protrusion is aiming to express the depth information of the remote video sowe expected that it enhances the remote conversation partner’s presence. We conductedexperiments in order to confirm an effect of the developed system as a telepresencesystem. The results of the experiment showed that the protrusion on the screen surfaceprovided by this system strengthens the presence of the object and remote person.Additionally, It was suggested that this system is more valid than the liquid crystaldisplay in despite of lower quality of the video.

Acknowledgments. This work was supported by JSPS KAKENHI Grant NumbersJP26280076, Telecommunication Advancement Foundation, and Tateishi Science and technol-ogy Foundation.

References

1. De Greef, P., Ijsselsteijn, W.A.: Social presence in a home tele-application. CyberPsychol-ogy Behav. 4(2), 307–315 (2001)

2. Nakanishi, H., Kato, K., Ishiguro, H.: Zoom cameras and movable displays enhance socialtelepresence. In: Proceedings of the SIGCHI Conference on Human Factors in ComputingSystems, pp. 63–72. ACM (2011)

3. Onishi, Y., Tanaka, K., Nakanishi, H.: Embodiment of video-mediated communicationenhances social telepresence. In: Proceedings of HAI 2016, pp. 171–178 (2016)

4. Wesugi, S., Miwa, Y.: Facilitating interconnectedness between body and space for full-bodied presence-utilization of “Lazy Susan” video projection communication system. In:Exploring the Sense of Presence 2004, pp. 208–215 (2004)

5. Leithinger, D., Follmer, S., Olwal, A., Ishii, H.: Physical telepresence: shape capture anddisplay for embodied, computer-mediated remote collaboration. In: Proceedings of UIST2014, pp. 461–470 (2014)

6. Prussog, A., Mühlbach, L., Böcker, M.: Telepresence in videocommunications. In:Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 38, no.3, pp. 180–184 (1994)

PopObject: A Robotic Screen for Embodying Video-Mediated Object Presentations 211

7. Misawa, K., Ishiguro, Y., Rekimoto, J.: Livemask: a telepresence surrogate system with aface-shaped screen for supporting nonverbal communication. Inf. Media Technol. 8(2), 617–625 (2013)

8. Wesugi, S., Ishikawa, K., Suzuki, N., Miwa, Y.: Interactive spatial copy wall for embodiedinteraction in a virtual co-existing space. In: 2004 13th IEEE International Workshop onRobot and Human Interactive Communication, ROMAN 2004, pp. 265–270. IEEE (2004)

9. Follmer, S., Leithinger, D., Olwal, A., Hogge, A., Ishii, H.: inFORM: dynamic physicalaffordances and constraints through shape and object actuation. In: Proceedings of the 26thAnnual ACM Symposium on User Interface Software and Technology, pp. 417–426 (2013)

10. Holman, D., Vertegaal, R.: Organic user interfaces: designing computers in any way, shape,or form. Commun. ACM 51(6), 48–55 (2008)

11. Rasmussen, M.K., Pedersen, E.W., Petersen, M.G., Hornbæk, K.: Shape-changinginterfaces: a review of the design space and open research questions. In: Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems, pp. 735–744. ACM(2012)

12. Sahoo, D.R., Hornbæk, K., Subramanian, S.: TableHop: an actuated fabric display usingtransparent electrodes. In: Proceedings of the 2016 CHI Conference on Human Factors inComputing Systems, pp. 3767–3780. ACM (2016)

13. Dand, D., Hemsley, R.: Obake: interactions on a 2.5 D elastic display. In: Proceedings of theAdjunct Publication of the 26th Annual ACM Symposium on User Interface Software andTechnology, pp. 109–110. ACM (2013)

14. Peschke, J., Göbel, F., Gründer, T., Keck, M., Kammer, D., Groh, R.: DepthTouch: anelastic surface for tangible computing. In: Proceedings of the International WorkingConference on Advanced Visual Interfaces, pp. 770–771. ACM (2012)

15. Troiano, G.M., Pedersen, E.W., Hornbæk, K.: User-defined gestures for elastic, deformabledisplays. In: Proceedings of the 2014 International Working Conference on Advanced VisualInterfaces, pp. 1–8. ACM (2014)

16. Brave, S., Ishii, H., Dahley, A.: Tangible interfaces for remote collaboration andcommunication. In: Proceedings of the 1998 ACM Conference on Computer SupportedCooperative Work, pp. 169–178. ACM (1998)

17. Müller, M., Knöfel, A., Gründer, T., Franke, I., Groh, R.: FlexiWall: exploring layered datawith elastic displays. In: Proceedings of the Ninth ACM International Conference onInteractive Tabletops and Surfaces, pp. 439–442. ACM (2014)

212 K. Kushida and H. Nakanishi