A Multi-Display System forDeploying and Controlling HomeAutomation

Yucheng Jinfortiss GmbHGuerickestr. 25Muenchen, 80805 Germanyjin@fortiss.org

Christian Prehoferfortiss GmbHGuerickestr. 25Muenchen, 80805 Germanyprehofer@fortiss.org

Chi Tai DangAugsburg UniversityUniversitaetsstr. 6aAugsburg, 86159 Germanydang@informatik.uni-augsburg.de

Elisabeth AndréAugsburg UniversityUniversitaetsstr. 6aAugsburg, 86159 Germanydang@informatik.uni-augsburg.de

Permission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.

Copyright is held by the owner/author(s).

ITS ’14 , Nov 16–19 2014, Dresden, GermanyACM 978-1-4503-2587-5/14/11.http://dx.doi.org/10.1145/2669485.2669553

AbstractIn this paper, we present a concept of using a homedevices mashup tool to wire home devices on a tabletopdisplay in combination with web based UIs on mobiledevices to control home devices. This concept is realizedby a multi-display system supported by the open-sourceframework Environs. The mashup tool on the tabletopenables multiple people to deploy home networked devicesco-located collaboratively with a more natural andintuitive interface. Moreover, web based UIs on mobiledevices enable individuals to control home devicesuniversally with high accessibility and mobility.

Author KeywordsInternet of Things; Networked devices; Multi-displayenvironment; Tabletop; Mobile devices

ACM Classification KeywordsH.5.2 [User Interfaces]: User interface managementsystems (UIMS).

IntroductionHome automation is featured by connected homeappliances and automatic awareness of the surroundingenvironment employing the Internet of Things (IoT)technology. Within recent years, the IoT technology hasincreased the available networked and intelligent devices

for such home appliances, such as Nest LearningThermostat [1], Philips Hue Connected Bulb [2], or SmartTVs. However, due to heterogeneity of platforms anddiversity of home device controllers, deploying andcontrolling home networked devices is still cumbersome.The emergence of IoT mashup tools, such as ClickScript[3], or WotKit [4], enable people to connect devices andcompose services to create IoT applications by means of aweb based graphical editor. However, these tools aredesigned for PCs which do not offer the advantages ofhorizontal tabletop displays, such as co-locatedcollaborative or natural interaction, or a large designspace. Therefore, we intend to combine a visual mashuptool with a tabletop display to wire home devices.Thereby, enable people to deploy home networked devicescollaboratively with less effort. Furthermore, current homeappliances require people to use different controllers tointeract with home devices. Our concept replaces thosecontrollers with familiar and personal mobile devices, suchas smartphones or tablets, which play as universalcontrollers for the home networked devices.

This paper describes a concept build on top of amulti-display system consisting of a tabletop display andmobile devices. The tabletop shows a visual mashup toolfor deploying home networked devices whereas the mobiledevices make use of web based UIs for controlling aparticular home device. The system aims to assist endusers to create personalized home automation easily andto control home devices universally using their mobiledevice.

Related WorksIn the light of this paper’s aim, several researchersinvestigated the idea of using mobile devices to controlhome appliances. Suo et al. [11] showed the system

House Genie to control and monitor home networkeddevices on smartphones universally. Nichols et al. [8]presented Huddle which can generate task-based UIs onhandheld devices for controlling multiple connected homeappliances. Pan et al. [9] demonstrated a handheld devicecalled GeeAirfor which can control home appliancesremotely through multimodal interaction using gestures,physical buttons, speech, and light. However, the usabilityof these existing works is restricted by limited designspace of mobile devices, especially when people wiredevices to deploy home automation. Moreover, they donot support multiple people to work collaboratively.

In fact, several research projects have investigated theusage of visual tools for end users to deploy and controlhome automation. The interactive home networkmanagement system Eden [12] helps people understandhow a home network works and configure network andnetworked devices intuitively. The application OSCAR [7]supports end users to monitor and manipulate connecteddevices, wherein people can save reusable configurationsfor frequent activities. Despite the benefits of these toolsfor home automation, so far there is still no one properplatform to support multiple people to wire home devicesconveniently and collaboratively, i.e. on a tabletopsurface. Kim et al. [6] presented an Ambient Wallinterface to monitor and control surrounding home devicesby simple gestures, but the workload of building such asystem is not trivial.

Seifried et al. [10] showed the system CRISTAL whichuses a tabletop display to enable gestural interaction onvideo images of the surrounding environment. It supportsmultiple people to control home devices and mediauniversally. However, controlling home devices only on atabletop display may be restricted by its limited mobility.

The Concept of our SystemOur system enables multiple people to manage and controlhome networked devices in a multi-display environmentcomprised of mobile devices and a tabletop display. To bespecific, this system provides the following features:

Figure 1: An example of a homeheating system in ClickScript.

Figure 2: UIs of home devices onmobile devices.

• It enables multiple people to deploy home networkeddevices collaboratively on a tabletop display.

• It allows people to interact with a particular homedevice through their mobile devices.

• It supports people to interact with home devices viamore natural and intuitive touch-enabled userinterfaces.

• It is based on a multi-display environment with highresponsiveness through low latency communicationbetween the components.

System ArchitectureThe tabletop display shows a visual mashup tool formanaging networked devices, whereas mobile devicespresent web based user interfaces for controlling homedevices as sketched in Figure 3.

Figure 3: Multi-display system architecture.

In order to achieve low latency communication and aresponsive system, our system makes use of theopen-source framework Environs [5], which is amulti-platform software framework that enables spanningof heterogeneous multi-display environments. We use thisframework to span an application environment thatincludes the tabletop and mobile devices in order toextend the possibilities of the tabletop application andco-located interactions with mobile devices. Allparticipating devices in the environment are automaticallyidentified and managed by the framework which enablesmobile devices to participate in an ad-hoc way.

Furthermore, the framework enables interactive videoportals between devices where touch-input on the mobiledevice is put through to the tabletop application andhappens there in realtime. By this way, users are able tomanipulate the mashup tool on the tabletop using theirmobile devices. Our system thereby allows managing andcontrolling of the home networked devices through mobiledevices and the tabletop application at the same time.

Tasks of Deployment and ControlOur concept focuses on two tasks that occur in deployingof home automation. One of the tasks is to manageconnections between physical devices through visualmashup tools as exemplarily depicted in Figure 1 forClickScript. In our system, such an application runs onthe tabletop display allowing people to plan and connecthome devices together by means of prevalent and familiardirect touch manipulations.

The other task is to control a particular device via its userinterface which is realized in our system by providingweb-based UIs of the home devices on users’ mobiledevices, for example as sketched in Figure 2.

Users who want to control a specific home device eitherselect the particular device on the mobile device or put themobile on or near to the particular device’s icon in thevisual mashup tool on the tabletop display, see Figure 4.

Figure 4: A mobile tablet placedon the air conditioner icon,whereupon the UI of the airconditioner is shown on themobile tablet.

Figure 5: The user controls theair conditioner by means of themobile tablet.

In our setup, each mobile device has a visual markermounted at the bottom-side which enables the tabletop todetect and identify the mobile device on the tabletopsurface. Upon placement of the mobile device, thetabletop creates an interactive video portal to the mobiledevice in order to visually replicate the tabletop areacovered by the mobile device. This video portal allowsusers to open the user interface of a home networkeddevice on their mobile device by touching on the particularicon. Afterwards, the mobile device keeps showing theuser interface even if it is lift off from the tabletop display,thus people can control the home devices from mobiledevices anywhere, see Figure 5.

ConclusionIn this paper, we presented a concept of a system tocontrol home networked devices in a multi-displayenvironment. Tabletop displays and mobile devices areused to manage and control home networked devicescollaboratively and conveniently. Future works include thedesign and implementation of a visual mashup tool for thetabletop display and web-based UIs for home networkeddevices in order to conduct research regarding useable UIdesigns or guidelines for tabletop mashup tools, orappropriate interaction techniques for collaborative orsynchronized manipulations.

AcknowledgmentsThe work described in this paper is co-funded byOC-Trust (FOR 1085) of the DFG.

References[1] https://nest.com/.[2] http://www2.meethue.com/.[3] http://clickscript.ch/.[4] Blackstock, M., and Lea, R. Wotkit: a lightweight

toolkit for the web of things. In Proc. of WoT,[5] Dang, C. T., and André, E. A framework for the

development of multi-display environmentapplications supporting interactive real-time portals.In Proc. of EICS, ACM Press (2014), 45–54.

[6] Kim, H., Jeong, K., Kim, S., and Han, T. Ambientwall: Smart wall display interface which can becontrolled by simple gesture for smart home. In Proc.of SIGGRAPH Asia,

[7] Newman, M. W., Elliott, A., and Smith, T. F.Providing an integrated user experience of networkedmedia, devices, and services through end-usercomposition. In Pervasive Computing. Springer.

[8] Nichols, J., Rothrock, B., Chau, D. H., and Myers,B. A. Huddle: automatically generating interfaces forsystems of multiple connected appliances. In Proc. ofUIST,

[9] Pan, G., Wu, J., Zhang, D., Wu, Z., Yang, Y., andLi, S. Geeair: a universal multimodal remote controldevice for home appliances. Pers Ubiquit Comput.

[10] Seifried, T., Haller, M., Scott, S. D., Perteneder, F.,Rendl, C., Sakamoto, D., and Inami, M. Cristal: acollaborative home media and device controller basedon a multi-touch display. In Proc. of ITS,

[11] Suo, Y., Wu, C., Qin, Y., Yu, C., Zhong, Y., and Shi,Y. Housegenie: Universal monitor and controller ofnetworked devices on touchscreen phone in smarthome. In Proc. of UIC-ATC,

[12] Yang, J., Edwards, W. K., and Haslem, D. Eden:supporting home network management throughinteractive visual tools. In Proc. of UIST,

https://nest.com/http://www2.meethue.com/http://clickscript.ch/

IntroductionRelated WorksThe Concept of our SystemSystem Architecture

Tasks of Deployment and ControlConclusionAcknowledgmentsReferences