The  PuReWidgets  toolkit  for  interac6ve  public  display  

applica6ons  

Jorge  C.  S.  Cardoso  jorgecardoso@ieee.org  

 Ubicomp@UMinho    

May  30,  2012  

1  

Content  

•  Mo6va6on  •  PuReWidgets  features  •  Architecture  •  Applica6on  models  •  Programming  library  overview  •  HelloWorld  

2  

Mo6va6on  Complexity  of  incorpora6ng  interac6ve  features  in  public  display  applica6ons.  Too  many  things  to  answer  when  designing  the  interac6on  for  a  public  display  applica6on:  •  What  controls  can  we  use  to  offer  interac6ve  features  to  users?  

–  No  standard  interac6ve  controls  for  public  displays  •  What  interac6on  mechanisms  can  be  used  to  provide  those  features?  

–  Various  possible  interac6on  mechanisms  (SMS,  BT  naming,  OBEX,  Email,  IM,  mobile  app,  desktop,  QR  codes,  touch,  etc.)  

–  Different  loca6ons  may  have  different  interac6on  resources  •  How  will  those  features  work?  

–  Desktop  interac6on  paradigm  does  not  generally  apply  –  Applica6ons  may  not  always  be  visible  

•  How  can  the  applica6on  react  to  different  users?  –  The  interac6on  environment  is  mul6-­‐user  –  We  need  to  iden6fy/differen6ate  users  

•  How  can  users  tell  what  they  can  do  with  the  applica6on?  –  Where  are  the  features  presented?  

3  

Mo6va6on  We  can  solve  all  those  ques6ons  for  ad  hoc  applica6ons,  but  

•  This  means  wasted  effort  in  developing  applica6ons  –  Programmers  must  deal  with  issues  extraneous  to  the  main  applica6on  logic  

•  Will  certainly  lead  to  inconsistencies  in  the  interac6on  model  –  Each  solu6on  will  behave  slightly  differently  –  Will  confuse  users  

We  need  a  programming  toolkit  for  interac6ve  public  display  applica6ons!  

4  

Toolkit  

•  The  toolkit  should  provide  a  consistent  model  for  user  interac6on  and  applica6on  development  – But  s6ll  allow  some  diversity  in  solu6ons  

5  

PuReWidgets  

•  PuReWidgets  tries  to  solve  these  problems.  •  To  make  sure  it  covers  a  wide  range  of  applica6ons,  we  looked  at  various  interac6ve  public  display  systems  to  learn  – General  requirements  – Types  of  controls  – Types  of  input  mechanisms  

6  

PuReWidgets  Widget-­‐based  toolkit.    

•  A  widget  represents  an  interac6ve  feature.  •  Is  represented  by  class  in  an  object-­‐oriented  programming  model.  •  Applica6ons  instan6ate  widgets,  define  a  callback,  and  receive  

interac6on  event  through  the  callback    •  Main  features  

–  Mul6ple,  extensible,  controls  –  Independence  of  input  mechanisms  and  modali6es  –  Automa6cally  generated  graphical  interfaces  –  Asynchronous  interac6on  –  Concurrent  interac6on        –  Graphical  affordances  –  Server  &  client  libraries  

7  

Features  Mul6ple,  extensible,  controls.      •  Allows  developers  to  create  rich  interac6ve  applica6ons.  Controls  can  be  extended  with  custom  

func6onality.  •  We  studied  various  interac6ve  display  systems  to  define  a  set  of  basic  interac6ve  controls,  common  to  the  

majority  of  applica6ons:  

•  Ac6on  bugon  –  Trigger  an  ac6on  in  the  applica6on,  e.g.,  play  a  video  

•  Op6on  selec6on  –  Selects  among  a  set  of  op6ons,  e.g.,  to  vote  

•  Text  entry  –  Sends  text  to  the  applica6on,  e.g.,  a  comment,  tag,  search  keyword  

•  Download  –  Receives  a  media  file  from  the  applica6on,  e.g.,  download  poster  

•  Upload  –  Uploads  a  media  file  to  the  applica6on,  e.g.,  a  photo  to  be  displayed  

•  Check-­‐in  –  Says  “I’m  here”  

8  

Features  

Independence  of  input  mechanisms  and  modali6es.    •  Allows  developers  to  focus  on  the  core  applica6on  func6onality  and  not  on  the  low-­‐level  interac6on  details.  

•  Allows  applica6ons  to  work  in  heterogeneous  loca6ons,  with  different  mechanism    

•  Interac6on  can  be  accomplished  with  various  mechanisms  (SMS,  Bluetooth  naming,  QR  codes,  and  automa6cally  generated  graphical  interfaces).  

9  

Features  

Automa6cally  generated  graphical  interfaces.    •  Provide  a  richer  interac6on  for  desktop  and  mobile  devices.  

•  PuReWidgets  generates  web-­‐based  interfaces  for  each  applica6on.    

•  It  also  generates  QR  codes  for  each  applica6on's  feature    

10  

Features  

Asynchronous  interac6on.    •  Allows  applica6ons  to  receive  input  events  that  were  generated  when  the  applica6on  was  not  listening.  

•  PuReWidgets  provides  a  persistent  input  queue  for  each  applica6on  

•  Applica6ons  can  request  past  input  at  any  6me  

11  

Features  

Concurrent  interac6on.    •  Allows  various  users  to  interact  at  the  same  6me,  while  providing  applica6ons  with  iden6ty  informa6on  that  allows  them  to  differen6ate  users.  

•  Every  input  event  carries  a  user  id  (if  available)  – Where  possible,  anonymous  interac6ons  are  s6ll  “iden6fied”  (more  on  this  later)  

12  

Features  Graphical  affordances.    •  Allows  users  to  immediately  recognize  interac6ve  features  and  input  feedback.  

•  Controls  have  an  op6onal  graphical  representa6on  that  can  be  used  on  the  public  display  interface.  – Applica6ons  may  not  show  the  features  on  the  public  display,  but  they  will  s6ll  be  available    

•  Input  feedback  is  also  (op6onally)  displayed  on  the  public  display.  

13  

Features  

Server  &  client  libraries.    •  Programmers  can  choose  to  use  just  the  server,  just  the  client,  or  both  libraries  together.  

•  Allows  developers  to  choose  the  best  applica6on  model.  

14  

PuReWidgets  Architecture  

15  

Interac6on  services  •  PuReWidgets  service  –  Keeps  informa6on  about  every  widget  –  Generates  graphical  interfaces  for  desktop,  mobile,  touch  plaporms    •  These  allow  anonymous  interac6ons,  but  they  generate  persistent  cookie-­‐based  random  ids.  

–  Generates  QR  codes  for  every  widget  –  Generates  unique  textural  reference  codes  for  data-­‐based  interac6ons  (SMS,  BT  naming,  OBEX,  etc.)  

•  IO  infrastructure  –  (from  Instant  places)  –  Provides  low-­‐level  data-­‐based  interac6on  

16  

PuReWidgets  library  

•  Provides  an  object-­‐oriented  library  of  widgets  •  Hides  the  communica6on  details  with  the  PuReWidgets  service  

•  Provides  other  features,  unrelated  to  interac6on,  such  as  – Applica6on  “place-­‐aware”  storage  

•  PuReWidgets  provides  a  name-­‐value  server  datastore  for  easy  storage  of  place-­‐specific  seqngs/state  

–  Skeleton  admin  interface  for  place-­‐owners  •  To  configure  place-­‐specific  seqngs  

17  

Local  applica6on  model  

18  

Local  applica6on  model  

•  Applica6on  logic  on  the  public  display  (player)  – Subject  to  the  schedule  6mings  

•  May  not  be  able  to  react  (or  update  data  structures)  immediately  when  user  interacts  – But  will  eventually  (when  it  is  displayed  again)  

•  When  displayed,  the  toolkit  periodically  asks  the  service  for  input  events  – And  triggers  the  appropriate  widget  event  

19  

Remote  applica6on  model  

20  

Remote  applica6on  model  

•  Applica6on  logic  on  the  server  •  Can  react  (or  update  data  structures)  immediately  to  user  input  – Even  if  not  on  the  public  display,  the  reac6on  can  be  visible  on  other  channels  

•  The  toolkit  periodically  asks  the  service  for  input  events  – And  triggers  the  appropriate  widget  event  

21  

PuReWidgets  Implementa6on  

•  Google  Appengine  (server)  •  Google  Web  Toolkit  –  GWT  (client)  

•  Overview  of  the  library  

22  

Using  PuReWidgets  

•  Hello  World  applica6on  (local  model)  •  Video  

23  

Ini6al  Evalua6on/Development  process  

•  Two  applica6ons  developed  so  far  (ignore  the  “graphic  design”,  please)  –  Youtube  player,  media  intensive,  complex  graphical  components  (several  panels  that  change  over  6me),  local  applica6on  model  

–  Everybody  votes,  based  on  display  owner  created  content,  simpler  graphical  component,  combines  remote  and  local  applica6on  models  

24  

Ini6al  Evalua6on/Development  process  

•  Con6nuous  refinement  – Develop  interac6ve  public  display  applica6ons  – Gain  more  insight  about  the  difficul6es      – Refine  the  toolkit  to  address  the  iden6fied  problems  

– Refactor  the  applica6ons  to  include  the  toolkit  changes  

25  

Summary  

•  PuReWidgets  facilitates  applica6on  development  – Provides  ready-­‐to-­‐use  interac6ve  controls  –  Integrates  various  input  mechanisms  via  an  I/O  infrastructure  

– Generates  applica6on  interfaces  for  desktop,  mobile,  and  QR  codes  

– Provides  client  and  server  applica6on  development  models  

26  

The  end  

27