InterfaceSAYA Web Project

Vladimir Postel

Edward Rafaelov Advisors: Prof. Shlomi Dolev & Mr. Michael Orlov

Introduction

Saya is a receptionist robot of the computer science department at Ben Gurion University of the Negev.

The mechanical control of Saya is realized via pneumatic actuators, and additionally via neck and eyes DC motors (simultaneously controlled by a microprocessor receiving simple commands via RS-232 port).

Additionally, Saya has a video camera with regular USB interface in her left eye, and similarly transparent connections to microphone and speakers.

Two computers connected to Saya, one is responsible for speech synthesis and the other is for video capture, for face recognition and additional features.

The goal of our project is to build a web site in order to make it possible to control remotely some of Saya's features and devices from the web in addition to increase popularity of Saya – robot receptionist project, by providing information about the robot, and make some of Saya's features accessible to everyone. In order to make the robot more attractive it's possible to make Saya also remotely interactive by adding the possibility to transmit the video from the Saya's left eye camera and audio from the microphone. The site should supply the ability to control remotely those devices that are vital parts of Saya, and also create a possibility to communicate remotely by sending requests and messages to Saya from the site.

Feature that enables changing Saya's facial expressions remotely can be added as well.

Implementation

The site was created by using several development tools: Html / Dhtml and CSS were used in order to create and design the web pages. Javascript was used at some parts of the web site pages. PHP was used at pages where remote connections were needed, some of the functions were implemented in PHP. For example: ssh2 connections, socket connections with TCP/IP. Java - transmitting audio and video servers were implemented in Java using JMF and RTP. Servers on the computers of Saya that communicate with the web site are implemented in Java as well. Connection to neck and eyes DC motors via RS232 and initialization is also implemented in Java with RxTx library. Java Applet is receiving video and audio streams from the camera and microphone and displaying them in real time (the applet is signed).

The web interface project consists of several parts:

1. Sentry power controller of eyes and compressor

Connection to sentry (device that control the power of Saya's camera and compressor for pneumatic actuators that are used to create facial expressions) enables control of the power supply for eyes and compressor. In order to achieve our purpose we use php5 with ssh2 library for connecting to the sentry device that has unique ip address. We create a remote shell on sentry and perform the commands. After the execution of a command, the status of the eyes and compressor devices is performed to the user. PHP - a widely-used general-purpose scripting language that is especially suited for Web development and can be embedded into HTML. PHP generally runs on a web server, taking PHP code as its input and creating Web pages as output. However, it can also be used for command-line scripting and client-side GUI applications. PHP can be deployed on most web servers and on almost every operating system and platform free of charge. 2. Eyes initialization Before turning on the eyes, an initialization should take place. We wrote tcp/ip server on java that runs on a computer that responsible for eyes (Saya1 – computer that is responsible for video capturing, from video camera located in Saya's left eye). When a user wants to initialize the eyes, the server receives a message, turns on a thread that listens to com2 port. The connection to the com2 port implemented by RXTX java library. RXTX is a native library providing serial and parallel communication for the Java Development Toolkit (JDK). All the process is automatic, the thread gets questions about initial values from the device and sends proper values back.

3. Video and audio transmission Two computers are responsible for audio and video streaming, thus two servers were created. The camera is connected to Saya1 computer – transmitting the video stream (video server). The microphone is connected to Saya2 – transmitting the audio stream (audio server). Transmission process consists from the server and client sides.

Server side: the transmitter is located on the server side, it captures the stream (video or audio) from the device (video camera or microphone) that is connected to each computer (saya1 and saya2) respectively. When a client connects to the site in order to view the video and audio, the ip of his computer is sent to the transmitting servers. The servers add the new client to their transmitting lists, create new session and start transmitting the audio and video streams to the client. When a client closes the page, the servers get a "bye message" (RTCP) and automatically delete the client from their lists and close the sessions that were created for him.

Client side: when a client enters the web page to view the video

and audio, his computer's ip is sent, by using php, to the transmitting servers (Saya1 and Saya2). After that, an applet that receives and presents the media streams, is downloaded to the client's computer. The applet connects to the transmitting servers, receives the video and audio streams and performs them to the user in real time. In order to transmit/receive the real time video and audio streams, JMF (Java Media Framework) library of Java is used. The JMF is a Java Library that enables audio, video and other time-based media to be added to Java applications and applets. This optional package, which can capture, playback, stream, and transcode multiple media formats, extends the Java Platform, Standard Edition (Java SE) and allows development of cross-platform multimedia applications. The communication is established by using the Real-time Transport Protocol (RTP) that defines a standardized packet format for delivering audio and video over the Internet. RTP can carry any data with real-time characteristics, such as interactive audio and video. It goes along with the RTCP and it's built on top of the User Datagram Protocol (UDP). Applications using RTP are less sensitive to packet loss, but typically very sensitive to delays.

RTCP provides out-of-band control information for an RTP flow. It partners RTP in the delivery and packaging of multimedia data, but does not transport any data itself. It is used periodically to transmit control packets to participants in a streaming multimedia session. The primary function of RTCP is to provide feedback on the quality of service being provided by RTP. 4. Communicating with Saya The Part that is responsible for the communication with Saya consists from the server and client sides as well. Server side: TCP/IP server runs on the computer that is responsible for the speech synthesis. The server was implemented in java . The server receives messages from clients, pushes them to queue and then delivers them to the speech recognizer engine one by one, so Saya can answer the messages and client can hear the answer by the audio stream via the applet. Client side: Implemented in php and html. The web page, that is responsible for sending messages to Saya, has a text input field. The client writes a message and clicks the "send" button on the page. When the "send" button is clicked, tcp/ip socket connection to the server is created and the message is transferred to the server on Saya2 computer. A possibility to add face expressions to messages is added as well. So, when Saya answers to the request or a message, an expression can be obtained on her face. The protocol we use is Transmission Control Protocol (TCP) that is one of the core protocols of the Internet protocol suite. TCP provides reliable, in-order delivery of a stream of bytes. 5. Security Some pages are secured by using .htaccess and .htpasswd for authorization, authentication. .htaccess (hypertext access) is the default name of Apache's directory-level configuration file. .htaccess files are often used to specify the security restrictions for the particular directory, hence the filename "access". The .htaccess file is often accompanied by an .htpasswd file which stores valid usernames and their passwords.

6. Design The web site pages were implemented with Css, Dhtml, Html, Javascript and Php. Cascading Style Sheets (CSS) is a stylesheet language used to describe the presentation of a document written in a markup language. Its most common application is to style web pages written in HTML. It is designed primarily to enable the separation of document content (written in HTML or a similar markup language) from document presentation (written in CSS). This separation can improve content accessibility, provide more flexibility and control in the specification of presentational characteristics, and reduce complexity and repetition in the structural content. JavaScript is a scripting language most often used for client-side web development. It is a dynamic, weakly typed, prototype-based language with first-class functions. The language is best known for its use in websites (as client-side JavaScript), but is also used to enable scripting access to objects embedded in other applications. Dynamic HTML or DHTML is a collection of technologies used together to create interactive and animated web sites by using a combination of a static markup language (such as HTML), a client-side scripting language (such as JavaScript), a presentation definition language (Cascading Style Sheets, CSS), and the Document Object Model. A DHTML webpage is any webpage in which client-side scripting changes variables of the presentation definition language, which in turn affects the look and function of otherwise "static" HTML page content, after the page has been fully loaded and during the viewing process. Thus the dynamic characteristic of DHTML is the way it functions while a page is viewed, not in its ability to generate a unique page with each page load.

Control of the web interface: Running Video transmission server on Saya1: Enter the C:\TxVideoServer library and enter java Tx command. Running eyes initialization on Saya1: Enter the C:\rxtxserver library and type java server. Running Audio transmission server on Saya2 Enter the C:\Tx library and type java Tx command. Running text messages processor on Saya2: Enter the C:\chatServer library and type java ChatServer command.

Screen Shots

Sentry- power controller of eyes and compressor:

Authentication using .htaccess

Viewing video and audio streams from Saya's left eye camera and microphone and the ability to send messages with face expressions to Saya.

Media library containing pictures, Press releases and TV videos about Saya.

Main page:

.rchitecture of Audio And Video transmission processesA

4) Gets message with client’s IP, creates connection & start transmitting. When client exits closes the connection.

3) Client’s IP is sent to the transmitting computers

6) The applet runs, and presents the streams, (audio and video), from both transmitters.

1) Client enters the site that is on the server.

2) The Applet is downloaded to client’s host.

RTP R

TP

Saya1

Audio

Saya2

Video

5) Audio is transmitted to clients IP.

Architecture of connection to Sentry, Execution of commands and eyes initialization

User

Site on apache status

Sentry

Eyes Compressor

Com2 Rx-Tx

Saya1

3) Server that runs on Saya1 gets eyes initialization request, connects to eyes via com2 port and runs initialization

1) Client enters the site that is on the server, Opens a page that provides control over Sentry.

2) Commands that client chooses are executed on sentry.

Video

Architecture of 'Communication' with Saya

Audio & speech recognition

Answer

Message to Saya

2) Message is received by the server on Saya2 and processed by Saya2

1) Client enters the site, Opens a page that provide ability to send messages to Saya.

Site on apache

Saya 2

User

ionConclus

Humanoid robots are very attractive. Most people are excited by communicating with the robots.

Making Saya more realistic and practical will receive more attention and interest, so many additional helpful features could be developed.

For future development of the project, additional options can be added:

1) Ability to recognize people who have Bluetooth technology cell phones, by managing a database of people with cell phone numbers. When a person with Bluetooth passes near Saya, his name can be found in the data base and he will be greeted by Saya.

2) The site can grow to a portal.

3) Links to other projects about Saya can be added.

4) Control and Configuration of Saya’s features can be done via the web.

References HTML, Dhtml, JAVASCRIPT, CSS tutorials - http://www.w3schools.com . Java Media Framework API - java.sun.com/products/java-media/jmf/ Javaª Media Framework API Guide - ©1998-99 Sun Microsystems, Inc. 2 l 550 Garcia Avenue, Mountain View, California 94043-1100 U.S.A.

library providing serial and parallel communication for the Java Development Toolkit – www.rxtx.org PHP Tutorial – http://php.eitan.ac.il. php5 Secure Shell2 Functions manual - http://www.php.net/manual/en/ref.ssh2.php ssh2 connection with php5 - http://kevin.vanzonneveld.net/techblog/article/make_ssh_connections_with_php .htaccess password protection - http://tools.dynamicdrive.com/password/ definitions are taken from: http://en.wikipedia.org/wiki/PHPhttp://en.wikipedia.org/wiki/Cascading_Style_Sheets http://en.wikipedia.org/wiki/Javascript http://en.wikipedia.org/wiki/Real-time_Transport_Protocol http://en.wikipedia.org/wiki/Real-time_Transport_Control_Protocolhttp://en.wikipedia.org/wiki/Java_Media_Framework Saya's technical and mechanical information reference: http://www.cs.bgu.ac.il/~orlovm /teaching/saya http://www.free-css-templates.com/- CSS template. http://randomibis.com/coolclock/ - javascript clock feature. http://www.cs.bgu.ac.il/information/photo_gallery/saya/index.html - photo gallery of Saya in computer science department.

Index Introduction ………………………………………………...2 Implementation……………………………………………..3

Sentry power controller of eyes and compressor........4 Eyes initialization……………………………………4 Video and audio transmission…………………….....5 Communicating with Saya………………………......6 Security………………………………………………6 Design………………………………………………..7

Control of the web interface………………………………...8 Screen Shots…………………………………..……………9-12 Architecture Architecture of Audio And Video transmission processes…..13 Architecture of connection to Sentry, Execution of commands and eyes initialization……………..14 Architecture of 'Communication' with Saya………………....15 Conclusion…………………………………………………...16 References………………………………………….………...17