Customizable Virtual Keyboard

8/3/2019 Customizable Virtual Keyboard

1/13

Customizable Virtual Keyboard

Introduction High Level Design Software Implementation Hardware Implementation Testing Conclusion Appendix

Please send any comments or questions to:

Naweed Paya ([email protected]) Venkat Ganesh ([email protected])

Introduction

It is becoming increasingly difficult for users to interact with the slew of portable gadgetsthey carry, especially in the area of text entry. Although miniature displays and keyboardsmake some portable devices, such as cell phones and PDAs, amazingly small, users hands

do not shrink accordingly.

To solve this problem, we proposed a Virtual Keyboard. This device will replace a physicalkeypad with a customizable keyboard printed on a standard A3 size paper whosekeystrokes are read and translated to real input. This virtual keyboard can be placed on anyflat surface, such as desktops, airplane tray tables, kitchen counters, etc. and can theoreticallybe interfaced with any computing device that requires text entry. This would eliminate theneed to carry anything around and also prevent any chance of mechanical damage to thekeypad in harsh environments if a simple lamination is used to protect the paper. In addition,
http://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#introhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#introhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#designhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#designhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#hardwarehttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#hardwarehttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#softwarehttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#softwarehttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#testinghttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#testinghttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#conclusionhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#conclusionhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#appendixhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#appendixhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#appendixhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#conclusionhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#testinghttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#softwarehttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#hardwarehttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#designhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/index.html#intro


2/13

buttons on this device can be reconfigured on-the-fly to give a new keyboard layout using aGUI we built in Java and then transferring that data to the device using a computers serialport.

High Level Design

Figure 1: Finger shining bright red when passing through the laser plane

The Virtual Keyboard has three main components: the laser, camera, and printed keyboard.The laser beam is simply a conventional off-the-shelf red laser with a line-generatingdiffractive optical element attached to it. This assembly generates an invisible plane of redlight hovering a few millimeters above the typing surface. When a finger passes through thisplane, it shines bright red in that region.

Figure 2: C3038 image sensor module mounted on a custom PCB

The CMOS camera continuously captures images of the region where the printed keyboard issupposed to be placed and checks these images for red color data above a specified threshold.The threshold concept works in this case because the laser shining on a typical human fingergenerates saturating values of red color data, which is very easily distinguishable from its

surroundings.


3/13

Figure 3: Comparing an actual keyboard with a printed keyboard

Lastly, the printed keyboard is simply a standard A3 size paper that contains a customkeyboard layout. After rigorous testing, we decided to use a black background and blueletters for the printed keyboard because our device doesnt use its own light source.

Therefore, proper contrast is necessary to distinguish the typing finger from the surroundingarea in various lighting conditions. The actual programming of the printed keyboard layoutinto the device can be done using a serial port and a GUI we developed in Java. This GUIbasically gives the user a blank grid of buttons and the user can choose to assign any buttonto any letter or number he/she desires.

Software Implementation

The software component was split into 5 main components:

Implementing the I2C protocol to read and write registers from camera Reading values from camera to obtain 6 frames every second Processing the images to obtain a pressed key Converting the pressed key into a scan code which is then transmitted using the PS/2

protocol Sending serial data from a java application to update the array of scan codes in the

Mega32

Main Operation

At first we initialize PORTA on the Mega32 to take UV input from the camera and PORTCto communicate with the camera over the I2C interface. The baud rate is set to 19,200bps forserial communication. We then run the calibrate function on the camera, which looks at ablack keyboard to determine a distinguishable value for red color threshold. Then we call afunction called "init_cam" which performs a soft reset on the camera before writing therequired values to corresponding camera registers. These registers change the frame size to176x144, turn on auto white balance, set the frame rate to 6 fps, and set the output format to16-bit on the Y/UV mode with Y=G G G G and UV = B R B R. The code then enters aninfinite loop which checks for the status of the PS2 transmitting queue and tries to process the

next captured frame if the queue is empty. If not, the queue is updated and the PS2transmission is allowed to continue.


4/13


5/13


6/13

Laser

Figure 6: Red laser module with a line-generating DOE attached

Our original plan, at the time of the project proposal, was to use an infrared laser to detectbutton presses using the CMOS camera, but we realized that user safety would be a majorissue in that case. The user would never know even if he/she is staring directly at the laserand, therefore, there would be no way to prevent eye damage. In addition, we also realizedthat the CMOS camera were using (OV6630) is not very effective at detecting infrared light.

Hence, we decided to use a Class II 635nm red laser instead.

The laser module we bought came with a built-in driver; therefore, we didnt have to worryabout biasing the laser properly to make it operational. All we had to do was to connect thelaser to a 3V power source, which we obtained using a simple 3V voltage regulator.

Figure 7: Laser line generation calculation

The laser module also came with a line-generating diffractive optical element attached to it.However, since we didnt know the fan-angle for this DOE, we had to experiment withvarious distances in order to obtain a line length of at least 8.5, which was required to cover

the entire width of our printed keypad. In the end, we had to place the laser at a distance ofapproximately 12.5 to obtain good results.

Camera


7/13

Figure 8: C3038-4928IR 1/4 Color Sensor Module

For this project we decided to use the C3038 1/4 color sensor module with digital output,which uses OmniVisions CMOS image sensor OV6630. The two primary reasons why we

chose this specific camera module were low cost and the fact that it is capable of outputtingimage color data in progressive scan mode. Progressive scanning was an importantconsideration for us since we dont have enough computational power available on the

16Mhz Mega32 microcontroller to process entire frames at once; however, we can certainly

process images line-by-line as they come in. After rigorous testing and a lot of research, werealized that we could work with only the red channel data from the camera and still be ableto identify keystrokes accurately. Hence, we connected the 8-bit red channel output from thecamera (UV[7:0]) to PORTA[7:0] on the Mega32.

Casing

The hardware assembly for our device is designed to hold the camera at a fixed position suchthat it looks over the appropriate region of the printed keyboard. In addition, it also holds thelaser module at a fixed position such that the plane of red light completely covers the areaabove the printed keyboard. In order to ensure that new custom-printed keyboards can be

swapped in-and-out of the device while maintaining proper distances, we permanentlyattached a piece of black poster board of the right length to the assembly and mounted 4photo-corners on it.

Figure 9: Hardware casing for Virtual Keyboard device

Testing

Keystroke Accuracy:


8/13

As a result of the limited viewing angle of the camera and positioning of the laser, we had todesign and calibrate with various keypad layouts to make sure we could detect all of thebuttons with reasonable accuracy. Our final design for the generic keypad and testing results(percentage accuracy) for this layout are given in Figure 3. For the testing, we tried 100keystrokes per key and set the acceptance threshold at 70% for side areas and 80% for the

central area. This means that if we can recognize a certain key accurately at least 70 or 80times, respectively, out of the 100 times that its pressed, that key passes the test.

Figure 10: Testing results for keystroke detection accuracy

Conclusion

Although the final project was very satisfying, our results did not completely meet ourexpectations. The keyboard worked as we predicted but typing speed was minimal (about 60characters per minute) due to limited processing capabilities of the Mega32 microcontroller.

If we had more time, we would have liked to increase the theoretical maximum typing speedby possibly using another microcontroller in parallel or maybe even an external FPGA to doextra image processing. In addition, we would also like to include sound effects forkeystrokes and a dynamic calibration algorithm which can be used to orient the custom-printed keyboard in any direction. This sort of functionality would require performing 2Dimage transforms on-the-fly, which is not feasible with the existing microcontroller. Last but

not least, we could certainly try to improve our current keystroke detection algorithm toimprove typing accuracy.

Appendix

Standards

I2C-Bus Specification Version 2.1:


9/13

Two wires in an I2C bus, serial data (SDA) and serial clock (SCL), carry informationbetween the devices connected to the bus. Each device is recognized by a unique address andcan operate as either a transmitter or receiver, depending on the function of the device. Inaddition to transmitters and receivers, devices can also be considered as masters or slaveswhen performing data transfers. A master is the device which initiates a data transfer on the

bus and generates the clock signals to permit that transfer. At that time, any device addressedis considered a slave.

PS/2 Keyboard Protocol:

The PS/2 keyboard interface typically uses a bidirectional synchronous serial protocol, butfor our implementation we do not need the computer (host) to communicate with themicrocontroller (device). Therefore, for our purposes, the device always generates the clocksignal and all data is transmitted one byte at a time. Each byte is sent in a frame consisting of11 bits, in the following order:

1 start bit. This is always 0. 8 data bits, least significant bit first. 1 parity bit (odd parity). 1 stop bit. This is always 1.

The parity bit is set to 1 if there is an even number of 1's in the data bits and set to 0 if there isan odd number of 1's in the data bits. The number of 1's in the data bits plus the parity bitalways add up to an odd number (odd parity.) This is used for error detection. Data sent fromthe device to the host is read on the falling edge of the clock signal; the clock frequency mustbe in the range 10 - 16.7 kHz. This means clock must be high for 30-50 s and low for 30-50s.

Ethical and Legal Considerations

Throughout the final project, we committed ourselves to the highest ethical and professionalconduct and closely adhered to the IEEE Code of Ethics. We placed an extra emphasis on thefollowing points mentioned in the Code of Ethics:

1. To accept responsibility in making decisions consistent with the safety, health andwelfare of the public, and to disclose promptly factors that might endanger the publicor the environment.

2. To avoid real or perceived conflicts of interest whenever possible, and to disclosethem to affected parties when they do exist.3. To be honest and realistic in stating claims or estimates based on available data.4. To improve the understanding of technology, its appropriate application, and potential

consequences.5. To maintain and improve our technical competence and to undertake technological

tasks for others only if qualified by training or experience, or after full disclosure ofpertinent limitations.

6. To seek, accept, and offer honest criticism of technical work, to acknowledge andcorrect errors, and to credit properly the contributions of others.

7. To avoid injuring others, their property, reputation, or employment by false ormalicious action.


10/13

8. To assist colleagues and co-workers in their professional development and to supportthem in following this code of ethics.

Since we were using a Class II laser device, we always made sure to keep the laser targetedaway from other individuals in the Lab. In addition, we designed an enclosure for our device

such that the laser would not be visible to the user.

We did not have any legal considerations since we did not use code or algorithms from othersources, did not use parts regulated by federal agencies, and did not infringe upon anyexisting patents. Although a commercial product similar to our device, called the VirtualLaser Keyboard, is currently manufactured by a company known as I-Tech, we believe thatwe have significantly distinguished our product such that we will not encounter any copyrightissues. The commercial product is not dynamically reconfigurable, uses a red laser to projecta standard QWERTY keyboard pattern onto a surface, and uses an infrared laser forkeystroke detection. Our product, on the other hand, uses a customizable printed keyboard, ared laser for keystroke detection, and custom keystroke detection algorithms.

Safety

Figure 11: EM radiation absorption characteristics of the human eye

The human body is vulnerable to the output of certain lasers, and under certaincircumstances, exposure can result in damage to the eye and skin. Research relating to injurythresholds of the eye and skin has been carried out in order to understand the biologicalhazards of laser radiation. It is now widely accepted that the human eye is almost alwaysmore vulnerable to injury than human skin. The cornea (the clear, outer front surface of theeyes optics), unlike the skin, does not have an external layer of dead cells to protect it from

the environment. Hence, the cornea absorbs the laser energy and may be damaged. The figurebelow illustrates the absorption characteristics of the eye for different laser wavelengthregions. Since we only used a Class II laser in this project and provided a proper enclosurefor the device such that it isnt directly visible to the user, special protection is not requiredfor normal users. People with sensitive eyesight or other severe vision problems, however,might want to take some precautionary measures and should not use the device for extendedperiods of time. In addition, users are strongly advised not to look directly into the laser beamat any time.

Budget

Our total budget for this project was $75.00, and we easily managed to keep our costs lessthan that.

PART COST SOURCE


11/13

PART COST SOURCE

Total $48.41 --

RS-232 Serial Port $1.00 ECE 476 Digital Lab

MAX233CPP -- Sampled

Red LED -- ECE 476 Digital Lab

Jumpers -- ECE 476 Digital Lab

Surface mount capacitors, resistors -- ECE 476 Digital Lab

16 MHz Crystal Oscillator -- ECE 476 Digital Lab

LM340T5 Voltage Regulator -- Sampled

Slide Switch -- ECE 476 Digital Lab

Atmel ATMega32 $8.00 ECE 476 Digital Lab

40 pin DIP Socket $2.00 ECE 476 Digital Lab

8 pin DIP Socket $0.40 ECE 476 Digital Lab

635nm Laser $8.00 Ebay

9V Power supply -- ECE 476 Digital Lab

OV6630 CMOS Camera $25.03 Electronics123

PS/2 cable and USB adapter -- Previously owned

Poster board $3.99 Walmart

Wood -- Scrap

Task Distribution

Naweed Paya:

Solder prototype board and C3088 camera module Camera implementation Design and assemble hardware casing PS/2 communication Testing and debugging Final report

Venkat Ganesh:

Laser and camera implementation

Java applet I2C communication


12/13

Testing and debugging Final report

Acknowledgements

We would like to thank Prof. Bruce Land and the ECE 476 staff for their continual support,insightful comments, and suggestions which altogether made this project possible. We wouldalso like to thank Ruibing Wang, an ECE M. Eng student at Cornell University, for hisassistance in getting the I2C protocol to work. This protocol was necessary to communicatewith the camera, and was therefore an integral component of our project.

In addition, we would like to reference two past ECE 476 projects. To get an initial idea ofthe camera settings required for proper operation of the OV6630 camera using a Mega32microcontroller, we looked at the camera implementation in the project titled AutonomousSearchBot by John and Diego. In addition, we used the Wireless Keyboard project by

Luke Hejnar and Sean Leventhal as an example to implement the PS/2 keyboard protocolusing a Mega32 microcontroller.

Code Listing

Custom keyboard design GUI Serial communication from PC to Virtual Keyboard Camera operation and Image processing PS/2 keyboard protocol I2C communication

Schematic

Download in PDF format Download in SCH format

Datasheets

Mega32 microcontroller C3038 camera module OV6630 optical sensor ADP3300ART-32 voltage regulator

Code References

AVRCam Bayer color pattern

Vendor Sites

Digikey Electronics123
http://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/keyboardGUI.javahttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/SerialComm.javahttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/camera1.chttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/camera1.chttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/keyboard.chttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/keyboard.chttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/twi_master.hhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/twi_master.hhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/schematic.pdfhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/schematic.pdfhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/Schematic.schhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/Schematic.schhttp://www.atmel.com/dyn/resources/prod_documents/doc2503.pdfhttp://www.atmel.com/dyn/resources/prod_documents/doc2503.pdfhttp://www.electronics123.net/amazon/datasheet/c3038.pdfhttp://www.electronics123.net/amazon/datasheet/c3038.pdfhttp://www.digchip.com/datasheets/download_datasheet.php?id=728465&part-number=OV6630http://www.digchip.com/datasheets/download_datasheet.php?id=728465&part-number=OV6630http://www.digchip.com/datasheets/download_datasheet.php?id=121974&part-number=ADP3300ART-32http://www.digchip.com/datasheets/download_datasheet.php?id=121974&part-number=ADP3300ART-32http://www.jrobot.net/Projects/AVRcam.htmlhttp://www.jrobot.net/Projects/AVRcam.htmlhttp://en.wikipedia.org/wiki/Bayer_filterhttp://en.wikipedia.org/wiki/Bayer_filterhttp://www.digikey.com/http://www.digikey.com/http://www.electronics123.com/http://www.electronics123.com/http://www.electronics123.com/http://www.digikey.com/http://en.wikipedia.org/wiki/Bayer_filterhttp://www.jrobot.net/Projects/AVRcam.htmlhttp://www.digchip.com/datasheets/download_datasheet.php?id=121974&part-number=ADP3300ART-32http://www.digchip.com/datasheets/download_datasheet.php?id=728465&part-number=OV6630http://www.electronics123.net/amazon/datasheet/c3038.pdfhttp://www.atmel.com/dyn/resources/prod_documents/doc2503.pdfhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/Schematic.schhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/schematic.pdfhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/twi_master.hhttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/keyboard.chttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/camera1.chttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/SerialComm.javahttp://instruct1.cit.cornell.edu/courses/ee476/FinalProjects/s2008/nap33_vsg3/nap33_vsg3/final_web_page_files/keyboardGUI.java


13/13

Background Sites

ECE 476 AVR serial communication PS/2 Keyboard Protocol
http://www.electronics123.com/http://www.electronics123.com/http://www.electronics123.com/http://instruct1.cit.cornell.edu/courses/ee476/Serialcom/http://instruct1.cit.cornell.edu/courses/ee476/Serialcom/http://www.computer-engineering.org/ps2protocol/http://www.computer-engineering.org/ps2protocol/http://www.computer-engineering.org/ps2protocol/http://instruct1.cit.cornell.edu/courses/ee476/Serialcom/

Customizable Virtual Keyboard

Documents

Transcript of Customizable Virtual Keyboard