A Study on the Tele-medicine Robot System with
Face to Face Interaction
Dae Seob Shin*★
Abstract
Consultation with the patient and doctor is very important in the examination. However, if the consultation
cannot be done directly, such as corona virus, it is difficult for the doctor to determine the patient’s condition
more accurately. Recently, an image counseling system has been developed based on the Internet, but in the case
of heart disease, remote medical counseling cannot be performed because it is not possible to stethoscope the
heart sounds remotely. In order to solve this problem, it is necessary to develop an interactive mobile robot
capable of remote medical consultation, and a doctor and a patient should be able to set a planting sound during
consultation and transmit it in real time. In this paper, we developed a robot that can remotely control a medical
counseling robot to move to a hospital room where patients are hospitalized, and to consult a patient in the room
remotely from a doctor’s office. A remote medical imaging stethoscope system for real-time heart sound
transmission is presented. The proposed system is a kind of P2P communication that transmits video information,
audio information, and control signal independently through webRTC platform, so that there is no data loss.
Consults and sees doctors in real time and finds it more effective than traditional methods for patient security.
The system implemented in this paper will be able to perform remote medical care in the place where the
spread of diseases between humans like the recent corona 19 as well as the remote medical care of heart disease
patients in the future.
Key words:Telemedicine, Face-to-face, WebRTC, Stethoscope, Mobile robot
* Dept. of Electronic Information Communication Division, Shin Ansan University
★ Corresponding author
E-mail:[email protected], Tel:+82-2-2679-8556
※ Acknowledgment
Manuscript received Mar. 10, 2020; revised Mar. 19, 2020; accepted Mar. 24, 2020.
This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License
(http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction
in any medium, provided the original work is properly cited.
Ⅰ. Introduction
Recently, with the emergence of smartphones
and tablet PCs in line with 5G technology, as the
information communication technology has been
developed and advanced, the interface technology
between human and computer has been developed,
and the interface technology between human and
computer has been diversified in various forms.
As 5G technology develops, u-health technology
is attracting attention as it is being combined
with an increase in the elderly population. Advances
in communication also mean that there is a
foundation for providing health care anywhere,
anytime[1][2].
As communication technology improves, efforts
to create added value in telemedicine using video
communication have been attempted in various
fields. In recent years, telemedicine has been
revisited due to u-health. There is a need for a
service that can receive medical services anywhere
outside the hospital, measure biometric information,
ISSN:1226-7244 (Print)ISSN:2288-243X (Online) j.inst.Korean.electr.electron.eng.Vol.24,No.1,293~301,March 2020논문번호 20-01-40 http://dx.doi.org/10.7471/ikeee.2020.24.1.293
293
(293)
294 j.inst.Korean.electr.electron.eng.Vol.24,No.1,293~301,March 2020
and provide consultation with a doctor without
going to the hospital.
In particular, as the elderly population increases,
the demand for remote monitoring of patients
without going to the hospital is increasing. In
addition, when a virus that spreads between humans
has appeared recently, such as a corona virus,
there is a need for a system that enables a doctor
to perform a medical examination remotely without
directly meeting a patient.
Even in hospitals, doctors need a system that
enables them to remotely manage their patients
without visiting the patient’s room. Types of
telemedicine are shown in Fig. 1. It can be divided
into three as follows. (a) may be a medical
consultation between the medical institution’s
ward and the physician. (b) may provide consultation
between school facilities and physicians. (c) enables
telemedicine, such as homes and health care
facilities. This study is an effective robotic system
for conducting medical consultations with patients
and medical institutions such as (a) and (c).
Fig. 1. Telemedicine Service Classification and Structure.
Fig. 2. Appearance of Remote Video Robot.
In medical consultations, doctors need a minimum
of photos and a stethoscope to accurately determine
a patient’s condition. Observation of the trauma
of the patient is possible through a remote imaging
robot. However, patients with heart disease are
difficult to consult because they cannot remotely
listen to heart sounds. It should be possible to
measure the heart sounds during the consultation
between the doctor and the patient and transmit
them in real time. If a system capable of measuring
such sound can be transmitted in conjunction
with a remote video robot, effective telemedicine
will be possible.
In this paper, we propose an video robot system
that can be operated remotely to check the condition
of the patient. Remote control is possible not
only from PC but also from smart device, and it
is configured so that doctors can conveniently
control the image robot using various sensors
and multi-touch technology built in smartphone
or tablet PC as well as from fixed PC[3]. In
addition, we designed, implemented and implemented
a telemedicine video consultation system for
heart sound transmission over the Internet. For
video communication, we use WebRTC platform,
which is more effective than the existing P2P
communication, to transmit video information, audio
information, and data information separately[4].
Ⅱ. Related Work
1. Remote Video Robot System Configuration
The remote image robot system is designed
and manufactured to be composed of three types.
* Robot Mechanism Design
First of all, the design of the robot’s external
mechanical part were carried out. In general, the
robot was designed with a low cost and simple
structure so that it can be used from homes to
medical institutions without burden.
In addition, the robot has two wheels, a 12-inch
tablet PC monitor with a height of 120cm, and
(294)
A Study on the Tele-medicine Robot System with Face to Face Interaction 295
Fig. 5. Robot base manufacturing process.
the screen can be moved by Fan / Tilt. Fig. 3
shows the drawings of the remote imaging robot
mechanism and modeled in 3D. In Fig. 4, the
robots have a friendly feeling by covering various
types of character dolls on the remote video robot.
Fig. 3. designed remote video robot base.
Fig. 4. Appearance design on remote video robot.
* Robot base production
The base part of the robot was manufactured
to fit the designed robot. Two motors and ball
casters were used. Fig. 5 shows the manufacturing
process of the robot base.
We manufactured the robot base, developed the
robot drive unit, and installed the ultrasonic
sensor around the robot to move the robot safely.
We also developed a control algorithm to drive
two motors.
① Development of Autonomous Driving Control
Algorithm for Mobile Robot
- Intelligent control algorithm and control method
for remote control driving of Mobile Robot
driven by 2 axis DC motor of Differential
Driving type were developed.
- The sensor part of Mobile Robot adopts IR
sensor and ultrasonic sensor to enable real-time
collision avoidance while the robot is driving,
and the camera is mounted on the upper part
of the robot so that the image of the robot in
front of the robot can be remotely monitored
It was sent to the operator.
Fig. 6. Autonomous Driving and Collision Avoidance Structure
of Mobile Robot.
* Robot Monitor Structure
LCD monitor for video communication was
designed. The monitor was designed with the
fan / tilt moving structure and driven by two
servomotors.
(295)
296 j.inst.Korean.electr.electron.eng.Vol.24,No.1,293~301,March 2020
Fig. 7. Robot Monitor Fan / Tilt Design.
Fig. 8. Fan / Tilt using servo motor.
Fig. 9. Remote Video robot Appearance.
We designed and built the robot to the world
level to move the robot safely, and experimented
to confirm the range of motion.
(Main Function Spec) Unit World LevelDeveloped
Level
Speed control error cm/sec within ±10 ±10
Positioning errors cm within ±15 ±10
Ultrasonic sensor interface cm 5 2
Light sensor interface o o
Collision avoidance o o
Table 1. Acdeptable Goal
2. Electronic Stethoscope System
The remote image transmission robot was
designed to work with a stethoscope to check
not only the image but also the patient’s condition.
Auscultation is a diagnostic procedure performed
to detect heart failure and digestive status defects.
The doctor listens to the sound of the stethoscope
on the heart, lungs or intestine, the source of the
sound. Fig. 10 shows the structure of the
stethoscope.
Fig. 10. the structure of the stethoscope.
The stethoscope is transmitted to the ear
through the earpieces as the sound source
captured by the diaphragm moves up the tube.
This measured heart rate goes up the tube and
you hear only low notes due to the HPF
(Hight-Pass Filter). As a result, doctors cannot
hear the original sound of the patient’s measured
heart sound, and the sound of the low frequency
band is mixed with the heart sound every time.
Fig. 11. control structure of the electronic stethoscope.
Recently, the electronic stethoscope is widely
used. It was solved by filtering and amplification
(296)
A Study on the Tele-medicine Robot System with Face to Face Interaction 297
Fig. 14. Working Process of WebRTC Peer to Peer
Communication.
as shown in Fig. 11. And while traditional
stethoscopes only listen to doctors, electronic
stethoscopes have the advantage of being able to
store and record.
Fig. 12. Connect to the tablet of the electronic stethoscope.
The existing electronic stethoscope was connected
to the robot and transmitted in real time as an
video, and the receiver realized the medical
consultation at the receiving end to realize a
remote medical service requiring a stethoscope.
Fig. 12 shows the structure of an electronic
stethoscope connected to a tablet. This system is
installed on the video transmission robot manufactured
earlier and transmits the video and the audio
signal of the electronic stethoscope to the doctor.
3. Program development
It is important to obtain accurate patient’s
medical information in real time for the treatment
between the patient and the doctor with a remote
video robot. Hardware configuration is important
for accurate medical information, but software
configuration is most important. Existing video
communication is the structure that sends and
receives the control signal of the robot through
the TCP Socket while controlling by UDP Socket
communication. However, it is true that accurate
examination is difficult because of various
problems in communication. However, in this
study, we conducted experiments by securing
secure communication using WebRTC platform.
Fig. 13 shows the structure of WebRTC.
Fig. 13. Architecture of WebRTC.
Browsers and mobile applications use Audio
and Video RTC (real-time communication) using
WebRTC through a simple API. The WebRTC
component has been optimized to best suit this
purpose. WebRTC-based web applications provide
rich real-time multimedia capabilities (think video
chat) on the web without plug-ins, downloads, or
installations, and help build a robust WebRTC
platform that works across platforms in multiple
(297)
298 j.inst.Korean.electr.electron.eng.Vol.24,No.1,293~301,March 2020
web browsers. Fig. 14 shows a tank that transmits
video and audio signals with the WebRTC Peer
to Peer Communication procedure.
WebRTC Benefits:
• WebRTC is In-built in Firefox browser.
• Improved video and audio streaming.
• VP8 video codec and OPUS audio codec
provides much less data transmission without
packet loss.
Fig. 15 shows the process of running an Android
program using Eclipse.
Fig. 15. Android video control program development.
Fig. 16. Android tablet Control screen.
Fig. 17. Android smartphone screen.
Fig. 18. video transfer experiment after mounting on robot.
After completing the development of the remote
video robot, we made a character with a doll to
have a friendly relationship with the patient when
performing remote medical examination using a
real robot.
Fig. 19. Equipped the character on the developed guide
robot.
Ⅲ. Implementation and evaluation
In order to perform the experiment of the
implemented system, the experiment is composed
of the experiment of the robot control unit and
the hardware control experiment.
1. Control board and video transmission experiment
In order to test the robot controller, a program
written in C language was downloaded to the
ATmega128 board. After the Bluetooth pairing
was performed, the program was set up to 0×31,
0×32, 0×34 ... 0×38 and the servo was driven
(298)
A Study on the Tele-medicine Robot System with Face to Face Interaction 299
using data such as 0×40, 0×41, 0×42, 0×43. Then,
a motion control experiment was performed using
a remote control app.
Fig. 20. Experimental procedure of video transmission and
motor drive.
2. Evaluation item experiment of development
technology
For the evaluation of the developed product, the
experiment was carried out at the temperature:
(20 ± 2) ℃ and humidity: (56 ± 5)% R. H. In this
section, we will briefly summarize the results and
data of the experiment.
Table 2-1. Experiment item and evaluation method.
No. Test Item Evaluation mothod
1Video transmission
speed⦁Video transmission time
measurement.
2 Sound size capacity⦁Speaker output noise
measurement.⦁Measure 1m from the front.
3 Operating time⦁Check the remaining charge
after operation / operation when fully charged.
4 Max Speed⦁Maximum moving speed
measurement
5 Battery usage indicator⦁LED indication.⦁Display remaining battery
Check.
6 weight ⦁Weight measurement.
7 Robot height ⦁Robot height measurement.
8After sensor response
Stop speed
⦁Stop speed measurement after sensor response.⦁Stop after detecting a forward
object while moving
9Stethoscope Frequency
MeasurementStored Stethoscope
Measurements
Sample photo (front) Sample photo (rear)
Table 2-2. Evaluation results.
No. Test Items Target value Evaluation results
1Video transmission
speed20 Frame
Video call with WebRTC platform
2 Sound size capacity 60 dBA/m 70.8 dBA
3 Operating time 8H 10H
4 Max Speed 30 cm/s 40.62 cm/s
5Battery usage
indicatorLED display LED display
6 weight 38 kg 17.80 kg
7 Robot height 120 cm 132.6 cm
8After sensor response
Stop speed1 s 0.20 s
9Stethoscope Frequency
Measurement8000Hz 8000Hz
3-3. Experiment content
The experiment was carried out according to
the evaluation method for each item, and the
evaluation of the sound and speed of the robot
and the frequency of the stethoscope sound were
summarized.
3-3-1. Robot output sound measurement
experiment
Speaker maximum output noise was measured
during video call with tablet PC. The noise
measuring room (width×length×height) was 4.98
×3.1×3,42 m and the distance from the robot was
measured at 1 m from the front. The noise
measuring instrument was performed by NA-27
(RION) and there were 10 measuring functions,
which made it easy to measure. The experiment
was performed three times to find the average
value.
(299)
300 j.inst.Korean.electr.electron.eng.Vol.24,No.1,293~301,March 2020
[Measurement Data]
NO Room noise (dBA) Noise measurement (dBA)
1
41.4
70.1
2 69.5
3 71.4
Average value 70.8
[Evaluation results]
- Speaker maximum output noise result (69.5~
71.4) dBA.
Noise meter Noise measurement pictures
Fig. 21. Experimental process photo.
3-3-2. Stop speed experiment after sensor response
In order to improve the interaction between the
patient and the doctor through the robot, the
doctor responds quickly to the robot’s sensor
response to speed up the interaction and environment
recognition of the remote robot. The instrument
used was a stopwatch and a HS-6 (CASIO)
instrument.
[Measurement Data]
NO Reaction time (s) etc
1 0.22
2 0.18
3 0.17
Average value 0.20
[Evaluation results]
- Result of measurement of downtime after reaction
of object detection sensor (0.17~0.22) s.
Photos before the moveStill photo after detecting an
object while moving
Fig. 22. Experimental process photo.
3-3-3. Stethoscope storage experiment
Received the stethoscope sound transmitted from
the remote video robot and added the function to
store and play the stethoscope sound on the tablet
PC, and experimented through the storage and
playback screen on the screen. Fig. 23 shows a
screen for storing and playing stethoscope
sounds on the tablet screen.
Fig. 23. Experimental process photo.
Since the stethoscope sound must be transmitted
and stored, it is encoded to 8000Hz 8bit mono
type and it is configured to provide the function
of playing and recording the stethoscope sound
received by the doctor. The stethoscope sound
sent from the patient can be stored in the doctor’s
monitor and managed for video consultation
history. The following figure shows the playing
and stopping of the stored stethoscope. Through
experiments, we could hear and diagnose a
stethoscope sound through a remote imaging
robot. In addition, since the images are transmitted
using WebRTC, more than 20 frames of images
have been acquired for remote medical examination,
and it was confirmed through experiments that
the images were transmitted safely without any
breaks in the image system.
Ⅳ. Conclusions
In this study, we developed an video transmission
robot system that can move freely in the structure
proposed in the fixed PC for telemedicine service.
Even if the doctor and the patient are far away,
the video call is performed in real time as if they
are close, and the doctor can operate the robot
remotely to examine the patient’s condition in
various ways. For the configuration of the video
(300)
A Study on the Tele-medicine Robot System with Face to Face Interaction 301
system, real-time video transmission is implemented
using the H263 codec, which is a video communication.
Unlike the method of providing the login function
to the server agent by using the TCP socket, the
webRTC platform is configured to separate the
video signal, the audio signal, and the control
signal so that the video is not interrupted and is
transmitted stably at a transmission speed of 20
frames or more per second. It confirmed that it
became. In addition, since the electronic stethoscope
system is mounted on the medical robot, the
doctor will consult with the patient to check the
planting in real time and receive and examine the
patient.
Using the system implemented in this study, it
is possible to monitor and remotely refer to
elderly patients at home or patients discharged
after heart disease surgery. Since the minimum
sound quality required for accurate diagnosis by
medical staff should be guaranteed, future research
projects will compare the sound quality before
and after the transmission of the stethoscope
sound, and conduct continuous experiments and
analysis for the effectiveness of the stethoscope
image counseling system.
In addition, various researches will be needed,
such as interactions that can safely control remote
robots using various sensors and multi-touch
technology mounted on smartphones or tablet PCs,
and environmental awareness technology of remote
robots.
References
[1] T. Cohen, “Medical and Information technologies.
converge,” IEEE Engineering in Medicine and
Biology Magazine, Vol.23, No.3 pp.59-65, 2004.
DOI: 10.1109/MEMB.2004.1317983
[2] W. Yuan, D. H. Guan, S. Y. Lee, and H. J. Lee,
“Using Reputation System in Ubiquitous Healthcare,”
The Proc. of 9th IEEE Int’l Conference on
e-Health Networking, pp.182-186, 2007.
DOI: 10.1109/HEALTH.2007.381626
[3] E. Pacchierotti, H. I. Christensen and P. Jensfelt,
“Human-Robot Embodied Interaction in Hallway
Settings: a Pilot User Study,” IEEE International
Workshop on Robots and Human Interactive
Comm, pp.164-171, 2005.
DOI: 10.1109/ROMAN.2005.1513774
[4] S. J Baek, R. H. Lee, C. S. Yi, “Design and
Development of A Systemic Structure to Ensure
the Interoperability between the WebRTC-based
Video Conferencing Systems and Heterogeneous
Terminals,” KIISE Transactions on Computing
Practices, Vol.23, No.4, pp.238-243. 2017.
DOI: 10.5626/KTCP.2017.23.4.238
BIOGRAPHY
Dae Seob Shin (Member)
1996:BS degree in Electronics
Engineering, Howon University.
1998:MS degree in Electronics
Engineering, Inha University.
2014:Ph. D. degree in Electrical and
Biomedical Engineering, Hanyang
University
2019~Present:adjunct Professor, Shin Ansan University
<Research Interests>
Image processing, neural network, Adaptive control, Signal
Processing, Embedded Control, Rehabilitation robots.
(301)
Top Related