Development of living body information and behavior ... · lying in bed and measures his...
Transcript of Development of living body information and behavior ... · lying in bed and measures his...
15
Journal of Engineering Education ResearchVol. 17, No. 4, pp. 15~20, July 2014
Development of living body information and behavior monitoring system for nursing personAi Ichiki*,† ․Hidetoshi Sakamoto* ․ Yoshifumi Ohbuchi**Kumamoto University
ABSTRACTThe non-contact easy detecting system of nursing person's body vital information and their behaviors monitoring system are
developed, which consist of “Kinect” sensor and thermography camera. The “Kinect” sensor can catch the body contour and the body moving behavior, and output their imaging data realtime. The thermography camera can detect respiration state and body temperature, etc. In this study, the practicability of this system was verified.
Keywords: Engineering Education
I. Introduction1)
The items of the respiratory state, the body temperature
and the behaviors for the nursing are very important
observation items. Especially, the state of respiratory
monitoring is an important item which is indispensable in
the detection of the “Sudden Infant Death Syndrome”(SIDS)
and the apnea syndrome. Polysomnograph(PSG) is widely
used in determining respiratory states. However, the largest
shortcoming of PSG is that it is expensive and its low
tolerance for the nursing patients and infants by relatively
high invasiveness of the PSG .
In this study, the easy monitoring system of non-contact
respiratory state and body temperature for early detecting
the SIDS and the apnea syndrome was proposed .
The validation for practical use as follows was carried out.
1) Verification of the posture detecting performance of
“Kinect” sensor.
2) Automatically detecting the state of respiratory by
thermography camera.
3) Development and verification of the new monitoring
system with “Kinect” sensor.
Received 21 June, 2014; Revised 21 June, 2014Accepted 30 July, 2014† Corresponding Author: [email protected]
II. Non-contact monitoring system of state of
respiratory state
The monitoring system outline of non-contact respiration
is shown in Fig 1. The initial system consists of the
infrared camera, CCD camera, room temperature measurement
unit and two personal computers. The CCD camera
detects a patient posture and his face contour, who is
lying in bed. The camera'a informations are used for the
thermography camera control. The thermography camera
measures his respiratory and his body temperature.
Fig. 2 shows an example of the tracking image of the
face detected with CCD camera. Recognizing his head area
by CCD camera, the face area is automatically displayed by
image processing. Next, the face area is extracted from
head area by skin color image matching. The mouth area
is initially set up manually. The face position is decided
Fig. 1 The body vital information monitoring system
Ai Ichiki․ Hidetoshi Sakamoto․ Yoshifumi Ohbuchi
공학교육연구 제17권 제4호, 201416
Fig. 2 The example of Posture tracking system by CCD camera
on position of mouth area by the face area's pattern
matching.
Fig. 3 shows the picture of image processing of
infrared image. The infrared camera detects the thermal
information around the face. By image processing of these
thermal information, the state of respiratory and the body
temperature can be obtained. The head region and face
area are detected by binary pictures of the face in infrared
camera. In setting up the mouth area in the face, the
face digital image obtained from CCD camera was used.
The state of respiratory is detected by using temperatures
fluctuate in the current of air by breath around the mouth.
However, this CCD camera has some problems about
the face recognition and the posture detection. So, we
replaced this camera with “Kinect” sensor for solving these
problems. Because this sensor can recognize the shape
and depth of the head, the image of the head can be
quickly monitored. The “Kinect” sensor is shown in Fig. 4.
Fig. 3 The picture of image processing by infrared camera
Fig. 4 “Kinect” sensor
III. Validation of the monitoring system
1. Verification of the posture detecting performance
of “Kinect” sensor
a. Detection ability of posture by “Kinect” sensor.
In this study, CCD camera was replaced with “Kinect”
sensor, and the verification about two items, that is, the
face recognition and the posture detection, were carried
out. Table 1 shows the comparison between CCD camera
and “Kinect” sensor. The front face tracking image is
shown in Fig. 5 By using “Kinect” sensor, the following
three items were improved.
• Auto detection of mouth area.
• Skip of skin color image process.
• Wide traceability of face rotation.
Table 1 Comparison between CCD camera and “Kinect” sensor
CCD camera "Kinect" sensor
Face recognitionSetting skin color
extractionInfrared distance sensor
Posture detectionRelative position in
mouth area to face area
The angle of face
rotation
Fig. 5 The front face tracking image by "Kinect" sensor
Development of living body information and behavior monitoring system for nursing person
Journal of Engineering Education Research, 17(4), 2014 17
Table 2 Face recognition angle
Distance from sensor and face/
Rotation velocity60cm 70cm 80cm 90cm 100cm 110cm 120cm
45deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚
90deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚
180deg/s 45~50˚ 45~50˚ 50~55˚ 50˚ 50˚ 50˚ 45~50˚
b. Evaluation of posture recognition ability by “Kinect”
sensor
We evaluated relationship of distance from “Kinect”
sensor to his face. The distance from "Kinect" sensor to
the face was changed, and the face rotation speed and
the face recognition angle were examined. Table 2
shows the results. From this table, the recognition angle
is independent of the distance from sensor and face.
2. Evaluation of automatically respiratory state
detecting by infrared camera.
a. Automatic setting of threshold by infrared camera.
• Threshold 1(two-valued of head area)
The histogram of two-valued image shows the
bimodality. The two-valued image separate the face area
and the background clearly. We calculated of threshold of
the valley part in histogram with mode method. Fig. 6 shows
an image processing as an example of thermography
camera. Fig. 6 (a) and (b) show the origin picture and the
picture after image processing respectively.
• Threshold 3( Breath detection)
The value of threshold 3 fluctuates widely by room
temperature and body temperature. We tried every threshold
value from 100 to 200. When the breath detection flag
appeared, we decided the value of threshold 3 and
determined as threshold. Fig. 7 shows an example of
image processing in infrared camera.
b. Validation of threshold 1 by infrared camera.
Here, when the body temperature change, the variance
of the threshold 1 was examined. In threshold 1
obtained by above trial, the face recognition ratio is
100% when the body temperatures are from 30 to 37
degree Celsius.
(a) The face origin picture
(b) The picture after face image processing
Fig. 6 Image processing example of thermo camera
Fig. 7 Nasal breathing
Ai Ichiki․ Hidetoshi Sakamoto․ Yoshifumi Ohbuchi
공학교육연구 제17권 제4호, 201418
(a) RGB image
(b) Depth image
(c) Infrared image
Fig. 8 Image processing example of “Kinect” sensor
IV. Development of the new monitoring system
by using “Kinect” sensor
In this research, CCD camera was replaced with “Kinect”
sensor (4.1.1) for the system improvement because of
there were some problems in the face recognition and
the posture detection. In the result, using “Kinect” sensor,
it was found that the face recognition, posture and
respiration are non-contact measurable by non-contact .
So, the new monitoring system by using only “Kinect” sensor
was developed(System 2) and compared with conventional
monitoring system (System1).
Fig. 8 shows image processing of “Kinect” sensor. Fig. 8(a)
is RGB image, Fig. 8(b) is Depth image and Fig. 8(c) is
Infrared image, respectively.
1. The monitoring system of “Kinect” sensor
“Kinect” sensor detects the patient's posture who is
lying in bed and measures his respiratory state with RGB
camera and depth sensor. In the posture measurement,
the multi-“Kinect” sensors are available. This system
(a) Posture measurement
(b) Respiration measurement
Fig. 9 The body vital information monitoring system (System2)
use two sensor and one is set at foot side and the other
is head side. Fig. 9 shows the body vital information
monitoring system (System 2). Fig. 9(a) is posture measurement,
Fig. 9(b) is respiration measuring, respectively.
2. Development of the monitoring system with
“Kinect” sensor(System2)
The posture measurement system tracks the body
motion and saves the image when the body moves.
Respiration measuring system detects the breathing state
and records the breathing count every minute.
a. Development of the posture measurement system
The posture is obtained by measuring the shortest
distance from “Kinect” sensor to nursing person. This
detection use the distance measuring function of “Kinect”
sensor. Fig. 10 shows the image processing of the
posture measurement system. The RGB camera of “Kinect”
sensor measures the body from foot side and head side.
The light blue point of the depth image is the minimum
Fig. 10 The image processing of the posture measurement system
Development of living body information and behavior monitoring system for nursing person
Journal of Engineering Education Research, 17(4), 2014 19
Fig. 11 The night measuring with the infrared camera
Fig. 12 The recoded data
depth and detects his posture. In the measurement at
night, RGB camera changes into the infrared camera
equipped with “Kinect” sensor. The night measuring
image with the infrared camera is shown in Fig. 11.
Also, this system records the RGB processing image
when the body moves. RGB image recording is decided
by the movement of the minimum depth. The recording
data is recorded the current data and time. Fig. 12 shows
the recorded data.
b. Development of the respiration measuring system
We developed the respiration measuring system by
using RGB camera and depth camera of “Kinect” sensor.
The image processing is shown in Fig. 13. The breathing
state is detected by movement of depth change of the
chest. Fig. 14 shows the flowchart of the breathing detection.
In the measurement at night, RGB camera changes into
the infrared camera. In the night measuring, RGB camera
is switched the infrared camera. The night measuring
with the infrared camera is shown in Fig. 15. Also, the
system records the breathing count every minute. Fig. 16
shows the logging data of the starting measurement time
and the breathing count.
Fig. 13 The image processing
Fig. 14 The flowchart of the breathing detection
Fig. 15 The night measuring with the infrared thermo camera
Fig. 16 The logging data of the breathing count
Ai Ichiki․ Hidetoshi Sakamoto․ Yoshifumi Ohbuchi
공학교육연구 제17권 제4호, 201420
V. The comparison of system 1 with system 2
In the posture measurement of the system 1, the traceable
limit of face rotation is 45~55 degree. In the system 2,
the traceable limit of face and body rotation are 90 degree.
In the respiration measuring system, we evaluated the
precision of the breathing state. The breathing data of system
2 corresponded to the standard breathing. The accuracy of
measuring the breathing improved by using “Kinect” sensor.
VI. Conclusions
<System 1>
• Using “Kinect” sensor instead of CCD camera, the system
improvement was carried out for practical use as follows.
(a) Automatic detection of mouth area and face area.
(b) Skip of the skin color detection process .
(c) Improvement of traceable limit of face rotation.
• By calculating threshold 1 and 3 automatically, the setting
time for measurement has been greatly shortened.
<System 2>
• “Kinect” sensor is an effective sensor to the posture
measurement and the respiration measurement.
• The performance of the posture measurement and the
respiration measuring is improved by replacing the
CCD camera into “Kinect” sensor.
• By combining the system 1 and system 2 organically, the
higher precise non-contact monitoring system is obtained.
References
1. Nobuhiro YOSHITAKE, An Implementation of Posture
Detection Functions to Inpatient Monitoring Systems using
Kinect, IPSJ SIG Technical Report(2013), 1-8
2. Yuhki TAKAHASHI, Development of System for Prevention
of Midnight Prowl Using Kinect, 2013 The Institute of
Industrial Applications Engineers Japan(2013), 42-43
3. Mariko AKIMOTO, A sheet-type device for home-monitoring
sleep apneas in children, Sleep and Biological Rhythms
(2011), 103-111
4. Tomohito HAYASHI, Study of Non-Restrictive Sleep Monitor
With Air-Matt Sensor, The Japan Society Mechanical Engineers
(2002), 71-74
5. Shunji HYUGA,”Let’s make the Kinect for Windows
application! ”(2012)
6. Tomoaki UEDA, The sensing world changing by “Kinect”
sensor, http://www.neo-tech-lab.co.uk/arsensing/
Ai Ichiki
Engineer of TOSHIBA Cooperation LTd.
Received BS (2012), MS (2014) in Mechanical System
Engineering from Kumamoto University, Japan. Her work
experiences are Product Engineer of TOSIBA Coopration LTd.
(2014), Her current research focuses on the production
technology of turbine for power electronics.
Phone: +81963423735
Fax: +81963423729
E-mail: [email protected]
Hidetoshi Sakamoto
Professor, Doctor of Engineering, at the Mechanical System
Division, Graduate School of Science and Technology, Kumamoto
University, Japan. He is received Master degree of Mechanical
Engineering by Kumamoto University, Japan in 1977, and got
his doctor's degree of Engineering from Kyushu University, Japan
in 1991. He was the Kyushu Branch Head of The Society of
Materials Science, Japan, 2006-2008 and now the Director of Infrared Thermography
Committee of The Society of Experimental Mechanics, Japan. He is also a member
of WIT international Science committee of “Computational Mechanics and Experimental
Methods”, “High performance Structures and Materials”, “Contact and Surface
Treatments”, and an editorial board member of WIT International Journal of Modeling
and Simulation He is the director of International conference on Far East Fracture
of Strength. The field of his research includes Solid Mechanics, Computer mechanics,
Sheet metal forming, High-speed fracture and deformation analysis, Biomaterial materials
strength evaluation and Engineering educational support program development, etc.
Phone: +81-96-342-3735
Fax: +81-96-342-3729
E-mail: [email protected]
Yoshifumi Ohbuchi
Associate Professor, Doctor of Engineering, Creative Engineering
and Design Education Center Faculty of Engineering, Kumamoto
University, Japan. He is received Master degree of Mechanical
Engineering by Kumamoto University, Japan in 1985, and got
his doctor's degree of Engineering from Tokyo Institute of
Technology, Japan in 2002. He was a member of Kumamoto
University from 1985 to 1993. He was Visiting Researcher of Tokyo Institute of Technology
from 1993 to 1994. He was Associate Professor of Fukuoka Institute of Technology,
Japan from 2003 to 2005. Since 2005, he is a Creative Engineering & Design Education
Center Faculty of Engineering, Kumamoto University, Japan. Interesting Research Area
are Engineering Education, Creative Engineering and Design, Succession Methods of
Traditional Craftsmanship and Skill, Metal cutting and grinding simulation.
Phone: +81-96-342-3732
Fax: +81-96-342-3729
E-mail: [email protected]