Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko...
-
Upload
spencer-hodges -
Category
Documents
-
view
216 -
download
4
Transcript of Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko...
![Page 1: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/1.jpg)
Ubiquitous Home: Retrieval of Experiences in a Home
Environment
Gamhewage C. DE SILVA
Toshihiko YAMASAKI
Kiyoharu AIZAWA
![Page 2: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/2.jpg)
Outline
• Introduction• Ubiquitous Home
– Sensors and Data Acquisition– Data Collection
• Retrieval– Footstep segmentation, Video and Audio
Handover– Key frame Extraction, Audio Segmentation
• User Interaction• User Study• Discussion• Future Work
![Page 3: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/3.jpg)
Introduction
![Page 4: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/4.jpg)
Introduction
• Automated capture of experience taking place at home is interesting.– Ex. first footstep of a child– Something is so important that people
have a strong desire to include themselves in the experience, rather than carry a camera and shoot photos.
![Page 5: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/5.jpg)
Introduction
• Capture and retrieval of experience in a home like environment is extremely difficult.– Large number of cameras and microphones– Continuous recording of data result in a very
large amount of data– Level of privacy
• Most difficult– Retrieval and summarization of captured data– Queries for retrieval could be at vary different
levels of complexity
![Page 6: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/6.jpg)
Introduction
• Multimedia retrieval for ubiquitous environments based solely in content analysis is neither efficient nor accurate– Make use of supplementary data from
other sensors for easier retrievalex. Proximity sensor, domain knowledge
![Page 7: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/7.jpg)
Introduction
• The research combines two main areas:– Ubiquitous Environment– Multimedia Retrieval
![Page 8: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/8.jpg)
Ubiquitous Environment
• Providing services to the people in the environment by detecting and recognizing their actions.
• Storing and retrieval of media, in different levels from photos to experiences
![Page 9: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/9.jpg)
Multimedia Retrieval
• Common approach is content analysis
• The use of context data where available can improve the performance greatly
![Page 10: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/10.jpg)
Main work on this article
• Capturing and retrieval of personal experiences in a ubiquitous environment that simulates a home.
• Create electronic chronicle for capturing video using interactive queries
• Main data: Video and Audio– Context data from pressure based floor
sensors to achieve fast and effective retrieval and summarization of video and audio data.
– Audio analysis and segmentation are used to complement context based retrieval.
![Page 11: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/11.jpg)
Ubiquitous Home
![Page 12: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/12.jpg)
Ubiquitous Home
• Sensors and Data Acquision
• Data Collection
![Page 13: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/13.jpg)
Sensors and Data Acquisition
• Layout of ubiquitous home.
![Page 14: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/14.jpg)
Sensors and Data Acquisition
• Images are recorded at the rate of five frames per second and stored in JPEG file format.
• Audio is sampled at 44.1kHz from each microphone and record into audio clip in mp3 file format and the duration is 1 minute.
• The floor sensors are point-based pressure sensors spaced by 180mm in a rectangular grid. The sample rate is 6Hz.– Start state=0, pressure over a threshold
state=1
![Page 15: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/15.jpg)
Data Collection
• Students’ experiment– Acquiring training data for actions and
events– Audio data are not available during the
experiment• Real-life experiment
– No manual monitoring of video was performed during the experiment
• The processing and analysis were performed offline
![Page 16: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/16.jpg)
Retrieval
![Page 17: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/17.jpg)
Retrieval
• Footstep Segmentation
• Video Handover
• Audio Handover
• Key Frame Extraction
• Audio Segmentation for Retrieval
![Page 18: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/18.jpg)
Retrieval
• Only a few data sources will convey useful information at any given time.
• Automatically select sources that will convey the most amount of information based on context data.
• Only the selected sources will be queried to retrieve data and these data will be analyzed further for retrieval.
![Page 19: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/19.jpg)
Retrieval
![Page 20: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/20.jpg)
Footstep Segmentation
• Noise– When there are footsteps on adjacent
sensors (very small duration)
– Relatively small weight such as a leg of a stool is placed in a sensor. (periodically)
• Kohonen Self Organizing Maps (SOM)
•
![Page 21: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/21.jpg)
Footstep Segmentation
• 3-stage Agglomerative Hierarchical Clustering (AHC) algorithm is used to segment sensor activations into footstep sequences of different persons
![Page 22: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/22.jpg)
Agglomerative Hierarchical Clustering algorithm
• First stage– Combine to form single footsteps– Distance function for clustering is
based on connectedness and overlap of duration
![Page 23: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/23.jpg)
Agglomerative Hierarchical Clustering algorithm
• Second stage– Combine to form path sequences
based on physiological constraints– Ex. Range of distance between steps,
overlap of duration in two steps, constraints on direction change
![Page 24: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/24.jpg)
Agglomerative Hierarchical Clustering algorithm
• Third stage– Compensate for the frgmentation of
individual path due to the absence of sensors in some areas
– Starting and ending timestamp, locations of the doors and furniture and information about places where floor sensors are not installed
![Page 25: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/25.jpg)
Agglomerative Hierarchical Clustering algorithm
![Page 26: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/26.jpg)
Footstep Segmentation
• Errors– Some paths are still fragmented after
clustering in the third stage– There are some cases of swapping in
paths between two persons when they walk close to each other
![Page 27: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/27.jpg)
Video Handover
• Select cameras in a way that a “good” video sequence can be constructed.
• Position-based handover– Based on simple view model, where the
viewable region for each camera is specified in terms of floor sensor coordinates.
![Page 28: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/28.jpg)
Position-based handover
• Create a video sequence that has the minimum possible number of shots.
• If the person can be seen from the previous camera, then that camera is selected.
• Otherwise, the viewable regions for the cameras are examined in a predetermined order and the first match is selected.
![Page 29: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/29.jpg)
Position-based handover(1) The change of color of the arrow indicates how the camera changes with the position of the person.
(2) It is possible to acquire a frontal view due to the positioning and orientation of cameras.
![Page 30: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/30.jpg)
Audio Handover
• Dub the video sequences
• Not necessary to use all of them since a microphone can cover a larger region compared to a camera
![Page 31: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/31.jpg)
Audio Handover
• Each camera is associated with one microphone for audio retrieval.
• Camera installed in a room– From the microphone that is located in the
center of that room
• Camera installed in the corridor– From the microphone that is closet to the
center of the region seen by that camera is selected
![Page 32: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/32.jpg)
Audio Handover(1) Minimize transitions between microphones
(2) Uniform amplitude level
![Page 33: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/33.jpg)
Video & Audio Handover
![Page 34: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/34.jpg)
Key Frame Extraction
• The video sequence constructed using video handover has be sample to extract key frames.
• For complete and compact– Minimize the number of redundant key
frames while ensuring that important key frames are not missed
![Page 35: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/35.jpg)
Key Frame Extraction
T is a constant time interval.
![Page 36: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/36.jpg)
Key Frame Extraction
• Adaptive spatio-temporal sampling algorithm– The time interval for sampling the next
key frame is reduced with footstep, thereby sampling more key frames when there are more footsteps
![Page 37: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/37.jpg)
Key Frame Extraction
• Evaluation– The subjects extracted key frames form
four video clips according to their own choice.
– Create average key frame sets which are used as ground truth for evaluation
– They voted for the key frame set that summarized the sequence best.
![Page 38: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/38.jpg)
Key Frame Extraction
![Page 39: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/39.jpg)
Key Frame Extraction
![Page 40: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/40.jpg)
Audio Segmentation for Retrieval
• The floor sensors are unable to capture data when people are not treading on a floor area with sensors.
• They are not activated if the pressure on the sensors is not sufficiently large.
• Audio-based retrieval can also be conducted independently to support various types of queries.
![Page 41: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/41.jpg)
Audio Segmentation for Retrieval
• The amount of audio to be processed is quite large.
• Tread-off– Utilizing the redundancy to improve the
accuracy of retrieval– Minimizing processing by removing
redundancy
![Page 42: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/42.jpg)
Audio Segmentation for Retrieval
• Eliminate audio corresponding to silence.– Compare the RMS power of the audio
signal against a threshold value.– RMS(Root Mean Square) is a statistical
measure of the magnitude of a varying quantity.
![Page 43: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/43.jpg)
Audio Segmentation for Retrieval
• Audio clips with one hour were extracted from different times of day.
• These clips were partitioned into frames having 300 samples.
• Adjacent frames had a 50% overlap.• The RMS value of each frame is
calculated and recorded, and the statistics obtained for each clip.
![Page 44: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/44.jpg)
Audio Segmentation for Retrieval
• Probabilistic distribution of the RMS values for different audio clips were not significantly different.
• Combine to a single probabilistic model for silence and noise
![Page 45: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/45.jpg)
Audio Segmentation for Retrieval
• The threshold for each microphone is estimated by analyzing audio data for silence and noise for that microphone.
• Threshold value was selected to be at 99% level of confidence according to this distribution.– Below 100% because false negatives
(sound misclassified as silence) are more costly than false positives(silence misclassified as sound).
![Page 46: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/46.jpg)
Silence Elimination
• First stage – based on individual microphone– If RMS value of each frame is large
than the threshold, the frame is considered to contain sound.
• Sets of contiguous frames with duration less than 0.1s are removed.
• Sets of contiguous frames with duration less than 0.5s apart are combined together to form single segment.
![Page 47: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/47.jpg)
Silence Elimination
• Second stage – based on multiple microphones in close proximity to reduce false positives.
• For each microphone– B(n) : Binary sound segment function– C(n) : Cumulative sound segment
function
![Page 48: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/48.jpg)
Silence Elimination
• Binary sound segment function– B(n) = 1 if there is sound in the n-th
second of audio stream– B(n) = 0 otherwise– For the set of microphones in the same
room
![Page 49: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/49.jpg)
Silence Elimination
• Noise– random
• It is less likely that noise in sound segments from different microphones occur simultaneously.
– Small duration
![Page 50: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/50.jpg)
Silence Elimination
• Voting algorithm to determine the sound segment function - S(n)– S(n) = 1 if C(n) convolution M(n) >=
ceil(k/2)– S(n) = 0 otherwise– M(n) = [111]– K= number of microphones installed in
the location
![Page 51: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/51.jpg)
Audio Segmentation for Retrieval
• Video is retrieved from all cameras in the room for each sound segment.
• The video created by handover is extended to include the time during which sounds were present before the start of the footstep sequence
![Page 52: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/52.jpg)
User Interaction
![Page 53: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/53.jpg)
User Interaction
![Page 54: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/54.jpg)
User Study-Real-Life Experiment
![Page 55: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/55.jpg)
User Study
• 1st requirement study
• 2nd – Given a demonstration on how to use
the system– Summit their own queries– Select video clips that they would like
to keep
• 3rd feedback about the system
![Page 56: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/56.jpg)
Discussion
![Page 57: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/57.jpg)
Discussion
• Issues Related to Capture
• Algorithm for Retrieval
• Real-Life Experiment
![Page 58: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/58.jpg)
Issues Related to Capture
• Continuous capture– The research was carried out at a
different location from the home-like environment.
– Experiments with families are quite difficult to arrange and the cost of losing important data due to algorithms with sufficient accuracy is quite high.
– Problem : large amount of disk space
![Page 59: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/59.jpg)
Issues Related to Capture
• Some of microphones seem to be redundant, given their range and directivity.– Save disk space
• Floor sensors are more expensive and difficult to maintain– Movement of furniture
![Page 60: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/60.jpg)
Algorithm for Retrieval
• The accuracy of footstep segmentation deteriorates when the number of persons in the house is large and with the movement of furniture
• Video handover can be improved by considering occlusion by other persons when selecting the camera.
• For audio handover, smoother transitions are possible by looking for silence near the point of microphone change.
![Page 61: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/61.jpg)
Algorithm for Retrieval
• Key frame extraction– Human-human and human-object
interaction
• Audio-based video retrieval will retrieved false result if the house is located at a place where loud sounds can enter the house from outside
![Page 62: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/62.jpg)
Real-Life Experiment
• The subjects in students’ experiments were independent in their actions.
• The behavior of the family in the real-life experiment was in the form of a group.– Accuracy of footstep segmentation is
decreased.
![Page 63: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/63.jpg)
Future Work
![Page 64: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/64.jpg)
Future Work
• Further clustering of floor sensor data and classification of audio data.
• Face detection
![Page 65: Ubiquitous Home: Retrieval of Experiences in a Home Environment Gamhewage C. DE SILVA Toshihiko YAMASAKI Kiyoharu AIZAWA.](https://reader035.fdocuments.in/reader035/viewer/2022070400/56649f125503460f94c251c3/html5/thumbnails/65.jpg)
Thank you