PIVOT VECTOR SPACE APPROACH FOR AUDIO - VIDEO

GUIDED BY : Ms NAGASHREE K T (Lecturer Of CSE/ISE)

PRESENTED BY : VINAY N (1CC08CS058)

CONTENTS INTRODUCTION AESTHETIC ASPECTS VIDEO AESTHETIC FEATURES AUDIO AESTHETIC FEATURES PIVOT REPRESENTATION MEDIA SEQUENCE MATCHING ADVANTAGES APPLICATIONS CONCLUSION

INTRODUCTION Audio-Video mixing is an important aspect of cinematography. An audio-mixing artist usually adds the musical accompaniment to video. Employing such artists is expensive & not feasible for a Home video presentation. Our automatic audio-video mixing technique is suited for Home videos. It uses a Pivot Vector space mapping method that matches video shots with music segments based on aesthetic cinematographic heuristics.

INTRODUCTION cont.. The PIVOT VECTOR SPACE APPROACH in audio mixing is a novel technique that automatically picks the best audio clip to mix with the given video shot. This technique uses pivot vector space mixing frame work & High level perceptual descriptors of audio & video characteristics.

AESTHETIC ASPECTS Movies comprise images ; graphic traces ; recorded speech, music, and noises; and sound effects. The different roles of music in movies can be categorized into:Setting the scene (create atmosphere of time and place), Adding emotional meaning , Serving as a background filler, Creating continuity across shots or scenes, and Emphasizing climaxes.

The table shows Asthetic Features that correspond in video & music

VIDEO AESTHETIC FEATURES a set of attributed features (such as color and motion) required to describe videos. This consists of features which required to describe videos. Light falloff brightness contrast between the light and shadow sides of an object Color features Motion vectors To measure the video segments motion intensity

VIDEO AESTHETIC FEATURES Cont color features extracted from a video shot consists of four features:1) Saturation : -the dominance of hue in the color 2) Hue : - a gradation or variety of a color 3) Brightness 4) Energy

AESTHETIC ATTRIBUTED FEATURE FORMATION we obtained the mean and standard deviation for estimating the confidence level of the Video & audio attributed features for any test shot.

AUDIO AESTHETIC FEATURES LOW-LEVEL FEATURES 1. Spectral centroid (brightness) : measure of a sounds brightness 2. Zerocrossing : measure of the frequency content of the signal 3. Volume (loudness) :

PERCEPTUAL FEATURES EXTRACTION Dynamics the volume of musical sound related to the musics loudness or softness Tempo features the music flow unique and differentiates it from other types of audio signal is temporal organization (beat rate). Perceptual pitch feature it is an important role in human hearing, and the auditory system apparently assigns a pitch to anything that comes to its attention.

PIVOT REPRESENTATION A vector space P acts as a pivot between the audio and video representation. Independent of any media. This space is defined with some aesthetic features in which music M and videos V are mapped. It consists of 2 types 1. media representation and 2. Pivot space mapping

MEDIA REPRESENTATION OF BASE VECTOR SPACES V AND M we consider how to represent video and audio clips into their aesthetic spaces V or M. In the two spaces, a dimension corresponds to an attributed feature, It includes brightness_high, brightness_low, and so on. One video shot is associated with one vector in the V space. Obtaining the values for each dimension resembles handling fuzzy linguistic variables,

MEDIA REPRESENTATION Cont The aesthetic feature playing the role of a linguistic variable and the attribute descriptor acting as a linguistic value which is represented using diagram. The X-axis refers to the actual computed feature value and the Y-axis simultaneously indicates the aesthetic label and the confidence value.

PIVOT SPACE MAPPING The mapping going from video-attributed aesthetic feature space to the pivot-attributed aesthetic feature space. Cause rotation or projection The mapping is incremental. The extraction of new video features, the modification of the transformation matrix Tv preserve all the existing mapping of music parts. We directly extrapolated the transformation matrices Tv and Tm form

PIVOT VECTOR REPRESENTATIONThe Figure shows the mapping process

MEDIA SEQUENCE MATCHINGIn the below figure shows that a) Matching between the video L02_44 & the music T05_1 b) Sample frame the video

Audio Video mixing results The fig shows the mixing of video shots from L02_44 to L02_51 The upper line presents the video shots sequence along the time axis. The lower line shows the musical audio tracks assocated with the shots according to the timeline.

ADVANTAGES Before the development of the PIVOT VECTOR SPACE APPROACH, audio-video mixing process can be carried out only by professional mixing artists. The Pivot vector space approach enables all the home video users and amateur video enthusiasts to give a professional look and feel to their videos. This technique also eliminates the need for professional mixing artists, thereby significantly reducing the cost, time and labour involved.

APPLICATIONS a large amount of home video footage is being produced due to products such as Digital video camcorders, Handicams etc. Hence, this technique will be of great use to all the amateur video enthusiasts and home video users.

CONCLUSIONS This is a technique that all amateur and home video artists can use in the creation of video footage that gives a professional look and feel. Since it is fully automatic, the user need not worry about his aesthetic capabilities.

THANK YOU