User-Guided Variable-Rate Time-Stretching Via Stiffness...
Transcript of User-Guided Variable-Rate Time-Stretching Via Stiffness...
![Page 1: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/1.jpg)
Nicholas J. Bryan, Jorge Herrera, and Ge Wang
Stanford University | CCRMA
User-Guided Variable-Rate Time-Stretching Via Stiffness Control
CCRMA DSP Seminar, November 6th 2012
![Page 2: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/2.jpg)
I Introduction
II Proposed Method III Numerical Optimization IV Time-Stretch Processor V Results VI Conclusions
Outline
![Page 3: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/3.jpg)
Introduction
• User control over variable-rate time-stretch processing • Stretch some regions more than others (e.g. stretchability, stiffness)
• Transient preservation, rhythmic warping, emphasis modification, etc.
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
![Page 4: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/4.jpg)
Demo - Rhythmic Warping
Original
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
1.5
2
Time (s)
Ampl
itude
s(t)k(t)a(t)
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
1.5
2
Ampl
itude
Time (s)
Stretched (2x)
No Stiffness / Stiffness
Swung + Stretched (2x)
![Page 5: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/5.jpg)
Demo - Emphasis Modification
0 0.5 1 1.5 2 2.5−1
0
1
2
3
4
5
Time (s)
Ampl
itude
I’m gonna make him an offer he can’t refuse
s(t)k(t)a(t)
original
warped
warped + stretched (slowed by 1.3x)
“I’m gonna make him an offer he can’t refuse.”
“I’m gonna maaake him an offer he caaaan’t refuse.”
![Page 6: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/6.jpg)
Motivation
• Automatic methods have no mechanism for user input • Direct manipulation of the stretch rate is hard!
2x
?
![Page 7: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/7.jpg)
Flexures and Pivots
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
[ProTools, Logic Pro, FL Studio, etc.]
00.2
0.4
0.6
0.8
11.2
1.4
−1
−0.50
0.51
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
[Nielson and Brandorff, 2002]
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
![Page 8: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/8.jpg)
Coupled Constraints
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
![Page 9: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/9.jpg)
I Introduction II Proposed Method III Numerical Optimization IV Time-Stretch Processor V Results VI Conclusions
Outline
![Page 10: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/10.jpg)
Proposed Method
1. User annotates stiffness (+ timing constraints)
2. Solve optimization problem to convert stiffness to stretch factor 3. Use pre-existing time-stretch processor to stretch sound
Stiffness Stretch Factor Processor
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
![Page 11: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/11.jpg)
Step I: Spring Chain
• Model audio as chain of springs
• Solve for equilibrium via Hooke’s Law
• Spring stiffness offers an intuitive measure (i.e. proportional)
Fi = �kixi
…
![Page 12: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/12.jpg)
Initial Formulation
find x
subject to f = 0
x
T1 = L
fi = ki+1xi+1 � kixi
L = final length
x = spring displacement
f = spring forces
k = spring sti↵ness
![Page 13: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/13.jpg)
• Violates intuition: no initial rest length
Problem
� = 2k1 = 100 k2 = 1 k2 = 1
k1 = 100
![Page 14: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/14.jpg)
Reformulation
find x
subject to f = 0
(x+ x0)T1 = L
x+ x0 � 0
find x
subject to f = 0
x
T1 = L
x0 = Initial Rest Length
![Page 15: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/15.jpg)
Reformulation
• Minimize the force disturbance from equilibrium (smoothly) • Minimize ≈ potential energy
minimize
x
||f ||2 + µ||x||2
subject to (x+ x0)T1 = L
x+ x0 � 0
µ = Penalty Weightx0 = Initial Rest Length
![Page 16: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/16.jpg)
Extensions
• Rhythmic warping, smoothing of user input, max stretching limits
• Example: Straight-to-Swing
minimize
x
||f ||2 + µ||x||2
subject to (x+ x0)T1 = L
x+ x0 � 0
(x
1+ x
10)
T1 =
23L/2
(x
2+ x
20)
T1 =
13L/2
(x
3+ x
30)
T1 =
23L/2
![Page 17: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/17.jpg)
Step II: Stiffness to Stretch Factor
• Given input and output lengths, compute stretch factor as simple ratio
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
1.5
2
2.5
3
Time (s)
Ampl
itude
s(t)k(t)a(t)
� = x+x0x0
![Page 18: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/18.jpg)
Step III: Time-Stretching
• Given optimal variable-rate stretch factor, process sound
• Phase Vocoder (PV)
• Pitch synchronous overlap add (PSOLA)
![Page 19: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/19.jpg)
Method Recap
• Annotation method for user-annotated time stretching
• Decouples timing constraints and stiffness • Solve optimization problem to get time-variable stretch-rate
![Page 20: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/20.jpg)
I Introduction II Proposed Method III Numerical Optimization IV Time-Stretch Processor V Results VI Conclusions
Outline
![Page 21: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/21.jpg)
• Solve the linearly constrained quadratic program
• Need iterative, numerical optimization method
• See CVX [Grant and Boyd, 2008] and [Boyd & Vandenberghe]
Numerical Computation I
minimize
x
||f ||2 + µ||x||2
subject to (x+ x0)T1 = L
x+ x0 � 0
minimize
x
(1/2)xTPx+ q
Tx+ r
subject to Ax = b
Cx d
![Page 22: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/22.jpg)
• Initialize x • Repeat Until Convergence { }
• General Descent Methods
• Gradient Descent
• Newton’s Method
• Newton’s Method w/Equality Constraints
• Interior-Point Barrier Method (+inequality constraints)
Descent-Based Methods
x
n+1 = x
n + ⌧
n�x
n
x
n+1 = x
n � ⌧
nrf(xn)
x
n+1 = x
n � ⌧
nr2f(xn)�1rf(xn)
x
n+1 = x
n + ⌧
n�x
n
r2f(x) A
T
A 0
� �x
w
�=
�rf(x)
0
�
![Page 23: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/23.jpg)
Descent-Based Methods Graphically I
x
n+1 = x
n � ⌧
nrf(xn)x
n+1 = x
n � ⌧
nr2f(xn)�1rf(xn)
Gradient Descent Newton’s Method
y = 2x2 � 2x+ 5
![Page 24: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/24.jpg)
Descent-Based Methods Graphically II
x
n+1 = x
n � ⌧
nrf(xn)x
n+1 = x
n � ⌧
nr2f(xn)�1rf(xn)
Gradient Descent Newton’s Method
y = x
TAx� x
TB + C
A =�
1 .5.5 1
�y = x
TAx� x
TB + C
x + 5�13
�T
(� 13 ,
53 ) (� 1
3 ,53 )
![Page 25: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/25.jpg)
Descent-Based Methods Graphically III Newton’s Method w/Equality Constraints
x
n+1 = x
n + ⌧
n�x
n
subject to
y = x
TAx� x
TB + C
A =�
1 .5.5 1
�y = x
TAx� x
TB + C
x + 5�13
�T
(�1, 1)
x1 = �x2
![Page 26: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/26.jpg)
Barrier Interior-Point (Newton) Method
• Move inequalities to main objective via log barrier
t > 0 t ! 1
minimize
x
(1/2)xTPx+ q
Tx+ r
subject to Ax = b
Cx d
minimize
x
(1/2)xTPx+ q
Tx+ r� (1/t)1T
log(�Cx+ d)
subject to Ax = b
![Page 27: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/27.jpg)
Solve Sequential Optimizations
......
t = 1minimize
x
(1/2)xTPx+ q
Tx+ r� (1/t)1T
log(�Cx+ d)
subject to Ax = b
t = 5minimize
x
(1/2)xTPx+ q
Tx+ r� (1/t)1T
log(�Cx+ d)
subject to Ax = b
t = 10 minimize
x
(1/2)xTPx+ q
Tx+ r� (1/t)1T
log(�Cx+ d)
subject to Ax = b
t ! 1minimize
x
(1/2)xTPx+ q
Tx+ r� (1/t)1T
log(�Cx+ d)
subject to Ax = b
![Page 28: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/28.jpg)
I Introduction II Proposed Method III Numerical Optimization IV Time-Stretch Processor V Results VI Conclusions
Outline
![Page 29: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/29.jpg)
Phase Vocoder
• Frequency domain method • Time-stretch • Pitch-shift
• General procedure • Filter Banks/STFT Analysis • Frequency domain processing • Synthesis
• Good for processing full-bandwidth audio signals
![Page 30: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/30.jpg)
Phase Vocoder Graphically
Figure from [Sethares 2007]
(a) Analysis (b) Processing (c) Synthesis
![Page 31: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/31.jpg)
Vocoder Time-Stretching
• Variable-rate time-stretching • Changing analysis hopsize, constant synthesis hopsize
• Linearly resample stretch-rate control signal as needed
• See websites for phase-vocoders in Matlab • [Dan Ellis 2002, Peter Moller-Nielsen 2002, William Sethares 2007]
![Page 32: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/32.jpg)
Synchronous Overlap and Add (SOLA)
1. Segment signal into overlapping blocks (fixed length/step size) 2. Shift the blocks according to scale factor 3. Compute cross-correlation 4. Find where cross-correlation is maximum 5. Adjust the shift to synchronize blocks with maximum similarity 6. Cross-fade blocks
[Roucos et al., 1985 and Makhoul et al., 1986.]
![Page 33: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/33.jpg)
SOLA Graphically
Figure from [Dutilleux et al. 2011]
![Page 34: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/34.jpg)
Pitch Synchronous Overlap and Add (PSOLA)
• Same idea as SOLA, but consider and preserve the local pitch [Moulines et al., 1989, 1990] • Developed specifically for speech signals.
• The algorithm is decomposed into two phases – Analysis: analyze and segment input – Synthesis: generate output by shifting and overlap-adding segments
from analysis
![Page 35: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/35.jpg)
Figure from [Dutilleux et al. 2011]
PSOLA Graphically
(a) Analysis (b) Synthesis
![Page 36: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/36.jpg)
• Determine the pitch period and pitch marks – Pitched parts
• spaced in time at the pitch rate • should correspond to glottal pulse
– Un-pitched parts • spaced at a constant rate
• Extract segments centered at every pitch mark , using a Hanning window of length (two pitch periods).
PSOLA Analysis
P (t) ti
titi
Li = 2P (ti)ti
ti
![Page 37: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/37.jpg)
1. Choice of corresponding analysis segment i minimizing the time distance
2. Overlap and add the selected segment: repeated for or discarded when
3. Determine the time where the next synthesis segment will be
centered (preserving local pitch)
PSOLA Synthesis
|↵ti � t̃k|
↵ > 1↵ < 1
t̃k+1
t̃k+1 = t̃k + P̃ (t̃k) = t̃k + P (ti)
![Page 38: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/38.jpg)
I Introduction II Proposed Method III Numerical Optimization IV Time-Stretch Processor V Results VI Conclusions
Outline
![Page 39: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/39.jpg)
Effect of Stretch Length
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Time (s)
a(t)
K = .5K = .75K = 1K = 1.5K = 2.0
• Varying the overall stretch factor gives smooth, intuitive stretch factors
minimize
x
||f ||2 + µ||x||2
subject to (x+ x0)T1 = L
x+ x0 � 0
� = .5
� = .75
� = 1.5
� = 2.0
� = 1.0
![Page 40: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/40.jpg)
Effect of Regularization
0 0.05 0.1 0.15 0.2 0.25 0.3 0.350.5
1
1.5
2
2.5
3
3.5
Time (s)
a(t)
µ = .001µ = .01µ = .1µ = 1
• Regularization penalty smooths the time-varying stretch factor
minimize
x
||f ||2 + µ||x||2
subject to (x+ x0)T1 = L
x+ x0 � 0
µ = .001
µ = .01
µ = .1
µ = 1
![Page 41: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/41.jpg)
Demo - Emphasis Modification
0 0.5 1 1.5 2 2.5−1
0
1
2
3
4
5
Time (s)
Ampl
itude
I’m gonna make him an offer he can’t refuse
s(t)k(t)a(t)
original
warped
warped + stretched (slowed by 1.3x)
“I’m gonna make him an offer he can’t refuse.”
“I’m gonna maaake him an offer he caaaan’t refuse.”
![Page 42: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/42.jpg)
Demo - Rhythmic Warping
Original
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
1.5
2
Time (s)
Ampl
itude
s(t)k(t)a(t)
0 0.2 0.4 0.6 0.8 1 1.2 1.4−1
−0.5
0
0.5
1
1.5
2
Ampl
itude
Time (s)
Stretched (2x)
No Stiffness / Stiffness
Swung + Stretched (2x)
![Page 43: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/43.jpg)
I Introduction II Proposed Method III Numerical Optimization IV Time-Stretch Processor V Results VI Conclusions
Outline
![Page 44: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/44.jpg)
• Method of user control over variable-rate time-stretching
• Decouples stiffness control + timing constraints to user
• Converts user control into optimal time-dependent stretch rate
• Agnostic to time modification algorithms
Conclusions
![Page 45: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/45.jpg)
• P. M. Nielson and S. Brandorff, “Time-stretching with a time dependent stretch factor,” University of Aarhus, 2002. http://www.daimi.au.dk/˜pmn/sound/.
• M. Grant and S. Boyd, “CVX: Matlab software for disciplined convex programming, version 1.21,” Apr. 2011.
• S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, NY, NY, USA, 2004.
• D. P. W. Ellis, “A phase vocoder in Matlab,” 2002, http://www.ee.columbia.edu/˜dpwe/ resources/matlab/pvoc/
• W. Sethares. Rhythm and Transforms, Springer, 2007. • S. Roucos and A. Wilgus. “High quality time-scale modification for speech.” ICASSP 1985. • J. Makhoul and A. El-Jaroudi. “Time-scale modification in medium to low rate speech coding.”
ICASSP, 1986. • E. Moulines and F. Charpentier. “Pitch-Synchronous waveform processing techniques for text-
to-speech synthesis using diphones.” Speech communication 9.5, 1990. • P. Dutilleux, G. De Poli, A. von dem Knesebeck, and U. Zoelzer, “Time-segment
processing,” in Dafx: Digital Audio Effects, Udo Zoelzer, Ed. Wiley & Sons, Inc., NY, NY, USA, 2nd edition, 2011.
References
![Page 46: User-Guided Variable-Rate Time-Stretching Via Stiffness ...njb/research/stiffness-ccrma-dsp-semin… · • Direct manipulation of the stretch rate is hard! 2x ? Flexures and Pivots](https://reader036.fdocuments.in/reader036/viewer/2022071102/5fdc13309deaa4744a18e288/html5/thumbnails/46.jpg)
Nicholas J. Bryan, Jorge Herrera, and Ge Wang
Stanford University | CCRMA
User-Guided Variable-Rate Time-Stretching Via Stiffness Control
CCRMA DSP Seminar, November 6th 2012