Scalable kernel methodshelper.ipam.ucla.edu/publications/bdcws2/bdcws2_15020.pdfScalable kernel...
54
Scalable kernel methods (aka N-body problems, radial basis functions) with Dhairya Malhotra, Chenhan Yu, James Levitt, Severin, Reiz Bill March, and Bo Xiao GEORGE BIROS padas.ices.utexas.edu
Transcript of Scalable kernel methodshelper.ipam.ucla.edu/publications/bdcws2/bdcws2_15020.pdfScalable kernel...
Scalable kernel methods (aka N-body problems, radial basis functions)
with Dhairya Malhotra, Chenhan Yu, James Levitt, Severin, Reiz Bill March, and Bo Xiao
GEORGE BIROSpadas.ices.utexas.edu
Outline
• Overview of kernel methods
• High-dimensions
• Extensions to positive definite matrices
• HPC
2
N-body problem
3
kernel/Grammatrix
kernel function
Input
N points in Rd: x1, . . . , xN
N densities in R : w1, . . . , wN
Output
N potentials in R : u1, . . . , uN
ui =NX
j=1j 6=i
G(xi, xj)wj
G(xi, xj) =1
kxi � xjk2<latexit sha1_base64="P5i4Q0BwZDZw07KZdahEJq1Z7+E=">AAAD5nicdVLbTtwwEM3u9kLTG7SPfTElILbAarNcCpWQUPvQ9oXSqlwqvESO44AXxwmxw7Iy/oW+VX3tB/UH+jed7IZyEbUUaXxmzszx5ISZ4Eq3239q9cadu/fujz1wHz56/OTp+MSzHZUWOWXbNBVpvhcSxQSXbFtzLdheljOShILthsfvyvzuKcsVT+VXPchYNyGHksecEg1QMFH7jTOScxkxqVE7066LQ3bIpdEkLATJrRHUugaHMfoos0JbNIOx6216KEu51ApxiTycEH0UhuaLPYi8N2gGeWeBP49FlGo1fxZAccWBKYprzm7SPDRk9S9Z/Yo1nPyp0DdGa5DLibi1j1dctimgDZAwk9G/F5X3/SLgaB0NmWlmsCqSwPTWfXuwaQODJ/FkD0t2grhF72fPAg6v6DVRP+jhrjvtjcgXCb8JuudGA+cuQMMXfAsJw+d8ewWFWzOq0BuUzWb5aBefqoxQ9sosdJLEgtQrAkBxnBNqfGvwOaBoAQGOz4OOBWGuG4xPtVv+2pK/1kHtVqezsry6CMHKsr+8uoT8Vnt4ppzqbMH/r+EopUUC+6SCKLXvgwm6huSaU8FgfKEYyDkmh2wfQkkSprpm6DuLpgGJUJzm8IF9huhVhiGJUoMkhMpy0epat1iwE9ktS8rcrSWFjle7hpe+Y5KO5sWFQDpFpZdRxHNGtRhAQGgOxqKIHhFYkAbHuzhnkvVpmiRERq9wTBIuBhGLSSE0/PG4Ci0s7WIz6P/BTqflQ/y5M7XxtlrfmPPCeenMOr7z2tlwPjhbzrZD64v1b/WwThtHje+NH42fo9J6reI8d66dxq+/+Rc4aQ==</latexit><latexit sha1_base64="P5i4Q0BwZDZw07KZdahEJq1Z7+E=">AAAD5nicdVLbTtwwEM3u9kLTG7SPfTElILbAarNcCpWQUPvQ9oXSqlwqvESO44AXxwmxw7Iy/oW+VX3tB/UH+jed7IZyEbUUaXxmzszx5ISZ4Eq3239q9cadu/fujz1wHz56/OTp+MSzHZUWOWXbNBVpvhcSxQSXbFtzLdheljOShILthsfvyvzuKcsVT+VXPchYNyGHksecEg1QMFH7jTOScxkxqVE7066LQ3bIpdEkLATJrRHUugaHMfoos0JbNIOx6216KEu51ApxiTycEH0UhuaLPYi8N2gGeWeBP49FlGo1fxZAccWBKYprzm7SPDRk9S9Z/Yo1nPyp0DdGa5DLibi1j1dctimgDZAwk9G/F5X3/SLgaB0NmWlmsCqSwPTWfXuwaQODJ/FkD0t2grhF72fPAg6v6DVRP+jhrjvtjcgXCb8JuudGA+cuQMMXfAsJw+d8ewWFWzOq0BuUzWb5aBefqoxQ9sosdJLEgtQrAkBxnBNqfGvwOaBoAQGOz4OOBWGuG4xPtVv+2pK/1kHtVqezsry6CMHKsr+8uoT8Vnt4ppzqbMH/r+EopUUC+6SCKLXvgwm6huSaU8FgfKEYyDkmh2wfQkkSprpm6DuLpgGJUJzm8IF9huhVhiGJUoMkhMpy0epat1iwE9ktS8rcrSWFjle7hpe+Y5KO5sWFQDpFpZdRxHNGtRhAQGgOxqKIHhFYkAbHuzhnkvVpmiRERq9wTBIuBhGLSSE0/PG4Ci0s7WIz6P/BTqflQ/y5M7XxtlrfmPPCeenMOr7z2tlwPjhbzrZD64v1b/WwThtHje+NH42fo9J6reI8d66dxq+/+Rc4aQ==</latexit>
Applications• Gravity & Coulomb
• Waves & scattering
• Fluids & transport
• Kernel methods in machine learning
• Approximation/Geostatistics
• Non-parametric statistics 4
Sim
ulat
ion
Dat
a an
alys
is
Relation to partial differential equations
5
6
��v +rp = 0, x 2 ⌦
divv = 0, x 2 ⌦
v = g, x 2 �
JvK = 0, x 2 �
J(�pI +rv +rvT )nK = f , x 2 �
dx
dt= v, 2 �
Kernel density estimation
7
Given {xj}Nj=1
pN (x) =1
N
NX
j=1
G(x, xj)
<latexit sha1_base64="7qlohvBXC9x3vGvEUfNW2Yue/R8=">AAACrnicdZFNbxMxEIad5assXykcuVhESClC0W5oSnqoFAFSuRCKRNJK8bLyOrOJW9u72N6SyNr+Mv4IV67wI3DSgKgEI9l6NfOOPX6clYIbG0XfGsG16zdu3tq6Hd65e+/+g+b2w7EpKs1gxApR6JOMGhBcwchyK+Ck1EBlJuA4O3u9qh+fgza8UB/tsoRE0pniOWfU+lTaHJMJkVmxcIf8HNTFRY2JW6SnpE7d6UFcfxpikoRkEpbpsL3YwQeY5JoyF9duWBNTyT+2w/biuW/cCb0/DNNmK+rE+7vxfhdHnW53r9d/4cVeL+71d3HcidbRQps4SrcbAzItWCVBWSaoMZM4Km3iqLacCahDUhkoKTujM5h4qagEk7g1gBo/9Zkpzgvtl7J4nf27w1FpzFJm3impnZsrp+UCPqtkZVnV/mmpbN5PHFdlZUGxy/vySmBb4BVUPOUamBVLLyjT3I+M2Zx6UNajD4kGBV9YISVV02ckp5KL5RRyWgnriMk30r/RgP9ANbNzR0qquZp6GrXzIHztDXguGt754d6XoKkttG/mM1W79b5i/hss/r8Ydzux1x+6rcGrDf0t9Bg9QW0Uo5dogN6iIzRCDH1F39EP9DOIgnGQBOmlNWhseh6hKxHMfwEcgdY2</latexit><latexit sha1_base64="7qlohvBXC9x3vGvEUfNW2Yue/R8=">AAACrnicdZFNbxMxEIad5assXykcuVhESClC0W5oSnqoFAFSuRCKRNJK8bLyOrOJW9u72N6SyNr+Mv4IV67wI3DSgKgEI9l6NfOOPX6clYIbG0XfGsG16zdu3tq6Hd65e+/+g+b2w7EpKs1gxApR6JOMGhBcwchyK+Ck1EBlJuA4O3u9qh+fgza8UB/tsoRE0pniOWfU+lTaHJMJkVmxcIf8HNTFRY2JW6SnpE7d6UFcfxpikoRkEpbpsL3YwQeY5JoyF9duWBNTyT+2w/biuW/cCb0/DNNmK+rE+7vxfhdHnW53r9d/4cVeL+71d3HcidbRQps4SrcbAzItWCVBWSaoMZM4Km3iqLacCahDUhkoKTujM5h4qagEk7g1gBo/9Zkpzgvtl7J4nf27w1FpzFJm3impnZsrp+UCPqtkZVnV/mmpbN5PHFdlZUGxy/vySmBb4BVUPOUamBVLLyjT3I+M2Zx6UNajD4kGBV9YISVV02ckp5KL5RRyWgnriMk30r/RgP9ANbNzR0qquZp6GrXzIHztDXguGt754d6XoKkttG/mM1W79b5i/hss/r8Ydzux1x+6rcGrDf0t9Bg9QW0Uo5dogN6iIzRCDH1F39EP9DOIgnGQBOmlNWhseh6hKxHMfwEcgdY2</latexit><latexit sha1_base64="7qlohvBXC9x3vGvEUfNW2Yue/R8=">AAACrnicdZFNbxMxEIad5assXykcuVhESClC0W5oSnqoFAFSuRCKRNJK8bLyOrOJW9u72N6SyNr+Mv4IV67wI3DSgKgEI9l6NfOOPX6clYIbG0XfGsG16zdu3tq6Hd65e+/+g+b2w7EpKs1gxApR6JOMGhBcwchyK+Ck1EBlJuA4O3u9qh+fgza8UB/tsoRE0pniOWfU+lTaHJMJkVmxcIf8HNTFRY2JW6SnpE7d6UFcfxpikoRkEpbpsL3YwQeY5JoyF9duWBNTyT+2w/biuW/cCb0/DNNmK+rE+7vxfhdHnW53r9d/4cVeL+71d3HcidbRQps4SrcbAzItWCVBWSaoMZM4Km3iqLacCahDUhkoKTujM5h4qagEk7g1gBo/9Zkpzgvtl7J4nf27w1FpzFJm3impnZsrp+UCPqtkZVnV/mmpbN5PHFdlZUGxy/vySmBb4BVUPOUamBVLLyjT3I+M2Zx6UNajD4kGBV9YISVV02ckp5KL5RRyWgnriMk30r/RgP9ANbNzR0qquZp6GrXzIHztDXguGt754d6XoKkttG/mM1W79b5i/hss/r8Ydzux1x+6rcGrDf0t9Bg9QW0Uo5dogN6iIzRCDH1F39EP9DOIgnGQBOmlNWhseh6hKxHMfwEcgdY2</latexit><latexit sha1_base64="7qlohvBXC9x3vGvEUfNW2Yue/R8=">AAACrnicdZFNbxMxEIad5assXykcuVhESClC0W5oSnqoFAFSuRCKRNJK8bLyOrOJW9u72N6SyNr+Mv4IV67wI3DSgKgEI9l6NfOOPX6clYIbG0XfGsG16zdu3tq6Hd65e+/+g+b2w7EpKs1gxApR6JOMGhBcwchyK+Ck1EBlJuA4O3u9qh+fgza8UB/tsoRE0pniOWfU+lTaHJMJkVmxcIf8HNTFRY2JW6SnpE7d6UFcfxpikoRkEpbpsL3YwQeY5JoyF9duWBNTyT+2w/biuW/cCb0/DNNmK+rE+7vxfhdHnW53r9d/4cVeL+71d3HcidbRQps4SrcbAzItWCVBWSaoMZM4Km3iqLacCahDUhkoKTujM5h4qagEk7g1gBo/9Zkpzgvtl7J4nf27w1FpzFJm3impnZsrp+UCPqtkZVnV/mmpbN5PHFdlZUGxy/vySmBb4BVUPOUamBVLLyjT3I+M2Zx6UNajD4kGBV9YISVV02ckp5KL5RRyWgnriMk30r/RgP9ANbNzR0qquZp6GrXzIHztDXguGt754d6XoKkttG/mM1W79b5i/hss/r8Ydzux1x+6rcGrDf0t9Bg9QW0Uo5dogN6iIzRCDH1F39EP9DOIgnGQBOmlNWhseh6hKxHMfwEcgdY2</latexit>
Silverman’99, “Density estimation”
Radial basis approximation
8
f(x) =NX
j=1
G(x, xj)wj
<latexit sha1_base64="f511wN8iJtXdSoI6f9BlNaEoePo=">AAACgXicdZHfbtMwFMbdDNgIf9bBJTcWFVKLUJVk6+g0TaoGEtwAQ6LbpCZEjnPSurOdzHZYqyjPsKfhFp6Dt8HpisQkOJKtT+d8xz7+OSk408bzfrWcjTt3721u3XcfPHz0eLu98+RU56WiMKY5z9V5QjRwJmFsmOFwXiggIuFwlly8aepn30BplssvZllAJMhUsoxRYmwqbvfCiZt1Fz18hENdiriaH/n114/4XXfxahHPe/gqnrth5Lpu3O54ff9gzz8IsNcPgv3BcNeK/YE/GO5hv++tooPWcRLvtEZhmtNSgDSUE60nvleYqCLKMMqhdsNSQ0HoBZnCxEpJBOioWr2pxi9sJsVZruySBq+yf3dURGi9FIl1CmJm+tZpGYdLGTWWpvZPS2myYVQxWZQGJL25Lys5NjluOOGUKaCGL60gVDE7MqYzogg1lqYbKpBwRXMhiExfhhkRjC9TyEjJTRXqbC3tGzXYP5FTM6vCgigmU0ujriwIW3sLlouCD3a4TwUoYnJlm9lU1tVqb5j/AYv/L06Dvm/156AzOl7T30LP0HPURT56jUboPTpBY0TRNfqOfqCfzobTczwnuLE6rXXPU3QrnMPfkKDDMQ==</latexit><latexit sha1_base64="f511wN8iJtXdSoI6f9BlNaEoePo=">AAACgXicdZHfbtMwFMbdDNgIf9bBJTcWFVKLUJVk6+g0TaoGEtwAQ6LbpCZEjnPSurOdzHZYqyjPsKfhFp6Dt8HpisQkOJKtT+d8xz7+OSk408bzfrWcjTt3721u3XcfPHz0eLu98+RU56WiMKY5z9V5QjRwJmFsmOFwXiggIuFwlly8aepn30BplssvZllAJMhUsoxRYmwqbvfCiZt1Fz18hENdiriaH/n114/4XXfxahHPe/gqnrth5Lpu3O54ff9gzz8IsNcPgv3BcNeK/YE/GO5hv++tooPWcRLvtEZhmtNSgDSUE60nvleYqCLKMMqhdsNSQ0HoBZnCxEpJBOioWr2pxi9sJsVZruySBq+yf3dURGi9FIl1CmJm+tZpGYdLGTWWpvZPS2myYVQxWZQGJL25Lys5NjluOOGUKaCGL60gVDE7MqYzogg1lqYbKpBwRXMhiExfhhkRjC9TyEjJTRXqbC3tGzXYP5FTM6vCgigmU0ujriwIW3sLlouCD3a4TwUoYnJlm9lU1tVqb5j/AYv/L06Dvm/156AzOl7T30LP0HPURT56jUboPTpBY0TRNfqOfqCfzobTczwnuLE6rXXPU3QrnMPfkKDDMQ==</latexit><latexit sha1_base64="f511wN8iJtXdSoI6f9BlNaEoePo=">AAACgXicdZHfbtMwFMbdDNgIf9bBJTcWFVKLUJVk6+g0TaoGEtwAQ6LbpCZEjnPSurOdzHZYqyjPsKfhFp6Dt8HpisQkOJKtT+d8xz7+OSk408bzfrWcjTt3721u3XcfPHz0eLu98+RU56WiMKY5z9V5QjRwJmFsmOFwXiggIuFwlly8aepn30BplssvZllAJMhUsoxRYmwqbvfCiZt1Fz18hENdiriaH/n114/4XXfxahHPe/gqnrth5Lpu3O54ff9gzz8IsNcPgv3BcNeK/YE/GO5hv++tooPWcRLvtEZhmtNSgDSUE60nvleYqCLKMMqhdsNSQ0HoBZnCxEpJBOioWr2pxi9sJsVZruySBq+yf3dURGi9FIl1CmJm+tZpGYdLGTWWpvZPS2myYVQxWZQGJL25Lys5NjluOOGUKaCGL60gVDE7MqYzogg1lqYbKpBwRXMhiExfhhkRjC9TyEjJTRXqbC3tGzXYP5FTM6vCgigmU0ujriwIW3sLlouCD3a4TwUoYnJlm9lU1tVqb5j/AYv/L06Dvm/156AzOl7T30LP0HPURT56jUboPTpBY0TRNfqOfqCfzobTczwnuLE6rXXPU3QrnMPfkKDDMQ==</latexit><latexit sha1_base64="f511wN8iJtXdSoI6f9BlNaEoePo=">AAACgXicdZHfbtMwFMbdDNgIf9bBJTcWFVKLUJVk6+g0TaoGEtwAQ6LbpCZEjnPSurOdzHZYqyjPsKfhFp6Dt8HpisQkOJKtT+d8xz7+OSk408bzfrWcjTt3721u3XcfPHz0eLu98+RU56WiMKY5z9V5QjRwJmFsmOFwXiggIuFwlly8aepn30BplssvZllAJMhUsoxRYmwqbvfCiZt1Fz18hENdiriaH/n114/4XXfxahHPe/gqnrth5Lpu3O54ff9gzz8IsNcPgv3BcNeK/YE/GO5hv++tooPWcRLvtEZhmtNSgDSUE60nvleYqCLKMMqhdsNSQ0HoBZnCxEpJBOioWr2pxi9sJsVZruySBq+yf3dURGi9FIl1CmJm+tZpGYdLGTWWpvZPS2myYVQxWZQGJL25Lys5NjluOOGUKaCGL60gVDE7MqYzogg1lqYbKpBwRXMhiExfhhkRjC9TyEjJTRXqbC3tGzXYP5FTM6vCgigmU0ujriwIW3sLlouCD3a4TwUoYnJlm9lU1tVqb5j/AYv/L06Dvm/156AzOl7T30LP0HPURT56jUboPTpBY0TRNfqOfqCfzobTczwnuLE6rXXPU3QrnMPfkKDDMQ==</latexit>
Buhmann’04, Wedland’04, Fornberg & Flyer’12Koumoutsakos & Cottet’01
Gaussian processes
9
f(x) ⇠ N (µ,C)
µ = G(:, x)Tw
C = G(x, x)�G(:, x)TG(:, :)�1G(:, x)<latexit sha1_base64="oAWE+2zTXXnB/PrHi93ZlVRllxg=">AAAC43icdVHbbtNAEN2YWzG3FB544GUhokpQG8WhKWklpIggwQtQpKatlHWjzXqcrLpem901TWTlC/qGeOXD+Az+gHEaLpVgJFtH55wZj8+MMyWta7W+V7wrV69dv7F20791+87de9X1+4c2zY2AgUhVao7H3IKSGgZOOgXHmQGejBUcjU/7pX70GYyVqT5w8wzChE+0jKXgDqlR9ZwN/bg+a1BmZUJZwt1UcFW8X9RZkm/SfsNnoY8eNoaJ1IXFndzCR41uvKRv6nubs8bJAT2jjPl9SukGLdkZsnTrj1yCvcZJsRUsViTaGejo97zQxxpVa61msLsd7LZpq9lu73S6zxHsdIJOd5sGzdayamRV+6P1So9FqcgT0E4obu0waGUuLLhxUijAubmFjItTPoEhQs0TsGGxTG1BnyIT0Tg1+GhHl+zfHQVPrJ0nY3SWsdhL02IFn3RYWkrtn5bcxd2wkDrLHWhx8b04V9SltLwEjaQB4dQcARdG4spUTLnhwuG9fGZAw5lIk4Tr6BmLeSLVPIKY58oVzMYriP9oAa+uJ25asIwbqSNMY1FgEKi9BszFwDtc7kMGhrvUYLOc6EWxfJeZ/wqW/h8ctpsB4o/tWu/VKv018og8IXUSkBekR96SfTIggvyoPKzQymMPvHPvi/f1wupVVj0PyKXyvv0EEcLf1A==</latexit><latexit sha1_base64="oAWE+2zTXXnB/PrHi93ZlVRllxg=">AAAC43icdVHbbtNAEN2YWzG3FB544GUhokpQG8WhKWklpIggwQtQpKatlHWjzXqcrLpem901TWTlC/qGeOXD+Az+gHEaLpVgJFtH55wZj8+MMyWta7W+V7wrV69dv7F20791+87de9X1+4c2zY2AgUhVao7H3IKSGgZOOgXHmQGejBUcjU/7pX70GYyVqT5w8wzChE+0jKXgDqlR9ZwN/bg+a1BmZUJZwt1UcFW8X9RZkm/SfsNnoY8eNoaJ1IXFndzCR41uvKRv6nubs8bJAT2jjPl9SukGLdkZsnTrj1yCvcZJsRUsViTaGejo97zQxxpVa61msLsd7LZpq9lu73S6zxHsdIJOd5sGzdayamRV+6P1So9FqcgT0E4obu0waGUuLLhxUijAubmFjItTPoEhQs0TsGGxTG1BnyIT0Tg1+GhHl+zfHQVPrJ0nY3SWsdhL02IFn3RYWkrtn5bcxd2wkDrLHWhx8b04V9SltLwEjaQB4dQcARdG4spUTLnhwuG9fGZAw5lIk4Tr6BmLeSLVPIKY58oVzMYriP9oAa+uJ25asIwbqSNMY1FgEKi9BszFwDtc7kMGhrvUYLOc6EWxfJeZ/wqW/h8ctpsB4o/tWu/VKv018og8IXUSkBekR96SfTIggvyoPKzQymMPvHPvi/f1wupVVj0PyKXyvv0EEcLf1A==</latexit><latexit sha1_base64="oAWE+2zTXXnB/PrHi93ZlVRllxg=">AAAC43icdVHbbtNAEN2YWzG3FB544GUhokpQG8WhKWklpIggwQtQpKatlHWjzXqcrLpem901TWTlC/qGeOXD+Az+gHEaLpVgJFtH55wZj8+MMyWta7W+V7wrV69dv7F20791+87de9X1+4c2zY2AgUhVao7H3IKSGgZOOgXHmQGejBUcjU/7pX70GYyVqT5w8wzChE+0jKXgDqlR9ZwN/bg+a1BmZUJZwt1UcFW8X9RZkm/SfsNnoY8eNoaJ1IXFndzCR41uvKRv6nubs8bJAT2jjPl9SukGLdkZsnTrj1yCvcZJsRUsViTaGejo97zQxxpVa61msLsd7LZpq9lu73S6zxHsdIJOd5sGzdayamRV+6P1So9FqcgT0E4obu0waGUuLLhxUijAubmFjItTPoEhQs0TsGGxTG1BnyIT0Tg1+GhHl+zfHQVPrJ0nY3SWsdhL02IFn3RYWkrtn5bcxd2wkDrLHWhx8b04V9SltLwEjaQB4dQcARdG4spUTLnhwuG9fGZAw5lIk4Tr6BmLeSLVPIKY58oVzMYriP9oAa+uJ25asIwbqSNMY1FgEKi9BszFwDtc7kMGhrvUYLOc6EWxfJeZ/wqW/h8ctpsB4o/tWu/VKv018og8IXUSkBekR96SfTIggvyoPKzQymMPvHPvi/f1wupVVj0PyKXyvv0EEcLf1A==</latexit><latexit sha1_base64="oAWE+2zTXXnB/PrHi93ZlVRllxg=">AAAC43icdVHbbtNAEN2YWzG3FB544GUhokpQG8WhKWklpIggwQtQpKatlHWjzXqcrLpem901TWTlC/qGeOXD+Az+gHEaLpVgJFtH55wZj8+MMyWta7W+V7wrV69dv7F20791+87de9X1+4c2zY2AgUhVao7H3IKSGgZOOgXHmQGejBUcjU/7pX70GYyVqT5w8wzChE+0jKXgDqlR9ZwN/bg+a1BmZUJZwt1UcFW8X9RZkm/SfsNnoY8eNoaJ1IXFndzCR41uvKRv6nubs8bJAT2jjPl9SukGLdkZsnTrj1yCvcZJsRUsViTaGejo97zQxxpVa61msLsd7LZpq9lu73S6zxHsdIJOd5sGzdayamRV+6P1So9FqcgT0E4obu0waGUuLLhxUijAubmFjItTPoEhQs0TsGGxTG1BnyIT0Tg1+GhHl+zfHQVPrJ0nY3SWsdhL02IFn3RYWkrtn5bcxd2wkDrLHWhx8b04V9SltLwEjaQB4dQcARdG4spUTLnhwuG9fGZAw5lIk4Tr6BmLeSLVPIKY58oVzMYriP9oAa+uJ25asIwbqSNMY1FgEKi9BszFwDtc7kMGhrvUYLOc6EWxfJeZ/wqW/h8ctpsB4o/tWu/VKv018og8IXUSkBekR96SfTIggvyoPKzQymMPvHPvi/f1wupVVj0PyKXyvv0EEcLf1A==</latexit>
Rasmussen & Williams’06, “Gaussian processes for machine learning”
10
Siggraph 01, Carr, Beatson et al
Logistic regression / SVM
11
Hastie & Tibshirani & Friedman, “The elements of Statistical Learning”
Kernel binary classification
12
Inference:
c(x) = signPN
j=1 G(x, xj)wj<latexit sha1_base64="7j2L1nrgyi0PCU/ErxsWAYrnFh4=">AAADXXicdVJdTxNBFN2loFgRQR988OVGSgKmNN1KEUhIiJqoDyoavhKmbGZnZ8vA7OwyMyttJvsjfNVf5pN/xbulNUL0Jrs5uffcO2fO3CiXwth2+6c/VZueuXN39l79/tyD+YcLi48OTVZoxg9YJjN9HFHDpVD8wAor+XGuOU0jyY+ii9dV/egr10Zkat8Oc95LaV+JRDBqMRUu+tOOjKa4SFJ2UcJ7lXDNFePbUAIhJ93c9uoNYCuDVdgBYkRfEVOkoTvfCcrTj/B2ZdAchOercBWeNyb8KpZvDd7XVKjtckJZhgZxMAgFEKGApNSeRZH7Up7GTWCTrFsLmgEpQyeuDyNl40Y/nllVx1K2AW5LC8Ufcage5zaBXBY0BpJkmkoJotFCreHCUrsVbK0HWx1otzqdje7mCwQb3aC7uQ5Bqz2KJW8ce+jbLokzVqRcWSapMSdBG1U5qq1gkpd1Uhie471pn58gVDTlpudGhpSwjJkYUAF+ysIo+3eHo6kxwzRCZuWLuTEtkfxS9SpKVfsnpbDJZs8JlRcWX/L6vKSQYDOodgBioTmzcoiAMi1QMrAzqimzuCl1gs/Pr1iWplTFz0lCUyGHMU9oIa0jJhlDvKPhuG+qb88cyakWKkY3SodGYO0NR180/4DiPuVcU4ubMNqe0o3+lecTY+H/4LDTChB/7iztvhq7P+s99Z55K17gvfR2vXfennfgMf/C/+Z/939M/arN1OZq89fUKX/c89i7EbUnvwH9Vwvf</latexit><latexit sha1_base64="7j2L1nrgyi0PCU/ErxsWAYrnFh4=">AAADXXicdVJdTxNBFN2loFgRQR988OVGSgKmNN1KEUhIiJqoDyoavhKmbGZnZ8vA7OwyMyttJvsjfNVf5pN/xbulNUL0Jrs5uffcO2fO3CiXwth2+6c/VZueuXN39l79/tyD+YcLi48OTVZoxg9YJjN9HFHDpVD8wAor+XGuOU0jyY+ii9dV/egr10Zkat8Oc95LaV+JRDBqMRUu+tOOjKa4SFJ2UcJ7lXDNFePbUAIhJ93c9uoNYCuDVdgBYkRfEVOkoTvfCcrTj/B2ZdAchOercBWeNyb8KpZvDd7XVKjtckJZhgZxMAgFEKGApNSeRZH7Up7GTWCTrFsLmgEpQyeuDyNl40Y/nllVx1K2AW5LC8Ufcage5zaBXBY0BpJkmkoJotFCreHCUrsVbK0HWx1otzqdje7mCwQb3aC7uQ5Bqz2KJW8ce+jbLokzVqRcWSapMSdBG1U5qq1gkpd1Uhie471pn58gVDTlpudGhpSwjJkYUAF+ysIo+3eHo6kxwzRCZuWLuTEtkfxS9SpKVfsnpbDJZs8JlRcWX/L6vKSQYDOodgBioTmzcoiAMi1QMrAzqimzuCl1gs/Pr1iWplTFz0lCUyGHMU9oIa0jJhlDvKPhuG+qb88cyakWKkY3SodGYO0NR180/4DiPuVcU4ubMNqe0o3+lecTY+H/4LDTChB/7iztvhq7P+s99Z55K17gvfR2vXfennfgMf/C/+Z/939M/arN1OZq89fUKX/c89i7EbUnvwH9Vwvf</latexit><latexit sha1_base64="7j2L1nrgyi0PCU/ErxsWAYrnFh4=">AAADXXicdVJdTxNBFN2loFgRQR988OVGSgKmNN1KEUhIiJqoDyoavhKmbGZnZ8vA7OwyMyttJvsjfNVf5pN/xbulNUL0Jrs5uffcO2fO3CiXwth2+6c/VZueuXN39l79/tyD+YcLi48OTVZoxg9YJjN9HFHDpVD8wAor+XGuOU0jyY+ii9dV/egr10Zkat8Oc95LaV+JRDBqMRUu+tOOjKa4SFJ2UcJ7lXDNFePbUAIhJ93c9uoNYCuDVdgBYkRfEVOkoTvfCcrTj/B2ZdAchOercBWeNyb8KpZvDd7XVKjtckJZhgZxMAgFEKGApNSeRZH7Up7GTWCTrFsLmgEpQyeuDyNl40Y/nllVx1K2AW5LC8Ufcage5zaBXBY0BpJkmkoJotFCreHCUrsVbK0HWx1otzqdje7mCwQb3aC7uQ5Bqz2KJW8ce+jbLokzVqRcWSapMSdBG1U5qq1gkpd1Uhie471pn58gVDTlpudGhpSwjJkYUAF+ysIo+3eHo6kxwzRCZuWLuTEtkfxS9SpKVfsnpbDJZs8JlRcWX/L6vKSQYDOodgBioTmzcoiAMi1QMrAzqimzuCl1gs/Pr1iWplTFz0lCUyGHMU9oIa0jJhlDvKPhuG+qb88cyakWKkY3SodGYO0NR180/4DiPuVcU4ubMNqe0o3+lecTY+H/4LDTChB/7iztvhq7P+s99Z55K17gvfR2vXfennfgMf/C/+Z/939M/arN1OZq89fUKX/c89i7EbUnvwH9Vwvf</latexit><latexit sha1_base64="7j2L1nrgyi0PCU/ErxsWAYrnFh4=">AAADXXicdVJdTxNBFN2loFgRQR988OVGSgKmNN1KEUhIiJqoDyoavhKmbGZnZ8vA7OwyMyttJvsjfNVf5pN/xbulNUL0Jrs5uffcO2fO3CiXwth2+6c/VZueuXN39l79/tyD+YcLi48OTVZoxg9YJjN9HFHDpVD8wAor+XGuOU0jyY+ii9dV/egr10Zkat8Oc95LaV+JRDBqMRUu+tOOjKa4SFJ2UcJ7lXDNFePbUAIhJ93c9uoNYCuDVdgBYkRfEVOkoTvfCcrTj/B2ZdAchOercBWeNyb8KpZvDd7XVKjtckJZhgZxMAgFEKGApNSeRZH7Up7GTWCTrFsLmgEpQyeuDyNl40Y/nllVx1K2AW5LC8Ufcage5zaBXBY0BpJkmkoJotFCreHCUrsVbK0HWx1otzqdje7mCwQb3aC7uQ5Bqz2KJW8ce+jbLokzVqRcWSapMSdBG1U5qq1gkpd1Uhie471pn58gVDTlpudGhpSwjJkYUAF+ysIo+3eHo6kxwzRCZuWLuTEtkfxS9SpKVfsnpbDJZs8JlRcWX/L6vKSQYDOodgBioTmzcoiAMi1QMrAzqimzuCl1gs/Pr1iWplTFz0lCUyGHMU9oIa0jJhlDvKPhuG+qb88cyakWKkY3SodGYO0NR180/4DiPuVcU4ubMNqe0o3+lecTY+H/4LDTChB/7iztvhq7P+s99Z55K17gvfR2vXfennfgMf/C/+Z/939M/arN1OZq89fUKX/c89i7EbUnvwH9Vwvf</latexit>
Training:
Given {xi 2 Rd, ci 2 {�1, 1}}Ni=1
Find {wj}Nj=1 such thatPN
j=1 G(xi, xj)wj = ci, 8i.<latexit sha1_base64="+mjO9w8qXy7vabM2Z74RV2AnXk8=">AAADcHicdVLbTtRAGG5ZD7ieQC808cJfWRIg62a7sggkJESN6IWKhlPCLM10Ot0dmE5LZwq7mdSn8Vbfx9fwCfy7ByJEJ2nz5T9+880XpFJo02z+cqcq167fuDl9q3r7zt1792dmH+zpJM8Y32WJTLKDgGouheK7RhjJD9KM0ziQfD84eVPm9894pkWidswg5Z2YdpWIBKMGQ/6s+2jekuEYG0jKTgr4oCKeccX4OhRAyGE7NZ3qfA3YQn8RNoBo0VVE57Fvjze84ugTbC30633/eBHO/ePapKFavTJ2J6NCCdUt1iclW+KMK6gRC31fABEKSExNLwjs1+IorAObRO0Lr+6RghS+FaOVF2veCRVCOQN3l/kRpRronPXA9KjB3BWuvrhgi9fBJXUgpzkNgURJRqUEUWsgfX9mrtnw1pa9tRY0G63WSnv1JYKVttdeXQav0RyeOWd8tlHKTRImLI+5MkxSrQ+9JlK0NDOCSV5USa55ilrQLj9EqGjMdccORSpgHiMhIAP8lIFh9O8OS2OtB3GAlaVI+tK0SPJT1SlLytw/S3ITrXasUGlu8G1H+6JcgkmgtAWEIuPMyAECyjKBlIH1aEaZQfNUCRqCn7MkjqkKl0hEYyEHIY9oLo0lOhpDvKPmaEHVNT1LUprh66AahUUhMPeWoy4Z/4jkPqc8owbdMbRTYYf/UvOJsPB/sNdqeIi/tOY2X4/Vn3aeOM+dBcdzXjmbzntn29l1mPvN/e7+cH9O/a48rjytPBuVTrnjnofOpVNZ+gMOhBSu</latexit><latexit sha1_base64="+mjO9w8qXy7vabM2Z74RV2AnXk8=">AAADcHicdVLbTtRAGG5ZD7ieQC808cJfWRIg62a7sggkJESN6IWKhlPCLM10Ot0dmE5LZwq7mdSn8Vbfx9fwCfy7ByJEJ2nz5T9+880XpFJo02z+cqcq167fuDl9q3r7zt1792dmH+zpJM8Y32WJTLKDgGouheK7RhjJD9KM0ziQfD84eVPm9894pkWidswg5Z2YdpWIBKMGQ/6s+2jekuEYG0jKTgr4oCKeccX4OhRAyGE7NZ3qfA3YQn8RNoBo0VVE57Fvjze84ugTbC30633/eBHO/ePapKFavTJ2J6NCCdUt1iclW+KMK6gRC31fABEKSExNLwjs1+IorAObRO0Lr+6RghS+FaOVF2veCRVCOQN3l/kRpRronPXA9KjB3BWuvrhgi9fBJXUgpzkNgURJRqUEUWsgfX9mrtnw1pa9tRY0G63WSnv1JYKVttdeXQav0RyeOWd8tlHKTRImLI+5MkxSrQ+9JlK0NDOCSV5USa55ilrQLj9EqGjMdccORSpgHiMhIAP8lIFh9O8OS2OtB3GAlaVI+tK0SPJT1SlLytw/S3ITrXasUGlu8G1H+6JcgkmgtAWEIuPMyAECyjKBlIH1aEaZQfNUCRqCn7MkjqkKl0hEYyEHIY9oLo0lOhpDvKPmaEHVNT1LUprh66AahUUhMPeWoy4Z/4jkPqc8owbdMbRTYYf/UvOJsPB/sNdqeIi/tOY2X4/Vn3aeOM+dBcdzXjmbzntn29l1mPvN/e7+cH9O/a48rjytPBuVTrnjnofOpVNZ+gMOhBSu</latexit><latexit sha1_base64="+mjO9w8qXy7vabM2Z74RV2AnXk8=">AAADcHicdVLbTtRAGG5ZD7ieQC808cJfWRIg62a7sggkJESN6IWKhlPCLM10Ot0dmE5LZwq7mdSn8Vbfx9fwCfy7ByJEJ2nz5T9+880XpFJo02z+cqcq167fuDl9q3r7zt1792dmH+zpJM8Y32WJTLKDgGouheK7RhjJD9KM0ziQfD84eVPm9894pkWidswg5Z2YdpWIBKMGQ/6s+2jekuEYG0jKTgr4oCKeccX4OhRAyGE7NZ3qfA3YQn8RNoBo0VVE57Fvjze84ugTbC30633/eBHO/ePapKFavTJ2J6NCCdUt1iclW+KMK6gRC31fABEKSExNLwjs1+IorAObRO0Lr+6RghS+FaOVF2veCRVCOQN3l/kRpRronPXA9KjB3BWuvrhgi9fBJXUgpzkNgURJRqUEUWsgfX9mrtnw1pa9tRY0G63WSnv1JYKVttdeXQav0RyeOWd8tlHKTRImLI+5MkxSrQ+9JlK0NDOCSV5USa55ilrQLj9EqGjMdccORSpgHiMhIAP8lIFh9O8OS2OtB3GAlaVI+tK0SPJT1SlLytw/S3ITrXasUGlu8G1H+6JcgkmgtAWEIuPMyAECyjKBlIH1aEaZQfNUCRqCn7MkjqkKl0hEYyEHIY9oLo0lOhpDvKPmaEHVNT1LUprh66AahUUhMPeWoy4Z/4jkPqc8owbdMbRTYYf/UvOJsPB/sNdqeIi/tOY2X4/Vn3aeOM+dBcdzXjmbzntn29l1mPvN/e7+cH9O/a48rjytPBuVTrnjnofOpVNZ+gMOhBSu</latexit><latexit sha1_base64="+mjO9w8qXy7vabM2Z74RV2AnXk8=">AAADcHicdVLbTtRAGG5ZD7ieQC808cJfWRIg62a7sggkJESN6IWKhlPCLM10Ot0dmE5LZwq7mdSn8Vbfx9fwCfy7ByJEJ2nz5T9+880XpFJo02z+cqcq167fuDl9q3r7zt1792dmH+zpJM8Y32WJTLKDgGouheK7RhjJD9KM0ziQfD84eVPm9894pkWidswg5Z2YdpWIBKMGQ/6s+2jekuEYG0jKTgr4oCKeccX4OhRAyGE7NZ3qfA3YQn8RNoBo0VVE57Fvjze84ugTbC30633/eBHO/ePapKFavTJ2J6NCCdUt1iclW+KMK6gRC31fABEKSExNLwjs1+IorAObRO0Lr+6RghS+FaOVF2veCRVCOQN3l/kRpRronPXA9KjB3BWuvrhgi9fBJXUgpzkNgURJRqUEUWsgfX9mrtnw1pa9tRY0G63WSnv1JYKVttdeXQav0RyeOWd8tlHKTRImLI+5MkxSrQ+9JlK0NDOCSV5USa55ilrQLj9EqGjMdccORSpgHiMhIAP8lIFh9O8OS2OtB3GAlaVI+tK0SPJT1SlLytw/S3ITrXasUGlu8G1H+6JcgkmgtAWEIuPMyAECyjKBlIH1aEaZQfNUCRqCn7MkjqkKl0hEYyEHIY9oLo0lOhpDvKPmaEHVNT1LUprh66AahUUhMPeWoy4Z/4jkPqc8owbdMbRTYYf/UvOJsPB/sNdqeIi/tOY2X4/Vn3aeOM+dBcdzXjmbzntn29l1mPvN/e7+cH9O/a48rjytPBuVTrnjnofOpVNZ+gMOhBSu</latexit>
c(x) = sign(xTw + b)<latexit sha1_base64="XhMHbDhcyF+UlFXDkBO5DC6s+L4=">AAACeHicdZFdb9MwFIbdjI8RPtaNS24sKkQLUpWEdXQXSBVwwQ1iSOs2qQnViXPSWrOdYDtsVdRbfg238F/4K1zhZEViEhzJ1qtz3mMfP05LwY0Ngp8db+vGzVu3t+/4d+/df7DT3d07MUWlGU5ZIQp9loJBwRVOLbcCz0qNIFOBp+n5m6Z++gW14YU6tqsSEwkLxXPOwLrUvEvjmc/6lwP6isaGL1T/8tMxvaDPaTrw48T3/Xm3FwzDw/3wMKLBMIoORuMXThyMwtF4n4bDoI0e2cTRfLczibOCVRKVZQKMmYVBaZMatOVM4NqPK4MlsHNY4MxJBRJNUrdPWdMnLpPRvNBuKUvb7N8dNUhjVjJ1Tgl2aa6dlgv8rJLG0tT+aalsPk5qrsrKomJX9+WVoLagDR6acY3MipUTwDR3I1O2BA3MOoh+rFHhBSukBJU9i3OQXKwyzKESto5NvpHujQbdV6iFXdZxCZqrzNFY1w6Eq71Fx0XjezfchxI12ELXLfx13e4N8z9g6f/FSTQMnf4Y9SavN/S3ySPymPRJSF6SCXlHjsiUMPKVfCPfyY/OL496T73BldXrbHoekmvhRb8B5fTANg==</latexit><latexit sha1_base64="XhMHbDhcyF+UlFXDkBO5DC6s+L4=">AAACeHicdZFdb9MwFIbdjI8RPtaNS24sKkQLUpWEdXQXSBVwwQ1iSOs2qQnViXPSWrOdYDtsVdRbfg238F/4K1zhZEViEhzJ1qtz3mMfP05LwY0Ngp8db+vGzVu3t+/4d+/df7DT3d07MUWlGU5ZIQp9loJBwRVOLbcCz0qNIFOBp+n5m6Z++gW14YU6tqsSEwkLxXPOwLrUvEvjmc/6lwP6isaGL1T/8tMxvaDPaTrw48T3/Xm3FwzDw/3wMKLBMIoORuMXThyMwtF4n4bDoI0e2cTRfLczibOCVRKVZQKMmYVBaZMatOVM4NqPK4MlsHNY4MxJBRJNUrdPWdMnLpPRvNBuKUvb7N8dNUhjVjJ1Tgl2aa6dlgv8rJLG0tT+aalsPk5qrsrKomJX9+WVoLagDR6acY3MipUTwDR3I1O2BA3MOoh+rFHhBSukBJU9i3OQXKwyzKESto5NvpHujQbdV6iFXdZxCZqrzNFY1w6Eq71Fx0XjezfchxI12ELXLfx13e4N8z9g6f/FSTQMnf4Y9SavN/S3ySPymPRJSF6SCXlHjsiUMPKVfCPfyY/OL496T73BldXrbHoekmvhRb8B5fTANg==</latexit><latexit sha1_base64="XhMHbDhcyF+UlFXDkBO5DC6s+L4=">AAACeHicdZFdb9MwFIbdjI8RPtaNS24sKkQLUpWEdXQXSBVwwQ1iSOs2qQnViXPSWrOdYDtsVdRbfg238F/4K1zhZEViEhzJ1qtz3mMfP05LwY0Ngp8db+vGzVu3t+/4d+/df7DT3d07MUWlGU5ZIQp9loJBwRVOLbcCz0qNIFOBp+n5m6Z++gW14YU6tqsSEwkLxXPOwLrUvEvjmc/6lwP6isaGL1T/8tMxvaDPaTrw48T3/Xm3FwzDw/3wMKLBMIoORuMXThyMwtF4n4bDoI0e2cTRfLczibOCVRKVZQKMmYVBaZMatOVM4NqPK4MlsHNY4MxJBRJNUrdPWdMnLpPRvNBuKUvb7N8dNUhjVjJ1Tgl2aa6dlgv8rJLG0tT+aalsPk5qrsrKomJX9+WVoLagDR6acY3MipUTwDR3I1O2BA3MOoh+rFHhBSukBJU9i3OQXKwyzKESto5NvpHujQbdV6iFXdZxCZqrzNFY1w6Eq71Fx0XjezfchxI12ELXLfx13e4N8z9g6f/FSTQMnf4Y9SavN/S3ySPymPRJSF6SCXlHjsiUMPKVfCPfyY/OL496T73BldXrbHoekmvhRb8B5fTANg==</latexit><latexit sha1_base64="XhMHbDhcyF+UlFXDkBO5DC6s+L4=">AAACeHicdZFdb9MwFIbdjI8RPtaNS24sKkQLUpWEdXQXSBVwwQ1iSOs2qQnViXPSWrOdYDtsVdRbfg238F/4K1zhZEViEhzJ1qtz3mMfP05LwY0Ngp8db+vGzVu3t+/4d+/df7DT3d07MUWlGU5ZIQp9loJBwRVOLbcCz0qNIFOBp+n5m6Z++gW14YU6tqsSEwkLxXPOwLrUvEvjmc/6lwP6isaGL1T/8tMxvaDPaTrw48T3/Xm3FwzDw/3wMKLBMIoORuMXThyMwtF4n4bDoI0e2cTRfLczibOCVRKVZQKMmYVBaZMatOVM4NqPK4MlsHNY4MxJBRJNUrdPWdMnLpPRvNBuKUvb7N8dNUhjVjJ1Tgl2aa6dlgv8rJLG0tT+aalsPk5qrsrKomJX9+WVoLagDR6acY3MipUTwDR3I1O2BA3MOoh+rFHhBSukBJU9i3OQXKwyzKESto5NvpHujQbdV6iFXdZxCZqrzNFY1w6Eq71Fx0XjezfchxI12ELXLfx13e4N8z9g6f/FSTQMnf4Y9SavN/S3ySPymPRJSF6SCXlHjsiUMPKVfCPfyY/OL496T73BldXrbHoekmvhRb8B5fTANg==</latexit>
Logistic regression
13
logp(x)
1� p(x)= wTx
logp(x)
1� p(x)=
NX
j=1
G(x, xj)w
<latexit sha1_base64="P25/nT10RgLWMXDCgeYdkvk5bNE=">AAACyHicfVFdb9MwFHUzNkb4WAePvFhUSB1iVVLW0T1MqgAJhAQMad0mNVnlOjetN8fxbGdtZeWFH4nEf+EBJysSk4Ar2T66Puf6+tyJ5EybIPjR8NburG/c3bzn33/w8NFWc/vxic4LRWFIc56rswnRwJmAoWGGw5lUQLIJh9PJ5dvq/vQalGa5ODZLCXFGpoKljBLjUuOmjEZ+xPMpjlJFqJXtxU5pw936xId4fn6MF34U+9G1loSC3e1JU/r/F0W6yMb24jAszz/j9+3Fy8X4YgfPqzK+P262gk54sBcedHHQ6Xb3e/1XDuz3wl5/D4edoI4WWsXReLsxiJKcFhkIQznRehQG0sSWKMMoB9dMocH1dkmmMHJQkAx0bGtrSvzcZRKc5sotYXCd/VNhSab1Mps4ZkbMTN+qlnK4EnFFqe7+SilM2o8tE7IwIOjNe2nBsclxZTdOmAJq+NIBQhVzLWM6I84144biRwoEzGmeZUQkL6KUZIwvE0hJwY2NdLqC7o8a3GjF1MxsJIliInFulDaoh/EOnC8KPrnmvkhQxOTKidlUlLbeK89/G4v/DU66ndDhr93W4M3K/U30FD1DbRSi12iAPqAjNEQUfUc/G+uNDe+jJ725t7yheo2V5gm6Fd63X20I298=</latexit><latexit sha1_base64="P25/nT10RgLWMXDCgeYdkvk5bNE=">AAACyHicfVFdb9MwFHUzNkb4WAePvFhUSB1iVVLW0T1MqgAJhAQMad0mNVnlOjetN8fxbGdtZeWFH4nEf+EBJysSk4Ar2T66Puf6+tyJ5EybIPjR8NburG/c3bzn33/w8NFWc/vxic4LRWFIc56rswnRwJmAoWGGw5lUQLIJh9PJ5dvq/vQalGa5ODZLCXFGpoKljBLjUuOmjEZ+xPMpjlJFqJXtxU5pw936xId4fn6MF34U+9G1loSC3e1JU/r/F0W6yMb24jAszz/j9+3Fy8X4YgfPqzK+P262gk54sBcedHHQ6Xb3e/1XDuz3wl5/D4edoI4WWsXReLsxiJKcFhkIQznRehQG0sSWKMMoB9dMocH1dkmmMHJQkAx0bGtrSvzcZRKc5sotYXCd/VNhSab1Mps4ZkbMTN+qlnK4EnFFqe7+SilM2o8tE7IwIOjNe2nBsclxZTdOmAJq+NIBQhVzLWM6I84144biRwoEzGmeZUQkL6KUZIwvE0hJwY2NdLqC7o8a3GjF1MxsJIliInFulDaoh/EOnC8KPrnmvkhQxOTKidlUlLbeK89/G4v/DU66ndDhr93W4M3K/U30FD1DbRSi12iAPqAjNEQUfUc/G+uNDe+jJ725t7yheo2V5gm6Fd63X20I298=</latexit><latexit sha1_base64="P25/nT10RgLWMXDCgeYdkvk5bNE=">AAACyHicfVFdb9MwFHUzNkb4WAePvFhUSB1iVVLW0T1MqgAJhAQMad0mNVnlOjetN8fxbGdtZeWFH4nEf+EBJysSk4Ar2T66Puf6+tyJ5EybIPjR8NburG/c3bzn33/w8NFWc/vxic4LRWFIc56rswnRwJmAoWGGw5lUQLIJh9PJ5dvq/vQalGa5ODZLCXFGpoKljBLjUuOmjEZ+xPMpjlJFqJXtxU5pw936xId4fn6MF34U+9G1loSC3e1JU/r/F0W6yMb24jAszz/j9+3Fy8X4YgfPqzK+P262gk54sBcedHHQ6Xb3e/1XDuz3wl5/D4edoI4WWsXReLsxiJKcFhkIQznRehQG0sSWKMMoB9dMocH1dkmmMHJQkAx0bGtrSvzcZRKc5sotYXCd/VNhSab1Mps4ZkbMTN+qlnK4EnFFqe7+SilM2o8tE7IwIOjNe2nBsclxZTdOmAJq+NIBQhVzLWM6I84144biRwoEzGmeZUQkL6KUZIwvE0hJwY2NdLqC7o8a3GjF1MxsJIliInFulDaoh/EOnC8KPrnmvkhQxOTKidlUlLbeK89/G4v/DU66ndDhr93W4M3K/U30FD1DbRSi12iAPqAjNEQUfUc/G+uNDe+jJ725t7yheo2V5gm6Fd63X20I298=</latexit><latexit sha1_base64="P25/nT10RgLWMXDCgeYdkvk5bNE=">AAACyHicfVFdb9MwFHUzNkb4WAePvFhUSB1iVVLW0T1MqgAJhAQMad0mNVnlOjetN8fxbGdtZeWFH4nEf+EBJysSk4Ar2T66Puf6+tyJ5EybIPjR8NburG/c3bzn33/w8NFWc/vxic4LRWFIc56rswnRwJmAoWGGw5lUQLIJh9PJ5dvq/vQalGa5ODZLCXFGpoKljBLjUuOmjEZ+xPMpjlJFqJXtxU5pw936xId4fn6MF34U+9G1loSC3e1JU/r/F0W6yMb24jAszz/j9+3Fy8X4YgfPqzK+P262gk54sBcedHHQ6Xb3e/1XDuz3wl5/D4edoI4WWsXReLsxiJKcFhkIQznRehQG0sSWKMMoB9dMocH1dkmmMHJQkAx0bGtrSvzcZRKc5sotYXCd/VNhSab1Mps4ZkbMTN+qlnK4EnFFqe7+SilM2o8tE7IwIOjNe2nBsclxZTdOmAJq+NIBQhVzLWM6I84144biRwoEzGmeZUQkL6KUZIwvE0hJwY2NdLqC7o8a3GjF1MxsJIliInFulDaoh/EOnC8KPrnmvkhQxOTKidlUlLbeK89/G4v/DU66ndDhr93W4M3K/U30FD1DbRSi12iAPqAjNEQUfUc/G+uNDe+jJ725t7yheo2V5gm6Fd63X20I298=</latexit>
Spectral clustering
14
V : largest eigenvectors of GNormalize V (:, 1 : )K-means cluster V (:, 1 : )
<latexit sha1_base64="O2HfXf8JditjEs6hyNTp63cMooU=">AAACxXicdZFdaxNBFIYnWz9q/Er10pvBrFBFw25sapqroEIFUSvYtNAN4WT27GbIfKwzs5G4BH+kV/4U75xNIlioB2Z4Oec9Z2aemRaCWxdFvxrBzrXrN27u3mrevnP33v3W3oOR1aVheMq00OZ8ChYFV3jquBN4XhgEORV4Np2/qetnCzSWa/XFLQscS8gVzzgD51OTlgxH4YCGyRyKAkIqwORoHUWeo1ogc9pYqjMaHodJ0vyojQTBvyMNR/uD5/Fg0/a0rr1/IRGUpUyU1qG5wuFj0mpHnfjoID7q0qjT7R72+i+9OOzFvf4BjTvROtpkGyeTvcYwSTUrJSrHBFh7EUeFG1dgHGcCV82ktFgAm0OOF14qkGjH1ZrLij7xmZRm2vilHF1n/+2oQFq7lFPvlOBm9tK0TOBXNa4tde1KS+my/rjiqigdKrY5LysFdZrWrGnKjUcoll4AM9xfmbIZGGAekJ9kUOE3pqUElT5LMpBcLFPMoBSuSmy2lf6NFv2/qtzNqqQAw1XqaawqD8LX3qLnYvCDv9ynAg34H/PNPFerar3XzP+Cpf8Xo24n9vpztz18vaW/Sx6Rx2SfxOQVGZJ35IScEkZ+kt+NoLETHAcycMFiYw0a256H5FIEP/4AyUPZ5g==</latexit><latexit sha1_base64="O2HfXf8JditjEs6hyNTp63cMooU=">AAACxXicdZFdaxNBFIYnWz9q/Er10pvBrFBFw25sapqroEIFUSvYtNAN4WT27GbIfKwzs5G4BH+kV/4U75xNIlioB2Z4Oec9Z2aemRaCWxdFvxrBzrXrN27u3mrevnP33v3W3oOR1aVheMq00OZ8ChYFV3jquBN4XhgEORV4Np2/qetnCzSWa/XFLQscS8gVzzgD51OTlgxH4YCGyRyKAkIqwORoHUWeo1ogc9pYqjMaHodJ0vyojQTBvyMNR/uD5/Fg0/a0rr1/IRGUpUyU1qG5wuFj0mpHnfjoID7q0qjT7R72+i+9OOzFvf4BjTvROtpkGyeTvcYwSTUrJSrHBFh7EUeFG1dgHGcCV82ktFgAm0OOF14qkGjH1ZrLij7xmZRm2vilHF1n/+2oQFq7lFPvlOBm9tK0TOBXNa4tde1KS+my/rjiqigdKrY5LysFdZrWrGnKjUcoll4AM9xfmbIZGGAekJ9kUOE3pqUElT5LMpBcLFPMoBSuSmy2lf6NFv2/qtzNqqQAw1XqaawqD8LX3qLnYvCDv9ynAg34H/PNPFerar3XzP+Cpf8Xo24n9vpztz18vaW/Sx6Rx2SfxOQVGZJ35IScEkZ+kt+NoLETHAcycMFiYw0a256H5FIEP/4AyUPZ5g==</latexit><latexit sha1_base64="O2HfXf8JditjEs6hyNTp63cMooU=">AAACxXicdZFdaxNBFIYnWz9q/Er10pvBrFBFw25sapqroEIFUSvYtNAN4WT27GbIfKwzs5G4BH+kV/4U75xNIlioB2Z4Oec9Z2aemRaCWxdFvxrBzrXrN27u3mrevnP33v3W3oOR1aVheMq00OZ8ChYFV3jquBN4XhgEORV4Np2/qetnCzSWa/XFLQscS8gVzzgD51OTlgxH4YCGyRyKAkIqwORoHUWeo1ogc9pYqjMaHodJ0vyojQTBvyMNR/uD5/Fg0/a0rr1/IRGUpUyU1qG5wuFj0mpHnfjoID7q0qjT7R72+i+9OOzFvf4BjTvROtpkGyeTvcYwSTUrJSrHBFh7EUeFG1dgHGcCV82ktFgAm0OOF14qkGjH1ZrLij7xmZRm2vilHF1n/+2oQFq7lFPvlOBm9tK0TOBXNa4tde1KS+my/rjiqigdKrY5LysFdZrWrGnKjUcoll4AM9xfmbIZGGAekJ9kUOE3pqUElT5LMpBcLFPMoBSuSmy2lf6NFv2/qtzNqqQAw1XqaawqD8LX3qLnYvCDv9ynAg34H/PNPFerar3XzP+Cpf8Xo24n9vpztz18vaW/Sx6Rx2SfxOQVGZJ35IScEkZ+kt+NoLETHAcycMFiYw0a256H5FIEP/4AyUPZ5g==</latexit><latexit sha1_base64="O2HfXf8JditjEs6hyNTp63cMooU=">AAACxXicdZFdaxNBFIYnWz9q/Er10pvBrFBFw25sapqroEIFUSvYtNAN4WT27GbIfKwzs5G4BH+kV/4U75xNIlioB2Z4Oec9Z2aemRaCWxdFvxrBzrXrN27u3mrevnP33v3W3oOR1aVheMq00OZ8ChYFV3jquBN4XhgEORV4Np2/qetnCzSWa/XFLQscS8gVzzgD51OTlgxH4YCGyRyKAkIqwORoHUWeo1ogc9pYqjMaHodJ0vyojQTBvyMNR/uD5/Fg0/a0rr1/IRGUpUyU1qG5wuFj0mpHnfjoID7q0qjT7R72+i+9OOzFvf4BjTvROtpkGyeTvcYwSTUrJSrHBFh7EUeFG1dgHGcCV82ktFgAm0OOF14qkGjH1ZrLij7xmZRm2vilHF1n/+2oQFq7lFPvlOBm9tK0TOBXNa4tde1KS+my/rjiqigdKrY5LysFdZrWrGnKjUcoll4AM9xfmbIZGGAekJ9kUOE3pqUElT5LMpBcLFPMoBSuSmy2lf6NFv2/qtzNqqQAw1XqaawqD8LX3qLnYvCDv9ynAg34H/PNPFerar3XzP+Cpf8Xo24n9vpztz18vaW/Sx6Rx2SfxOQVGZJ35IScEkZ+kt+NoLETHAcycMFiYw0a256H5FIEP/4AyUPZ5g==</latexit>
Examples in CSE• Potential and approximation theory (fractional PDEs)
• Uncertainty quantification with Gaussian processes
• Kernel PCA for model reduction
• Diffusion maps / manifold learning
• Compression of covariance matrices
• Construction of priors in inverse problems
• Identifying patterns in spatial fields (structural defects in materials)
• Chemistry (atomization energies, accelerate MD) 15
Summary
• Several methods for approximation, supervised and unsupervised learning
• Rich theory related to reproducing kernels, learning theory, approximation theory, convergence with respect the sample size, stability with respect perturbations
• Wasserman’06, Rasmussen’06, Taylor & Cristianini’05Scholkopf & Smola’02
16
Major shortcomings
• Choice of the kernel function
• Choice of the distance metric
• Computational complexity
17
G(xi, xj) = exp
✓� 1
�2kxi � xjk2
◆
<latexit sha1_base64="6eM8uwkuuCyLEckgluFNVT/N9aM=">AAACY3icdZBdb9MwFIad8DUyPrrBHUKyqJA6xKokrKO7QJrgAsTVkOg2qQ6R6xy3ZrYTbAdaZflP/BokruCG/4HbFYlJcCRLr97znmP7mVRSWBfH34PwytVr129s3Iw2b92+c7eztX1sy9owGLFSluZ0Qi1IoWHkhJNwWhmgaiLhZHL2atk/+QzGilK/d4sKMkWnWnDBqPNW3nkbkfHr3jwXT+f5xx38AhOYV0QCd71dwg1lTdI2xIqpoh/Slpz7JN7FPkvO8xQTI6Yzt4NJFkVR3unG/eRgLzlIcdxP0/3B8JkX+4NkMNzDST9eVRet6yjfCgJSlKxWoB2T1NpxElcua6hxgkloI1JbqCg7o1MYe6mpAps1q0+3+LF3CsxL4492eOX+PdFQZe1CTXxSUTezl7ZxCZ90towse/+M1I4Ps0boqnag2cV9vJbYlXgJEhfCAHNy4QVlRvgnYzajnpjzuCNiQMMXVipFdfGEcKqEXBTAaS2d58nXsvXQ/pDB/xfHaT/x+l3aPXy5xreBHqBHqIcS9BwdojfoCI0QQ1/RN/QD/Qx+hZvhdnj/IhoG65l76FKFD38D0Sq3sw==</latexit><latexit sha1_base64="6eM8uwkuuCyLEckgluFNVT/N9aM=">AAACY3icdZBdb9MwFIad8DUyPrrBHUKyqJA6xKokrKO7QJrgAsTVkOg2qQ6R6xy3ZrYTbAdaZflP/BokruCG/4HbFYlJcCRLr97znmP7mVRSWBfH34PwytVr129s3Iw2b92+c7eztX1sy9owGLFSluZ0Qi1IoWHkhJNwWhmgaiLhZHL2atk/+QzGilK/d4sKMkWnWnDBqPNW3nkbkfHr3jwXT+f5xx38AhOYV0QCd71dwg1lTdI2xIqpoh/Slpz7JN7FPkvO8xQTI6Yzt4NJFkVR3unG/eRgLzlIcdxP0/3B8JkX+4NkMNzDST9eVRet6yjfCgJSlKxWoB2T1NpxElcua6hxgkloI1JbqCg7o1MYe6mpAps1q0+3+LF3CsxL4492eOX+PdFQZe1CTXxSUTezl7ZxCZ90towse/+M1I4Ps0boqnag2cV9vJbYlXgJEhfCAHNy4QVlRvgnYzajnpjzuCNiQMMXVipFdfGEcKqEXBTAaS2d58nXsvXQ/pDB/xfHaT/x+l3aPXy5xreBHqBHqIcS9BwdojfoCI0QQ1/RN/QD/Qx+hZvhdnj/IhoG65l76FKFD38D0Sq3sw==</latexit><latexit sha1_base64="6eM8uwkuuCyLEckgluFNVT/N9aM=">AAACY3icdZBdb9MwFIad8DUyPrrBHUKyqJA6xKokrKO7QJrgAsTVkOg2qQ6R6xy3ZrYTbAdaZflP/BokruCG/4HbFYlJcCRLr97znmP7mVRSWBfH34PwytVr129s3Iw2b92+c7eztX1sy9owGLFSluZ0Qi1IoWHkhJNwWhmgaiLhZHL2atk/+QzGilK/d4sKMkWnWnDBqPNW3nkbkfHr3jwXT+f5xx38AhOYV0QCd71dwg1lTdI2xIqpoh/Slpz7JN7FPkvO8xQTI6Yzt4NJFkVR3unG/eRgLzlIcdxP0/3B8JkX+4NkMNzDST9eVRet6yjfCgJSlKxWoB2T1NpxElcua6hxgkloI1JbqCg7o1MYe6mpAps1q0+3+LF3CsxL4492eOX+PdFQZe1CTXxSUTezl7ZxCZ90towse/+M1I4Ps0boqnag2cV9vJbYlXgJEhfCAHNy4QVlRvgnYzajnpjzuCNiQMMXVipFdfGEcKqEXBTAaS2d58nXsvXQ/pDB/xfHaT/x+l3aPXy5xreBHqBHqIcS9BwdojfoCI0QQ1/RN/QD/Qx+hZvhdnj/IhoG65l76FKFD38D0Sq3sw==</latexit><latexit sha1_base64="6eM8uwkuuCyLEckgluFNVT/N9aM=">AAACY3icdZBdb9MwFIad8DUyPrrBHUKyqJA6xKokrKO7QJrgAsTVkOg2qQ6R6xy3ZrYTbAdaZflP/BokruCG/4HbFYlJcCRLr97znmP7mVRSWBfH34PwytVr129s3Iw2b92+c7eztX1sy9owGLFSluZ0Qi1IoWHkhJNwWhmgaiLhZHL2atk/+QzGilK/d4sKMkWnWnDBqPNW3nkbkfHr3jwXT+f5xx38AhOYV0QCd71dwg1lTdI2xIqpoh/Slpz7JN7FPkvO8xQTI6Yzt4NJFkVR3unG/eRgLzlIcdxP0/3B8JkX+4NkMNzDST9eVRet6yjfCgJSlKxWoB2T1NpxElcua6hxgkloI1JbqCg7o1MYe6mpAps1q0+3+LF3CsxL4492eOX+PdFQZe1CTXxSUTezl7ZxCZ90towse/+M1I4Ps0boqnag2cV9vJbYlXgJEhfCAHNy4QVlRvgnYzajnpjzuCNiQMMXVipFdfGEcKqEXBTAaS2d58nXsvXQ/pDB/xfHaT/x+l3aPXy5xreBHqBHqIcS9BwdojfoCI0QQ1/RN/QD/Qx+hZvhdnj/IhoG65l76FKFD38D0Sq3sw==</latexit>
Computational challenges
• N points
N2 work for matvec
N3 work for factorization
18
Related work — low dimensions• Barnes & Hut’86 — treecodes
• Greengard & Rokhlin’87 — FMM
• Rokhlin’90 — high-frequency FMM
• Hackbush & Novak’89 — panel clustering
• Benderdorf’08 & Hackbush’99,’15 — H-matrices
• Greengard & Gropp’91 — parallel shared memory
• Warren & Salmon’93 — parallel distributed memory
19
Software libraries• FMM3D - Greengard (NYU)
• ExaFMM - Yokota
• BBFMM - Darve
• FastCap2 - White
• ScalFMM - INRIA
• KIFMM, PVFMM - Ying, B., Malhotra
• ChaNGa, GADGET, LAMMPS and other application-specific packages(Many others based on Ewald sums)
20
Idea I: far-field —> low rank
21
xwxj
xi
Idea II: Near/Far field split
22
Idea III: recursion
23
1 2
3 4
Hierarchical matrices, basic idea
24
Questions
• Accurate far-field approximation
• Optimal complexity
• Error bounds
• HPC
• For d=4 these have been answered
25
High dimensions
26
Related work — high dimensions
• Griebel et al’12 — Fast Gauss transform
• Duraiswami’06 — Improved Fast Gauss transform
• Lee, Vuduc & Gray’12 — Treecode (parallel)
• Kondor et al’16 — Wavelets in high dimensions
• Mahoney and Darve’15 — HSS matrices
27
Challenges in high-dimensions• Constructing the far-field approximations
polynomial in ambient-D
• Near-far field decompositionpolynomial in ambient-D
• No scalable algorithms (other than Nystrom)
• Nystrom method assumes low rankprovably not the case with increasing N
28
Randomized linear algebra — Nystrom method
• Low-rank decomposition of G
• Random sampling of O(s) points, s: target rank
• Work
• Error
29
Compute a rank r approximation of offdiagonal block G.G[:,S] is an N-by-r column submatrix of G (skeleton columns).P is an r-by-m matrix of interpolation coefficients.
G ≈ G[:,S]
P
RRQR and TRSM → O(N*m*r + m*r2) complexityCheng, Gimbutas, Martinsson, Rokhlin SISC’05
Approximating off-diagonal blocks
Nm
Nr
Alternative algorithms for IDRandomized methods- Random matrix Ω- Compute ID on product ΩG- Use the same skeletons and P for an ID of G- O(N*m*r + m*r2) or O(N*m*log(r) + m*r2)
Sampling- Sample ℓ rows of G- Compute ID on Gsamples- Use the same skeletons and P for an ID of G- O(ℓ*m*r + m*r2)⇒ Better than random if ℓ << N
Wolfe et al. ACHA’08Martinsson et al. ACHA’11Liberty et al. PNAS’07Halko, Martinsson, Tropp SIREV’11
References:Achlioptas & McSherry JACM’07Drineas, Kannan, Mahoney SISC’06Frieze, Kannan, Vempala JACM’04Deshpande & Vempala ’06Deshpande et al. SIMAX’08and others
Related Work in Sampling•Uniform sampling - Gittens ’11, B. & March’17
• Leverage score sampling - Drineas, Mahoney, Muthukrishnan ’08
• Row ℓ2 norm sampling - Drineas, Kannan, Mahoney ’06
• Adaptive ℓ2 norm sampling - Deshpande & Vempala ’06
• Elementwise sampling - Achlioptas & McSherry’07, Keshavan etal’10
• Equivalent densities - Ying, Biros, Zorin ’04
•Matrix completion also related - Candes, Recht’09
•Deterministic vs. probabilistic
How many samples?• Uniform sampling to construct r-rank approximation to G
G is N-by-m matrix
33
kG� G̃k �r+1
p1 + 6N/`
` > coherence(G)m log�r�
�
�: probability of failure<latexit sha1_base64="t6RysNW8cVU/IAE4z6AlPVhx2Uo=">AAACy3icdZHbbhMxEIa9y6kspxSuEDcWCVJL1bAbmpBwARVcpNxURSK0Uh1FXu84sWqvt7YXCJu95Il4Gt4G54BEJRjJ0u/xP2PP57SQwro4/hWE167fuHlr63Z05+69+w8a2w8/W10aBiOmpTZnKbUgRQ4jJ5yEs8IAVamE0/Ti/fL89AsYK3T+yc0LGCs6zQUXjDqfmjR+koIakWeQOxwXLopaZDHcJ07IDKphTRZEwiX2QayYKjqpzF5SE7+7NK5K9nr4+AUBKWvcImRZ7DV+490q1d8qpmdgIGdQ7wx3scJE6qnvx90O4YayytQVyUA6WhMjpjO3u+rSWudar3FhdEpTIYWbY80xp0KWBqJJoxm3k8FBMujguN3p9Lr9l170ukm3f4CTdryKJtrEyWQ7CEimWan8lExSa88TP+u4osYJJqGOSGmhoOyCTuHcy5wqsONqRbfGz3wmw1wbvzylVfbviooqa+cq9U5F3cxe6cY9vny8tCzP/mkpHe+PK5EXpfOs1vfxUmKn8fLHcCYMMCfnXlBmhH8yZjPq+Tn/rxHxgOEr00rRPHtOOFVCzjPgtJSuIpZvZO2h/SGD/y9GnfagnXzsNA/fbehtoSfoKdpBCXqFDtEROkEjxILHwdvgKPgQHocu/B4u1tYw2NQ8Qlci/PEbzvbb2w==</latexit><latexit sha1_base64="t6RysNW8cVU/IAE4z6AlPVhx2Uo=">AAACy3icdZHbbhMxEIa9y6kspxSuEDcWCVJL1bAbmpBwARVcpNxURSK0Uh1FXu84sWqvt7YXCJu95Il4Gt4G54BEJRjJ0u/xP2PP57SQwro4/hWE167fuHlr63Z05+69+w8a2w8/W10aBiOmpTZnKbUgRQ4jJ5yEs8IAVamE0/Ti/fL89AsYK3T+yc0LGCs6zQUXjDqfmjR+koIakWeQOxwXLopaZDHcJ07IDKphTRZEwiX2QayYKjqpzF5SE7+7NK5K9nr4+AUBKWvcImRZ7DV+490q1d8qpmdgIGdQ7wx3scJE6qnvx90O4YayytQVyUA6WhMjpjO3u+rSWudar3FhdEpTIYWbY80xp0KWBqJJoxm3k8FBMujguN3p9Lr9l170ukm3f4CTdryKJtrEyWQ7CEimWan8lExSa88TP+u4osYJJqGOSGmhoOyCTuHcy5wqsONqRbfGz3wmw1wbvzylVfbviooqa+cq9U5F3cxe6cY9vny8tCzP/mkpHe+PK5EXpfOs1vfxUmKn8fLHcCYMMCfnXlBmhH8yZjPq+Tn/rxHxgOEr00rRPHtOOFVCzjPgtJSuIpZvZO2h/SGD/y9GnfagnXzsNA/fbehtoSfoKdpBCXqFDtEROkEjxILHwdvgKPgQHocu/B4u1tYw2NQ8Qlci/PEbzvbb2w==</latexit><latexit sha1_base64="t6RysNW8cVU/IAE4z6AlPVhx2Uo=">AAACy3icdZHbbhMxEIa9y6kspxSuEDcWCVJL1bAbmpBwARVcpNxURSK0Uh1FXu84sWqvt7YXCJu95Il4Gt4G54BEJRjJ0u/xP2PP57SQwro4/hWE167fuHlr63Z05+69+w8a2w8/W10aBiOmpTZnKbUgRQ4jJ5yEs8IAVamE0/Ti/fL89AsYK3T+yc0LGCs6zQUXjDqfmjR+koIakWeQOxwXLopaZDHcJ07IDKphTRZEwiX2QayYKjqpzF5SE7+7NK5K9nr4+AUBKWvcImRZ7DV+490q1d8qpmdgIGdQ7wx3scJE6qnvx90O4YayytQVyUA6WhMjpjO3u+rSWudar3FhdEpTIYWbY80xp0KWBqJJoxm3k8FBMujguN3p9Lr9l170ukm3f4CTdryKJtrEyWQ7CEimWan8lExSa88TP+u4osYJJqGOSGmhoOyCTuHcy5wqsONqRbfGz3wmw1wbvzylVfbviooqa+cq9U5F3cxe6cY9vny8tCzP/mkpHe+PK5EXpfOs1vfxUmKn8fLHcCYMMCfnXlBmhH8yZjPq+Tn/rxHxgOEr00rRPHtOOFVCzjPgtJSuIpZvZO2h/SGD/y9GnfagnXzsNA/fbehtoSfoKdpBCXqFDtEROkEjxILHwdvgKPgQHocu/B4u1tYw2NQ8Qlci/PEbzvbb2w==</latexit><latexit sha1_base64="t6RysNW8cVU/IAE4z6AlPVhx2Uo=">AAACy3icdZHbbhMxEIa9y6kspxSuEDcWCVJL1bAbmpBwARVcpNxURSK0Uh1FXu84sWqvt7YXCJu95Il4Gt4G54BEJRjJ0u/xP2PP57SQwro4/hWE167fuHlr63Z05+69+w8a2w8/W10aBiOmpTZnKbUgRQ4jJ5yEs8IAVamE0/Ti/fL89AsYK3T+yc0LGCs6zQUXjDqfmjR+koIakWeQOxwXLopaZDHcJ07IDKphTRZEwiX2QayYKjqpzF5SE7+7NK5K9nr4+AUBKWvcImRZ7DV+490q1d8qpmdgIGdQ7wx3scJE6qnvx90O4YayytQVyUA6WhMjpjO3u+rSWudar3FhdEpTIYWbY80xp0KWBqJJoxm3k8FBMujguN3p9Lr9l170ukm3f4CTdryKJtrEyWQ7CEimWan8lExSa88TP+u4osYJJqGOSGmhoOyCTuHcy5wqsONqRbfGz3wmw1wbvzylVfbviooqa+cq9U5F3cxe6cY9vny8tCzP/mkpHe+PK5EXpfOs1vfxUmKn8fLHcCYMMCfnXlBmhH8yZjPq+Tn/rxHxgOEr00rRPHtOOFVCzjPgtJSuIpZvZO2h/SGD/y9GnfagnXzsNA/fbehtoSfoKdpBCXqFDtEROkEjxILHwdvgKPgQHocu/B4u1tYw2NQ8Qlci/PEbzvbb2w==</latexit> B. & March ACHA’17
ASKIT: N-body code, high-D• Randomized Linear Algebra — far field
approximation
• Parallel binary trees — permutation, partitioning
• Nearest neighbors — pruning and sampling
• Treecode / FMM
• MPI / OpenMP / SIMD / GPU acceleration
34
SISC’15,16 ACHA’15 KDD’15 SC’15
IPDPS’15,16,17
Evaluation
35
Evaluation
36
Complexity and error
• Work
• Error
• Nystrom
37
off-diagonal
diagonal
Parallel complexity
38
Points per MPI task n = Np Tree depth D = log N
s
Tree construction (ts + tw) log2p logN + (tw log p) (d+ k)n
Skeletonization tf⇣ns+ log p
⌘s3
Evaluation tsp+ (tw + tf )d k sD n
Gaussian
39
3D, 1M points
64D/20D intr, 1M points
Kernel regression
40
Kernel acceleration
41
Kernel regression scaling
42
MNIST dataset for OCR strong scaling, 8M points d=784
Extensions to arbitrary SPD matrices• Construct HSS approximation for a generic dense SPD
matrix FMM / N-body acceleration but no points are given
• Four components- Permute order to expose low-rank structure — O(N) - Compress blocks — O(N logN)- Fast Matvec — O(N)- HPC implementation (task par/ async + ARM, x86/KNL, GPUs)
43
44
Representative results
Geometry oblivious
• Permute matrix to expose low-rank structure
• Geometry-based algorithms: need distance between indices i,j
• Gram vectorsdistance(i,j) = function(Gii, Gij, Gjj)
Geometry oblivious
• Gram vectors (G is SPD):
• Distances
46
Euclidean k�i � �jk22 Gii +Gjj � 2Gij
Angle sin⇣
�i·�j
k�ik2k�jk2
⌘1� Gij2
GiiGjj<latexit sha1_base64="B2tyN6shuNkRYHVavAO9qNg9hf8=">AAADF3icdVLLbtQwFE3Cq4TXFJZsLGaoCqijJHTKdFdAqCyLRB9SPR05zs2Mp44TbAc0SvMDbGHD17BDbFnyMwg7SRGV4EqRj889vvf6OHHBmdJB8NP1Ll2+cvXaynX/xs1bt+/0Vu8eqLyUFPZpznN5FBMFnAnY10xzOCokkCzmcBifvrT5w/cgFcvFW70sYJKRmWApo0Qbatr7hQsimUhAaBQU2vdxDDMmKk3ikhNZV5zS2n9VUs4SIAKtoQE+Q7iYsylDGy1YIHw2jU6igc3uTivG6idmWSxqo4gsYdAAYew/FzMOyMaaJRQTmEOq1xFOJaFVVxbTJNdd6boyxVveNjnfLOymRliy2Vw/ahqHdpymzK5teBLVVTtLO0rdDIBBJH/u5vv+tNcPhuH2ZrgdoWAYRVuj8VMDtkbhaLyJwmHQRN/pYm+66ro4yWmZGcMoJ0odh8a2SUWkZpRD7eNSQUHoKZnBsYGCZKAmVfNQNXpomASluTSfMbxh/z5RkUypZRYbZUb0XF2olnJ4JyZWYnP/lJQ6HU8qJopSg6Btv7TkSOfIPj5KmASq+dIAQiUzIyM6J8YzbX4RH0sQ8IHmWUZE8hinJGN8mUBKSq4rrNIO1sa0c2fQ/8FBNAwNfhP1d1509q04950HzroTOs+cHee1s+fsO9RN3I/uJ/ez98X76n3zvrdSz+3O3HMuhPfjNy1c9wk=</latexit><latexit sha1_base64="B2tyN6shuNkRYHVavAO9qNg9hf8=">AAADF3icdVLLbtQwFE3Cq4TXFJZsLGaoCqijJHTKdFdAqCyLRB9SPR05zs2Mp44TbAc0SvMDbGHD17BDbFnyMwg7SRGV4EqRj889vvf6OHHBmdJB8NP1Ll2+cvXaynX/xs1bt+/0Vu8eqLyUFPZpznN5FBMFnAnY10xzOCokkCzmcBifvrT5w/cgFcvFW70sYJKRmWApo0Qbatr7hQsimUhAaBQU2vdxDDMmKk3ikhNZV5zS2n9VUs4SIAKtoQE+Q7iYsylDGy1YIHw2jU6igc3uTivG6idmWSxqo4gsYdAAYew/FzMOyMaaJRQTmEOq1xFOJaFVVxbTJNdd6boyxVveNjnfLOymRliy2Vw/ahqHdpymzK5teBLVVTtLO0rdDIBBJH/u5vv+tNcPhuH2ZrgdoWAYRVuj8VMDtkbhaLyJwmHQRN/pYm+66ro4yWmZGcMoJ0odh8a2SUWkZpRD7eNSQUHoKZnBsYGCZKAmVfNQNXpomASluTSfMbxh/z5RkUypZRYbZUb0XF2olnJ4JyZWYnP/lJQ6HU8qJopSg6Btv7TkSOfIPj5KmASq+dIAQiUzIyM6J8YzbX4RH0sQ8IHmWUZE8hinJGN8mUBKSq4rrNIO1sa0c2fQ/8FBNAwNfhP1d1509q04950HzroTOs+cHee1s+fsO9RN3I/uJ/ez98X76n3zvrdSz+3O3HMuhPfjNy1c9wk=</latexit><latexit sha1_base64="B2tyN6shuNkRYHVavAO9qNg9hf8=">AAADF3icdVLLbtQwFE3Cq4TXFJZsLGaoCqijJHTKdFdAqCyLRB9SPR05zs2Mp44TbAc0SvMDbGHD17BDbFnyMwg7SRGV4EqRj889vvf6OHHBmdJB8NP1Ll2+cvXaynX/xs1bt+/0Vu8eqLyUFPZpznN5FBMFnAnY10xzOCokkCzmcBifvrT5w/cgFcvFW70sYJKRmWApo0Qbatr7hQsimUhAaBQU2vdxDDMmKk3ikhNZV5zS2n9VUs4SIAKtoQE+Q7iYsylDGy1YIHw2jU6igc3uTivG6idmWSxqo4gsYdAAYew/FzMOyMaaJRQTmEOq1xFOJaFVVxbTJNdd6boyxVveNjnfLOymRliy2Vw/ahqHdpymzK5teBLVVTtLO0rdDIBBJH/u5vv+tNcPhuH2ZrgdoWAYRVuj8VMDtkbhaLyJwmHQRN/pYm+66ro4yWmZGcMoJ0odh8a2SUWkZpRD7eNSQUHoKZnBsYGCZKAmVfNQNXpomASluTSfMbxh/z5RkUypZRYbZUb0XF2olnJ4JyZWYnP/lJQ6HU8qJopSg6Btv7TkSOfIPj5KmASq+dIAQiUzIyM6J8YzbX4RH0sQ8IHmWUZE8hinJGN8mUBKSq4rrNIO1sa0c2fQ/8FBNAwNfhP1d1509q04950HzroTOs+cHee1s+fsO9RN3I/uJ/ez98X76n3zvrdSz+3O3HMuhPfjNy1c9wk=</latexit><latexit sha1_base64="B2tyN6shuNkRYHVavAO9qNg9hf8=">AAADF3icdVLLbtQwFE3Cq4TXFJZsLGaoCqijJHTKdFdAqCyLRB9SPR05zs2Mp44TbAc0SvMDbGHD17BDbFnyMwg7SRGV4EqRj889vvf6OHHBmdJB8NP1Ll2+cvXaynX/xs1bt+/0Vu8eqLyUFPZpznN5FBMFnAnY10xzOCokkCzmcBifvrT5w/cgFcvFW70sYJKRmWApo0Qbatr7hQsimUhAaBQU2vdxDDMmKk3ikhNZV5zS2n9VUs4SIAKtoQE+Q7iYsylDGy1YIHw2jU6igc3uTivG6idmWSxqo4gsYdAAYew/FzMOyMaaJRQTmEOq1xFOJaFVVxbTJNdd6boyxVveNjnfLOymRliy2Vw/ahqHdpymzK5teBLVVTtLO0rdDIBBJH/u5vv+tNcPhuH2ZrgdoWAYRVuj8VMDtkbhaLyJwmHQRN/pYm+66ro4yWmZGcMoJ0odh8a2SUWkZpRD7eNSQUHoKZnBsYGCZKAmVfNQNXpomASluTSfMbxh/z5RkUypZRYbZUb0XF2olnJ4JyZWYnP/lJQ6HU8qJopSg6Btv7TkSOfIPj5KmASq+dIAQiUzIyM6J8YzbX4RH0sQ8IHmWUZE8hinJGN8mUBKSq4rrNIO1sa0c2fQ/8FBNAwNfhP1d1509q04950HzroTOs+cHee1s+fsO9RN3I/uJ/ez98X76n3zvrdSz+3O3HMuhPfjNy1c9wk=</latexit>
Gij = �i · �j<latexit sha1_base64="prE+m2VpcaPRhURxAThTUYxEoRs=">AAACT3icdZBNb9QwEIad5as1X1s4crFYISEOqyR0l+0BqYIDHIvEtpXWUTRxJl23thNsB7SK9p/wa7jChSO/hBvCuw0SlWAkS49n3vGM36JR0vk4/hENrl2/cfPWzi69fefuvfvDvQfHrm6twLmoVW1PC3CopMG5l17haWMRdKHwpLh4vamffETrZG3e+1WDmYYzIyspwIdUPpzyBqw0JRrP4sZTyheMvsk7eb5mLxlvljKXjIuy9peXc8ozSmk+HMXj5GA/OUhZPE7T6WT2PMB0kkxm+ywZx9sYkT6O8r0o4mUtWh0GCQXOLZIwLuvAeikUrilvHTYgLuAMFwENaHRZt/3gmj0JmZJVtQ0nLLrN/t3RgXZupYug1OCX7sprlcIPJttINrV/SlpfzbJOmqb1aMTlvKpVzNdsYxorpUXh1SoACCvDykwswYLwwVrKLRr8JGqtwZTPeAVaqlWJFbTKd9xVPa6DaX+cYf+H43ScBH6Xjg5f9fbtkEfkMXlKEvKCHJK35IjMiSCfyRfylXyLvkc/o1+DXjqIenhIrsRg9zfpt7GM</latexit><latexit sha1_base64="prE+m2VpcaPRhURxAThTUYxEoRs=">AAACT3icdZBNb9QwEIad5as1X1s4crFYISEOqyR0l+0BqYIDHIvEtpXWUTRxJl23thNsB7SK9p/wa7jChSO/hBvCuw0SlWAkS49n3vGM36JR0vk4/hENrl2/cfPWzi69fefuvfvDvQfHrm6twLmoVW1PC3CopMG5l17haWMRdKHwpLh4vamffETrZG3e+1WDmYYzIyspwIdUPpzyBqw0JRrP4sZTyheMvsk7eb5mLxlvljKXjIuy9peXc8ozSmk+HMXj5GA/OUhZPE7T6WT2PMB0kkxm+ywZx9sYkT6O8r0o4mUtWh0GCQXOLZIwLuvAeikUrilvHTYgLuAMFwENaHRZt/3gmj0JmZJVtQ0nLLrN/t3RgXZupYug1OCX7sprlcIPJttINrV/SlpfzbJOmqb1aMTlvKpVzNdsYxorpUXh1SoACCvDykwswYLwwVrKLRr8JGqtwZTPeAVaqlWJFbTKd9xVPa6DaX+cYf+H43ScBH6Xjg5f9fbtkEfkMXlKEvKCHJK35IjMiSCfyRfylXyLvkc/o1+DXjqIenhIrsRg9zfpt7GM</latexit><latexit sha1_base64="prE+m2VpcaPRhURxAThTUYxEoRs=">AAACT3icdZBNb9QwEIad5as1X1s4crFYISEOqyR0l+0BqYIDHIvEtpXWUTRxJl23thNsB7SK9p/wa7jChSO/hBvCuw0SlWAkS49n3vGM36JR0vk4/hENrl2/cfPWzi69fefuvfvDvQfHrm6twLmoVW1PC3CopMG5l17haWMRdKHwpLh4vamffETrZG3e+1WDmYYzIyspwIdUPpzyBqw0JRrP4sZTyheMvsk7eb5mLxlvljKXjIuy9peXc8ozSmk+HMXj5GA/OUhZPE7T6WT2PMB0kkxm+ywZx9sYkT6O8r0o4mUtWh0GCQXOLZIwLuvAeikUrilvHTYgLuAMFwENaHRZt/3gmj0JmZJVtQ0nLLrN/t3RgXZupYug1OCX7sprlcIPJttINrV/SlpfzbJOmqb1aMTlvKpVzNdsYxorpUXh1SoACCvDykwswYLwwVrKLRr8JGqtwZTPeAVaqlWJFbTKd9xVPa6DaX+cYf+H43ScBH6Xjg5f9fbtkEfkMXlKEvKCHJK35IjMiSCfyRfylXyLvkc/o1+DXjqIenhIrsRg9zfpt7GM</latexit><latexit sha1_base64="prE+m2VpcaPRhURxAThTUYxEoRs=">AAACT3icdZBNb9QwEIad5as1X1s4crFYISEOqyR0l+0BqYIDHIvEtpXWUTRxJl23thNsB7SK9p/wa7jChSO/hBvCuw0SlWAkS49n3vGM36JR0vk4/hENrl2/cfPWzi69fefuvfvDvQfHrm6twLmoVW1PC3CopMG5l17haWMRdKHwpLh4vamffETrZG3e+1WDmYYzIyspwIdUPpzyBqw0JRrP4sZTyheMvsk7eb5mLxlvljKXjIuy9peXc8ozSmk+HMXj5GA/OUhZPE7T6WT2PMB0kkxm+ywZx9sYkT6O8r0o4mUtWh0GCQXOLZIwLuvAeikUrilvHTYgLuAMFwENaHRZt/3gmj0JmZJVtQ0nLLrN/t3RgXZupYug1OCX7sprlcIPJttINrV/SlpfzbJOmqb1aMTlvKpVzNdsYxorpUXh1SoACCvDykwswYLwwVrKLRr8JGqtwZTPeAVaqlWJFbTKd9xVPa6DaX+cYf+H43ScBH6Xjg5f9fbtkEfkMXlKEvKCHJK35IjMiSCfyRfylXyLvkc/o1+DXjqIenhIrsRg9zfpt7GM</latexit>
GOFMM compression algo outline
• Distance(i,j) defined using matrix entries
• Symmetrically permute matrix using binary tree construction
• Use randomized nearest neighbors to find Neighbors(i) - Direct interactions (strong admissibility)
- Sampling to construct far-field approximation
47
48
GOFMM vs Other implementations
MATVEC Dependencies
50
Computational primitives
• Geometric — distance calculation / projections
• Analytic — special functions / fast transforms
• Algebraic — BLAS / QR / SVD / Cholesky
• Combinatorial — hash / sort / merge / select / search
• Memory — permute / pack / gather / scatter
51
Multiple architectures
52
GOFMM on Stampede 2
53
Summary
• Kernel methods in CSE
• Generalization to SPD matrices and high dimensions
• References: IPDPS’17, SC’17, SISC’16, ACHA’17, SC’18
54
