Learning Kernel Classifiers: R. Herbrich (Ed.); MIT Press, 2002, ISBN: 0-262-08306-X, $40

1
Learning Kernel Classi®ers R. Herbrich (Ed.); MIT Press, 2002, ISBN: 0-262-08306-X, $40 Kernel methods are currently a hot topic in machine learning. They are based on the idea of an implicitly computed feature mapping which allows one to implement `nonlinear' classi®ers using linear techniques. Several obvious factors have contributed to their popularity: they are theoretically far more tractable than methods such as neural networks, their statistical performance is very good, and there are a large number of ef®cient algorithms and implementations. Their popularity has given rise to many publications: several special issues of journals (including Machine Learn- ing, Journal of Machine Learning Research, and IEEE Transactions on Neural Networks); several edited collec- tions of papers (based on NIPS workshops, and published by MIT press); a comprehensive public website http:// www.kernel-machines.org and a number of monographs. The book under review is the second or third (depending on how you count) to come out of the learning with kernels community in the last couple of years. Learning Kernel Classi®ers is based on the author's PhD Thesis, but if the author did not reveal this, a reader could be forgiven for not noticing. It suffers none of the usual vices such books typically have (lack of balance and perspective, and poor writing style). The book focuses on classi®cation learning and comprises ®ve main chapters (some 200 pages) and a number of technical appendices (130 pages). It covers both algorithmic and theoretical aspects. Chapter 1 sets up the learning problems to be considered in an easy going way and delineates the content of the rest of the book. Chapter 2 introduces Kernel Classi®ers within a classical machine learning perspective covering such topics as the principal of risk minimization, essentials of kernels, the representer theorem, support vector machines and adaptive margin machines. Chapter 3 provides a complementary (Bayesian) perspec- tive on Kernel Classi®ers, introducing the essentials of Gaussian processes (the Bayesian viewpoint of kernels), relevance vectors machines, and Bayes point machines. Chapter 4 moves onto general learning theory, covering in an elegant and concise way classical Vapnik±Chervonekis theory, structural risk minimization, the luckiness frame- work and margin bounds. Chapter 5 considers theoretical approaches to the analysis of learning algorithms that are more algorithm speci®c. The author provides a very nice presentation of the PAC- Bayesian framework developed recently by David McAllester and extends it to provide bounds for large margin classi®ers. The chapter also covers compression bounds and algorithmic stability bounds. Overall this book is very clearly written. The author has taken considerable care to stick to a coherent system of notation and has managed to explain complex ideas in a concise and effective way. He has achieved considerable precision throughout which is of great help to a beginning student. The author has focused on the essence of the various techniques without undue detail (but this does mean it is not suitable in itself as a cookbook). The book contains a very nice balance between algorithms and theory, and the focus of the theory is on really why the various algorithms work. The carefully written code on the web site will help the reader who wishes to experiment with the algorithms discussed in the book get going quite quickly. The book contains new results due to the author (for example the PAC-Bayesian margin bound). The book can be read sequentially (with pleasure!), which is not true of all technical books. Whilst the author has clearly gone to considerable effort to explain the various ideas as simply as possible, he has not `dumbed down' the results; indeed some of the technical arguments are quite sophisticated and intricate. All the necessary technical argumentation is there (much of it in the appendices, which maintain the high level of writing quality apparent in the main text). As well as detailed technical proofs, the appendices include Pseudocode for all of the algorithms discussed in the text. Working code in the R language is available on the associated web site http://www.learning-kernel-classi®ers. org. The web site also contains a list of errata (mostly typos). The book does not try to be encyclopedic, and that is one of its strengths. Reading Herbrich's book would be a good preparation for the more comprehensive, contemporaneous Learning with Kernels, by Bernhard Scholkopf and Alex Smola (MIT Press, 2002) http://www.learning-with-kernels. org. I have recommended the book to my own PhD students to learn about the ®eld of kernel methods to a positive response (they have said it is a very good book for beginners). I highly recommend it to any serious student of kernel methods in machine learning with a reasonable mathematical background. Robert Williamson * Department of Telecommunications Engineering, Research School of Information Sciences and Engineering, Australian National University, Canberra 0200, Australia E-mail address: [email protected] Book reviews / Neural Networks 15 (2002) 927±930 930 PII: S0893-6080(02)00086-2 * Tel.: 161-2-6125-0079; fax: 161-2-6125-8623.

Transcript of Learning Kernel Classifiers: R. Herbrich (Ed.); MIT Press, 2002, ISBN: 0-262-08306-X, $40

Page 1: Learning Kernel Classifiers: R. Herbrich (Ed.); MIT Press, 2002, ISBN: 0-262-08306-X, $40

%������� &���� ��$$�'��$� �������" �= ��C F#8 ���� !! � #$�%? !& ) &!7*!)&M�

-0!

E����� ���"� ��� �/������� � "�� ��.�� �� ���"���

��������� 8"�� ��� ��� �� �"� � �� �3 �� ��.�������

���./�� 3���/�� ��..��� �"��" ����� ��� �� ��.������

4���������5 ����;�� /��� ������ ���"��</�� $������

�����/ 3����� "��� �������/�� �� �"��� .�./������? �"��

��� �"����������� 3�� ���� ��������� �"�� ���"� /�" �

��/��� �������� �"��� ��������� .��3������� � ���� ��� �

�� �"��� ��� � ����� �/���� �3 �3;����� �������"� ��

��.������������

8"��� .�./������ "� ����� ��� �� ���� ./���������?

������ .����� �/� �3 9�/���� �����/ ���1�$��� <����#

�)� 2������ �� 1�$��� <�����) *�����$�� �� �"""

�����$���� �� ������ ���������C ������ � ��� ������&

���� �3 .�.�� ���� �� %#�$ ����"�.� �� ./���"�

�� F#8 .���C � ���.��"����� ./���� ������ "��.?BB

����������&���"������� �� � �/���� �3 �������."�

8"� ���� /� �� ������ � �"� ���� �� �"�� � �.�� ���

�� "�� ��/ ��/��� �� ���� �/� �3 �"� �������� ���" ������

����/���� �� �"� ��� ��/.�� �3 �����

<�����) 5����� %����=��� � ��� �� �"� �/�"��5 �"K

8"��� �/� �3 �"� �/�"�� � ��� ������ �"�� � ��� �� ��/� ��

3������� 3�� ��� ��������� #� /33�� ���� �3 �"� //�� ����

/�" ���� ��.������ "��� ����� �3 ������� �� .��.�������

�� .��� ������� ������

8"� ���� 3��/� �� ����;������ �������� ��

���.��� ;�� ���� �"�.��� ���� !! .���� �� �

�/���� �3 ���"����� �..�� ��� �(*! .����� #� �����

���" �������"��� �� �"��������� �.����

�"�.��� ( �� /. �"� �������� .������ �� �� ���� ���

�� �� ��� ����� ��� �� �������� �"� ������� �3 �"� ��� �3

�"� �����

�"�.��� ����� /�� E����� ����;�� ���"�� � �������

���"��� �������� .��.������ �������� /�" ��.�� � �"�

.�����.�� �3 ��� ������:������ ������� �3 ������� �"�

��.������� �"������ /..��� ������ ���"��� �� � �.����

������ ���"����

�"�.��� * .���� � � ���.��������� ��������� .��.��&

���� �� E����� ����;��� ����� /���� �"� ������� �3

G�/��� .����� ��"� ������� ����.���� �3 ��������

��������� ������ ���"���� �� ���� .���� ���"����

�"�.��� 0 ���� ���� ������� �������� �"����� ��������

�� �� ������� �� ��������� ������� N�.���A�"��������

�"����� ��/��/��� ��� ������:������ �"� �/����� 3����&

���� �� ������ ��/� �

�"�.��� ' ���� �� �"��������� �..����"� �� �"� ������

�3 �������� �������"� �"�� ��� ���� �������"� .���;��

8"� �/�"�� .���� � � ���� ���� .���������� �3 �"� �1�&

������� 3�������� �����.� �������� �� K���

F�1������ �� �2��� �� �� .���� � ��/� 3�� �����

������ ����;��� 8"� �"�.��� ��� ����� ���.�����

��/� �� �������"��� �������� ��/� �

6������ �"� ���� � ���� ������� �������� 8"� �/�"�� "�

����� ���� ������ ���� �� ���� �� � ��"����� ���� �3

�������� �� "� ������ �� �2.���� ���.��2 � �� �� �

������ �� �33������ ���� �� "� ��"���� ���� ������

.������� �"��/�"�/� �"��" � �3 ����� "��. �� � ���������

�/ ����

8"� �/�"�� "� 3��/� �� �"� ����� �3 �"� �����/

���"��</� ���"�/� /� /� ����� ��/� �"� �� ���� �� �

��� /������ �� ����3 � � ���������� 8"� ���� ������� �

���� ���� ������� ������� �������"� �� �"����� �� �"�

3��/ �3 �"� �"���� � �� ������ �"� �"� �����/ �������"�

����� 8"� ����3/��� ������� �� � �� �"� ��� ��� ���� "��.

�"� ��� �� �"� ��"� �� �2.������� ���" �"� �������"�

��/� �� �"� ���� ��� ����� </��� </������

8"� ���� ������� ��� ��/�� /� �� �"� �/�"�� �3��

�2��.�� �"� �1�&������� ������ ��/� �� 8"� ���� ���

�� ��� �</�������� ����" .���/��O�� �"��" � ��� ��/� �3 ���

���"����� �����

I"��� �"� �/�"�� "� ������� ���� �� ���� ������ �33���

�� �2.���� �"� �����/ � �� � ��.�� � .������ "� "� ���

4 /��� ���5 �"� ��/��C �� �� ��� �3 �"� ���"�����

���/���� ��� </��� �."������� �� ���������� 1�� �"�

������� ���"����� ���/��������� � �"��� ��/�" �3 �� ��

�"� �..�� ���� �"��" �������� �"� "��" ����� �3 �������

</����� �..����� �� �"� ���� ��2���

1 ���� � ������ ���"����� .���3� �"� �..�� ���

����/ � ��/ ��� � 3�� ��� �3 �"� �������"� ��/� ��

�"� ��2�� I������ �� � �� �"� ����/��� � ��������� �� �"�

������� ��� ��� "��.?BB������������&������&����;���

���� 8"� ��� ��� ��� ������� � ��� �3 ������ ������

��.���

8"� ���� �� ��� ��� �� �� �������.� ��� �� �"�� � ���

�3 �� ������"� �� ��� �������"5 ���� ��/� �� � ���

.��.������� 3�� �"� ���� ���.��"������ ������.������/

@������� ���" E������ �� ����"�� $�"����.3 �� 1��2

$���� �F#8 ���� !! � "��.?BB������������&���"&�������

����

# "��� �������� � �"� ���� �� �� ��� �"K �/ ���

�� ����� ���/� �"� ;�� �3 ������ ���"� �� � .������

��.��� ��"�� "��� �� �� � � ���� ��� ���� 3��

���������� # "��"�� �������� �� �� ��� ����/ �/ ���

�3 ������ ���"� �� ���"��� �������� ���" � ���������

���"�������� �������/� �

����� I��������P

8�&������� �� ���$�����$����� "�)�����)- *�����$�

+$���� �� ���������� +$��$�� ��� "�)�����)-

0�������� ������� 9������ -

%��6���� ����- 0�������

"#��� �������; ���L����"�������� /

���� ������ � ������ �������� �� ������ ��� �!�+*!

�##? $!7+*&)!7!�! �!!!7)&

P 8���? �)(& &)( '&!!,+C 3�2? �)(& &)( '&7) *�