Difference Of Convex (DC) Functions and DC Programming

Songcan Chen

Outline

1. A Brief History2. DC Functions and their Property3. Some examples4. DC Programming5. Case Study6. Our next work

1. A Brief History• 1964, Hoang Tuy, (incidentally in his convex

optimization paper),• 1979, J. F. Toland, Duality formulation• 1985, Pham Dinh Tao, DC Algorithm• 1990 --, Pham Dinh Tao, et al• …

H. Tuy, Concave programming under linear constraints, Translated Soviet Mathematics 5 (1964), 1437-1440.J. F. Toland, On subdi®erential calculus and duality in nonconvex optimization,Bull. Soc. Math. France, M¶emoire 60 (1979), 173-180.Pham Dinh Tao, Duality in d.c. (di®erence of convex functions) optimization.Subgradient methods, Trends in Mathematical Optimization, International Series of Numer Math. 84 (1988), Birkhauser, 277-293.

Applicable Fields

• For smooth/non-smooth and convex/non-convex optimization problems, especially,

• For large-scale DC problems Robust and efficient in solving!

Hence• Machine learning (Clustering, Kernel

optimization, Feature selection,…)• Engineering (Quality control,...)• …

2.1 DC Functions• Definition 2.1. Let C be a convex subset of Rn.

A real-valued function f : C R is called DC on C, if there exist two convex functions g, h: C R such that f can be expressed in the form

f(x)=g(x)-h(x) (1)h(x) convex -h(x) concave.

If C= Rn, then f is simply called a DC function.

Notice: DC representation for f is NOT unique, in fact, can have infinite decompositions!

2.2 Their Properties

Let f and fi, i = 1 , . . . , m, be DC functions. Then, the following functions are also DC:

1)1

, , 1, 2,..., .m

i i ii

f R i mλ λ=

∈ =∑

2)1,2,...,1,2,...,

max { } min { }i ii mi mf and f

==

3) |f(x)|

4)1

m

ii

f=∏

2.2 Their Properties (Cont’d)

1) Every function f: Rn R whose second partial derivatives are continuous everywhere is DC.

2) Let C be a compact convex subset of Rn. Then for any continuous function c: C R and for any ε >0, there exists a DC function f: C R such that

|c ( x ) - f ( x ) | < ε, for any x in C.3) Let f: Rn R be DC, and let g: R R be convex.

Then, the composite function (g o f )(x)=g(f(x)) is DC.

3. Some simple examples

1) xtQx, Q=A-B, A and B are positive semi-definite.2) xty, 3) Let dM be a distance function, then

dM(x)=inf{||x-y||: y in M}.

Proof of 3)

4. DC Programming

• 4.1 Primal Problem• 4.2 Dual Problem• 4.3 DC Algorithm (DCA)

4.1 Primal Problem

• A general form

0inf{ ( ) : , ( ) 0, 1,2,..., }nif x x X R f x i mα = ∈ ⊆ ≤ =

Where fi=gi-hi, i=1,2,…,m are DC functions and X is a closed convex subset of Rn.

Constrained (closed) Set X can be represented by a convex indicator function which is added to the g0(x) (f0=g0-h0): IX(x)=0 if x in X, +∞ otherwise.

(Pdc)

4.1 Primal Problem (Cont’d)

When X is constrained by a set of linear inequality equations and the objective function is linear, the optimization problem is called polyhedral DC, solving it amounts to solving a linear programming. For example, selecting features based on SVM and l0 norm [5].

Notations1) The conjugate function g* of g is defined by

2) Support Domain of g(x)

3) ε-subdifferential of g(x) at x0, when ε=0, simply called

subdifferential.

Notations (Cont’d)Support Domain of the subdifferential ∂g

Range Domain of ∂g

4.2 Dual Problem

Using the definition of conjugate functions, we have

with

4.2 Dual Problem (Cont’d)Dual Formulation:

Where Y= dom ∂h*.

A perfect symmetry exists between the primal and its dual programs (P) and (D):

the dual program to (D) is exactly (P).

4.2 Dual Problem (Cont’d)• The necessary local optimality condition for Pdc, is

∂h(x*) in ∂g(x*)

• A point that x* that verifies the generalized Kuhn-Tucker condition

∂h(x*)∩∂g(x*) ≠ ∅

is called a critical point of g−h.

4.3 DCA

Affine majorization of the concave part –h(x)!

4.3 DCA (Cont’d)

• Different decompositions → thus make trade-off between Complexity of each step,• number of iterations.• Local convergence, empirically: “good” optima.

4.3 DCA (Cont’d)Convergence properties

– DCA is a descent method (i.e., the sequences{g(xk) − h(xk)} and {h*(yk) − g*(yk)} are both

decreasing) without linesearch;– If the optimal value α of problem (Pdc) is finiteand the infinite sequences {xk} and {yk} are bounded, then every limit point x* (resp. y*) of {xk} (resp. {yk}) is a critical point of g − h (resp. h* − g*), i.e., ∂h(x*) ∩ ∂g(x*) = ∅ (resp. ∂h*(y*) ∩ ∂g*(y*) = ∅).– DCA has a linear convergence for general DC programs.

5. Case Study

• 5.1 Fuzzy c-means Clustering • 5.2 Feature Selection and Classification

5.1 Fuzzy c-means Clustering

FCM (Cont’d)

• How to be changed to DC1) g and h?2) X – a convex set of variables (U, V)?

Characterization of Convex Set• From the centers’ solution V={vi ，i=1,2,…,c},

Leading to the Euclidean ball Ri with radius r. It is convex!

In fact, ‖vi‖≦max{‖xk‖, k=1, 2, …, n} for all i.

• For U, let

Characterization of Convex Set

• Constraints

Leading to the Euclidean sphere Sk with radius 1. It is NOT convex.

Equivalent Formulation to FCM

A DC decomposition of the above objective function

DC Formulation

For all

with

A Question: is H(T, V) unconditionally convex? No!

Condition ensuring H(T,V)

Notice here B denotes a Ball and thus is convex!

In fact, alpha can be 2 max{‖xk‖, k=1, 2, …, n} !

Proof (1)Proof: from

Just prove the function are convex for all i and k

Define

Its Hessian

For all (x, y):

Proof (2)

So f(x, y) is convex on

Proof (3)implying

is convex on

Further hi,k is convex

Finally, the function H(T,V ) is convex on B × C.

Proof (4)For all T ∈ B (closed ball) and a given matrix V ∈ C, the function J2m(T,V) is concave in variable T (since H(T,V ) is convex). Hence S (sphere, i.e., boundary) contains minimizers (reaching at boundary) of J2m(T,V ) on B, i.e.,

DC Formulation

where

Solving FCM by DCA (1)

A key: construct two sequences (Yl, Zl) ∈ ∂H (Tl, Vl) and

H is differentiable and its gradient at the point (Tl, Vl):

(14)

Algorithm 1. DCA applied to FCM

Solving FCM by DCA (2)

More precisely:

Accelerating DCA -- FCM-DCM (1)

Accelerating DCA -- FCM-DCM (2)

Accelerating DCA -- FCM-DCM (3)

Two phase algorithm 3

Partial results

(a) (resp. (b)) corresponds to the original image without (resp. with) noise;(c) (resp. (d)) represents the resulting image given by FCM Algorithm without

(resp. with) spatial information.(e) represents the resulting image given by Algorithm 2 without spatial information(f) (resp. (g)) represents the resulting image given by Algorithm 3 without

(resp. with) spatial information.

5.2 Feature Selection and Classification

Formulation of problem

• Given two finite point sets A and B in Rn represented by the matrices A ∈ Rm×n and B ∈ Rk×n, respectively. Discriminate these sets by a separating plane (w∈ Rn, γ∈ R)

which uses as few features as possible.

The optimization problem

Where yi, i=1,2,…,m and zj, j=1,2,…,k are non-negative slack variables, e is a vector with all entries of 1.

The zero-norm:

P. S. Bredley and O. L. Mangasarian, Feature Selection via concave minimization and support vector machines, ICML’08.

Optimization Difficulty of Zero-Norm

• Discontinuity at the origin• NP-Hard

Solution: Approximation to Zero-norm!for example,

Approximate Zero-norm

where

Reformulation of the optimization

where K is the polyhedral convex set defined by:

A DC decomposition of the approximation

where

They are both convex!

Illustration

DC Decomposition of the Objective

where:

A Final Formulation: FSC-DC

DCA Revisited

DCA for FSC

(15) And (16)

Feasible Domain

An Important Theorem

Experimental Result

Selection of the λ: CV

6. Our next work: Applying DCP

• (Structured) AUC-FSC-DC (based SVM)• Asymmetric (SVM) FSC-DC• FSC based on DR and LR• KFCM-DC and Combination• NMF-DC and Combination • SPP-DC (mainly in Sparse solving)• …

Reference[1] R. Horst, N. V. THOAI, DC Programming: Overview, JOURNAL OF OPTIMIZATION THEORY AND

APPLICATIONS: Vol. 103, No. 1, pp. 1-43, 1999.[2] PHAM DINH TAO AND LE THI HOAI AN, CONVEX ANALYSIS APPROACH TO D.C.

PROGRAMMING, ACTA MATHEMATICA VIETNAMICA, 22(1) 1997, pp. 289-355.[3] Le Thi Hoai An, D.C. Programming for Solving a Class of Global Optimization Problems via

Reformulation by Exact Penalty, C. Bliek et al. (Eds.): COCOS 2002, LNCS 2861, pp. 87–101, 2003.

[4] Julia Neumann, Christoph Schnorr, and Gabriele Steidl, SVM-Based Feature Selection by Direct Objective Minimisation, C.E. Rasmussen et al. (Eds.): DAGM 2004, LNCS 3175, pp. 212–219, 2004.

[5] Hoai An Le Thi, Hoai Minh Le, Van Vinh Nguyen, Tao Pham Dinh, A DC programming approach for feature selection in support vector machines learning, Adv Data Anal Classif (2008) 2:259–278.

[6] Le Thi Hoai An, M. Taye Belghiti, Pham Dinh Tao, Feature Selection via DC Programming and DCA, J Glob Optim (2007) 37:593–608.

[7] Hoai An Le Thi, Hoai Minh Le, Tao Pham Dinh, Fuzzy clustering based on nonconvex optimisationapproaches using difference of convex (DC) functions algorithms, ADAC (2007) 1:85–104.

[8] Le ThiHoai An, M. Tayeb Belghiti, Pham Dinh Tao, A new efficient algorithm based on DC programming and DCA for Clustering, J Glob Optim (2007) 37:593–608.

[9] LE THI Hoai An, LE Hoai Minh, NGUYEN Van Vinh, PHAMDINHTao, Combined Feature Selection and Classification using DCA, IEEE IC on Research, Innovation and Vision for the Future (RIVF), 2008, pp:233-239.

[10] Le Thi Hoai An, Van Vinh Nguyen, Samir Ouchani: Gene Selection for Cancer Classification Using DCA. ADMA 2008: 62-72 .

[11] Le Thi Hoai An, Le Hoai Minh, Nguyen Trong Phuc, Pham Dinh Tao: Noisy Image Segmentation by a Robust Clustering Algorithm Based on DC Programming and DCA. Industrial Conf. DM 2008: 72-86.

• Thanks a lot!

• Q & A