Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley

Practical Machine Learning, Fall 2009

Duchi (UC Berkeley)

Convex Optimization for Machine Learning

Fall 2009

1 / 53

Outline What is Optimization Convex Sets Convex Functions Convex Optimization Problems Lagrange Duality Optimization Algorithms Take Home Messages

Duchi (UC Berkeley)


Fall 2009

2 / 53

What is Optimization

What is Optimization (and why do we care?)

Duchi (UC Berkeley)


Fall 2009

3 / 53


What is Optimization?

◮

Finding the minimizer of a function subject to constraints: minimize f0 (x) x

s.t. fi (x) ≤ 0, i = {1, . . . , k}

hj (x) = 0, j = {1, . . . , l}

Duchi (UC Berkeley)


Fall 2009

4 / 53


What is Optimization?

◮

Finding the minimizer of a function subject to constraints: minimize f0 (x) x

s.t. fi (x) ≤ 0, i = {1, . . . , k}

hj (x) = 0, j = {1, . . . , l}

◮

Example: Stock market. “Minimize variance of return subject to getting at least $50.”

Duchi (UC Berkeley)


Fall 2009

4 / 53


Why do we care? Optimization is at the heart of many (most practical?) machine learning algorithms. ◮

Linear regression:

minimize kXw − yk2 w

◮

Classification (logistic regresion or SVM): minimize w

n X i=1

log 1 + exp(−yi xTi w)

or kwk2 + C

Duchi (UC Berkeley)

n X i=1

ξi s.t. ξi ≥ 1 − yi xTi w, ξi ≥ 0.


Fall 2009

5 / 53


We still care... ◮

Maximum likelihood estimation: maximize θ

◮

w

X i≺j

log 1 + exp(wT xi − wT xj )

k-means: minimize J(µ) = µ1 ,...,µk

◮

log pθ (xi )

i=1

Collaborative filtering: minimize

◮

n X

k X X

j=1 i∈Cj

kxi − µj k2

And more (graphical models, feature selection, active learning, control) Duchi (UC Berkeley)


Fall 2009

6 / 53


But generally speaking... We’re screwed. ◮ Local (non global) minima of f0 ◮ All kinds of constraints (even restricting to continuous functions): h(x) = sin(2πx) = 0

250 200 150 100 50 0 −50 3 2

3

1

2 0

1 0

−1

−1

−2

−2 −3

Duchi (UC Berkeley)

−3


Fall 2009

7 / 53


But generally speaking... We’re screwed. ◮ Local (non global) minima of f0 ◮ All kinds of constraints (even restricting to continuous functions): h(x) = sin(2πx) = 0

250 200 150 100 50 0 −50 3 2

3

1

2 0

1 0

−1

−1

−2

−2 −3

◮

−3

Go for convex problems! Duchi (UC Berkeley)


Fall 2009

7 / 53

Convex Sets

Convex Sets

Duchi (UC Berkeley)


Fall 2009

8 / 53

Convex Sets

Definition A set C ⊆ Rn is convex if for x, y ∈ C and any α ∈ [0, 1], αx + (1 − α)y ∈ C.

y x

Duchi (UC Berkeley)


Fall 2009

9 / 53

Convex Sets

Examples

◮

All of Rn (obvious)

Duchi (UC Berkeley)


Fall 2009

10 / 53

Convex Sets

Examples

◮

All of Rn (obvious)

◮

Non-negative orthant, Rn+ : let x 0, y 0, clearly αx + (1 − α)y 0.

Duchi (UC Berkeley)


Fall 2009

10 / 53

Convex Sets

Examples

◮

All of Rn (obvious)

◮

Non-negative orthant, Rn+ : let x 0, y 0, clearly αx + (1 − α)y 0.

◮

Norm balls: let kxk ≤ 1, kyk ≤ 1, then kαx + (1 − α)yk ≤ kαxk + k(1 − α)yk = α kxk + (1 − α) kyk ≤ 1.

Duchi (UC Berkeley)


Fall 2009

10 / 53

Convex Sets

Examples Affine subspaces: Ax = b, Ay = b, then A(αx + (1 − α)y) = αAx + (1 − α)Ay = αb + (1 − α)b = b.

1 0.8 0.6 0.4

x3

◮

0.2 0 −0.2 −0.4 1 0.8

1 0.6

0.8 0.6

0.4 0.4

0.2

x2 Duchi (UC Berkeley)

0.2 0

0

x1


Fall 2009

11 / 53

Convex Sets

More examples ◮

Arbitrary T intersections of convex sets: let Ci be convex for i ∈ I, C = i Ci , then x ∈ C, y ∈ C

⇒

αx + (1 − α)y ∈ Ci ∀ i ∈ I

so αx + (1 − α)y ∈ C.

Duchi (UC Berkeley)


Fall 2009

12 / 53

Convex Sets

More examples PSD Matrices, a.k.a. the positive semidefinite cone Sn+ ⊂ Rn×n . A ∈ Sn+ means xT Ax ≥ 0 for all x ∈ Rn . For A, B ∈ S+ n, xT (αA + (1 − α)B) x

= αxT Ax + (1 − α)xT Bx ≥ 0.

1

0.8

0.6

z

◮

0.4

0.2

0 1 1

0.5

◮

On right: y x z 2 S+ = 0 = x, y, z : x ≥ 0, y ≥ 0, xy ≥ z 2 z y

0.8

0

−0.5

0.6 0.4

0.2

−1

Duchi (UC Berkeley)


0

x

Fall 2009

13 / 53

Convex Functions

Convex Functions

Duchi (UC Berkeley)


Fall 2009

14 / 53

Convex Functions

Definition A function f : Rn → R is convex if for x, y ∈ dom f and any α ∈ [0, 1], f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y).

αf(x) + (1 - α)f(y)

f (y)

f (x)

Duchi (UC Berkeley)


Fall 2009

15 / 53

Convex Functions

First order convexity conditions Theorem Suppose f : Rn → R is differentiable. Then f is convex if and only if for all x, y ∈ dom f f (y) ≥ f (x) + ∇f (x)T (y − x)

f(y)

f(x) + ∇f(x)T (y - x) (x, f(x)) Duchi (UC Berkeley)


Fall 2009

16 / 53

Convex Functions

Actually, more general than that Definition The subgradient set, or subdifferential set, ∂f (x) of f at x is ∂f (x) = g : f (y) ≥ f (x) + g T (y − x) for all y . f (y)

Theorem f : Rn → R is convex if and only if it has non-empty subdifferential set everywhere.

(x, f(x))

f (x) + g T (y - x) Duchi (UC Berkeley)


Fall 2009

17 / 53

Convex Functions

Second order convexity conditions Theorem Suppose f : Rn → R is twice differentiable. Then f is convex if and only if for all x ∈ dom f , ∇2 f (x) 0.

10

8

6

4

2

0 2 2

1 1

0 0 −1

−1 −2

Duchi (UC Berkeley)

−2


Fall 2009

18 / 53

Convex Functions

Convex sets and convex functions

Definition The epigraph of a function f is the set of points

epi f

epi f = {(x, t) : f (x) ≤ t}. ◮

epi f is convex if and only if f is convex.

◮

Sublevel sets, {x : f (x) ≤ a} are convex for convex f .

Duchi (UC Berkeley)

a


Fall 2009

19 / 53

Convex Functions

Examples

Examples ◮

Linear/affine functions: f (x) = bT x + c.

Duchi (UC Berkeley)


Fall 2009

20 / 53

Convex Functions

Examples

Examples ◮

Linear/affine functions: f (x) = bT x + c.

◮

Quadratic functions: 1 f (x) = xT Ax + bT x + c 2 for A 0. For regression: 1 1 1 kXw − yk2 = wT X T Xw − y T Xw + y T y. 2 2 2

Duchi (UC Berkeley)


Fall 2009

20 / 53

Convex Functions

Examples

More examples ◮

Norms (like ℓ1 or ℓ2 for regularization): kαx + (1 − α)yk ≤ kαxk + k(1 − α)yk = α kxk + (1 − α) kyk .

Duchi (UC Berkeley)


Fall 2009

21 / 53

Convex Functions

Examples

More examples ◮


◮

Composition with an affine function f (Ax + b): f (A(αx + (1 − α)y) + b) = f (α(Ax + b) + (1 − α)(Ay + b)) ≤ αf (Ax + b) + (1 − α)f (Ay + b)

Duchi (UC Berkeley)


Fall 2009

21 / 53

Convex Functions

Examples

More examples ◮


◮

Composition with an affine function f (Ax + b): f (A(αx + (1 − α)y) + b) = f (α(Ax + b) + (1 − α)(Ay + b)) ≤ αf (Ax + b) + (1 − α)f (Ay + b)

◮

Log-sum-exp (via ∇2 f (x) PSD): f (x) = log

n X

exp(xi )

i=1

Duchi (UC Berkeley)

!


Fall 2009

21 / 53

Convex Functions

Examples

Important examples in Machine Learning

3

◮

◮

SVM loss: [1 - x]+

f (w) = 1 − yi xTi w +

Binary logistic loss:

f (w) = log 1 + exp(−yi xTi w)

Duchi (UC Berkeley)

log(1 + ex )

0 −2


3

Fall 2009

22 / 53

Convex Optimization Problems


Definition An optimization problem is convex if its objective is a convex function, the inequality constraints fj are convex, and the equality constraints hj are affine

minimize f0 (x) x

(Convex function)

s.t. fi (x) ≤ 0 (Convex sets) hj (x) = 0 (Affine)

Duchi (UC Berkeley)


Fall 2009

23 / 53


It’s nice to be convex Theorem If x ˆ is a local minimizer of a convex optimization problem, it is a global minimizer. 4

x∗

3.5

3

2.5

2

1.5

1

0.5

0

Duchi (UC Berkeley)

0.5

1

1.5

2

2.5


3

3.5

Fall 2009

24 / 53


Even more reasons to be convex

Theorem ∇f (x) = 0 if and only if x is a global minimizer of f (x). Proof. ◮

∇f (x) = 0. We have f (y) ≥ f (x) + ∇f (x)T (y − x) = f (x).

◮

∇f (x) 6= 0. There is a direction of descent.

Duchi (UC Berkeley)


Fall 2009

25 / 53


LET’S TAKE A BREAK

Duchi (UC Berkeley)


Fall 2009

26 / 53

Lagrange Duality

Lagrange Duality

Duchi (UC Berkeley)


Fall 2009

27 / 53

Lagrange Duality

Goals of Lagrange Duality

◮

Get certificate for optimality of a problem

◮

Remove constraints

◮

Reformulate problem

Duchi (UC Berkeley)


Fall 2009

28 / 53

Lagrange Duality

Constructing the dual ◮

Start with optimization problem: minimize f0 (x) x

s.t. fi (x) ≤ 0, i = {1, . . . , k}

hj (x) = 0, j = {1, . . . , l}

◮

Form Lagrangian using Lagrange multipliers λi ≥ 0, νi ∈ R L(x, λ, ν) = f0 (x) +

◮

k X

λi fi (x) +

l X

νj hj (x)

j=1

i=1

Form dual function g(λ, ν) = inf L(x, λ, ν) = inf x

Duchi (UC Berkeley)

x

  

f0 (x) +

k X

λi fi (x) +

i=1


l X j=1

  νj hj (x) 

Fall 2009

29 / 53

Lagrange Duality

Remarks

◮

Original problem is equivalent to " minimize x

◮

#

sup L(x, λ, ν)

λ0,ν

Dual problem is switching the min and max: h i maximize inf L(x, λ, ν) . λ0,ν

Duchi (UC Berkeley)

x


Fall 2009

30 / 53

Lagrange Duality

One Great Property of Dual Lemma (Weak Duality) If λ 0, then

g(λ, ν) ≤ f0 (x∗ ).

Proof. We have g(λ, ν) = inf L(x, λ, ν) ≤ L(x∗ , λ, ν) x

= f0 (x∗ ) +

k X i=1

Duchi (UC Berkeley)

λi fi (x∗ ) +

l X j=1

νj hj (x∗ ) ≤ f0 (x∗ ).


Fall 2009

31 / 53

Lagrange Duality

The Greatest Property of the Dual

Theorem For reasonable1 convex problems, sup g(λ, ν) = f0 (x∗ ) λ0,ν

1

There are conditions called constraint qualification for which this is true Duchi (UC Berkeley)


Fall 2009

32 / 53

Lagrange Duality

Geometric Look Minimize 12 (x − c − 1)2 subject to x2 ≤ c. 14

0.6

12

10 0.4

8 0.2

6 0

4

−0.2

2

0 −0.4

−2 −2

−1.5

−1

−0.5

0

x

0.5

1

1.5

2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

λ

True function (blue), constraint Dual function g(λ) (black), primal op(green), L(x, λ) for different λ timal (dotted blue) (dotted) Duchi (UC Berkeley)


Fall 2009

33 / 53

Lagrange Duality

Intuition Can interpret duality as linear approximation.

Duchi (UC Berkeley)


Fall 2009

34 / 53

Lagrange Duality

Intuition Can interpret duality as linear approximation. ◮

I− (a) = ∞ if a > 0, 0 otherwise; I0 (a) = ∞ unless a = 0. Rewrite problem as minimize f0 (x) + x

Duchi (UC Berkeley)

k X i=1

I− (fi (x)) +


l X

I0 (hj (x))

j=1

Fall 2009

34 / 53

Lagrange Duality

Intuition Can interpret duality as linear approximation. ◮

I− (a) = ∞ if a > 0, 0 otherwise; I0 (a) = ∞ unless a = 0. Rewrite problem as minimize f0 (x) + x

◮

k X i=1

I− (fi (x)) +

l X

I0 (hj (x))

j=1

Replace I(fi (x)) with λi fi (x); a measure of “displeasure” when λi ≥ 0, fi (x) > 0. νi hj (x) lower bounds I0 (hj (x)): minimize f0 (x) + x

Duchi (UC Berkeley)

k X

λi fi (x) +

i=1


l X

νj hj (x)

j=1

Fall 2009

34 / 53

Lagrange Duality

Example: Linearly constrained least squares minimize x

1 kAx − bk2 2

s.t. Bx = d.

Form the Lagrangian: L(x, ν) =

1 kAx − bk2 + ν T (Bx − d) 2

Take infimum: ∇x L(x, ν) = AT Ax − AT b + B T ν

⇒

x = (AT A)−1 (AT b − B T ν)

Simple unconstrained quadratic problem! inf L(x, ν) x

=

1

A(AT A)−1 (AT b − B T ν) − b 2 + ν T B((AT A)−1 AT b − B T ν) − dT ν 2 Duchi (UC Berkeley)


Fall 2009

35 / 53

Lagrange Duality

Example: Quadratically constrained least squares 1 kAx − bk2 2 Form the Lagrangian (λ ≥ 0): minimize x

L(x, λ) =

s.t.

1 kxk2 ≤ c. 2

1 1 kAx − bk2 + λ(kxk2 − 2c) 2 2

Take infimum: ∇x L(x, ν) = AT Ax − AT b + λI

⇒

x = (AT A + λI)−1 AT b

1

A(AT A + λI)−1 AT b − b 2 + λ (AT A + λI)−1 AT b 2 −λc x 2 2 One variable dual problem! inf L(x, λ) =

1 1 g(λ) = − bT A(AT A + λI)−1 AT b − λc + kbk2 . 2 2 Duchi (UC Berkeley)


Fall 2009

36 / 53

Lagrange Duality

Uses of the Dual

◮

Main use: certificate of optimality (a.k.a. duality gap). If we have feasible x and know the dual g(λ, ν), then g(λ, ν) ≤ f0 (x∗ ) ≤ f0 (x) ⇒ f0 (x∗ ) − f0 (x) ≥ g(λ, ν) − f0 (x)

⇒ f0 (x) − f0 (x∗ ) ≤ f0 (x) − g(λ, ν).

Duchi (UC Berkeley)


Fall 2009

37 / 53

Lagrange Duality

Uses of the Dual

◮

Main use: certificate of optimality (a.k.a. duality gap). If we have feasible x and know the dual g(λ, ν), then g(λ, ν) ≤ f0 (x∗ ) ≤ f0 (x) ⇒ f0 (x∗ ) − f0 (x) ≥ g(λ, ν) − f0 (x)

⇒ f0 (x) − f0 (x∗ ) ≤ f0 (x) − g(λ, ν).

◮

Also used in more advanced primal-dual algorithms (we won’t talk about these).

Duchi (UC Berkeley)


Fall 2009

37 / 53

Optimization Algorithms


Duchi (UC Berkeley)


Fall 2009

38 / 53


Gradient Methods

Gradient Descent

The simplest algorithm in the world (almost). Goal: minimize f (x) x

Just iterate xt+1 = xt − ηt ∇f (xt ) where ηt is stepsize.

Duchi (UC Berkeley)


Fall 2009

39 / 53


Gradient Methods

Single Step Illustration

f (x)

(xt , f(xt )) f (xt ) - η∇f(xt )T (x - xt )

Duchi (UC Berkeley)


Fall 2009

40 / 53


Gradient Methods

Full Gradient Descent f (x) = log(exp(x1 + 3x2 − .1) + exp(x1 − 3x2 − .1) + exp(−x1 − .1))

Duchi (UC Berkeley)


Fall 2009

41 / 53


Gradient Methods

Stepsize Selection How do I choose a stepsize? ◮

Idea 1: exact line search ηt = argmin f (x − η∇f (x)) η

Too expensive to be practical.

Duchi (UC Berkeley)


Fall 2009

42 / 53


Gradient Methods

Stepsize Selection How do I choose a stepsize? ◮

Idea 1: exact line search ηt = argmin f (x − η∇f (x)) η

Too expensive to be practical. ◮

Idea 2: backtracking (Armijo) line search. Let α ∈ (0, 21 ), β ∈ (0, 1). Multiply η = βη until f (x − η∇f (x)) ≤ f (x) − αη k∇f (x)k2 Works well in practice.

Duchi (UC Berkeley)


Fall 2009

42 / 53


Gradient Methods

Illustration of Armijo/Backtracking Line Search

f(x - η∇f(x))

f(x) - ηαk∇f(x)k2 η ηt = 0

η0

As a function of stepsize η. Clearly a region where f (x − η∇f (x)) is below line f (x) − αη k∇f (x)k2 . Duchi (UC Berkeley)


Fall 2009

43 / 53


Gradient Methods

Newton’s method

Idea: use a second-order approximation to function. 1 f (x + ∆x) ≈ f (x) + ∇f (x)T ∆x + ∆xT ∇2 f (x)∆x 2 Choose ∆x to minimize above: −1 ∆x = − ∇2 f (x) ∇f (x)

This is descent direction:

−1 ∇f (x)T ∆x = −∇f (x)T ∇2 f (x) ∇f (x) < 0.

Duchi (UC Berkeley)


Fall 2009

44 / 53


Gradient Methods

Newton step picture

fˆ

(x, f(x)) f (x + ∆x, f(x + ∆x))

fˆ is 2nd -order approximation, f is true function. Duchi (UC Berkeley)


Fall 2009

45 / 53


Gradient Methods

Convergence of gradient descent and Newton’s method

◮

Strongly convex case: ∇2 f (x) mI, then “Linear convergence.” For some γ ∈ (0, 1), f (xt ) − f (x∗ ) ≤ γ t , γ < 1. f (xt ) − f (x∗ ) ≤ γ t

◮

or t ≥

1 1 log ⇒ f (xt ) − f (x∗ ) ≤ ε. γ ε

Smooth case: k∇f (x) − ∇f (y)k ≤ C kx − yk. f (xt ) − f (x∗ ) ≤

◮

K t2

Newton’s method often is faster, especially when f has “long valleys”

Duchi (UC Berkeley)


Fall 2009

46 / 53


Gradient Methods

What about constraints?

◮

Linear constraints Ax = b are easy. For example, in Newton method (assume Ax = b): 1 minimize ∇f (x)T ∆x + ∆xT ∇2 f (x)∆x ∆x 2

s.t. A∆x = 0.

Solution ∆x satisfies A(x + ∆x) = Ax + A∆x = b.

Duchi (UC Berkeley)


Fall 2009

47 / 53


Gradient Methods

What about constraints?

◮

Linear constraints Ax = b are easy. For example, in Newton method (assume Ax = b): 1 minimize ∇f (x)T ∆x + ∆xT ∇2 f (x)∆x ∆x 2

s.t. A∆x = 0.

Solution ∆x satisfies A(x + ∆x) = Ax + A∆x = b. ◮

Inequality constraints are a bit tougher... 1 minimize ∇f (x)T ∆x + ∆xT ∇2 f (x)∆x ∆x 2

s.t. fi (x + ∆x) ≤ 0

just as hard as original.

Duchi (UC Berkeley)


Fall 2009

47 / 53


Gradient Methods

Logarithmic Barrier Methods Goal: minimize f0 (x) x

s.t. fi (x) ≤ 0, i = {1, . . . , k}

Convert to minimize f0 (x) + x

k X i=1

I− (fi (x))

Approximate I− (u) ≈ −t log(−u) for small t. minimize f0 (x) − t x

Duchi (UC Berkeley)

k X

log (−fi (x))

i=1


Fall 2009

48 / 53


Gradient Methods

The barrier function 10

0

−2 −3

u

0

1

I− (u) is dotted line, others are −t log(−u). Duchi (UC Berkeley)


Fall 2009

49 / 53


Gradient Methods

Illustration Minimizing cT x subject to Ax ≤ b.

c t=1

Duchi (UC Berkeley)

t=5


Fall 2009

50 / 53


Subgradient Methods

Subgradient Descent

Really, the simplest algorithm in the world. Goal: minimize f (x) x

Just iterate xt+1 = xt − ηt gt where ηt is a stepsize, gt ∈ ∂f (xt ).

Duchi (UC Berkeley)


Fall 2009

51 / 53


Subgradient Methods

Why subgradient descent?

◮

Lots of non-differentiable convex functions used in machine learning: f (x) = 1 − aT x + ,

f (x) = kxk1 ,

f (X) =

k X

σr (X)

r=1

where σr is the rth singular value of X. ◮

Easy to analyze

◮

Do not even need true sub-gradient: just have Egt ∈ ∂f (xt ).

Duchi (UC Berkeley)


Fall 2009

52 / 53


Subgradient Methods

Proof of convergence for subgradient descent Idea: bound kxt+1 − x∗ k using subgradient inequality. Assume that kgt k ≤ G. kxt+1 − x∗ k2 = kxt − ηgt − x∗ k2

= kxt − x∗ k2 − 2ηgtT (xt − x∗ ) + η 2 kgt k2

Duchi (UC Berkeley)


Fall 2009

53 / 53


Subgradient Methods

Proof of convergence for subgradient descent Idea: bound kxt+1 − x∗ k using subgradient inequality. Assume that kgt k ≤ G. kxt+1 − x∗ k2 = kxt − ηgt − x∗ k2

= kxt − x∗ k2 − 2ηgtT (xt − x∗ ) + η 2 kgt k2

Recall that f (x∗ ) ≥ f (xt ) + gtT (x∗ − xt ) so

⇒

− gtT (xt − x∗ ) ≤ f (x∗ ) − f (xt )

kxt+1 − x∗ k2 ≤ kxt − x∗ k2 + 2η [f (x∗ ) − f (xt )] + η 2 G2 .

Then f (xt ) − f (x∗ ) ≤ Duchi (UC Berkeley)

kxt − x∗ k2 − kxt+1 − x∗ k2 η 2 + G . 2η 2


Fall 2009

53 / 53


Subgradient Methods

Almost done... Sum from t = 1 to T : T X t=1

T i Tη 1 Xh f (xt ) − f (x ) ≤ kxt − x∗ k2 − kxt+1 − x∗ k2 + G2 2η 2

Duchi (UC Berkeley)

∗

t=1

1 Tη 2 1 kx1 − x∗ k2 − kxT +1 − x∗ k2 + G = 2η 2η 2


Fall 2009

54 / 53


Subgradient Methods

Almost done... Sum from t = 1 to T : T X t=1

T i Tη 1 Xh f (xt ) − f (x ) ≤ kxt − x∗ k2 − kxt+1 − x∗ k2 + G2 2η 2 ∗

t=1

1 Tη 2 1 kx1 − x∗ k2 − kxT +1 − x∗ k2 + G = 2η 2η 2

Now let D = kx1 − x∗ k, and keep track of min along run, f (xbest ) − f (x∗ ) ≤ Set η =

D √ G T

1 η D 2 + G2 . 2ηT 2

and

Duchi (UC Berkeley)

DG f (xbest ) − f (x∗ ) ≤ √ . T Convex Optimization for Machine Learning

Fall 2009

54 / 53


Subgradient Methods

Extension: projected subgradient descent Now have a convex constraint set X. Goal: minimize f (x)

xt

x∈X

ΠX (xt )

Idea: do subgradient steps, project xt back into X at every iteration. xt+1 = ΠX (xt − ηgt )

X x∗

Proof: kΠX (xt ) − x∗ k ≤ kxt − x∗ k if x∗ ∈ X. Duchi (UC Berkeley)


Fall 2009

55 / 53


Subgradient Methods

Projected subgradient example minimize x

Duchi (UC Berkeley)

1 kAx − bk 2

s.t. kxk1 ≤ 1


Fall 2009

56 / 53


Subgradient Methods


Duchi (UC Berkeley)

1 kAx − bk 2

s.t. kxk1 ≤ 1


Fall 2009

56 / 53


Subgradient Methods


Duchi (UC Berkeley)

1 kAx − bk 2

s.t. kxk1 ≤ 1


Fall 2009

56 / 53


Subgradient Methods


Duchi (UC Berkeley)

1 kAx − bk 2

s.t. kxk1 ≤ 1


Fall 2009

56 / 53


Subgradient Methods


Duchi (UC Berkeley)

1 kAx − bk 2

s.t. kxk1 ≤ 1


Fall 2009

56 / 53


Subgradient Methods


Duchi (UC Berkeley)

1 kAx − bk 2

s.t. kxk1 ≤ 1


Fall 2009

56 / 53


Subgradient Methods


Duchi (UC Berkeley)

1 kAx − bk 2

s.t. kxk1 ≤ 1


Fall 2009

56 / 53


Subgradient Methods


Duchi (UC Berkeley)

1 kAx − bk 2

s.t. kxk1 ≤ 1


Fall 2009

56 / 53


Subgradient Methods

Convergence results for (projected) subgradient methods ◮

Any decreasing, non-summable stepsize ηt → 0, f xavg(t) − f (x∗ ) → 0.

Duchi (UC Berkeley)


P∞

t=1 ηt

= ∞ gives

Fall 2009

57 / 53


Subgradient Methods


◮


P∞

t=1 ηt

= ∞ gives

√ Slightly less brain-dead analysis than earlier shows with ηt ∝ 1/ t C f xavg(t) − f (x∗ ) ≤ √ t

Duchi (UC Berkeley)


Fall 2009

57 / 53


Subgradient Methods


◮


P∞

t=1 ηt

= ∞ gives

√ Slightly less brain-dead analysis than earlier shows with ηt ∝ 1/ t C f xavg(t) − f (x∗ ) ≤ √ t

◮

Same convergence when gt is random, i.e. Egt ∈ ∂f (xt ). Example: n

X 1 1 − yi xTi w + f (w) = kwk2 + C 2 i=1

Just pick random training example. Duchi (UC Berkeley)


Fall 2009

57 / 53

Take Home Messages

Recap

◮

Defined convex sets and functions

◮

Saw why we want optimization problems to be convex (solvable)

◮

Sketched some of Lagrange duality

◮

First order methods are easy and (often) work well

Duchi (UC Berkeley)


Fall 2009

58 / 53

Take Home Messages

Take Home Messages

◮

Many useful problems formulated as convex optimization problems

◮

If it is not convex and not an eigenvalue problem, you are out of luck

◮

If it is convex, you are golden

Duchi (UC Berkeley)


Fall 2009

59 / 53