Seminar Notes, Lecture 2 (.pdf)

Denoising and edge detection Image modeling

Analytic Methods for Image Processing, Lecture II Analysis and Applied Math Seminar Spring, 2008 Joe Lakey

January 30, 2008 Joe Lakey

Computational Methods


Outline: Edge detection, Denoising and modeling

I

Soft thresholding

Joe Lakey




I

Soft thresholding

I

Edge detection

Joe Lakey




I

Soft thresholding

I

Edge detection

I

Modeling objects and textures (PDE)

Joe Lakey




I

Soft thresholding

I

Edge detection

I

Modeling objects and textures (PDE)

I

Solvability

Joe Lakey



Denoising Edge detection

Naive denoising: Thresholding

I

Transform image u 7→ U

Joe Lakey





I


I

Shrink transform coefficients according to noise level

Joe Lakey





I


I


I

Orthogonal transform: GWN 7→ GWN; compresses information

Joe Lakey





I


I


I


I

Justified by function spaces

Joe Lakey





I


I


I


I


I

Realistically, depends on dissimilarity of info vs noise

Joe Lakey





I


I


I


I


I

Realistically, depends on dissimilarity of info vs noise

I

Use wavelets

Joe Lakey




Thresholding decision

Hard:

D(U, λ) =

U if |U| > λ 0 else

Soft: D(U, λ) =

sgn (U)(|U| − λ)+ if |U| > λ 0 else

Joe Lakey




Parameters and variations I

√ Universal (Global) threshold: λ = σ 2 log # points

Joe Lakey




Parameters and variations I I

√ Universal (Global) threshold: λ = σ 2 log # points Optimality: asymptotic in expectation / L2

Joe Lakey




Parameters and variations I I I

√ Universal (Global) threshold: λ = σ 2 log # points Optimality: asymptotic in expectation / L2 VisuShrink (images): Universal with # pixels (oversmooths)

Joe Lakey




Parameters and variations I I I I

√ Universal (Global) threshold: λ = σ 2 log # points Optimality: asymptotic in expectation / L2 VisuShrink (images): Universal with # pixels (oversmooths) SUREShrink: Stein’s Unbiased Risk Estimator per subband.

Joe Lakey





√ Universal (Global) threshold: λ = σ 2 log # points Optimality: asymptotic in expectation / L2 VisuShrink (images): Universal with # pixels (oversmooths) SUREShrink: Stein’s Unbiased Risk Estimator per subband. I

Subband coefficients Xν , ν = 1, . . . , d

Joe Lakey







I

SURE (t, X ) = d − 2#{ν : |Xν | ≤ t} +

d X ν=1

Joe Lakey


min (Xν , t)2






I

SURE (t, X ) = d − 2#{ν : |Xν | ≤ t} +

d X ν=1

I

λ = arg mint SURE (t, X ).

Joe Lakey


min (Xν , t)2






I

SURE (t, X ) = d − 2#{ν : |Xν | ≤ t} +

d X ν=1

I I

λ = arg mint SURE (t, X ). Use as λ for subband soft thresholding.

Joe Lakey


min (Xν , t)2






I

SURE (t, X ) = d − 2#{ν : |Xν | ≤ t} +

d X

min (Xν , t)2

ν=1 I I

I


rationale: in applications noise shows up in fine scales.

Joe Lakey







I

SURE (t, X ) = d − 2#{ν : |Xν | ≤ t} +

d X

min (Xν , t)2

ν=1 I I

I I


rationale: in applications noise shows up in fine scales. BayesShrink. Estimates distribution in each subband Joe Lakey




Wavelets and Besov norms I

Besov space on Rn : Bps,q

=

0

f ∈ S : kφ∗f kLp +

X ∞

sj

2 kφj ∗f kLp

j=1

Joe Lakey


q

1/q

1/p, wavelet. θ: 2s R(N, Θsp,q ) ∼ N −r , r = 2s + 1 Bounded variation s = 1, p = 1 (not quite Besov) Donoho: Unconditional bases are best bases Joe Lakey




Wavelet shrinkage summary

I

Optimality

Joe Lakey





I

Optimality

I

Performance depends on accurate identification of noise characteristics

Joe Lakey





I

Optimality

I


I

Hypothesis on behavior of noise versus image under transform

Joe Lakey





I

Optimality

I


I

Hypothesis on behavior of noise versus image under transform

I

Models: make info vs noise precise

Joe Lakey




Noisy Homer

wavelet soft threshold

50

50

100

100

150

150

200

200

250

250 50

100

150

200

250

50

100

150

200

250

2 % wavecompress reconstruction 10 % curvecompress reconstruction 50

50

100

100

150

150

200

200

250

250 50

100

150

200

250 Joe Lakey

50

100


150

200

250


Original granite


Gaussian white noise, sigma = 50

50

50

100

100

150

150

200

200

250

100

50

100

150

200

250

Granite image Histogram

250

120

50

100

150

200

250

GWN image Histogram

100

80

80

60

60 40

40

20 0 0

20 100

200

300 Joe Lakey

0 −200

−100


0

100

200



Granite plus noise

Homer plus noise

50

50

100

100

150

150

200

200

250

50

100

150

200

Wavelet soft threshold

250

granite

50 100 150 200 250

50

100

150

200

250

250

50

50 100 150 200 250

50

Error 50 150 200 50

100

150

150

200

100

150

250

Homer

200

250

200

250

Error

100

250

100

Wavelet soft threshold

200

250 Joe Lakey

50 100 150 200 250

50

100


150



Cycle spinning

Joe Lakey




Homer plus sigma=25

50

50

100

100

150

150

200

200

250 50

100

150

200

250

250 50

100

Joe Lakey

150

200

250


50

100

150

200

250



50

50

100

100

150

150

200

200

250

250

300

300

350

350

400

400

450

450

500 50

100

150

200

250

300

350

400

450

500

500 50

100

150

200

250

Joe Lakey

300

350

400

450

500

50


100

150

200

250

300

350

400

450

500



Image objects: edge detection

I

Geometric objects: edges

Joe Lakey





I


I

How to detect and construct?

Joe Lakey





I


I


I

Gradient or Laplacian based approaches

Joe Lakey





I


I


I

Gradient or Laplacian based approaches

I

Large gradient or Laplacian alone not enough

Joe Lakey




Image objects: Canny edge detection

I

Canny edge detection “Canny, A computational approach to edge detection” IEEE Trans. Pattern Anal. Mach, Intell. 8, 679–698, 1986.

Joe Lakey





I

I

Canny edge detection “Canny, A computational approach to edge detection” IEEE Trans. Pattern Anal. Mach, Intell. 8, 679–698, 1986. Criteria

Joe Lakey





I

I

Canny edge detection “Canny, A computational approach to edge detection” IEEE Trans. Pattern Anal. Mach, Intell. 8, 679–698, 1986. Criteria I

Minimize errors (correctly identify edge set).

Joe Lakey





I

I

Canny edge detection “Canny, A computational approach to edge detection” IEEE Trans. Pattern Anal. Mach, Intell. 8, 679–698, 1986. Criteria I I

Minimize errors (correctly identify edge set). Localization (detected edge pixels close to actual edge)

Joe Lakey





I

I

Canny edge detection “Canny, A computational approach to edge detection” IEEE Trans. Pattern Anal. Mach, Intell. 8, 679–698, 1986. Criteria I I I

Minimize errors (correctly identify edge set). Localization (detected edge pixels close to actual edge) Uniqueness (one response per edge).

Joe Lakey




Canny algorithm Algorithmic steps: I Remove noise (presmooth): suppress false positives/negatives

Joe Lakey




Canny algorithm Algorithmic steps: I Remove noise (presmooth): suppress false positives/negatives I Compute gradient magnitude via Sobel differences: convolve with     −1 0 −1 1 2 1 0 0  δx =  −2 0 2  ; δy =  0 −1 0 1 −1 −2 −1

Joe Lakey




Canny algorithm Algorithmic steps: I Remove noise (presmooth): suppress false positives/negatives I Compute gradient magnitude via Sobel differences: convolve with     −1 0 −1 1 2 1 0 0  δx =  −2 0 2  ; δy =  0 −1 0 1 −1 −2 −1 I

Edge strength: magnitude of (δx ∗ I , δy ∗ I ) per pixel.

Joe Lakey




Canny algorithm Algorithmic steps: I Remove noise (presmooth): suppress false positives/negatives I Compute gradient magnitude via Sobel differences: convolve with     −1 0 −1 1 2 1 0 0  δx =  −2 0 2  ; δy =  0 −1 0 1 −1 −2 −1 I I

Edge strength: magnitude of (δx ∗ I , δy ∗ I ) per pixel. Edge direction θ = arctan(δy ∗ I /δx ∗ I ) per pixel.

Joe Lakey




Canny algorithm Algorithmic steps: I Remove noise (presmooth): suppress false positives/negatives I Compute gradient magnitude via Sobel differences: convolve with     −1 0 −1 1 2 1 0 0  δx =  −2 0 2  ; δy =  0 −1 0 1 −1 −2 −1 I I I

Edge strength: magnitude of (δx ∗ I , δy ∗ I ) per pixel. Edge direction θ = arctan(δy ∗ I /δx ∗ I ) per pixel. Predict at degenerate points

Joe Lakey




Canny algorithm Algorithmic steps: I Remove noise (presmooth): suppress false positives/negatives I Compute gradient magnitude via Sobel differences: convolve with     −1 0 −1 1 2 1 0 0  δx =  −2 0 2  ; δy =  0 −1 0 1 −1 −2 −1 I I I I

Edge strength: magnitude of (δx ∗ I , δy ∗ I ) per pixel. Edge direction θ = arctan(δy ∗ I /δx ∗ I ) per pixel. Predict at degenerate points Quantize θ: multiple of π/4 to trace edge path.

Joe Lakey




Canny algorithm Algorithmic steps: I Remove noise (presmooth): suppress false positives/negatives I Compute gradient magnitude via Sobel differences: convolve with     −1 0 −1 1 2 1 0 0  δx =  −2 0 2  ; δy =  0 −1 0 1 −1 −2 −1 I I I I I

Edge strength: magnitude of (δx ∗ I , δy ∗ I ) per pixel. Edge direction θ = arctan(δy ∗ I /δx ∗ I ) per pixel. Predict at degenerate points Quantize θ: multiple of π/4 to trace edge path. Suppress non-edge gradient pixels.

Joe Lakey




Canny algorithm Algorithmic steps: I Remove noise (presmooth): suppress false positives/negatives I Compute gradient magnitude via Sobel differences: convolve with     −1 0 −1 1 2 1 0 0  δx =  −2 0 2  ; δy =  0 −1 0 1 −1 −2 −1 I I I I I I

Edge strength: magnitude of (δx ∗ I , δy ∗ I ) per pixel. Edge direction θ = arctan(δy ∗ I /δx ∗ I ) per pixel. Predict at degenerate points Quantize θ: multiple of π/4 to trace edge path. Suppress non-edge gradient pixels. Hysteresis: eliminate streaking I I

start edge: low threshold stop edge: high threshold Joe Lakey




50

50

00

100

50

150

00

200

50

250 50

100

150

200

250 Joe Lakey

50

100


150

200

250



50

100

150

200

250 50

100

150

200

250

Joe Lakey

50

100


150

200

250



0

100

0

200

0

300

0

0

400

0

500

0

600

0

0

700

0 50

100

150

200

250

300

350

400

450

500

Joe Lakey

100

200

300


400

500

600

700

800



Canny edges of granite

50

100

150

200

250 50

100

150

200

250

Joe Lakey

50

100


150

200

250


Mumford-Shah functional Diffusion The Rudin-Osher-Fatemi model

Image models: energy functionals

I

Seek Energy function whose minimization preserves objects, suppresses noise/artifacts

Joe Lakey





I


I

Objects + noise E(u; f ) = arg min ϕ1 kukX1 + ϕ2 kf − ukX2

Joe Lakey





I


I

Objects + noise E(u; f ) = arg min ϕ1 kukX1 + ϕ2 kf − ukX2

I

Objects + texture + noise F(u, v ; f ) = arg min ϕ1 kukX1 +ϕ2 kv kX2 +ϕ3 kf −u−v kX3

I

Usually ϕi : multiplier and/or power

Joe Lakey




Mumford-Shah functional (CPAM, 1989)

I

EMS (u, Γ; f ) = µ

2

ZZ

2

ZZ

|∇u|2 dx dy +ν|Γ|

(u−f ) dx dy + Ω

Ω\Γ

(1)

Joe Lakey





I

EMS (u, Γ; f ) = µ

2

ZZ

2

ZZ



Ω\Γ

(1) I

µ, ν: weighting parameters

Joe Lakey





I

EMS (u, Γ; f ) = µ

2

ZZ

2

ZZ



Ω\Γ

(1) I

µ, ν: weighting parameters

I

|Γ|: length of edge set Γ.

Joe Lakey




Minimizer characteristics

I

Minimizer (u, Γ)

Joe Lakey





I

Minimizer (u, Γ)

I

u will approximate f in L2 norm on Ω

Joe Lakey





I

Minimizer (u, Γ)

I


I

Γ will have a small length

Joe Lakey





I

Minimizer (u, Γ)

I


I


I

u will not vary too much off Γ.

Joe Lakey





I

Minimizer (u, Γ)

I


I


I

u will not vary too much off Γ.

I

Solution is a cartoon

Joe Lakey




Minimization in general (univariate) I

Find zero(s) of ∂E /∂u

Joe Lakey




Minimization in general (univariate) I I

Find zero(s) of ∂E /∂u R1 1-D: u : [0, 1] → R; E = 0 F (u, u 0 ) dx

Joe Lakey




Minimization in general (univariate) I I I I

Find zero(s) of ∂E /∂u R1 1-D: u : [0, 1] → R; E = 0 F (u, u 0 ) dx Solve Euler-Lagrange equation ∂ ∂F ∂F − =0 ∂u ∂x ∂u 0

Joe Lakey





I

Find zero(s) of ∂E /∂u R1 1-D: u : [0, 1] → R; E = 0 F (u, u 0 ) dx Solve Euler-Lagrange equation ∂ ∂F ∂F − =0 ∂u ∂x ∂u 0 for derivation http://en.wikipedia.org/wiki/Euler-Lagrange equation

Joe Lakey





I I

Find zero(s) of ∂E /∂u R1 1-D: u : [0, 1] → R; E = 0 F (u, u 0 ) dx Solve Euler-Lagrange equation ∂ ∂F ∂F − =0 ∂u ∂x ∂u 0 for derivation http://en.wikipedia.org/wiki/Euler-Lagrange equation N-variables N ∂F X ∂ ∂F − =0 ∂u ∂xν ∂uν ν=1

Joe Lakey





Find zero(s) of ∂E /∂u R1 1-D: u : [0, 1] → R; E = 0 F (u, u 0 ) dx Solve Euler-Lagrange equation ∂ ∂F ∂F − =0 ∂u ∂x ∂u 0

I

for derivation http://en.wikipedia.org/wiki/Euler-Lagrange equation N-variables N ∂F X ∂ ∂F − =0 ∂u ∂xν ∂uν

I

Artificial Time gradient descent

I

ν=1

N

∂u ∂F X ∂ ∂F =− − ∂t ∂u ∂xν ∂uν ν=1

Joe Lakey




Solvability

I

Convex E admits unique minimizer

Joe Lakey




Solvability

I


I

EMS admits minimizers in its weak formulation

Joe Lakey




Solvability

I


I


I

Γ replaced by “jump set of u”; minimize over u

Joe Lakey




Solvability

I


I


I


I

Dal Maso, Morel and Solimini, “A variational method in image segmentation: existence and approximation results.” Acta Math. 168 (1992), no. 1-2, 89–151.

Joe Lakey




Solvability

I


I


I


I


I

based on ideas of E. De Giorgi

Joe Lakey




Solvability

I


I


I


I


I

based on ideas of E. De Giorgi

I

cf. Ambrosio, “A compactness theorem for a new class of functions of bounded variation.” Boll. Un. Mat. Ital. B (7) 3 (1989), no. 4, 857–881.

Joe Lakey




Meta-issues

I

EMS good for textureless images.

Joe Lakey




Meta-issues

I I

EMS good for textureless images. Is it the best one?

Joe Lakey




Meta-issues

I I

EMS good for textureless images. Is it the best one? I

u approximating f off γ

Joe Lakey




Meta-issues

I I

EMS good for textureless images. Is it the best one? I I

u approximating f off γ u not varying off Γ

Joe Lakey




Meta-issues

I I

EMS good for textureless images. Is it the best one? I I I

u approximating f off γ u not varying off Γ Is L2 the right norm?

Joe Lakey




Meta-issues

I I

EMS good for textureless images. Is it the best one? I I I

I

u approximating f off γ u not varying off Γ Is L2 the right norm?

Computational issues

Joe Lakey




Isotropic (heat) diffusion

I

∂I (x, t) = ∇ · ∇I (x, t) ∂t

Joe Lakey




Isotropic (heat) diffusion

I

I

∂I (x, t) = ∇ · ∇I (x, t) ∂t kills noise but does not preserve features

Joe Lakey




Noise diffusion

Joe Lakey




Granite diffusion

Joe Lakey




Homer diffusion

Joe Lakey




Anisotropic diffusion I

Perona and Malik. “Scale-space and edge detection using anisotropic diffusion.” IEEE Trans. on Pattern Analysis and Machine Intelligence, 12(7):629-639, July 1990

Joe Lakey






I

Smooth within partition regions, inhibit smoothing across regions

Joe Lakey






I


I

∂I (x, t) = ∇ · (c(x, t)∇I (x, t)) ∂t

Joe Lakey






I


I

I

∂I (x, t) = ∇ · (c(x, t)∇I (x, t)) ∂t 2

2

Diffusion function c(x, t). E.g., c(r ) = e−r /κ ; or c(r ) = (1 + (r /κ)1+α )−1 where r = |∇I (x, t)|.

Joe Lakey






I


I

∂I (x, t) = ∇ · (c(x, t)∇I (x, t)) ∂t 2

2

I


I

|c∇I | increases with |∇I (x, t)| until |∇I (x, t)| ≈ κ then → 0.

Joe Lakey






I


I

∂I (x, t) = ∇ · (c(x, t)∇I (x, t)) ∂t 2

2

I


I

|c∇I | increases with |∇I (x, t)| until |∇I (x, t)| ≈ κ then → 0.

I

Computational issues Joe Lakey




Anisotropic diffusion numerics: 1-D d d ∂I (x, t) = c(x, t) I (x, t) ∂t dx dx I (x, t + ∆t) − I (x, t) 1 dI ∆x dI ∆x ≈ , t −c ,t c x+ x− ∆t ∆x dx 2 dx 2 1 1 ∆x ∆x x+∆x/2 ≈ c I ·+ ,t − I · − ,t ∆x ∆x 2 2 x−∆x/2 h 1 ∆x I (x + ∆x, t) − I (x, t) ≈ c x+ ,t ∆x 2 ∆x ∆x I (x, t) − I (x − ∆x, t) i −c x − ,t 2 ∆x

Joe Lakey




Anisotropic diffusion numerics: 2-D

I

Take lattice structure into account

Joe Lakey





I


I

Account for nearest neighbors directly or

Joe Lakey





I


I


I

Allow NN to influence indirectly.

Joe Lakey





I


I


I


I

Account for boundaries

Joe Lakey





I


I


I


I

Account for boundaries

I

Inew = I + δt divergence c. ∗ gradient I .

Joe Lakey




Homer anisotropic diffusion

Joe Lakey




Diffusion with image forcing

I

Idea: add back in a little of image at each step

Joe Lakey





I


I

Numerical tradeoff between noise smoothing and noise/image forcing

Joe Lakey





I


I


I

Inew = (1 − ∆t) I + ∆t I0 + α∆t∇ · (c∇I );

Joe Lakey





I


I


I

Inew = (1 − ∆t) I + ∆t I0 + α∆t∇ · (c∇I ); I

Possible revision: define diffusion filter in terms of edge set

Joe Lakey




Isotropically forced granite

Play/PauseSlow

Joe Lakey




Anisotropically forced granite

Play/PauseSlow

Joe Lakey




Isotropically forced homer

Joe Lakey




Anisotropically forced noisy homer

Play/PauseSlow

Joe Lakey




Rudin-Osher-Fatemi model I

Rudin, Osher and Fatemi, “Nonlinear total variation based on noise removal methods,” Physica D., 60:259–268 (1992)

Joe Lakey




Rudin-Osher-Fatemi model I I

Rudin, Osher and Fatemi, “Nonlinear total variation based on noise removal methods,” Physica D., 60:259–268 (1992) EROF : minimization intended to separate objects, suppress noise

Joe Lakey




Rudin-Osher-Fatemi model I I I

Rudin, Osher and Fatemi, “Nonlinear total variation based on noise removal methods,” Physica D., 60:259–268 (1992) EROF : minimization intended to separate objects, suppress noise Z Z EROF (u; f ) = |∇u| + λ (f − u)2 (2) Ω

Joe Lakey

Ω






I

Ω

λ: tuning parameter

Joe Lakey






I I

Ω

λ: tuning parameter R Ω |∇u|: total variation kukBV ♠

Joe Lakey






I I I

Ω

λ: tuning parameter R |∇u|: total variation kukBV ♠ RΩ R 2 Ω |∇u| vs Ω\Γ |∇u| + |Γ|: regularizing

Joe Lakey






I I I I

Ω

λ: tuning parameter R |∇u|: total variation kukBV ♠ RΩ R 2 Ω |∇u| vs Ω\Γ |∇u| + |Γ|: regularizing Variation norm:

Joe Lakey






I I I I

Ω

λ: tuning parameter R |∇u|: total variation kukBV ♠ RΩ R 2 Ω |∇u| vs Ω\Γ |∇u| + |Γ|: regularizing Variation norm: R I

kukBV = supϕ~

Ω

u∇ · ϕ ~

Joe Lakey






I I I I

Ω

λ: tuning parameter R |∇u|: total variation kukBV ♠ RΩ R 2 Ω |∇u| vs Ω\Γ |∇u| + |Γ|: regularizing Variation norm: R I I

kukBV = supϕ~ Ω u∇ · ϕ ~ Sup over ϕ ~ ∈ C01 (Ω, R2 ).

Joe Lakey






I I I I

I I

I

Ω

λ: tuning parameter R |∇u|: total variation kukBV ♠ RΩ R 2 Ω |∇u| vs Ω\Γ |∇u| + |Γ|: regularizing Variation norm: R kukBV = supϕ~ Ω u∇ · ϕ ~ Sup over ϕ ~ ∈ C01 (Ω, R2 ).

Quasi-Alaoglu for k · k|BV + k · kL1 (strong L1 , weak BV) Joe Lakey




Minimization of EROF I

EROF (u; f ) = 12 kf − uk22 + αkukBV

Joe Lakey




Minimization of EROF I I

EROF (u; f ) = 12 kf − uk22 + αkukBV Euler-Lagrange equation ∇u α∇ · − (u − f ) = 0 |∇u|

Joe Lakey





I

EROF (u; f ) = 12 kf − uk22 + αkukBV Euler-Lagrange equation ∇u α∇ · − (u − f ) = 0 |∇u| Artificial time ut = α∇ ·

∇u − (u − f ); |∇u|

Joe Lakey

u(x, 0) = f (x)





I

EROF (u; f ) = 12 kf − uk22 + αkukBV Euler-Lagrange equation ∇u α∇ · − (u − f ) = 0 |∇u| Artificial time ut = α∇ ·

I

∇u − (u − f ); |∇u|

u(x, 0) = f (x)

Discrete time ∇u +f u(x, t + ∆t) = (1 − ∆t)u(x, t) + ∆t α∆ · |∇u|

I

Finite stepsize issues both spatially and temporally Joe Lakey
