Bernd Girod: EE368 Digital Image Processing. Scale-Space Feature Detection
no. 1. Scale-space feature detection. ▫ Image features can appear similarly on all
...
Scale-space feature detection
Chapter overview: scale-space feature detection
Image features can appear similarly on all scales
Scale space representation of images
In addition to shift-invariance, scale-invariance is often a desirable property of feature detection Scale-space representation of an image is useful to detect features which are both shift-invariant and scale-invariant Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 1
Family of signals generated by successive smoothing with a Gaussian filter
( )
( )
Harris-Laplacian SIFT SURF
Scale-Space Feature Detection no. 2
Bernd Girod: EE368 Digital Image Processing
[Witkin 1983]
Shift-invariance
Rotation-invariance
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 4
Bernd Girod: EE368 Digital Image Processing
Scale space: Laplacian images
Scale-Space Feature Detection no. 7
( )
t ⋅∇ 2 f t x, y
( )
f t x, y
t=4
( )
t ⋅∇ 2 f t x, y
t=4
t = 16
t = 64
2
)
(
)
H ω x ,ω y =
1
(k −1)σ 2
[G (ω ,ω ) − G (ω ,ω )] k 2σ 2
σ2
x
y
x
y
4.
( )
f t x, y Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 9
Scale space: Binarized Laplacian images
t = 16
(
) (
) (
)
)
Scale-Space Feature Detection no. 6
Difference of Gaussians
( )
t ⋅∇ 2 f t x, y
Bernd Girod: EE368 Digital Image Processing
( )
f t x, y
t = 64
x 2 + y2
DoG ( x, y ) =
2σ 2
1
⎛
1
( k − 1) σ 2 ⎜⎝ 2π k 2σ 2
e
−
x 2 + y2 2 k 2σ 2
−
− 1 e 2πσ 2
x 2 + y2
(
(
Harris
))
y
Harris
Apply Harris detector in a spatial neighborhood at scale th to refine keypoint location xh , yh Repeat 2. and 3. until convergence
x
( )
t ⋅∇ 2 f t x, y
Scale-Space Feature Detection no. 10
Non-creation of local extrema
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 13
Bernd Girod: EE368 Digital Image Processing
Keypoint detection with automatic scale selection (cont.)
t = 16
t=8
t=4 2
t=4
t=4
Scale-Space Feature Detection no. 14
SIFT blob detection
Harris-Laplacian example (200 strongest peaks)
t=8 2
t=4
SIFT - Scale-Invariant Feature Transform Decompose image into DoG scale-space representation Detect minima and maxima locally and across scales Fit 3-d quadratic function to 9 values to localize extrema with sub-pixel/sub-scale accuracy
t=2 2
t=2
t= 2 t =1
t = 16
t = 64
[Brown, Lowe, 2002]
[Lindeberg 1996] Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 11
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 12
Bernd Girod: EE368 Digital Image Processing
2σ 2
⎞ ⎟ ⎠
Scale-Space Feature Detection no. 8
some initial scale For each Harris corner xh , yh detect characteristic scale
Laplacian zero-crossings
t=1
(
t = σ2 = 1, k = 1.1
th = arg max t ⋅∇ 2 f t xh , yh 3.
) (
1 2 ω + ω 2y F t ω x ,ω y 2 x
)
Scale-space representation provides all scales; which scale is best for keypoint detection? Harris-Laplacian Harris 1. Detect Harris corners at scale t 2.
t=1
1 ⎛ x 2 + y2 ⎞ − e 1− πσ 4 ⎜⎝ 2σ 2 ⎟⎠
t
) (
=−
(
Keypoint detection with automatic scale selection
Low-gradient-magnitude edges removed
(
⎛ t ⎞ 1 2 ω + ω 2y exp ⎜ − ω x2 + ω 2y ⎟ F ω x ,ω y 2 x ⎝ 2 ⎠
Bernd Girod: EE368 Digital Image Processing
H ω x ,ω y = − ω x2 + ω y2 Gσ ω x ,ω y
)
=−
t = σ2 = 1
t = σ2 = 1, k = 1.1
t=1
) ) (
LoG vs. DoG
Zero crossings of Laplacian images
( )
) (
Laplacian of Gaussian
( )
Scale space: edge detection
Difference of Gaussians
f t x, y
(
Bernd Girod: EE368 Digital Image Processing
LoG ( x, y ) = −
LoG vs. DoG (cont.)
)
Original image f (x,y)
Zero-crossings of 2nd derivative: Fewer features at coarser scales
t = σ2 = 1
Scale-Space Feature Detection no. 5
Scale space as heat diffusion ( )
Parametric family of images smoothed by Gaussian filter
Laplacian of Gaussian
(
∂ t 1 f x, y = ∇ 2 f t x, y ∂t 2
Scale-Space Feature Detection no. 3
( )
∂ ∂ t F ω x ,ω y = G t ω x ,ω y F ω x ,ω y ∂t ∂t ⎛ t ⎞ ∂ = exp ⎜ − ω x2 + ω 2y ⎟ F ω x ,ω y ∂t ⎝ 2 ⎠
(
Coarser scales
Bernd Girod: EE368 Digital Image Processing
Separability
Increasing t
scale t
Non-creation of local extrema (for f (x,y) and all of its partial derivatives) since g t x, y ≥ 0 and unimodal. Solution to diffusion equation (heat equation) ∂ t 1 f x, y = ∇ 2 f t x, y ∂t 2
Scale-space representation of images
From an original signal f(x) generate a parametric family of signals f t (x) where fine-scale information is successively attenuated
Scale-space representation of images (cont.)
Bernd Girod: EE368 Digital Image Processing
Scale-space representation of a signal
Commutative semigroup property
Definition and useful properties Scale space and diffusion equation LoG vs DoG
Scale-space edge detection Scale-space keypoint detectors
Scale-space representation of images (cont.)
Scale-Space Feature Detection no. 15
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 16
SIFT scale space pyramid: octave 1
SIFT scale space pyramid: octave 2
SIFT scale space pyramid: octave 5
-
-
-
-
-
-
-
-
-
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 17
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 18
SIFT scale space pyramid: octave 3
SIFT scale space pyramid: octave 4
-
-
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 21
Robustness against scaling
SIFT blobs
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 22
Hessian keypoints in scale space
−g(x, y)
-
-
-
-
[Mikolajczyk, Schmid, 2001] Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 19
SURF blob detection
Scale-Space Feature Detection no. 20
SURF blob detection (cont.)
SURF – Speeded Up Robust Features [Bay, Tuytelaars, Van Gool, ECCV 2006] No subsampling – all resolution levels at full spatial resolution Simple approximation of scale space Gaussian derivatives using integral images
D tyy
Bernd Girod: EE368 Digital Image Processing
D txy
Determinant of Hessian
( ) t
t xx
t yy
(
det H ≈ D D − 0.9D
t xy
)
2
Non-maximum suppression in 3x3x3 [x,y,t] neighborhood
Interpolation of maximum of det(H) in image space x,y and scale t
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 25
SURF blob detection (cont.)
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 27
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 26
SIFT blobs
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 28
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 23
Bernd Girod: EE368 Digital Image Processing
Scale-Space Feature Detection no. 24