Keywords: Blocking artifacts, fast adaptive filter, hybrid filter, JPEG compression. Abstract. Blocking artifacts are the visual artifacts found at block boundaries of ...
Blocking artifacts removal by a hybrid filter method Tuan Q. Pham
Lucas J. van Vliet
Quantitative Imaging Group, Faculty of Applied Sciences, Delft University of Technology, Lorentzweg 1, 2628 CJ Delft, The Netherlands {tuan, lucas}@qi.tnw.tudelft.nl http://www.qi.tnw.tudelft.nl/∼lucas Keywords: Blocking artifacts, fast adaptive filter, hybrid filter, JPEG compression.
Abstract Blocking artifacts are the visual artifacts found at block boundaries of DCT-based compressed images. The blocking artifacts removal method in this paper simultaneously performs an edge-preserving and a low-pass filtering of the degraded image. The deblocked result is then constructed from a soft blending of the two filtered outputs. The novel contribution here is the soft blending that avoids hard segmentation of the image into monotone and textured regions, thereby eliminating boundary artifacts after segment unification. The 1-dimensional oriented filter is an efficient adaptive filter that enhances lines and edges for textured regions. Results from experiments show that the proposed algorithm significantly reduces artifacts of compressed images from a wide bit-rate range in a reasonably short amount of time.
1
Introduction
Blocking artifact is one of the most annoying defects in DCT-based (Discrete Cosine Transform) image compression standards (e.g. JPEG, MPEG), especially when image quality is compromised for low bit-rates. The degradation is a result of a coarse quantization of the DCT coefficients of each image block without taking the inter-block correlations into account. Although more efficient compression schemes do exist, the DCT-based codec has become stable for such a long period of time that it has spread its artifacts over almost all digital images we encounter today. The need for a fast blocking artifact removal technique is therefore a motive that constantly drives new ideas and implementations in this field. There are generally three common post-filtering techniques in the literatures: spatial filtering, DCTbased filtering and hybrid filtering methods. In [7], the gradient energy is used to classify the image into
homogeneous and highly textured regions. Heavily smoothing is performed on the homogeneous regions to reduce staircase blocking artifacts. Textured regions get less filtering because the artifacts sometimes are indistinguishable from the underlying high frequency signal. DCT-based methods such as projection onto convex sets (POCS) [13] try to alleviate the artifacts by applying smoothness constraints on the DCT coefficients. POCS-based methods, however, are notorious for their high computational complexity, which is not practically suitable for mass processing. Other DCTbased methods such as [8] suppress the non-zero DCT coefficients of the in-between blocks. These methods, though good for eliminating artifacts in monotone regions, still need an extra spatial filtering step for textured regions. Recently, a hybrid method that combines frequency and spatial information into a single algorithm [12] is proposed with a good reported result. The method first reconstructs the DCT coefficients based on their observed probability distribution. A spatial filtering step with kernels adapted to local signal further removes block discontinuity, at the same time enhances lines and edges. The adaptive filter, originated from [9], is known to give excellent filtering result but it is complicated and slow. In general, most current blocking artifact removal techniques follow the same scheme depicted in figure 1. The compressed image may undergo a preprocessing stage before being segmented into monotone and complex regions. Each segment is then subjected to a different filter. The filtered segments are finally joined to form the output image. Two problems exist with this common model. First, the input image is hard segmented into two types of patches. The joined output therefore still contains artifacts in the segmentation border. Second, the filtering for textured regions is either done very little
2D edgeadaptive filtering Edgeness segmentation Low-pass filtering
Figure 1: Common deblocking approach classifies the input image into homogeneous and textured regions. to keep the perceptual sharpness [7], or is over done at a high computational cost [12]. In this paper, we propose a solution to each of these problems. As can be seen from our modified filtering scheme in figure 2, a soft blending of two filtered results replaces the hard segmentation. In addition, a fast and effective edge enhancing technique stands in place of the edge preserving technique in the first diagram. 1D edgeadaptive filtering
T = ∇I∇I T = λu uuT + λv vvT A = (λu − λv )/(λu + λv ) E = λu + λv
φ = arg(u) κ = ∂φ/∂v (1)
weight
Low-pass filtering
image gradient is one crude estimator of the local orientation, this estimator is very susceptible to noise. To reduce the effect of noise, multiple gradient vectors in a local neighborhood are gathered for a Principal Component Analysis (PCA). The outcome of this regularization is not just a robust orientation estimate but also a confidence measure: the anisotropy (the degree of elongation of the ellipse in figure 3). The anisotropy takes values from 0 to 1, where 0 means no dominant local orientation exists and 1 signifies the presence of a strong dominant orientation.
1-weight
Figure 2: Our deblocking approach by soft blending of filtered outputs without texture classification. The remaining part of the paper is organized as follows: section 2 introduces a hybrid adaptive filtering technique which includes a fast 1-D oriented filter for lines and edges enhancement. The algorithm, results and commercial applicabilities of our deblocking technique are presented in section 3. Section 5 finally concludes the paper.
Figure 3: The gradient structure tensor as a principal component analysis of local gradient vectors. The gradient structure tensor T is computed from local gradient vector ∇I = [∂I/∂x ∂I/∂y]T as in equation (1), where the eigenvector u corresponding to the largest eigenvalue λu determines the local orientation φ; E, A and κ denote gradient energy, anisotropy and curvature, respectively. Sobel operator [3] is used to compute the image gradients and Gaussian smoothing filter of scale 2 is used for tensor regularization (∇I∇I T ). Curvature is defined as the rate of change of orientation along the dominant linear structure. Note that a proper mapping of φ which contains jumps between ±π must be employed to compute κ [2].
2.2 One-dimensional oriented filter 2
Structure Adaptive Filtering
This section presents a fast algorithm for edgepreserving adaptive filtering. The filter is a weighted average of a one-dimensional edge-adaptive filter and a low-pass filter. The adaptive parameters are computed from the image gradient information using the Gradient Structure Tensor (GST) [15].
2.1 Gradient structure tensor Before any structure adaptive adaptive filtering can be applied, we need to know the edgeness and orientation of every pixel in the image. The GST, also known as the second moment matrix [12], is employed for this purpose (equation 1). Although the direction of
Anisotropic Gaussian filtering whose kernel is proportional to the the inverse of the eigenvalues of the GST has been widely used in the literature [12][9]. However, the drawback of this technique lies in its speed. Since a different rotated and shifted anisotropic Gaussian kernel must be computed at each location, its computational complexity is high. Instead of using a full two-dimensional adaptive kernel, we propose to filter the image in only one orientation along the local linear structure. This 1D filter aligns with local orientation and it is slightly bent according to the local curvature to better fit the image structure (figure 4a). The scale of the 1D filter (σ1D ) varies according to the anisotropy to minimize over-smoothing of minute details and maximize noise reduction along
ϕ u
v
y = 12 κ x2 (a) noisy input (SNR=14dB) & a 1D oriented & curved filter
(b) 1D oriented filter does not filter flat regions very well
(c) low-pass filter reduces noise well but smoothes out edges
(d) hybrid filter retains edge sharpness and filters noise well
Figure 4: Performance of the hybrid adaptive filter. Note the edge continuity and sharpness in figure 4d. elongated structures. In equation (2), we set σ2D = 2 and an edge-enhancing parameter α=1. σ1D = σ2D Aα
(2)
The 1D oriented filter can be implemented in a simple and efficient manner. Rather than having to recompute the coefficients of a 2D adaptive filter at every pixel as done in [9], we use lookup filters of fixed sizes and resample the image at filtered positions instead. The coefficients of the lookup filters at discrete scales (0.5, 1, 1.5, 2, ...) are stored in a database. The image is resampled along the positions of the oriented and curved structure then a 1D convolution is performed per output pixel. Note that for a band-limited image, a good interpolator does not introduce any error. For an aliased image, there is a certain smoothing due to resampling. However, the error is negligible if the image is resampled with enough pixels (we use a resampling step ≤ 1 pixel.) A right combination of the resampling step and the lookup filter can filter the image one-dimensionally at any desired scale.
2.3
A hybrid adaptive filter
The 1D oriented filter, however, is not designed to be a stand-alone filter. Although it filters lines and edges well, flat regions are hardly filtered because of a small anisotropy. To handle these flat regions, we filter the image separately with a Gaussian filter of scale σ2D and later blend the two filtered results together
using the scheme in figure 2. Note that Gaussian filtering is separable and could be done very efficiently by a recursive implementation [14]. As a result, the computational complexity is dominated by the 1D filter only, which is much more efficient than the fullkernel adaptive 2D filter in [9]. The output of the hybrid filter is a linear blending of the edge-adaptive and low-pass filtered results using a variable blending function. This blending weight is based on gradient energy and is designed such that the output takes values from the edge-adaptive image for textured regions and values from the low-pass image for flat regions. This makes up for the drawbacks of both filter modes: edgeadaptive does not filter flat regions well (figure 4b), and low-pass filtering destroys edges (figure 4c). The blending weight in equation (3) maps the gradient energy E into a [0 1] interval. The normalization term E0 = mean(E) determines the soft transition point of the blending function from 0 to 1. Using this hybrid filter, a noisy image in figure 4a could be filtered very effectively as shown in figure 4d. w1 = exp(−E/E0 )
(3)
Interestingly, the hybrid filter can be seen as an extension to the local statistics filter developed by Lee [6] in the early 1980’s. The Lee filter performs a weighted average of the input image and its low-pass version. Its major drawback is that it fails to remove noise in the vicinity of lines and edges. We overcome this by pre-filter the input along local structure before the weighted average. As a result, our hybrid filter can produce a sharp output and at the same time reduce noise over the whole image.
3 Blocking Artifact Removal Algorithm Blocking is the most common compression artifact in JPEG/MPEG images. The artifact is caused by space-variant quantization noise of the low bit-rate Discrete Cosine Transform (DCT) compression algorithm [11]. This section uses the hybrid filter in section 2 to reduce quantization noise and re-establish image continuity across adjacent coded blocks. However, since quantization noise at low bit-rate creates false edges at block boundaries, these edges are preserved by the original hybrid filter. To correct for this undesirable effect, we assign low certainty values to pixels close to block boundaries. Robust orientation estimation is computed from a modified Gradient Structure Tensor (GST) for uncertain data. Given a correct orientation estimate at block boundaries, the 1D oriented filter is elongated to connect linear structures on both sides of the boundaries.
3.1
∇I∇I T in equation (1) is replaced with a normalized averaging [5]:
Blockiness and sample certainty
Since DCT compression at low bit-rate results in higher error variance for locations near block boundaries, the boundary pixels are generally less reliable than the pixels in the middle. We assign a certainty value to each boundary pixel based on its blockiness. The blockiness is defined for pixels on an 8x8 DCT block: vertical edge bx = | |Ix | − |Ix | | by = | |Iy | − |Iy | | horizontal edge b= max (bx , by ) corner pixels 0 middle pixels (4) where |Ix | and |Iy | are the gradient magnitudes in x and y-dimension; |Ix | and |Iy | are the average gradient magnitudes in a local neighborhood (σ = 1). In words, the blockiness equals signal activity at block boundary less the mean activity. The sample certainty is then defined as: c = exp(−b/b0 )
(5)
where b0 = mean(b | b 6= 0) determines the soft transition point of c from 0 to 1. The image c is further blurred by a Gaussian filter with a small scale (σ = 0.7) so that the certainty changes gradually from one in the middle to a smaller number at the boundary. Example of the blockiness and certainty of a portrait image compressed1 with JPEG quality 10 can be seen in figure 5.
T ∗ = ∇I∇I T c/c
(6)
where s is a Gaussian filtered version of an image s at scale σ2D = 2.
3.3 Hybrid filter for DCT-coded images With an estimation of certainty, orientation and gradient energy, the DCT-coded image can be deblocked using the hybrid filter. The only parameter changed from the original hybrid filter in section 2 is the scale of the one-dimensional oriented filter: ∗ σ1D = σ2D (1 − c)
(7)
This scale reflect a different edge-preserving filter for DCT-coded images: instead of scaling the filter according to the anisotropy as in (2), the filter is extended at block boundary pixels to connect linear structures on both side of the boundary. This edgeconnecting capability is achievable thanks to the modified orientation estimate that is robust against blocky quantization noise. C 1D oriented IEP w filtering ,E Deblocked image Low-pass ILP (1-w) filtering
Certainty & Orientation JPEG image
Figure 6: Our deblocking algorithm.
(a) blockiness 0 ≤ b ≤ ∞
(b) certainty 0 ≤ c ≤ 1
The presented deblocking algorithm requires only one parameter to be fine-tuned by user, that is the maximum scale σ2D . This scale controls the maximum smoothing for both the 1D oriented filtering and the low-pass filtering in figure 6. Depending on the compression quality of image, σ2D could take value from zero to three. One can even automate this parameter by a no-reference perceptual quality assessment of JPEG compressed images [16].
Figure 5: Blockiness and certainty of Lena at 0.1 bpp.
4 3.2
Orientation from uncertain data
Due to the space-variant quantization error, the gradient images are less accurate near block boundaries, which in turns causes erroneous orientation estimate. To reduce the influence of unreliable gradient vectors in the GST, the certainty developed in subsection 3.1 is assigned to each vector before the PCA. In other words, the Gaussian smoothing of the tensor 1 compressed
by Irfanview, a freeware graphic viewer.
Results and Analysis
The results of blocking artifacts removal by the hybrid filtering method can be found in figure 7. Images on the left column are JPEG compressed with quality 30 and 10, which corresponds to 0.2 and 0.1 bpp respectively. Corresponding images on the right column are the result of our deblocking algorithm. As can be seen in figure 7a, JPEG compression at a low bit rate (0.2 bpp) clearly shows some blockiness. These blocking artifacts are effectively removed in figure 7b. Although the deblocked image may appear a bit smoother than the blocky input, all linear edges
are perfectly retained. Small features like the pupils of the eyes are also still intact. At a very low bit rate (0.1 bpp), the compressed image in figure 7c shows severe staircase artifacts. By aggressively smoothing flat regions while restoring linear edges, the artifacts are eliminated altogether in figure 7d. Yet, not a single detail from the low-quality compressed image is missing from the deblocked result. A JPEG quality score [16] computed on the images in figure 7 further confirms the effectiveness of the hybrid filter method. The jpeg score ranging from zero to ten measures visual quality of images in terms of continuity across block boundaries. At 0.2 bpp, the Lena image in figure 7a scores only 3.2 points while its postprocessed image in figure 7b almost reaches top score. With a jpeg score of 9.9, the deblocked image shows no sign of artifacts as can also be seen in zoomed-in versions in figure 8. Not surprisingly, the very low quality JPEG image in figure 7c scores a merely 0.3 point. However, our deblocking algorithm restores it to a jpeg score of 9.3 in figure 7d. Comparisons with several deblocking methods such as Chou [1] and reapplication of JPEG [10] in figure 8 further confirm an excellent performance of the hybrid filter method. Unlike other methods, our deblocking algorithm does not smooth out the image after removing blocking artifacts. This is possible because the algorithm uses a small smoothing scale at high detailed pixels. More importantly, the smoothing is only one-dimensional along linear structures. Lines and edges are therefore re-connected rather than being diffused across. In addition to removing blocking artifacts, our algorithm also removes ringing effects around edges as well as noises in flat regions. All these favorable features make the algorithm versatile enough to be used as a post-processing filter across a wide bit-rate range.
5
Summary
In short, we have presented an efficient adaptive filtering technique that can be applied to removing blocking artifacts from DCT-based compressed images. The outputs of the deblocking algorithm show no perceptual artifacts for a wide range of compression bit-rates. What is more, the output images retain their sharp edges with both structured and unstructured noise being removed.
References [1] J. Chou, M. Crouse, and K. Ramchandran. A simple algorithm for removing blocking artifacts in block-transform coded images, in Proc. of ICIP’98, Chicago, pp. 377–380, 1998. [2] M. van Ginkel, J. van de Weijer, L.J. van Vliet and P.W. Verbeek. Curvature estimation from
orientation fields, in Proc. of SCIA’99, Greenland, pp. 545–551, 1999. [3] R. Gonzalez and R. Woods. Digital Image Processing, Addison Wesley, 1992, pp. 414–428. [4] L. Karray, P. Duhamel, and O. Rioul. Image coding with an L∞ norm and confidence interval criteria. IEEE TIP, 7(5):621–631, 1998. [5] H. Knutsson and C-F. Westin. Normalized and differential convolution: methods for interpolation and filtering of incomplete and uncertain data. In Proc. CVPR, 1993. pp. 515–523. [6] J.S. Lee. Digital image enhancement and noise filtering by use of local statistics. IEEE PAMI, 2:165–168, 1980. [7] Y.L. Lee, H.C.Kim and H.W. Park. Blocking effect reduction of JPEG images by signal adaptive filtering, IEEE TIP, 4(7):229–234, 1998. [8] Y. Luo and R.K. Ward. Removing the Blocking Artifacts of Blocked-Based DCT Compressed Images, IEEE TIP, 12(7):838–842, 2003. [9] M. Nitzberg and T. Shiota. Nonlinear image filtering with edge and corner enhancement, IEEE Trans. on PAMI, 14(8):826–833, 1992. [10] A. Nosratinia. Enhancement of JPEGcompressed images by re-application of JPEG, J. of VLSI Sig. Proc., 27:69–79, 2001. [11] M.A. Robertson and R.L. Stevenson. DCT quantization noise in compressed images Robertson, IEEE Trans. Circuits and Systems for Video Technology, 15(1):27–38, 2005. [12] G.A. Triantafyllidis, D. Tzovaras, D. Sampson, and M.G. Strintzis. Combined Frequency and Spatial Domain Algorithm for the Removal of Blocking Artifacts, EURASIP JASP, 2002(6):601–612, 2002. [13] Y. Yang, N.P. Galatsanos and A.K. Katsaggelos. Projection-Based Spatially-Adaptive Reconstruction of Block Transform Compressed Images, IEEE TIP, 4(7):896–908, 1995. [14] I.T. Young and L.J. van Vliet. Recursive implementation of the Gaussian filter, Signal Processing, 44(2):139–151, 1995. [15] L.J. van Vliet and P.W. Verbeek. Estimators for orientation and anisotropy in digitized images. In Proc. of ASCI’95, the Netherlands, 1995. pp. 442–450. [16] Z. Wang, H.R. Sheikh and A.C. Bovik. Noreference perceptual quality assessment of JPEG compressed images, Proc. of ICIP, Sept. 2002.
(a) JPEG quality 30 (0.2 bpp), jpeg score = 3.2
(b) Deblocking result of (a), jpeg score = 9.9
(c) JPEG quality 10 (0.1 bpp), jpeg score = 0.3
(d) Deblocking result of (c), jpeg score = 9.3
Figure 7: Results of JPEG/deblocked pairs using the hybrid filter deblocker.
input
Chou [1]
input
Chou [1]
reapply JPEG [10]
hybrid filter
reapply JPEG [10]
hybrid filter
Figure 8: Comparison of different deblocking results of a JPEG compressed input at quality 30.
