➠
➡ COLOR IMAGE CODING BY UTILIZING THE CROSSED MASKING Yan Meng
Linfeng Guo
Stevens Institute of Technology Hoboken, NJ 07030
[email protected] ABSTRACT A color image coding scheme that takes advantage of the chrominance-on-luminance masking effect is proposed in this paper. This coding scheme exploits the remarkable crossed masking effect, such that more distortion in luminance will be hidden in the places where the chrominance components are strong. First, the asymmetrical crossed masking effects are introduced. Next, proper evaluation of the luminance Just-Noticeable-Difference (JND) threshold lifting due to the chrominance masking is investigated. Based upon that, the crossed masking effect is utilized to achieve extra gain on color image compression under the dyadic wavelet transform (DWT) centric image coding framework without compromise of viewing quality. Meanwhile, no any overhead of side information is invoked in the proposed scheme.
1. INTRODUCTION Image compression technology has achieved great success in the senses of both technology and business. Since 1990s, Discrete Wavelet Transform (DWT) began to steal the thunder of the Discrete Cosine Transform (DCT) in the field of image compression and was finally adopted as the transformation of the second generation of the international still image compression standard – JPEG 2000 [1]. Since DWT mimics the early stage of human vision’s processing of the incoming visual stimulus to great extent, it provides a perfect platform for schemes of visual quality optimization. In short, three aspects of visual properties are utilized to make the compression more visually optimized [2-3]: contrast sensitivity, luminance adaptation, and masking effect. For instance, the JPEG2000 standard has integrated a handful of optional visual optimization tools that take advantage of both the contrast sensitivity and the masking effect [4]. Although most of the digital images we deal with are color images, the compression technique behind them just simply treats each color image as the compound of three independent grayscale images and compresses them separately (Actually it does do something to take
0-7803-8874-7/05/$20.00 ©2005 IEEE
BrainMedia LLC New York, NY 10016
[email protected]
advantages of the dependencies amongst the different color components, such as the conversion from RGB space to YCrCb space and the down-sampling of Cr and Cb components). In fact, the strong perceptual dependences amongst the luminance and the chrominance components are well-known in the psychovisual research community. However, image compression researchers either are not quite aware of them or just simply tend to ignore them historically. In this paper, we will try to exploit the perceptual interactions amongst the superimposed luminance and chrominance components, and utilize them into the image compression. 2. CROSSED AND UNCROSSED MASKING 2.1. Crossed Masking between Luminance and Chrominance Visual masking is a psychological phenomenon that has been investigated for decades [5]. Crossed masking between luminance and chrominance denotes the masking between the luminance masker and the chrominance target, or vice versa. Extensive researches have been conducted on this phenomenon. Although the actual experimental results vary to different extents, researchers have drawn some consensuses: x Low spatial frequency chrominance maskers have no effect on luminance patterns [6-7]. x At low spatial frequency, there is no remarkable asymmetry of the crossed masking effects [8-9] (Luminance masks color and color masks luminance). x At medium to high spatial frequency, there is asymmetry between the crossed masking effects. Specifically, the masking on the luminance target by the chrominance masker is much stronger than vice versa. x Over a wide range of contrast, luminance masks facilitate the detection of both chrominance gratings and colors [8]. x The chrominance-on-luminance masking is quite similar to the luminance-on- luminance masking. To give an illustration, a curve of threshold versus frequency (TvF) function of crossed masking [10] is
II - 389
ICASSP 2005
➡
➡ reprinted here under the permission of the author. The x-axis denotes the ratio of the spatial frequencies between the masker and the target, and the y-axis denotes the difference (in dB) of the detection thresholds (represented as the contrast) with and without the masker. The solid line is for the masking of the red-green gratings by the yellow-black gratings, and the dashed line is for the masking of the yellow-black gratings by the red-green gratings. The result is averaged from the data of two subjects, including all masker and target frequency combinations of 1, 2, 4 and 8-cpd gratings at the relative phase of 0o, 90o, and 180o.
One probe pattern is masked by more than one mask pattern is called simultaneous masking. As an important aspect of masking effect, the simultaneous masking was also investigated in the psychovisual science [12-13]. The general findings for the simultaneous masking by two maskers can be summarized as follows: z When one masker has high contrast and the other has low, the masking by the high contrast masker counteracts the facilitation effect of the low contrast masker, so that the facilitation disappears. z When both maskers have high contrast, the simultaneous masking effect is close to that of only a single masker. It’s very easy to notice that the masking effect on luminance by chrominance is as strong as the luminance-on-luminance masking (i.e., uncrossed masking), which has been extensively utilized in gray level image compression for long time [2-4]. However, the chrominance-on-luminance masking is seldom mentioned in color image compression. Since most of the digital images we deal with today are color image, utilizing this effect in compression is quite useful and makes lots of sense. 3. PROPOSED COLOR IMAGE CODING SCHEME 3.1. Color Space
Fig. 1. Crossed masking of color-on-luminance and luminance-on-color 2.2. Crossed Masking between Different Colors The crossed masking between two patterns with different colors is quite symmetric and similar to the masking between two patterns with the same color. The data of crossed-color masking from [8] is reprinted in Fig. 2, where a greenish (solid symbols) or reddish (open symbols) target was presented with a greenish (left half of graph) or reddish (right half of graph) masker. Data are plotted in units of “no masker” thresholds.
The color space conversion from RGB to YCrCb provides a fairly good framework for the utilization of chrominance-on-luminance masking effect, but not perfect. The reason of its non-perfection is that the chrominance the psycho-visual science researchers talk about is isoluminant chrominance, but the chrominance components of Cr and Cb are color difference signals, which still contain a fairly amount of luminance information. The influence of the inherent luminance information in Cr and Cb on the masking evaluation will be investigated in the future. As the pilot testing of the feasibility, we just simply ignore this non-perfection here. 3.2. Coding Framework
Fig. 2. Crossed-color masking effect 2.3. Simultaneous Masking
To incorporate the crossed masking effect in compression, the Dyadic Wavelet Transform (DWT) based image coding framework is adopted. First, a digital color image is converted to YCrCb space from RGB space, and the YCrCb is 4:1:1 sampled. Secondly, each component is transformed to 5-level subbands (totally 16 subbands) through successive DWT using biorthogonal 9/7 bases. The three subbands in each level are named as HH, HL, and LH, respectively. The four subbands in the lowest decomposition level are labeled as LL5, HL5, LH5, and HH5. Finally, the Cr and Cb are uniformly quantized and entropy coded by utilizing the zero-tree concept. Since
II - 390
➡
➡ this scheme can be tested in any DWT based image coder, to save the coding work, a simple image coder called MRWD [11] is adopted here. 3.3. Connection between Wavelet Coefficient and Visual Perception As illustrated below, a single coefficient of a subband in wavelet domain is corresponding to a contrast pattern in image space through the reconstruction.
masker can be fitted by a straight line. In order to utilize the chrominance-on-luminance masking, the quantized Cr and Cb components are de-quantized to compute the threshold elevation factor wCM of chrominance masking on the coefficients in Y, i.e., wCM (b, i, j ) max § § § v~ (b, i, j ) · 0.60 · § § v~ (b, i, j ) · 0.60 · · ¨ b r ¨ ¸ ¨ ¸¸ ¨ ¸ ¨¨ max¨1, ¨ t (b, i, j ) ¸ ¸, max¨1, ¨¨ t (b, i, j ) ¸¸ ¸ ¸¸, ¹ ¹¹ ¹ ¹ © © r © © b © (1)
Fig. 4. Detection contrast curve for a target in the presence of a masker.
Fig. 3. Reconstructed contrast patterns of wavelet coefficients in image space Top left: LL subband; top right: HL subband; bottom left: LH subband; bottom right: HH subband. The contrast pattern has the following characteristics: Its position in image space is determined by the position of the coefficient in wavelet subband. x The size of pattern produced by children coefficients is about ¼ of that produced by their parent. x Contrast patterns exhibit different orientations. It is easy for those researchers in psychovisual science to recognize that the above reconstructed patterns resemble the Gabor patterns used in psycho-visual experiments. In other words, the characteristics of human visual system exploited from the Gabor pattern based psychological experiments can be safely applied to the perception of the patterns reconstructed from wavelet coefficients. Therefore, the evaluation of crossed masking based on Gabor patterns can also be used on the wavelet coefficients. x
where v~b (b, i, j ) and v~r (b, i, j ) denote the de-quantized coefficients (i, j ) in subband b of chrominance Cb and Cr, respectively; t b (b, i, j ) and t r (b, i, j ) denotes the intensity threshold CT 0 in Fig. 4 for Cb and Cr components above which the masking effect becomes more and more obvious, and they have been measured through 2AFC psycho-visual experiments. Note that the adopted max function scheme (choosing the max masking of either Cr or Cb as the decisive impact factor) of utilizing simultaneous chrominance-on-luminance masking effects is supported by the experimental evidences explained in Sec. 2.3. Finally, the subbands of Y are quantized level by level and in raster scan within each subband, so that the cross-bands masking can be utilized. The quantization equation for the coefficient v(b, i, j ) in subband b of luminance Y is
3.4. Crossed Masking based Adaptive Quantization v q (b, i, j )
Normally, the relationship between the detection threshold of test pattern and the contrast of mask pattern can be depicted as the curve in Fig. 4. The x-axis in Fig. 4 denotes the contrast of the masker, and the y-axis denotes the detection threshold of the target contrast pattern. When the masker’s contrast is below C T 0 , the detection of the target is not influenced by the masker. When the contrast of the masker keeps increasing, the detection of the target is becoming more and more difficult. In a log-log plot like in Fig. 4, the relationship between the detection threshold of the target and the contrast of the
ª v (b, i, j ) º « » ' w ( b , i , j ) CM ¬ ¼
(2)
where ' denotes the step size of uniform quantizer, which is applied to each subband in Y component; wCM ( b, i , j ) denotes the threshold elevation factor on coefficient v(i, j ) due to the chrominance masking. Since there is no remarkable chrominance-on-luminance masking effect at low spatial frequency, no chrominance masking is applied to the subbands in the lowest frequency level of Y.
II - 391
➡
➠ 6. REFERENCES
3.5. Decoding Procedure During the decoding, the color components Cr and Cb are entropy decoded and de-quantized first. After that, the JND lifting of luminance coefficients due to chrominance masking is computed using the equation (1), such that the adaptive quantization step size for each wavelet coefficient in luminance Y can be derived. Then the Y component is entropy decoded and de-quantized. After that, the inverse dyadic wavelet transform is applied to Y, Cr, and Cb in successive way to covert the coefficients from wavelet domain to space domain. Finally, the Cr and Cb components are up-sampled and YCrCb components are converted back into RGB space. Apparently, there is no extra side information overhead for the utilization of the crossed masking based adaptive quantization. 4. EXPERIMENTAL RESULTS The performance of the proposed HVS-based wavelet color image coding method is evaluated through subjective evaluations. In order to do that, a couple of true color images have been encoded at several different bit rates. At each bit rate each image is compressed twice, one with the consideration of the extra JND lifting on luminance due to chrominance through the procedure described above, and the other without. The coding results show that at the bit rate of about 0.75 bpp, coding by utilizing the crossed masking can save 0.1 bpp while still maintain the same perceptual quality at the specified viewing distance. However, it is also noticed that the image quality through crossed-masking encoding is inferior to that through non-crossed-masking one at extremely low bit rate, e.g., at the compression ratio of 100:1. We believe the reason is that the proposed crossed-masking based coding algorithm still tends to reserve a significant amount of coding bits for the color components even at extremely low bit rate, even though the perceptual quality of luminance details cannot be kept any more. Apparently, it is a questionable decision. 5. CONCLUSIONS In this paper, we investigated the cross masking effect of human visual system, and used that to visually optimize the DWT based image compression. The subjective evaluations of the coded images proved that the proposed scheme has obtained extra visual quality enhancement. Meanwhile, since the adaptive quantization step sizes for Y coefficients can be derived from the decoded coefficients in Cr and Cb, no overhead of side information is invoked.
[1] ITU-T Recommendation T.800 | ISO/IEC 15444-1: JPEG 2000 Image Coding System: Core Coding System. [2] Ingo Hontsch and Lina J. Karam, “Locally adaptive perceptual image coding,” IEEE Trans. on Image Processing, vol. 9, no. 9, pp. 1472-1483, Sept. 2000. [3] A. B. Watson, etc, “Visibility of Wavelet Quantization Noise”, IEEE Transactions on Image Processing, vol. 6, no. 8, pp. 1164-1175, August, 1997. [4] W. Zeng, S. Daly and S. Lei, “An overview of the visual optimization tools in JPEG 2000," Special Issue on JPEG-2000 of Signal Processing: Image Communication, vol 17, no 1, pp. 85-104, Jan. 2002. [5] Brian A. Wandell, Foundations of Vision, Sinauer Associates, 1995. [6] A. E. Elsner, Joel Pokorny, and Stephen A. Burns, “Chromaticity discrimination: effects of luminance contrast and spatial frequency,” Journal of Optical Society of America A, vol. 3, no. 7, pp. 916-920, July 1986. [7] Eugene Switkes, Arther Bradley and Karen K. De Valois, “Contrast dependence and mechanisms of masking interaction among chromatic and luminance gratings,” Journal of Optical Society of America A, vol. 5, no. 7, pp. 1149-1162, July 1988. [8] G. R. Cole, C. F. Stromeyer III, and R. E. Kronauer, “Visual interaction with luminance and chromatic stimuli,” Journal of Optical Society of America A, vol. 7, pp. 128-140, 1990. [9] Ram L. Pandey Vimal, “Color-luminance interaction: data produced by oblique cross masking,” Journal of Optical Society of America A, vol. 15, no. 7, pp. 1756-1766, July 1998. [10] Eugene Switkes and Karen K. De Valois, “Luminance and chromaticity interactions in spatial vision,” Color Vision, pp. 465-470, edited by J. D. Mollon and L. T. Sharpe, Academic Press, 1983. [11] Sergio D. Servetto, Kannon Ramchandran, and Michael T. Orchard, “Image coding based on a morphological representation of wavelet data,” IEEE Trans. on Image Processing, vol. 8, no. 9, pp. 1161-1174, Sept. 1999. [12] John M. Foley, “Human luminance pattern-vision mechanisms: masking experiments require a new model,” Journal of Optical Society of America A, vol. 11, no. 6, pp. 1710-1719, June 1994. [13] Kathy T. Mullen and M. Angeles Losada, “Evidence for separate pathways for color and luminance detection mechanisms,” Journal of Optical Society of America A, vol. 11, no. 12, pp. 3136-3151, Dec. 1994.
II - 392
