proposed adaptation of MGS based SVC first set adaptation unit of AU (Access Unit) or .... SVC reference software JSVM (Joint Scalab. 9.14[7]. Assumption that ...
Adaptation of Scalable Video Coding to Packet Loss and its Performance Analysis Euy-Doc Jang *, Jae-Gon Kim *, Truong Thang**,Jung-won Kang**
*Korea Aerospace University, 100, Hanggongdae gil, Hwajeon-dong, Deogyang-city, Gyeonggi-do. 412-791, KOREA **Electronics and Telecommunications Research Institute, 138 Gajeongno. Yuseong-gu. Daejeon. 305-700. KOREA {jangeuydoc, jgkim}@kau.ac.kr, {tchang, jungwon}@etri.re.kr
Abstract— SVC (Scalable Video Coding) is a new video coding standard to provide convergence media service in heterogeneous environment with different networks and diverse terminals. This paper presents the performance analysis on packet loss in the delivery of SVC over IP networks and an efficient adaptation method to packet loss. In particular, SVC with MGS (Medium Grained Scalability) as well as spatial and temporal scalabilities is addressed in the consideration of packet-based adaptation since finer adaptation is possible with a sufficient numbers of quality layers in MGS. In order to minimize quality degradation resulted by packet loss, the proposed adaptation of MGS based SVC first set adaptation unit of AU (Access Unit) or GOP corresponding to allowed delay and then selectively discard packets in order of importance in terms of layer dependency. In the experiment, the effects of packet loss quantitative qualities are analyzed and the effectiveness of the proposed adaptation to packet loss is shown. Keywords— SVC, MGS, Adaptation, Packet Loss
I. INTRODUCTION In convergent media environments, users may access and interact with multimedia contents on different type of terminals and networks. Such an environment includes a variety of multimedia terminal such as PC, TV, PDA, or cellular phones. In such environment, much more attention is being focused on video adaptation as a promising approach for keeping the QoS (Quality of Service) of video contents.[1] In general, scalable format is required to provide efficient video adaptation that performed by using simply process. Scalable video coding (SVC), which is developed by the Joint Video Team (JVT) from ISO/IEC MPEG and the ITU-T VCEG as an extension of the H.264/AVC, is a new video coding standard to provide efficient video adaptation[2][3]. Scalable streams of SVC are composed of a base layer that is compatible to H.264/AVC and one or more enhancement layers, which may enhance the spatial-temporal resolution and/or quality of the base layer. Based on such scalable layered structure, SVC can be easily adapted to meet diverse constraints imposed by devices and connections by simple truncation.
In this paper, we consider the adaptation of SVC to packet loss in IP networks. In particular, SVC with MGS (Medium Grained Scalability) as well as spatial and temporal scalabilities is addressed in the consideration of packet-based adaptation since finer adaptation is possible with a sufficient numbers of quality layers in MGS. First, the effect of random packet losses on the received quality of SVC with MGS is evaluated in terms of spatial quality and temporal resolution. Based on the results, we proposes and adaptation method of SVC to reduce the quality degradation caused by packet losses. The adaptation method first set the adaptation unit as multiple times of AU (Access Unit) or GOP (Group of Picture) according to allowed delay or buffering, and then accommodating packet losses that may occurred by scheduling in a network node by discarding the packets in order of their importance in terms of layer dependences among scalable layers. The reminder of this paper is organized as follows. In Section II we briefly introduce the scalable structure of MGS. The proposed adaptation of SVC with MGS is presented in Section III. Section IV presents the experiments to evaluate the effect of packet loss and the performance improvement of the proposed adaptation. Finally, conclusion is provided in Section V. II. SCALABILITY : MGS The SVC bit stream may be composed of multiple spatial, temporal, and SNR layers in a combined way. Typical SVC includes multiple spatial layers, each of which has a base quality layer with multiple enhanced quality layers. In addition, temporal scalability is provided based on hierarchical-B picture structure. Quality scalability is provided by either of coarse-grain quality scalability (CGS) layer or medium-grain quality scalability (MGS). In this paper, we mainly deal with MGS in which refinement coefficients can be split in several fragments to cope with packet losses with sufficient number of quality layers[5]. In MGS, the coded data corresponding to a quantization step size can be fragmented into at most 15-layer. In other words, a base-layer with minimum quality is quantized with a larger step size and then at the quality enhancement layer, quantization error signal that is the difference between original image and the decoded image is quantized with a smaller step size. The set
of MGS layers corresponding to a quantization step size is named as “MGS stack”[4][5]. Illustration of an SVC AU with MGS coding is shown in Fig. 1. This example AU contains two spatial layers. Both of the low spatial layer and enhancement spatial layer are enhanced by two MGS stacks, each of which is composed 3 MGS layers. residual coeff motion data
4CIF
MGS layers corresponding to a quantization step size
2nd spatial layer D=1
MGS layers corresponding to a quantization step size quality base
MGS1
D=2
(2,2,0)
IDR
(1,1, 1)
(1,2,1)
(1,0,1) Q=1
CIF
IDR
D=1
(1,2,0) Q=0
IDR
Q=0
MGS1 NAL Type=20,idr_flag=1 QCIF IDR
Q=6 Q=5 Q=4 Q=3
IDR
Q=0
Q=2 Q=1 quality base
GOP AU
Q=6 Q=5 Q=4 Q=3
In the case of NAL unit loss caused by packet loss, NAL units that depend on the lost NAL unit can’t be decoded as well as the lost NAL unit itself. Therefore, it is necessary to modify the corrupted stream into a valid stream to be decoded correctly by discarding the dependent dent NAL NA unit additionally, prior to submitting to the decoder[6].
NAL Type=5 T=0
Q=1
NAL Type=20 20,idr_flag=0
NAL Type=1 Type 2
1
D=0
Q=0 2
0
tim e
1st spatial layer D=0
Q=2 Q=1 Q=0 Access unit
Fig. 1 MGS quality-layer layer structure in an access unit
III. EFFECT OF PACKET LOSS AND ADAPTATION TECHNIQUE In this section assume that SVC with MGS, MGS we evaluate the effect of packet losses on the quality of decoded SVC sequence. Based on the evaluation, we propose adaptation method to minimize the quality degradation caused by packet losses. A. Effect of Packet Loss The scalable layer of SVC is identified using a triple ID, consisting of the Dependency ID, Temporal ID, Quality ID, which is referred as triple (D,T,Q). For example, a base layer Network Abstraction Layer (NAL) unit of lowest temporal resolution and SNR scalability be identified as (D,T,Q) = (0,0,0). SVC layers are highly interdependent on each other, which means that the loss of a NAL unit of a certain layer may cause a severe degradation of quality or even prevent the correct decoding of other layer due to error propagation agation. For instance, Fig. 2 shows an interdependent structure in a SVC bit stream encoded with three spatial layers, three temporal layers and MGS layers. In a GOP, if a NAL unit (1,2,0) is lost, then all NAL units that depend on it (i.e., NAL unit (1,2,1)) can’t be decoded even if they are correctly received. In addition, other temporal and spatial layers also have interdependency, interdependency and thenn they are cause error propagation if the packet pay loading the reference data is lost.
Fig. 2 Interdependency structure of a SVC
B. Proposed SVC Adaptation Method As shown in the previous Section, ection, the SVC bit stream could constitute an interdependent scalable structure, which in turn leads to poor quality result when it is exposed to random packet loss. In order to alleviate this problem, in this section test several network based adaptation mechanism which selective discard packets, and evaluating reception quality qu using Y-PSNR. In adaptation of selectively discard packet, first we set adaptation unit of AU or GOP corresponding to allowed delay. When a longer delay is allowed, the adaptation unit can be extended to multiples of AU or GOP. In the case of adaptation unit is AU, random packet losses loss causes arbitrary packets discarding ing but the proposed method is able to selectively discard the packets that has lower dependency as shown in Fig. 3.
Fig. 3 Selectively electively discard packets in AU adaptation unit
When a packet loss is occurred over an adaptation unit, a scheduler selectively discards a packet that is the least important in terms of layer dependency in an adaptation unit. In the experiment to show the impact of random packet loss on SVC bit stream, video sequences is encoded by using the SVC reference software JSVM (Joint Scalable Video Model) 9.14[7]. Assumption that each scalable layer in the SVC bit
stream is composed in a NAL unit and each NAL unit is packetized into a RTP packet. In addition, the base layer of each AU is not affected from packet loss. Each NAL unit embedding coded data of a scalable layer has different packet length[8]. In experiment, by using the internet packet loss model be able to reflect packet loss in SVC video-stream[9]. Table I presents the test sequences and SVC coding parameters used in the experiment. The sequences are coded in SVC with two spatial layers, with five levels of temporal scalability and MGS quality scalability.
represented in Fig. 5 by an absence of PSNR values for the associated frames. Fig. 6 presents the reduction of the mean frame rate of City sequences, showing a pronounce decrease in this value as the PLR incases, reaching nearly 14fps for a PLR of 3.14%.
Table 1 . SVC Coding Parameter
Coding rate Spatial/Temporal resolution Coding option used Test sequences MGS QP
Base layer 748 bps Enhancement layer 1,815 bps Base layer QCIF, 30 Hz Enhancement layer CIF, 30 Hz Fast search mode, MV search range: 32, CAVLC, loop filter, adaptive inter-layer prediction City, Crew (134 frames) Base layer: 38 Enhancement layer: 26 ~ 42
C. Random Packet Loss Fig. 4 presents the results of random packet loss in the City sequence. As shown, a small increase packet loss rate (PLR) corresponds to a significant increase of the amount of data that is received but cannot be decoded as a consequence of interlayer dependency. In this figure, the sum of data loss at the receiver includes the discarded data caused by layer dependency and the absence of lower layer NAL unit.
Fig. 5 Impact of random packet loss in term of PSNR of decoded frames for a SVC Crew (CIF)
Fig. 6 Impact of random packet loss in term of mean frame rate of decoded frames for a SVC Crew sequence Fig. 4 Impact of random packet loss in the City sequence
Random packet losses also affect the PSNR and number of frames that may be decoded correctly. Fig. 5 shows the reduction of both these parameters as the PLR increases for the Crew sequence. It can be verified that not only the PSNR value of decoded frames decreases, but also that there is a reduction in the number of frames that are decoded, which is
D. Adaptation Method of Selectively Discard Proposed adaptation based on selective discarding considers the two adaptation units AU and GOP. The adaptation unit of AU discard packet that the lowest dependency in AU from the random packet loss. In other word, discard the highest quality layer NAL unit in MGS with SVC bit stream an AU. This
adaptation mechanism need to delay of an AU but it reflect nearly internet packet loss model pattern. However, in case of AU adaptation unit causes the quality degradation. Because, the loss of lower temporal layer packet lead to enhancement temporal layer packet cannot be decoded. Shown in Fig.7 increase in packet loss ratio corresponds to increase of loss of lower temporal layer AU that enhancement temporal layer AU reference it. It caused more quality degradation. GOP adaptation unit is extended AU adaptation unit, when need to packet loss in GOP discard the packet that the lowest interlayer dependency in quality layer as well as temporal layer and spatial layer. This mechanism prevent additional packet loss duo to interlayer dependency. At the experiment, first we consider temporal and quality layer packet dependency then to reflect the number of packet loss (
( Dd , Td , Qd )
decide discarded.
Nl
)
presented. Based on the evaluation, an adaption method of SVC to cope with packet losses is proposed. MGS with SVC as well as spatial and temporal scalabilities is addressed in the consideration of packet-based adaptation in IP network or time varying network since finer adaptation is possible with sufficient numbers of quality layers in MGS. The results of random packet loss in SVC lead to serious problem of quality degradation, but the method reduces the error propagation caused by packet loss by selectively discarding less important packets with the layer-dependency consideration. The effectiveness of the proposed adaptation is shown in the experiments with some SVC bit streams with MGS. In particular, adaptation unit of GOP provides efficient adaptation in sudden increase of packet loss ratio while preventing severe quality degradation. The proposed method can be used for efficient QoS adaptation in SVC delivery over error prone IP networks.
value of packet that should be
N
The number of packet in each temporal layer T , given expression (1), this present the number of packet that be able to discard at arbitrary temporal layer value of maximum quality layer.
T
in GOP.
N T = 2( T −1) × (Q max + 1)
Qmax
is
(1)
As can be seen, increases packet loss ratio corresponds to increase of packet that be discarded. Then maximum temporal layer
Tmax
cannot accommodate packet loss. In this case, we
T = (T
− 1)
max consider packet of lower temporal layer d to reflect packet loss. First decide temporal layer by using expression of (1), and then determine the quality layer that be able to reflect packet loss by using expression of (2).
N Qd = Qmax − (T −l 1) − 1 2
(2)
In case adaptation in GOP is more than AU when compare to PSNR performance at same packet loss ratio. Even though increases packet loss ratio, adaptation in GOP different from AU prevent radically quality degradation but need to long term delay of GOP. Proposed adaptation of selective discard mechanism (see Fig. 7) as compared with Random packet loss (see Fig. 5 and 6) efficiently cope with SVC quality degradation duo to packet loss on IP network. In particular, adaptation unit of GOP provide efficiently adaptation mechanism in sudden increase of packet loss ratio. IV. CONCLUSION In this paper, the performance evaluation of packet losses in SVC in terms of decoded spatial-temporal quality is
Fig. 7 Quality of selective discarding packet
REFERENCES [1]
[2]
[3]
[4]
S.-F Chang, A. Vetro, "Video adaptation: concepts, technologies, and open issues", Proceedings of the IEEE, vol. 93, pp.148 - 158, Jan. 2005. T. Wiegand, G. J. Sullivan, J. Scharz, and M. Wien, "Joint Draft 11 of SVC Amendment," Joint Video Team, doc. JVT-X201, Geneva, Switzerland, July 2007. H. Schwarz, D. Marpe, T. Wiegand, "Overview of the Scalable Video Coding Extension of the H.264/AVC Standard," IEEE Trans. CSVT, vol. 17, no. 9, pp.1103-1120, Sept. 2007. T. C. Thang, J. W. Kang, J.-J Yoo, J.-G. Kim, "Multilayer adaptation for MGS-based SVC bitstream," in Proc. ACM Multimedia 2008, Oct. 2008.
[5]
[6]
[7] [8]
[9]
T. C. Thang, J.-G. Kim, J. W. Kang, J.-J Yoo, "SVC adaptation: Standard tools and supporting methods," Signal Processing: Image Comm., vol. 24, no. 3, pp. 214-228, Oct. 2009. J. M. Monterio, C. T. Calafate, and M. S. Nunes, "Evaluation of the H.264 scalable video coding in error prone IP networks", IEEE Trans. Circuits Syst. Video Technol., vol. 54, no. 3, pp. 652-659, Sep. 2008. J. Reichel, H. Schwarz, and M. Wien, Joint Scalable Video Model 11 (JSVM 11), Joint Video Team, Doc. JVT-X202, Jul. 2007. A. Elleftheriadis and S. Wenger, "System and tranport Internet of SVC," IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 9, pp. 1149-1163, Sep. 2007. Y. Guo, H. Q. Li and Y. K. Wang, "SVC/AVC loss simulator," ISO/IEC JTC 1/SC 29/WG 11, JVT-Q069, Oct. 2005.
