applications in entertainment, education and business is driven by the ..... Journal of Educational Technology , 34(4), pp 393-406. ... and Expo, Lausanne, pp.
Changing Frame Rate, Changing Satisfaction? S.R. Gulliver
G. Ghinea
Department of Information Systems and Computing Brunel University, United Kingdom. {Stephen.Gulliver, George.Ghinea}@brunel.ac.uk
Abstract Perceptual indicators of multimedia quality have to be taken into account in order to build the truly end-toend communication architectures of the future. This paper investigates the perceptual impact of differing multimedia presentation frame rates on user information assimilation and level of satisfaction. We apply the Quality of Perception (QoP) concept, which encompasses not only a user’s ability to analyse, synthesise and assimilate informational content of multimedia but also his/her satisfaction with the quality of a multimedia presentation. Our results show that higher frame rates, although resulting in a betterperceived level of quality and enjoyment, across a range of video content, do not significantly increase the level of user information assimilation.
1. Introduction The effectiveness of distributed multimedia applications in entertainment, education and business is driven by the networking protocols and communications systems that deliver multimedia to the end-user. Research and development in network protocols is currently driven from a technical perspective, with little or no reference being given to the benefit to the user. However, we propose that the effectiveness of any multimedia presentation is ultimately measured by the user’s multimedia experience in terms of information assimilation and satisfaction, implying a quality definition that has two main facets: of perception (QoP) and of service (QoS) [3]. The former encompasses the infotainment (i.e. combined informational and entertainment) aspect of multimedia and characterises the human side of the distributed multimedia experience, that is, the user’s ability to analyse, understand and synthesise the informational content of a presentation as well as user satisfaction of the given presentation. The latter characterises the technical side of computer networking and represents the performance properties of the underlying network. User perception of multimedia has been studied extensively in the educational psychology [8] and HCI fields [2] [9]. However, these studies aim to improve user perception through the use of innovative
educational, ergonomic and presentational techniques, optimistically assuming that the underlying network is able to provide the optimum QoS that ultimately yields an excellent quality of multimedia presentation. In an ever-growing number of cases, though, the underlying communication system is unable to provide this assumed optimum QoS due to two competing factors: multimedia data sizes and network bandwidth. Accordingly the study described in this paper builds on previous work [3] that researched the impact of varying QoS on user QoP, and incorporates user perceived satisfaction – both the user’s level of enjoyment and the user’s judgement concerning the objective level of video quality. The structure of this paper is as follows. Section 2 presents research related to the Quality of Perception (QoP) concept; this is followed, in section 3, by a description of the empirical study undertaken. Section 4 presents the main results obtained and finally, in Section 5, conclusions are drawn and avenues for future research based on our findings are proposed.
2. Quality of Perception (QoP) 2.1 Why Quality of Perception? In a distributed setting, quality of digital multimedia has traditionally been measured using QoS technical parameters, such as jitter, delay, as well as loss and error rates. Although measurable, such objective parameters disregard the user’s perception of what defines multimedia quality [10]. To date, there has been a common assumption in the computer networking community that many quality issues will be resolved through objective solutions, such as increased bandwidth allocation [7][10]. As multimedia applications are produced for the enjoyment and/or education of human viewers, the user’s opinion of the presentation quality is important to any quality definition. Therefore, when evaluating multimedia quality, we propose that subjective testing by viewers must be considered in combination with objective testing of video quality. Apteker et al [1], studied video clips at different frame rates over a range of bandwidths, with user preference being used to determine ‘user watchability’. Results showed that ‘user watchability’ was significantly affected by the content of the video clips,
not just the level of available bandwidth, demonstrating the need for a user perspective when defining multimedia video quality. Ghinea and Thomas [3] presented candidates with a series of windowed video clips, shown at three frame rates (25, 15 and 5 frames per second). The clips were chosen to cover a broad spectrum of subject matter including: spatial parameters, temporal parameters, and importance of audio, video and textual information in context of the clip. Their results showed that a significant loss of frames (that is, a reduction in the frame rate) does not proportionally reduce the user's understanding and perception of the presentation. In fact, in some instances the user seemed to assimilate more information. Ghinea and Thomas introduced the concept of QoP (Quality of Perception), as they concluded that objective factors alone were incapable of defining the perceived quality of multimedia video. QoP has subsequently been used for tasks including: automatic configuration of protocol stacks [4] as well as measuring the perceptual impact of factors such as cognitive style [5] and type and level of deafness [6]. In order to explore the human side of the multimedia experience, we have used the notion of QoP in this study. QoP encompasses not only a user's satisfaction with the quality of multimedia presentations but also his or her ability to analyse, synthesise and assimilate the informational content of multimedia. Satisfaction is subjective in nature and, in this work, consists of two component parts: LOE (the user’s level of enjoyment whilst viewing a multimedia video clip) and LOQ (the user’s judgement concerning the objective level of quality assigned to the multimedia content being presented). Whilst LOQ was used in the original definition of QoP, LOE has been added to consider the impact of QoS variation on user enjoyment.
content. Thus, after watching a particular multimedia clip, the user was asked a number of questions (ten QoP-IA ques tions in this study for each video), which examined specific information being presented in the video. User IA is calculated as being the proportion of correct answers that were given to these ten questions. All IA questions have a definite answer, for e xample: (from a Rugby video clip used in our experiments) “What teams are playing?”. This question has an unambiguous answer (England and New Zealand) and was clearly presented in the multimedia clip, thus allowing us to determine if a participant had answered this question correctly or incorrectly. Since, in our experiment, questions could only be answered if certain information is assimilated from specific information sources (for example, the words of a song can only be gained from the audio stream), it is possible to determine the percentage of correctly answered questions that relate to the specific information sources within the multimedia video clip. For each feedback question the source of the answer was determined as having been assimilated from one of the following information sources: Video Information (V) relating specifically to the video window, for example, pertaining to the activity of lions in a documentary clip; Audio Information (A) presented in the audio stream; Textual information (T) contained in the video window, for example, information contained within a caption. Thus, by calculating the percentage of correctly absorbed information from different information sources, it is possible to determine the information source from which particip ants absorbed relatively the greatest quantity of information.
2.2. Measuring QoP
Information Assimilation (IA)
The subjective level of enjoyment (LOE) experienced by a user when watching a multimedia presentation, was polled by asking users to express, on a scale of 0 5, how much they enjoyed the presentation (with scores of 0 and 5 representing “no” and, respectively, “absolute” user satisfaction with the multimedia video presentation). The other component part of satisfaction, in our study, is the subjective level of quality (LOQ). In order to measure this, users were asked to indicate, on a scale of 0 - 5, how they judged, independent of subject matter, the presentation quality of multimedia content they had just seen (with scores of 0 and 5 representing “no” and, respectively, “absolute” user satisfaction with the multimedia presentation quality).
For our study, we expressed Information Assimilation (IA) as a percentage measure, which reflects the level of information assimilated by a user from multimedia
Both satisfaction questions were subsequently used to measure the impact of frame rate variation on LOE and LOQ.
To understand QoP, in the context of this study, it is important that the reader understands how QoP factors are defined and measured. Whilst there are a plethora of scales that are used to gauge subjective opinion of either video or audio material [10] [11], these ignore the infotainment characteristics of multimedia presentations (i.e. the entertainment-enjoyment angle). To the best of our knowledge, QoP is the only concept that addresses both sides of the infotainment spectrum by incorporating user level of information assimilation as one of its main components.
Satisfaction - LOE, LOQ
3. Experiments
4. Results
3.1 Participants
4.1 Objective QoP: IA
Our study involved 36 participants, who were evenly divided into six experimental groups. Participants were aged between 21 and 55 and were taken from a range of different nationalities, backgrounds, and spoke English as their first language or to a degree-level qualification. All were computer literate, and were presented with a series of 12 windowed MPEG video clips, between 31 and 45 seconds in length, depicting a broad range of infotainment content (see Table 1).
Average levels of video, audio, and textual information assimilated were calculated for each (frame_rate, video_clip) category. To measure whether any significant difference existed in level of IA, as a result of frame rate variation, an Analysis of Variance (ANOVA) test was carried out with frame rate as the independent variable and V, A, and T as dependent variables. This showed that the assimilation of video, audio and textual information was not significantly affected by frame rate variation. We therefore conclude that IA is not significantly affected by frame rate variation. This implies that video information is consistently assimilated irrespective of the presentation frame rate. This result supports previous work [3] demonstrating that low multimedia frame rates do not necessarily imply low IA, and that significant bandwidth capacity could be freed if perceptual considerations are universally integrated into the transmission of multimedia data.
3.2 Experimental Set-up In our experiment, only one QoS parameter (frame rate) was varied. We were particularly interested in frame rate, as multimedia video frame rate is the one parameter that has the greatest bandwidth implications in today’s distributed multimedia systems, and bandwidth is arguably the scarcest networking resource in such environments. Accordingly, a within -subjects design was chosen, where participants all viewed four video clips at 5 fps (frames per second), four at 15 fps, and four at 25 fps. Moreover, in order to counteract any possible order effects, the video clips were shown in a number of order and frame-rate combinations. To avoid audio and visual distraction a dedicated, uncluttered room was used throughout all experiments. To limit physical intrusion, tabletop multimedia speakers were used in stead of headphone speakers. A consistent audio level (70dB) was used.
3.3 Experimental Process Each participant was asked questions concerning their sight, which was followed by a basic eye-test. After each participant was given a brief introduction, the system was loaded. The viewing window was expanded and the relevant experiment order was activated. After showing each video clip, the video window was closed and the participant was asked 12 QoP questions about the video that they had just seen. QoP questions were chosen to encompass both information assimilation (10 questions) and satisfaction aspects of the information presented in the specific clip (LOE and LOQ questions). In order to be confident that the IA results were based purely on variations in the frame rate, questions were asked immediately after each clip was viewed. This was done to ensure that the assimilated information was still fresh in the memory of the participants.
4.2 Subjective QoP: LOE, LOQ ANOVA tests were carried out to statistically measure whether any significant difference occurred in user level of LOE and LOQ, as a result of frame rate variation. This analysis highlighted that both LOE {F(1,2) = 4.482, p=0.012} and LOQ {F(1,2) = 6.911, p=0.001} were significantly affected when the video is shown at different frame rates. Correlation tests between LOE results, for all frame rate groups, shows a correlation independent of frame rate. No correlation was found between LOQ results, suggesting that quality definition is more affected by frame rate variation that enjoyment definition. Post-hoc Tukey tests showed that the differences in LOE and LOQ were not significant between videos shown at 15 and 25 fps. This implies that quality degradation was only noticeable when a video was shown below 15fps. T-Tests for LOQ and LOE results, for all frame rate groups, showed significant variation (p=0.001) in LOQ and LOE as a result of the different video clips (see Table 1). This implies that the clip content, independent of QoS variation, impacts user satisfaction. Although the level of information assimilated from multimedia video was not significantly affected by frame rate variation, a user’s perception of quality and enjoyment is affected (Fig. 1). This shows that users are subjectively aware that a change in frame rate has occurred, yet this does not appear to cause a significant change in the objective level of information being assimilated (IA).
Band Commercial Chorus Cooking Animation Weather Documentary Pop Video News Rugby Snooker Space
2.79 2.79 2.08 2.91 3.04 2.75 2.5 3.08 2.41 2.625 3.208 2.83
LoQ (0-5) 15 2.83 3.33 2.83 2.916 3.458 2.708 3.208 3.33 3.166 2.583 2.75 3.33
25
2.708 3.541 2.625 3.166 3.25 2.958 2.708 3.708 3.58 2.958 3.083 3.041
5 2.583 2.625 2.458 2.541 3.5 2.5 3.208 3.08 3 1.833 2.5 2.583
LoE (%) 15 25 2.916 3.083 3.125 3 3.791 2.333 3.625 3.666 2.458 2.166 2.958 3.208
2.958 3.166 2.666 2.916 3.625 2.458 3.208 3.666 2.708 3.125 2.833 2.83
Table 1: Average IA score for all participant groups.
Although this shows that participants are aware of quality variation, this quality change does not seem to affect user IA, a finding of potential importance in distributed multimedia, especially in educational environments, where the primary goal of multimedia is one of information transfer.
5. Conclusions In this paper, we have shown that user level of information assimilation is unaffected by frame rate changes. We have also shown that, in our experiments, subjective enjoyment and perception of technical quality are significantly affected by the frame rate with which a presentation is being viewed. Our results highlight two major implications: firstly, that bandwidth-constrained multimedia does not necessarily mean enjoyment-constrained multimedia, especially, if perceptual considerations are taken into account; secondly, that closer attention needs to be given to the specific infotainment purposes of multimedia presentations when transmitting data, particularly in an educational context. In concluding, we identify that multimedia quality definition needs to be reconsidered. The fact that frame rate variation significantly impacts a user’s definition of quality and enjoyment, yet does no affect the level of objective information assimilation, has implications on using purely objective testing when defining multimedia quality. Indeed, further work is required considering multimedia quality definition, if future multimedia systems are not going to disregard a user’s definition of multimedia quality.
Average Satisfaction
5
3.2 3.1 3 2.9 2.8 2.7 2.6 2.5 2.4 5
Quality
15
25
Frame Rate
Enjoy
Figure 1. Average Subjective Ratings
6. References [1] Apteker, R.T., Fisher, J.A.., Kisimov, V.S., and Neishlos, H. “Video Acceptability and Frame Rate”, IEEE Multimedia, 2(3), pp. 32-40, 1995. [2] Faraday, P. and Sutcliffe A., “Authoring Animated Web Pages using Contact Points”, in Proceedings of ACM CHI ‘99, Pittsburgh, Pennsylvania, USA, pp. 458-465, 1999. [3] Ghinea G., and Thomas J.P., “QoS Impact on User perception and understanding of multimedia video clips”, in Proceedings of ACM Multimedia ‘98, Bristol, UK, pp. 49-54, 1998. [4] Ghinea G. and Thomas J. P., “User-Centric Communication Protocols and Architecture for the Web”, International Journal of Cognitive Ergonomics, 5(3), pp.237-260, 2001. [5] Ghinea G., and Chen S. The impact of cognitive styles on perceptual distributed multimedia quality, British Journal of Educational Technology , 34(4), pp 393-406. September 2003 [6] Gulliver S. R. and Ghinea G., “Impact of Captions on Deaf and Hearing Perception of Multimedia Video Clips”, IEEE International Conference on Multimedia and Expo, Lausanne, pp. 753-756, August 2002. [7] Kahneman, D. “Attention and Effort”, Prentice-Hall, Inc Englewood Cliffs, New Jersey, 1973, pp. 50-65. [8] Mayer R.E., “M ultimedia Learning: Are We Asking the Right Questions?”, Educational Psychologist, 32(1), pp. 1 -19, 1997. [9] Norman, D.A., “Cognitive Engineering”, in D.A. Norman and S.W. Draper (Eds.), “User-Centered System Design”, Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1986. [10] Steinmetz R. and Nahrstedt K., “Multimedia: Computing, Communications and Applications” Englewood Cliffs, NJ: Prentice-Hall, 1995. [11] Watson A and Sasse MA., “Measuring perceived quality and video in multimedia conferencing applications”, in Proceedings of ACM Multimedia ‘98, pp. 55-60, Bristol, UK, 1998
