House of Ads: a Multiplayer Action Game for Annotating Ad Video Content Katia Lida Kermanidis
Manolis Maragoudakis
Spyros Vosinakis
Department of Informatics Ionian University 49100, Corfu Greece
Department of Information and Communication Systems Eng University of the Aegean 83200, Samos, Greece
Department of Product and Systems Design Eng University of the Aegean 84100, Syros, Greece
[email protected]
[email protected]
[email protected]
ABSTRACT Content-based search in a large collection of media is a task that cannot be fully automated. Semantic annotations provided by humans through crowdsourcing techniques can be attached to media elements in the form of metadata and be used by search engines to provide the required results. However, the manual annotation of large collections is a tedious and time-consuming process and proper incentives need to be provided to attract contributors. Games with a purpose, or human computation games are based on the idea that people could solve computational tasks while playing online games. Numerous such games have been created and used in various areas, including media annotation. However, the majority of them contain limited game elements and the main challenge is the annotation task itself. This paper presents the design, implementation and early evaluation of a human computation game for supporting creative advertising. The game is part of a larger project for data mining and content-based searching in a rich collection of video ads aiming to serve as a creativity support tool for advertisers. Its main aim is to have players populate an ontology whilst playing a multiplayer game that includes an action and a quiz gameplay mode. The game incorporates a variety of elements and challenges found in modern action games and uses a metaphor of rooms and collectible items to represent the ontology concepts in the game world. The pilot study results indicate that players found the concept interesting, but there is still room for improvements in the gameplay and appearance.
Categories and Subject Descriptors • Crowdsourcing • Interactive Games • Ontology Engineering
Keywords Human computation games; creative advertising; creativity support tools; collaborative annotation.
1. INTRODUCTION The ubiquitous use of sound, image and video capturing devices such as cameras, computers and smartphones leads to a huge amount of media distributed over the web. As the size of shared digital collections increases exponentially, so does the need to Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. 16th EANN workshops, September 25 - 28, 2015, Rhodes Island, Greece © 2015 ACM. ISBN 978-1-4503-3580-5/15/09…$15.00 DOI: http://dx.doi.org/10.1145/2797143.2797174
accurately and efficiently search among them according to some criteria. E.g. the ability to search a large collection of videos to detect those that include a specific place, element or event would be beneficial to viewers and especially to professionals working in the creative industry. To fulfil such a need, a search engine would have to take advantage of high-level semantic descriptions of content associated with the respective media. Despite impressive advances in automatic media annotation (e.g. [14]), current algorithms and techniques are not sufficient enough to produce meaningful descriptions of the content in all cases. As a result, semantic annotation of media is mostly based on human contribution through metadata provided by the viewers in various forms, such as keywords, tags, etc. Media annotation can benefit from human contribution, provided that proper incentives are offered. The use of crowdsourcing approaches for tasks that are beyond the power of computers but can be performed well by humans has shown to be successful so far. However, the manual annotation of a large collection of content is a tedious and time-consuming process and, as such, it is unlikely to attract many contributors without providing some kind of motivation. A number of alternatives for motivating people to contribute their own work have been used so far, including altruism, monetary rewards, reputation, unlocking new content, etc. [1, 8] Most of these options are, nevertheless, either too costly or not as effective as required. On the other hand, digital games are capable of capturing users’ attention and letting them interact with a fictional environment for many hours. According to ESA statistics for 2014 [9] 59% of Americans play video games and 77% of gamers play at least an hour per week. Due to their popularity, digital games and especially online browser-based games, are regarded as the most promising means to attract contributors. The concept of ‘Games with a Purpose’ proposed by von Ahn [1] is based on the idea of using games as a frontend for content annotation tasks to attract more users. Numerous systems following this paradigm, currently termed as ‘human computation games’ [22], have emerged so far and applied in various areas including natural language processing, photo, sound and video annotation, protein folding, etc. However, the majority of these systems rely on simplified, quiz-based gaming rules and hardly include any of the game design patterns found in modern online games. There are only a few exceptions (e.g. [16, 26]) that provide more attractive gamelike interfaces. The focus of our work is in the area of creative advertising. We have designed and developed PromONTotion, a creativity support tool for advertisers, which provides access to a rich library of video ads and allows searching, retrieving statistical data and having access to consumers’ evaluations. The operations provided by
PromONTotion require the annotation of video files based on a fixed ontology hierarchy, and crowdsourcing is a reliable and effective means of achieving that. To that end, we have created a crowdsourcing tool for providing ad video annotations though online gaming. In this paper we present the design, development and early evaluation of ‘House of Ads’, an online multiplayer action game for annotating advertisement videos. The game introduces a novel design paradigm for human computation games that aims to offer a gaming experience similar to typical online action games. The rest of the paper is organized as follows. Section 2 presents the related work in creativity support tools for advertising and in human computation games for media annotation. In Section 3 we briefly present the PromONTotion project and its architecture. The next section presents in detail the design and implementation of House of Ads. A pilot study that has been setup for the initial evaluation of the game is described and its results are discussed in section 5. Finally, the conclusions of this work are presented in Section 6.
2. RELATED WORK 2.1 Creativity Support Tools for Advertising Creative advertising describes the process of capturing a novel idea/concept for an ad campaign and designing its implementation. It is governed nowadays by significant budget allocations and large investments. Several studies have been published regarding the impact of advertising [2, 3], as well as creativity in advertising [11]. A number of creativity support tools have been proposed, to help ad designers come up with novel ideas for setting up a new campaign. Such tools, that could enhance the development of creative ideas, are highly beneficial for the advertising industry. They usually focus on forcing upon the advertiser a certain restricted way of thinking, using creativity templates [10]. The methodology is based on the hypothesis that total freedom is not the most efficient way for enhancing the creative process, but constraining it with the use of a limited number of idea-forming patterns. Expert decision making systems, like ADCAD [5], have been proposed for triggering creative ideas. ADCAD relies on rules and facts that are in reality quite hard to provide as input to the decision making tool. Janusian wording schemata (the use of opposite words in taglines) have been used extensively [4] in advertising. IdeaFisher [7] is based on a database of predefined concepts and associations between them. GENI [19] guides the user to make connections and transformations between the entities of a brainstorming problem. Idea Expander [28] is a tool for supporting group brainstorming by showing pictures to a conversing group, the selection of which is triggered by the conversation context. Opas [21] presents a detailed overview of several advertising support tools. Most of the aforementioned creativity support tools make use of static non-expandable databases, term-relation dictionaries, handcrafted associations and trans-formations. Static, passive, expertdependent knowledge models can hurt creativity [21]. Unlike previous approaches to creativity support tool design, PromONTotion relies on no predefined or hand-crafted elements and associations, apart from an empty ontological backbone structure, that includes ad content concepts, consumer impact data slots, and taxonomic relations between them. The backbone can be populated by a large number of viewers through a multiplayer action game, the House of Ads, and the annotated videos can keep expanding without the requirement of experts. The knowledge available by the tool is data-driven and automatically derived,
making PromONTotion generic, dynamic, scalable, expandable, robust and therefore minimally restricting in the creative process and imposing minimal limitations to ideation or brainstorming.
2.2 Human Computation Games for Media Annotation The paradigm of using on-line games to collectively solve largescale computational problems was initially presented by von Ahn [1] under the term ‘Games with a Purpose’. The idea behind this approach is that fun and enjoyment can be used as an incentive for human participation in solving problems that are easy for humans but difficult for computers. People already spend much time playing digital games and, therefore, games designed to include the solution of a useful computational problem in their challenges may be both attractive to players and effective as solution providers. The idea has shown to be successful. One such game for labelling images, ESP has been played by hundreds of thousands of people generating millions of labels a few years after deployment [17]. Nowadays, the term ‘Human Computation Games’ is mostly used in the bibliography to describe this class of games. According to a recent review [22] a large number of implemented games has emerged in a wide variety of application areas, including content annotation, location-based content sharing, natural language processing, commonsense reasoning and ontology construction. Regarding the area of annotating multimedia content, most existing applications focus on image, music and web page annotations, and there are only a few examples of human computation games for video annotation. The first human computation games for annotating content focused purely on the computation task and had a quiz-like gameplay. In most of the cases two or more players had to agree on the same label that characterizes a resource presented to them. E.g. in OntoTube [25] players view a randomly chosen YouTube video and they must agree on answers from a set of questions about the video. The more questions the players answer consensually, the more points they earn. In TagATune [18] one player enters words that characterize the sound, whilst the other attempts to guess them. Players earn points based on the number of agreed descriptions per tune. In another example [27], players listen to a common piece of music, select good and bad semantic labels and get real-time feedback on the selection of all other players. Photoslap [13] is a multiplayer online game for the semantic annotation of images based on the rules of a popular card game named Snap. Players flip cards in turns and if a player detects that the last two consecutive cards are matching s/he may choose to slap, i.e. to declare that the images show the same content, and increase his/her score. If another player disagrees about the slap, s/he may object and if the objection is successful, s/he gains the points and the slapper looses them. Finally, players may initially set ‘traps’ by identifying matching cards before the game begins to prevent false objections. The rules of the game have been designed so that the only strategy guaranteed to increase the players’ score is to be honest about the content, i.e. to slap if the cards are matching and to object if they are not. In this game, the output is not the annotation of the image content, but the grouping of images according to their content. Furthermore, given that the game has a default strategy, the only real challenge for the players is pattern matching, i.e. to correctly identify whether two images have the same content. A more interesting game paradigm for image annotation is presented in KissKissBan [12].The game is played by three players, one of which has the role of the “blocker” and the other two are the
“couple”. In each game session the same image is presented to the players and the couple tries to “match” by typing the same word that describes the image. The blocker tries to stop them from matching by providing a list of banned words unknown to the couple. The game contains both collaborative and competitive gaming and through gameplay it collects labels that describe the image based on the players’ matching descriptions. The main challenge of the game for both the couple and the blocker is to try to find as many alternative descriptions for the image as possible to increase chances of matching or blocking respectively. SeaFish [26] is a human computation game for image annotation with some additional game-like characteristics. It is a single player game where players have to fish related images that are floating around. In each game round a single concept represented by an image is displayed to the players. Other images are also floating through the main screen using a metaphor of fishes swimming in the sea, and players have two minutes to mark those images as either related to the original concept or not. They can do so by catching the images with a fishing rod and dragging them to one of the two contained in the sea bottom: green if the image is related and red if it is not. The game decides about the related images based on the markings of the majority of the players. SeaFish bares more resemblance to regular games compared to previously mentioned approaches, in the sense that it presents a fantasy world and uses the fishing metaphor as a means for players to decide about matching images. However, the main challenge for the players remains the recognition of similar images. Recently, popular game design patterns and gaming elements have started to be taken into account in the design of human computation games. One such example is OntoGalaxy [6, 16], a single player space shooting game for finding synonyms to given words and for annotating resources like images or audio files. In the game players take the role of the commander of a spaceship and each game session is presented as a mission received from an imaginary headquarter. Missions are requiring different annotation tasks, e.g. players are asked to collect all freighter ships whose call sign is a synonym for a given word. While playing the shooting game, players can view the call sign of other spaceships and decide whether to collect them or not based on the mission statement. The efficient coupling of popular gameplay styles with computation tasks and the proper balance between pure gaming and problem solving still remain open research issues. The focus of our work is to explore an alternative paradigm for human computation games that includes a number of common gameplay elements and aims to enhance the fun and to capture players’ attention for longer time. The House of Ads is adopting a novel metaphor for the semantic annotation task: concept categories are represented as rooms and concepts as collectible bags of money within them. In each gaming session multiple annotations regarding the video can be provided due to the inclusion of multiple rooms in the gaming environment. Besides the necessary challenge of pattern recognition, i.e. to identify critical concepts in the video, the game incorporates further challenges, such as exploration, competitive gaming, player attacks, etc.
3. OVERVIEW OF THE PROMONTOTION PROJECT PromONTotion is a creative advertising support tool that relies on generic, automatically acquired knowledge. It incorporates collaboratively accumulated ad video content annotations, associations between them derived though reasoning within the ontological structure they form, and statistical information
regarding the impact an ad video has on consumers. Using mining techniques, it can extract knowledge concerning ad type, ad content, ad impact and the relations between them. Based on this semantic thesaurus of concepts, terms and relations, the support tool will provide ad designers with access to a rich library of video ads, in which they can search ads by content, retrieve statistical data, and access the consumers’ opinion about each ad. The architecture of PromONTotion is presented in Figure 1.
Figure 1. The Architecture of PromONTotion The ontology backbone includes concepts, categories and taxonomic (is-a, part-of etc.) relations between them, that are relevant to an advertising campaign. It has been sketched by marketing experts after watching approximately three hundred ad spots of particular product and service types. The backbone currently includes parameters that determine the message to the consumer (e.g. tag lines), filming technicalities, the ad content, as well as its artistic features. Also, subjective information regarding the impact of the ad to the consumers is included in the ontology. The ontology backbone is designed to be scalable, so it can be constantly enriched and updated. In order to support the generic, minimal human-expertisedemanding and data-driven nature of the proposed support tool, the ontology backbone is populated through crowdsourcing techniques. It is evident that the success of PromONTotion relies heavily on the plethora of provided annotations; therefore the annotation tool needs to be attractive, engaging, fun and addictive. To this end, the human computation game ‘House of Ads’ is designed and implemented especially for the task at hand. All player annotations are used to populate the ontology backbone and are stored in a database for further processing, that includes statistical as well as data mining techniques. The goal of the data processing phase is the detection of co-occurrence and correlation information between categories, terms and relations (e.g. in how many ads for cleaning products there is a housewife in the leading role etc.), and its statistical significance, as well as higher-level association information governing them. The ontological structure with all its content, objective (content annotations) and subjective (ad impact annotations from every player), along with the derived correlations and extracted knowledge mined in the previous phase, will be made usable to professionals in the advertising domain through a user-friendly interface. Advertisers will be able to see the content of old ads for related products, and thereby come up with new ideas, gain insight
regarding the impact of previous campaigns from the players’ evaluation, look for screenshots of videos using intelligent search, based not only on keywords, but on concepts. The architecture and early mining results of PromONTotion have been presented in [15, 20].
4. THE HOUSE OF ADS GAME House of Ads is an online multiplayer action game for annotating ad video content. Players find themselves in a competitive game environment, which has a retro / 8-bit look and feel. While concurrently watching the video of an advertisement, players have to choose the correct answers to questions about the video content faster than the others. The game includes two interchanging gameplay types: the action and the quiz mode. In the first case, the questions are represented as rooms and the answers as bags of money within them. Players have to explore the house and collect the appropriate bags whilst preventing the other players getting there first. The action mode allows player attacks through weapons that are available in the rooms to pick up. In the quiz mode, all players watch the same video and the goal is to answer a single question about the content faster than the others. The players that provide correct answers throughout the game are rewarded with money, which they can use to unlock new content and to enhance their abilities.
4.1 The Ontology Hierarchy The goal of the game as a human computation system is to populate the ontology backbone of PromONTotion with annotations of specific ad videos. The current ontology structure and categories are displayed in Figure 2.
The first type includes the majority of the ontology concepts, and they are represented in the game as questions regarding a specific video ad presented to the players. Information regarding the second type cannot be extracted by viewers of the ad; they have to be provided by experts in the field. Thus, they are excluded from the game. Finally, the viewers’ opinions are subjective, and as such, there are no correct and wrong answers. Therefore, these questions are not part of the main game; players are asked to enter their opinion about the ad after the end of each game session. We have converted all categories of the first type to questions and all subcategories and concepts to possible answers of the respective questions. E.g. the category “Product type” has two subcategories: “Product” and “Service”. From this category we have created the question “Which type of product is being advertised in the video?” and the possible answers are “a product”, “a service” and “both”. Each question and answer are represented in a full and a short form. The short versions are being used as floating names of game elements in the action mode to avoid extensive text display in the gaming environment. Some questions may have more than one appropriate answers for a specific ad. E.g. in a question about other key elements participating in the ad the respective answers include “a tool”, “a furniture”, “a vehicle”, etc. It may be possible that there are more than one of these elements included in the ad. Thus, the game does not make the assumption that there is a single correct answer for each questions, but decides on the appropriateness of each answer based on the percentage of players who favored that answer. Finally, questions based on concepts that are not in the top level of the hierarchy depend on the answers to other questions. E.g. a question asking which type of service is being advertised in the video is applicable only if the answer to another question regarding the type of the advertised product is “service” or “both product and service”. To address this issue we distinguish between top-level questions, which are applicable to any video, and contingency questions, which are unlocked only if the system has reached to a certain degree of confidence about the dependent answers.
4.2 Game Design The design goal of House of Ads is to be both effective as a human computation tool and fun as a game to play. Most existing human computation games have proven to be very effective in media content annotation. However, their gaming dimension is quite deficient. To explore new paradigms for human computation games that can possibly overcome this limitation, we decided to include more fun elements in the House of Ads game.
Figure 2. The ontology backbone. Regarding the categories and concepts of the ontology hierarchy, we can distinguish three different types: •
annotation of the content (features, location, product, etc)
•
production details, and
•
viewer’s opinion
Sid Meier, a successful game designer, claims that “a game is a series of interesting choices” [24]. He describes interesting choices as situations in which no single option is clearly better than any other, the options are not equally attractive, and the player must make an informed choice. Furthermore Adams and Rollings [23] argue that a game should avoid dominant strategies, i.e. strategies that reliably produce the best outcome a player may achieve, no matter what his opponent does. A dominant strategy makes all other choices pointless and, thus, limits the fun. If we take a deeper look at current human computation games, we will notice that there is a dominant strategy in most of them: to provide the correct annotations. No matter what the game rules are, if the player correctly solves the computational problem posed to him/her within the required time frame, it is guaranteed that s/he will have the best outcome. The only true challenge posed in these games is the content annotation itself, which is far from Sid Meier’s
definition of successful games. Based on this observation, our basic requirement was to avoid the problem of dominant strategy and to include multiple challenges and competing choices in the game.
regarding the video being watched. The goal of each player is to pick up the correct answer for as many questions as possible until the session is over. Each session lasts for three minutes.
There are plenty of successful patterns and paradigms for creating challenging online games, but the real challenge for human computation games is to efficiently couple the computation task with the gaming environment. Our goal was to find an appropriate metaphor that provides a good mapping between the player actions in the game and the annotation task. The metaphor should be able to fit well in a game story. Furthermore, the game story should be somehow relevant with the content that needs to be annotated. Krause et al [16] claim that players of OntoGalaxy found the decoupling of the game story (a space shooter) and the task odd. This requirement was, however, not easy to fulfil in our case, since the ads were not restricted to a specific product type.
The game environment presents a large house with multiple rooms displayed from a top-down perspective. (Fig. 3). Each room holds a question regarding the ad in the form of an entrance sign and includes the possible answers as bags of money with labels on them. The house also includes a number of weapons that can be picked up by players and used on other players to gain advantage over them. Each game session is played by two up to four players whose characters are initially placed in peripheral locations of the house, distant from each other, ensuring that everyone has almost equal chances to win the game.
The main design choices of House of Ads based on the aforementioned requirements were: •
multiplayer action game: action games is the third most popular category of online games [9] and they can be easily built as series of quick gaming sessions.
•
retro game look and feel: given that most of the ads in the collection are older the retro gaming environment looks more appropriate
•
a house with rooms as the main metaphor: each room represents a question and contains multiple collectible elements as possible answers
•
•
multiple challenges and competing choices: competition, time race, exploration, collection of weapons, attacks on other players
Each player character can freely move in the four directions using the keyboard. If a character approaches a question (room entrance) or an answer (bag of money), the full version of the question or answer is displayed as a tooltip. A player can select an answer by moving his/her character on top of it, an action that makes the character pick up the respective bag of money. If an answer has been picked, the other players cannot pick it up again. Furthermore, if a player has already picked an answer in a room, s/he is not allowed to pick up another or to change his/her selection in the same room. Players can also get weapons by moving their characters to their location. Each player can hold up to two weapons, one of which is the active one. S/he can switch active weapons or fire the active one using special keys. The available weapons are: •
a fence that is placed temporarily in front of the character who fires it and blocks the passage of others
•
a glue that spreads in a circular region when fired and makes characters who step on it walk slower
•
a laser gun that sends characters who get hit back to the starting position
•
a wine bottle that reverses the controls of the character being hit as an effect of getting drunk
reward mechanisms that motivate repeated visits
Finally, given that the main aim of House of Ads is to populate the ontology, we decided to include a quiz mode as well. This game mode is expected to produce quicker results and will also serve as a means to resolve conflicting player answers.
The game has been designed so as to avoid a dominant strategy. Each player choice has possible benefits and risks. If a player pays too much attention on the video, s/he increases the chances of finding the correct answers, but risks losing valuable time. If a player opts for the nearest room it is possible that s/he will be the first to pick up an answer, but the correct answer might not be easily detectable in the video. On the other hand, if s/he decides to explore the house first, s/he might detect easier questions but risks conflicting with other players. If s/he collects weapons, s/he will gain advantage over other players, but risks not being the first one to reach the correct answers in the near rooms. Each player might develop his/her own tactics and strategies and learn to outperform others as s/he gains more experience.
Figure 3. A screenshot of the action mode.
4.3 Gameplay In the action mode the players interact with two components displayed on the screen: the video device and the game environment. The ad video automatically starts playing when the game starts, and players may control the video presentation during the game session using the controls provided by the video device. The game environment includes questions and potential answers
The quiz mode (Fig.4) is a typical multiple choice selection game with a time challenge. Two up to four players simultaneously watch the same ad video and they have to answer a single question about the content. Players who provide the correct answer receive a fixed amount of money and the first one to answer correctly earns extra bonus.
4.4 Verification Mechanism Human computation games need to have a verification mechanism to prevent players from cheating with random or intentionally false answers. In House of Ads we adopt an output-agreement strategy.
If the majority of the players agree that a certain answer is more appropriate for a specific video, then it is assumed to be correct. Furthermore, players do not know each other’s identity when they join a multiplayer game and they cannot communicate. Therefore, the most promising strategy to earn points is to provide correct answers.
more specific and, as such, they are regarded more difficult for the players, because they need to watch the video more carefully. As more players are playing the game and more annotations get collected, the system progressively unlocks contingency questions for specific ads, allowing those questions to be included in a later session. The difficulty level of each gaming session is decided based on the mean experience of the players that participate in it. When a player enters the game, s/he participates in a series of game sessions that interchange between action mode and quiz mode. After the end of each session s/he is asked about his/her personal opinion regarding the ad, the elements s/he liked and those s/he would change. The money earned by a player can be used to unlock new abilities, such as to increase the speed of movement and the weapon range, or to change his/her appearance by adding new clothes or hairstyle. Also, the players get notified about blocked money that has been made available and about changes in their credibility. These game elements are expected to increase the players’ motivation to spend more time playing the game.
4.6 Ontology Population
Figure 4. A screenshot of the quiz mode. When a game session ends players earn the money they have collected. Each of the collected answers is transformed to a certain amount of money with an added bonus if it has been collected in the first half of the session. If the system decides that the selected answer is correct, then the earned money is available for the player to spend. On the other hand, if the answer is wrong, the player loses the money and his credibility is reduced. Finally, if the available data are not enough to decide whether the answer is correct or not, the earned money is blocked until a decision is reached. When this happens, the player is informed that his blocked money has been made available or that it has been lost, depending on the decision. The game also uses a simple credibility system to evaluate player input. Each player is assigned a credibility factor ranging from 0 to 1, which represents the percentage of correct answers to the total answers s/he has provided. Based on the value of this factor, players get informed about their own credibility in the game as low, medium or high. Initially all players have a credibility factor of 0.5 and when they provide a minimum threshold of answers, the actual credibility value is calculated. The game decides about correct annotations to a given ad video by calculating a score that depends on the percentage of people who selected an answer and their credibility. The score of each answer is calculated as the sum of the credibility factors of the players who selected it. It is then divided by the sum of all scores of the answers to the same question. If the score of the answer is significant (more than 3) and the resulting percentage is above a minimum threshold (initially set to 0.3), the answer is regarded as correct. On the other hand, if the number of available answers to a question is small or none of them has a significant score, then the correct results are not yet decidable.
4.5 Game Progression and Rewards Digital games should increase the level of difficulty as players gain more experience to ensure they remain in a state of flow. In House of Ads the difficulty of a level can be adapted by changing the size of the house and the type of question. Larger houses are harder to explore and the more rooms a house contains the more questions will be posed regarding the content of the video. Furthermore, questions related to concepts deeper in the ontology hierarchy are
The ontology of PromONTotion is populated based on the players’ choices in the game and on their personal opinion about each ad. Whenever there is a positive decision about an answer regarding a video, the respective concept is associated to that ad and added to the ontology to be further processed and used by the support tool. Questions for which a number of answers has been collected, but no certain decision can be reached yet, have a higher priority to appear in a future gaming session, preferably in quiz mode, so that the system reaches to a decision soon. The efficiency of House of Ads as a human computation system is satisfactory. Assuming a house consisting of six rooms with three players per session, each player provides 0.67 answers per minute. Assuming also that there are at least four answers needed to reach a satisfactory conclusion regarding a single concept, the number of queries resolved by a single person per minute is 0.17. This number is significantly higher in the quiz mode, where two up to four players resolve a single question in one minute time.
4.7 Implementation House of Ads has been implemented as an online browser-based game. The client application that creates the game interface has been developed in Adobe Flash, and the server-side environment that handles the multi-player synchronization in C# using Yahoo Games Network SDK. Each game session in action and quiz mode is played in individual ‘game rooms’, in which clients are connected and exchange data to synchronize the game environments. The game allows multiple parallel game rooms, so the number of concurrent players is restricted only by the server’s processing and network capacity. The game makes use of a database provided by Yahoo Games Network to store data such as the ontology backbone in the form of questions and answers, the player’s provided answers, the annotations identified as correct per ad, the unlocked contingency questions per ad, players’ earned money and experience, etc. Currently the database contains 138 video ads, 43 questions that can be asked about each single ad, and a total of 197 possible answers to these questions that are related to respective ontology concepts.
5. PILOT STUDY We prepared a pilot study of House of Ads in order to assess its impact on players’ enjoyment and its efficiency as a human computation tool, to gain empirical observations about its usage, and to discover critical design issues. The focus of our study was
not only on the implementation itself, but also on the introduced concept of action-based multiplayer games for content annotation. The pilot study took place in the Computer Laboratory of our department. 18 users (11 male and 7 female) participated in the evaluation process and all of them were computer science students. 10 users claimed that they were casual gamers (3 play games a few hours per day, 3 almost every day and 4 at least once a week), while the rest of them play rarely or never. After a 10 minute introduction to the main concept of our project and to the game aims and mechanics, users were asked to play consequent action mode sessions for 35 minutes. In each game session users were arranged in new game rooms of two up to four players without knowing each other’s identity. At the end of the study we asked the users to fill a questionnaire and conducted a wrap-up discussion with them. The recorded user annotations after the game sessions were 312 and involved 29 different ad videos. The number of results is satisfactory in terms of efficiency. However, an analysis of user responses revealed that almost half of the annotations were false. A possible explanation for this behavior is that the system did not penalize players providing false answers due to insufficient number of responses for each single question, and players felt free to experiment with the game rather than paying closer attention to the answers. The users’ opinion about the game were mixed. In a 5-point Likert scale question asking users to rate the game in terms of fun with 1 being very boring and 5 very entertaining, the mean value was 2.72. If we count the casual gamers’ answers only, the value slightly rises to 3. Looking closer at the responses of the experienced gamers who play almost daily, the results are improving: 3 of them found it neutral, 1 found it entertaining, one very entertaining, and one of them declared it was very boring and provided negative responses overall. Although the sample is too small, it may be possible that experienced gamers are more acquainted with a complicated game environment on one hand, but also have higher expectations from the gameplay on the other. The results in a question asking how possible it is to play the game again were similar with a mean value of 2.78. Four users found it possible or very possible to play the game again and 10 were neutral (not sure). We asked users to select one or more of the major game challenges and elements that they liked. The majority of them (12) liked the competitive environment and about a third of them liked the game weapons (7), the exploration challenge (7) and the content discovery in the ad video (7). Only 4 users responded that they liked the time challenge. Regarding the elements that they did not like, the first choice was the time challenge. Additional custom answers included that the game action was slow and the house was difficult to explore. In a question asking which elements of the game they would change, the majority of users (11) mentioned the game graphics. Other answers mentioned the game genre (5), whilst a few users mentioned again that they would like the game to have a faster pace. Based on the user responses and on the wrap-up discussion we had with them, it seems that users put great emphasis on the game content and challenges. A number of users would like to have the gameplay and the graphics improved. Some of them would like to have additional typical characteristics of action games included in the game, such as faster pace, multiple weapons, levels and rewards, etc. Others liked the fact that the game included funny and humorous elements, like the game weapons and the retro level design.
Finally, the majority of users found the idea of using an action game for collaborative video annotation interesting. In a question whether it is possible to use such types of games for media annotation, the responses were quite positive, with a mean value of 3.22.
6. CONCLUSIONS We have presented the design, development and pilot study of House of Ads, a multiplayer online action game for annotating ad videos. The game is part of wider project for the development of a creativity support tool for advertisers, and its main purpose is to populate a rich video library with collaborative annotations provided by players based on a defined ontology backbone. House of Ads incorporates gaming elements and gameplay styles found in popular online games, aiming to explore alternative paradigms to the typical quiz-based gaming mode of the majority of human computation games. The ontology concepts are transformed to questions and answers, and represented within the game as items to be collected in thematic rooms in a competitive action environment. The pilot study results indicate that users find the overall concept interesting, but have higher expectations from the graphics and gameplay quality of the game. Regarding our design and implementation, it seems that more emphasis needs to be put in the improvement of the game mechanics, look and feel, and reward mechanisms, which will probably lead to the reconsideration of some design choices. A future study will be based on the long-term use of the game and will include the verification mechanism and the reward system, in order to gain more insight about its effectiveness and about the appropriateness of multiplayer action games for human computation tasks.
7. ACKNOWLEDGEMENTS This Project is funded by the National Strategic Reference Framework (NSRF) 2007-2013: ARCHIMEDES III – Enhancement of research groups in the Technological Education Institutes.
8. REFERENCES [1]
Ahn, L. Von 2006. Games with a purpose. Computer. 39, 6, 96–98.
[2]
Aitken, R., Gray, B., and Lawson, R. 2008. Advertising effectiveness from a consumer perspective. International Journal of Advertising 27(2), 279-297.
[3]
Amos, C, Holmes, G., and Strutton, D. 2008. Exploring the relationship between celebrity endorser effects and advertising effectiveness: A quantitative synthesis of effect size. International Journal of Advertising, 27(2), 209-234.
[4]
Blasko, V. and Mokwa, M. 1986. Creativity in advertising: A Janusian perspective. Journal of Advertising, 15(4), 4372.
[5]
Burke, R.R., Rangaswamy, A., Wind, J., and Eliashberg, J. 1990. A Knowledge-Based System for Advertising Design. Marketing Science. 9, 3, 212–229.
[6]
Carranza, J. and Krause, M. 2012. Evaluation of game designs for human computation. AAAI Workshop Technical Report. WS-12-17, (2012), 9–15.
[7]
Chen, Z. 2010. Computational intelligence for decision support, CRC Press (2010).
[8]
Cuel, R. et al. 2011. Motivation mechanisms for participation in human-driven semantic content creation.
International Journal of Knowledge Engineering and Data Mining. 1, 4 (2011), 331. [9]
ESA 2014. Annual Report.
[10] Goldenberg, J., Mazursky, D., and Solomon, S. 1999. The fundamental templates of quality ads. Marketing science, 18(3), 333-351. [11] Hill, R. and Johnson, L. 2004. Understanding creative service: a qualitative study of the advertising problem delineation, communication and response (APDCR) process. International Journal of Advertising, 23(3), 285307. [12] Ho, C., Chang, T. H., Lee, J. C., Hsu, J. Y. J., and Chen, K. T. 2009. KissKissBan: a competitive human computation game for image annotation. Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP ’09. (2009), 11. [13] Ho, C., Chang, T. H., and Hsu, J. Y. J. 2007. Photoslap: A multi-player online game for semantic annotation. Proceedings of the National Conference on Artificial Inteligence. 22, (2007), 1359. [14] Hu, W., Xie, N., Li, L., Zeng, X., & Maybank, S. 2011. A survey on visual content-based video indexing and retrieval. IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews. 41, 6 (2011), 797–819. [15] Kermanidis, K. and Maragoudakis, M. 2013. Designing a Support Tool for Creative Advertising by Mining Collaboratively Tagged Ad Video Content: The Architecture of PromONTotion. Artificial Intelligence Applications and Innovations. Springer Berlin Heidelberg, 2013. 10-19. [16] Krause, M., Takhtamysheva, A., Wittstock, M., and Malaka, R. 2010. Frontiers of a Paradigm – Exploring Human Computation with Digital Games. Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP). (2010), 22–25. [17] Law, E. and von Ahn, L. 2009. Input-agreement: a new mechanism for collecting data using human computation games. Proceedings of the 27th international conference on Human factors in computing systems - CHI 09. (2009), 1197–1206.
[18] Law, E.L.M., Von Ahn, L., Dannenberg, R. B., and Crawford, M. 2007. Tagatune : a Game for Music and Sound Annotation. In International Conference on Music Information Retrieval (ISMIR’07). (2007), 361–364. [19] MacCrimmon, K. and Wagner, C. 1994. Stimulating ideas through creative software. Management Science, 40(11), 1514-1532. [20] Maragoudakis, M., Kermanidis, K. L., and Vosinakis, S. 2014. Extracting Knowledge from Collaboratively Annotated Ad Video Content. Artificial Intelligence Applications and Innovations (pp. 85-95). Springer Berlin Heidelberg. [21] Opas, T. 2008. An investigation into the development of a Creativity Support Tool for advertising. (2008). [22] Pe-Than, E.P.P., Goh, D. H. L., and Lee, C. S. 2012. A survey and typology of human computation games. Proceedings of the 9th International Conference on Information Technology, ITNG 2012. May 2015 (2012), 720–725. [23] Rollings, A. and Adams, E. 2003. Andrew Rollings and Ernest Adams on game design. New Riders. [24] Rollings, A. and Morris, D. 2004. Game architecture and design: a new edition. New Riders. [25] Siorpaes, K. and Hepp, M. 2008. OntoGame: Weaving the semantic web by online games. Lecture Notes in Computer Science 5021 LNCS, (2008), 751–766. [26] Thaler, S., Siorpaes, K., Mear, D., Simperl, E., and Goodman, C. 2011. SeaFish: A game for collaborative and visual image annotation and interlinking. Lecture Notes in Computer Science 6643 LNCS, PART 2 (2011), 466–470. [27] Turnbull, D., Liu, R., Barrington, L., and Lanckriet, G. R. 2007. A Game-Based Approach for Collecting Semantic Annotations of Music. Ismir. (2007), 2–5. [28] Wang, H., Cosley, D., and Fussell, S. R. 2010. Idea Expander: Supporting group brainstorming with conversationally triggered visual thinking stimuli. Proceedings of the 2010 ACM conference on Computer supported cooperative work (pp. 103-106). ACM.
