设计案例-ESP游戏.pdf

ESP游戏 徐迎晓 xuyingxiao@126.com PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn ESP Game • A new interactive system in the form of a game with a unique property: the people who play the game label images for us. • 用途 1. 有视觉障碍的人上网时可听到对图片的描述 2. 研究计算机视觉者可用大量有标签的图片训练机器学习算法 3. 图片搜索 4. 图片过滤 5. 个性化推荐（New） • 效果： – 5,000 people continuously playing the game could assign a label to all images indexed by Google in 31 days. PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn ESP（Extra Sensual Perception） PDF pdfFactory www.fineprint.cn Design Issues • Incremental success • Pass • Diversity • Ensure the quality of the labels PDF pdfFactory www.fineprint.cn When is an Image “Done”? 玩家数目不够？ PDF pdfFactory www.fineprint.cn • Cheating, 系统级解决方法是： 1.将不同地方的人配在一起，随机匹配，碰巧匹 配到身边的人概率小 2.记录IP地址，匹配者的IP地址必须不在不同网 段 3.系统检测到存在massive agreement strategy后， 放入大量的pre-recorded game-play，等尝试 cheating者放弃后，再逐步减少pre-recorded game-play 1.如何检查massive agreement strategy？—— 计算平均时间 PDF pdfFactory www.fineprint.cn Selecting the Images • blank images • images that consist of a single color, • images that are smaller than 20 pixels on either dimension, • images with an aspect ratio greater than 4.5 or smaller than 1 / 4.5 PDF pdfFactory www.fineprint.cn • Misspelling • Context-Specific Labels • Inappropriate Content PDF pdfFactory www.fineprint.cn EVALUATION • 如何证明玩家喜欢？ • 如何证明Label的质量好？ PDF pdfFactory www.fineprint.cn • Usage Statistics • 4个月内， A total of 13,630 people played the game during this time, generating 1,271,451 labels for 293,760 different images. Over 80% of the people played on more than one occasion (i.e., more than 80% of the people played on multiple dates). Furthermore, 33 people played more than 1,000 games (this is over 50 hours of playing!). • The average number of labels collected per minute by a pair of individuals is 3.89 (std. dev. = 0.69). PDF pdfFactory www.fineprint.cn • Quality of the Labels • Car dog, man, woman, stamp, Witherspoon (as in “Reese Witherspoon”), smiling, Alias (the TV show),cartoon, and green. PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn • Please type the six individual words that you feel best describe the contents of this image. Type one word per line below; words should be less than 13 characters PDF pdfFactory www.fineprint.cn • Current GWAP implementations rely on player collaborations to gather desired information – Players in the ESP Game are given the same image, and the descriptions which they both agree become the image labels – Players will be rewarded, e.g., gaining game points, for achieving consensus opinions PDF pdfFactory www.fineprint.cn Known Issue of ESP Game • Some known issues in this kind of collaborative mechanism. – First, players can benefit from cheating by forming coalitions – Second, players tend to give easier and more generic descriptions PDF pdfFactory www.fineprint.cn Solution-- KissKissBan • player-level cheating proof mechanism – Introduces an additional player, the blocker, whose objective is to stop the matching from happening PDF pdfFactory www.fineprint.cn Rule of Game • In the beginning of each round, the blocker has 7 seconds to provide blocked word list, which is the list of words he/she thinks couples might match on. These blocked words are not visible to the couples. • After the 7 seconds of entering blocked words, the couples have 30 seconds to match with each other. • The game time will decrease by 5 seconds if any couple types the blocked word, i.e., being blocked. Also, agreeing on the blocked word does not count as matching. • The couples win the round if they successfully match with each other within the time limit, otherwise the blocker wins. • Players switch roles every 5 rounds in 15 rounds of the game. PDF pdfFactory www.fineprint.cn Role • Blocker: • Each player will play the blocker for 5 rounds in the 15 rounds of the game. • Though the blocker only has 7 seconds to act in each round, he/she is able to see every word the couples are typing during the game. • Monitoring the actions of the couples not only makes the waiting process fun, but provides the blocker an opportunity to stop the couples from achieving some unified strategy. – For example, the blocker could give “a” as the blocked word if he/she founds the couples try to match on “a” in every round. PDF pdfFactory www.fineprint.cn • Couple: • The objective of the couples is the same as the players in the ESP Game: to guess what the partner is typing. • However, unlike the players in the ESP Game, the couples in KKB cannot see what the blocked words are. • Therefore, the couples are encouraged to guess harder words to avoid guessing the word in the blocked words list. PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn Incentive Structure • Zero sum game – the blocker loses 200 points and each couple gains 100 points when the couples win; – the blocker gains 200 points and each couple loses 100 points when the couples lose. • Available time period for entering words – In order to increase the possibility of blocking, the blocker will have to type as many words as possible in the short period of time, i.e., 7 seconds. PDF pdfFactory www.fineprint.cn Collected Labels • two kinds of labels, – matching label – blocking label • blocking label is the matching between the couple and the blocker. PDF pdfFactory www.fineprint.cn 与TabooWords区别 区别 • taboo words are generated from the statistics of past play history, but the blocked words are entered by the blocker in each play. • Second, the taboo words are visible by players but the blocked words are not. 优点 • provides more uncertainties • couples would be motivated to provide more diverse labeling • Not biasing player behavior. PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn Output agreement • ESP game • Squigl • PictureThis • Thumbs-Up PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn • Squigl – a game for gathering segmentation data for images in which two players are shown the same image and an associated label, then are asked to draw an outline around the object in the image with that label. – Points are awarded based on how much the two outlines of the object overlap PDF pdfFactory www.fineprint.cn • PictureThis – Players are shown a label and a list of images and asked to select the image that is the most relevant to that label. Players are again rewarded if their selections match. PDF pdfFactory www.fineprint.cn • Ontogame and Ontotube – players are given various types of input objects (e.g., Wikipedia excerpts, YouTube videos, eBay auctions) and an ontology, then asked to annotate the input object using the given ontology. PDF pdfFactory www.fineprint.cn Thumbs-Up: PDF pdfFactory www.fineprint.cn Game rules. • A player logs in and is randomly matched with another player. Both players are shown the same input query and images of two web pages deemed relevant to the query. To increase their scores, the players must agree on the same page as more relevant. PDF pdfFactory www.fineprint.cn Game features. • several well-known game features to make Thumbs-Up challenging and fun, which include: – a time limit (60 s), – score keeping (60 per successful match), – daily and all time high score lists, and – randomness in selecting partners, queries, and images. Features PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn Peekaboom PDF pdfFactory www.fineprint.cn Peekaboom • Improves on the data collected by the ESP Game and, for each object in the image, outputs precise location information as well as other information useful for training computer vision algorithms PDF pdfFactory www.fineprint.cn • The goal of the game is for Boom to reveal parts of the image to Peek, so that Peek can guess the associated word. • Boom reveals circular areas of the image by clicking. A click reveals an area with a 20-pixel radius. • Peek, on the other hand, can enter guesses of what Boom’s word is. • Boom can see Peek’s guesses and can indicate whether they are hot or cold. • When Peek correctly guesses the word, the players get points and switch roles PDF pdfFactory www.fineprint.cn elephant, trunk, tusk, ear? • Pings Component PDF pdfFactory www.fineprint.cn • Hints • give hints to Peek about how the word relates to the image PDF pdfFactory www.fineprint.cn • Game Points and the Bonus Round PDF pdfFactory www.fineprint.cn “bite-size” milestones PDF pdfFactory www.fineprint.cn Collecting Image Metadata • Input: an image-word pair • Peekaboom collects the following information: – How the word relates to the image – Pixels necessary to guess the word – The pixels inside the object, animal, or person – The most salient aspects of the objects in the image – Elimination of poor image-word pairs. PDF pdfFactory www.fineprint.cn Cheating • The player queue • IP address checks • Seed images • Limited freedom to enter guesses • Aggregating data from multiple players. PDF pdfFactory www.fineprint.cn 用途 • Improving Image-Search Results • Object Bounding-Boxes PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn • arrows or lines pointing to the objects PDF pdfFactory www.fineprint.cn Input Agreement PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn Input Agreement-TagATune • social tags – 28% of Internet users have tagged photos, news stories, or blog posts online – Flickr.com (photo sharing), Last.fm (music sharing) and YouTube.com (video sharing) PDF pdfFactory www.fineprint.cn known issues • Only the popular items are typically tagged, leaving a large proportion of the multimedia objects on the Web untagged • For multimedia objects with a time component, such as sound, music, and video clips, social tags found online often describe the object as a whole, making it difficult to link tags with specific content elements PDF pdfFactory www.fineprint.cn ESP Game? • output-agreement mechanism that works so well in many games failed to work for collecting data about sound clips in the TagATune prototype PDF pdfFactory www.fineprint.cn Reason • it can be very difficult for two players to agree on a description – such as “temperature” (e.g., chilly, warm), mood (e.g., dark, angry, mysterious), or the image it evokes (e.g., busy streets, festival), as well as categorizations that have no clearly defined boundaries (e.g., acid-jazz, jazz-funk, smooth jazz). PDF pdfFactory www.fineprint.cn New Game • two players are shown either the same object or different objects and each is asked to type a description of their given object • Unlike output-agreement games, where all communication is forbidden, all of the players’ descriptions are revealed to each other • Based on these descriptions, the players must decide whether they have been given the same object. PDF pdfFactory www.fineprint.cn TagATune interface PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn PDF pdfFactory www.fineprint.cn • 通过游戏判断文本中表达的情绪 • 游戏：Guesstiment – Guesstiment is a two-player asynchronous game – Aims at annotating a large corpus of text documents,similar to the goal of the ESP game – 与ESP的不同 • Rules • Asynchronous approach PDF pdfFactory www.fineprint.cn two players • Suggester – starts each round – Be given the whole text of a review document and he/she is supposed to: • Decide whether the whole text is positive or negative, i.e. the author is praising about a subject or criticising it. • Select a single word (or a sequence of words, as short as possible) which best describes the polarity (positive or negative) he has selected in part (1). – For example, when the negative polarity is chosen, the word ”terrible” would be a good choice for the representative word (provided that it is present in the text). PDF pdfFactory www.fineprint.cn • Guesser – be given only the word (or word sequence) suggested by the Suggester – guess polarity of the whole text just based on that single word. PDF pdfFactory www.fineprint.cn Suggester Role PDF pdfFactory www.fineprint.cn Guesser Role PDF pdfFactory www.fineprint.cn