Study: Predicting Player Behavior and How Zynga Profits from Data Analysis

An interesting front page story in The Wall Street Journal today by journalist Nick Wingfield discusses how casual gaming giant Zynga cashes in on their millions of players. After developing Fishville, following in the footsteps of highly successful titles like Farmville, managers noted players spending in-game currency on one type of fish more so than others. The “translucent angler fish” was being purchased more than 6 times the rate of other virtual fish. So the company quickly developed a whole line of translucent sea creatures, charging as much as $4 (this time, in real world money) for more exotic varieties.

This formula has been very successful for the company. Although only about five percent of Zynga’s player base spends serious money in their games, so many millions of people play that the company rakes in millions. They rake in even more by figuring out what the players want through data analysis.

Zynga is transforming the game industry. Traditional videogame companies create games they think players will like, then sell them. Zynga offers free games through Facebook Inc.’s social network, then studies data on how its audience plays them. It uses its findings to fiddle with the games to get people to play longer, tell more Facebook friends about them and buy more “virtual goods.” At the heart of the whole process is Zynga’s ability to analyze reams of data on how players are reacting to its games.

“We’re an analytics company masquerading as a games company,” said Ken Rudin, a Zynga vice president in charge of its data-analysis team, in one of a series of interviews with Zynga executives prior to the company’s July filing for an initial public offering.

This formula for financial success has other companies following Zynga’s lead. Rather than spending millions developing a title with a short shelf life, companies are turning to free games with extras that cost money. The primitive graphics Zynga uses are generally derided by serious gamers, but Zynga aims for the mass market, much the way American beer brewers produce bland beverages that appeal to the most palates.

All of Zynga’s games go through what amounts to a giant ongoing lab experiment involving players. Zynga conducts hundreds of “A-B tests” within its games, in which two sets of players see virtual goods on sale with, say, subtle color differences to see which color sells better…

Sizhao Yang, a former Zynga executive who helped create its virtual farming hit “FarmVille,” says his development team figured out by analyzing virtual-goods-sales data that “people buy animals a lot more than tractors and other inanimate objects.” The findings led the “FarmVille” team to more prominently feature animals in its online store, he says.

Interestingly, Wingfield reports there is considerable tension in the company between the data jockeys and the game designers. The game designers have a certain idea of how a game should look and function. The analysts drive the direction of game development based on the data, leading to tension. Some designers have quit the company in protest. Still, data remains the keystone in Zynga’s game plan for the foreseeable future.

The Zynga story on data analysis comes on the heels of the recent International Conference on the Foundations of Digital Games in Bordeaux this summer (fdg2011.org). There, Brent Harrison and David L. Roberts over at North Carolina State delivered an interesting paper, Using sequential observations to model and predict player behavior. Here’s their abstract:

In this paper, we present a data-driven technique for designing models of user behavior. Previously, player models were designed using user surveys, small-scale observation experiments, or knowledge engineering. These methods generally produced semantically meaningful models that were limited in their applicability. To address this, we have developed a purely data-driven methodology for generating player models based on past observations of other players. Our underlying assumption is that we can accurately predict what a player will do in a given situation if we examine enough data from former players that were in similar situations. We have chosen to test our method on achievement data from the MMORPG World of Warcraft. Experiments show that our method greatly outperforms a baseline algorithm in both precision and recall, proving that this method can create accurate player models based solely on observation data.

While not fixating on the profit motives that Zynga has in mind, Harrison and Roberts offer clues to game designers in guiding player behavior in-game. Educational games could become more engaging:

The ability to accurately predict a player’s behavior in a game has a number of applications. While these applications are beyond the scope of this paper, we discuss two of them briefly here to better situate and motivate our approach. With a model of player behavior, we can create an experience that is unique to a user’s tendencies or preferences. For example, if we predict that the user will choose to fight a certain non-player character (NPC) rather than talk to it, that NPC can be made more willing to fight. Another application involves guiding players to parts of games that they may enjoy. Modern games often take place in large, sandbox worlds where the player is given total freedom. It’s quite possible that players may never see content that they would like because the sandbox is just so big. Predictions about a player’s behavior can be used to guide her to the parts of the game that she would enjoy.

Eschewing surveys, the authors recommend a purely data-driven approach (as does Zynga):

We feel that a purely data-driven approach has significant promise for creating accurate predictive models of player behavior in games without the difficulties associated with earlier modeling techniques. Very little research has been done in this area to date.

Read the entire paper for further discussion of the algorithm they developed. Very interesting.

References:

Harrison, B & Roberts, D. L. (2011). Using sequential observations to model and predict player behavior. In Proceedings of the 2011 Foundations of Digital Games Conference. (FDG 2011), Bordeaux, France.

Wingfield, N. (2011, September 9). Virtual products, real profits. The Wall Street Journal, p.A1.
-

Study: Tough Times in RL Lead to Greater Second Life Satisfaction

An interesting study by Edward Castronova over at Indiana and Gert G. Wagner at Berlin University of Technology came out this summer in the social sciences journal Kyklos. Castronova and Wagner examined life satisfaction ratings from the 2005 World Values Survey and another survey of life satisfaction among Second Life players. Subjecting both sets of data to regression analysis showed correlations between difficult problems in real life leading to a more intense time online. Here’s their abstract:

We study life satisfaction data from the 2005 World Values Survey and a 2009 survey of users of the virtual world Second Life. Among Second Life users, satisfaction with their virtual life is higher than satisfaction with their real life. Regression analysis indicates that people in certain life situations, such as unemployment, gain more life satisfaction from “switching” to the virtual world than from changing their real-life circumstances. Thus, an unemployed person can become happier by visiting Second Life rather than finding a job. Correspondingly, problems in real life are positive predictors of intense use of virtual life.

It’s one of those “I could have told you that,” studies. The importance of the study is, now you can say “Research shows that people with real life difficulties tend to gain greater satisfaction in virtual worlds.” Castronova sums it nicely on the Terra Nova blog:

We are not finding any causal effects here, just correlations. What’s noteworthy is the magnitude of the correlations. Second Life is providing a big chunk of life satisfaction, just as big as the factors that previous researchers on life satisfaction have found were the “biggies,” like health, employment, and family relationships. (By the way, in case you didn’t know, money does not make you happy.)

He has a link to his copy of the study here. The official journal link is here.

References:

Castronova, E. & Wagner, G. E. (2011, August). Virtual life satisfaction. Kyklos  64(3). 313-328.
-

Are Old Video Games Still Art?

In a philosophical piece on Joystick Division, freelancer Dennis Scimeca ponders how old literature such as works of Shakespeare can still feel relevant, as can old paintings and even old movies. Yet somehow, old video games seem … old.

Check out his article, The Challenge of Taking Old Games seriously.

Frankenstein was written 166 years before I first cracked the pages of my copy …

Citizen Kane was shot 50 years before I entered Boston University’s film program …

Yet if I try to play Adventure on the Atari 2600 it’s nothing more than a quaint experience that I’m only interested in for about a minute.

This is a worthy entry in the ongoing discussion of video games as art.
 
 

Center for Children and Technology Reports on DS Games at AERA

I’m honored to be invited to participate in a discussion group this fall put together by the Education Development Center’s Center for Children and Technology (EDC/CCT). The research this group is involved with in the field of classroom gaming is impressive.

The list of projects EDC/CCT is working on is extensive. Among many, one project with the U.S. Dept. of Education involves the design of educational game modules for the Nintendo DSi handheld, aimed at middle school science and literacy.

A paper by Marion Goldstein, Marian Pasquale, and Katie McMillan Culp, members of the Possible Worlds team at CCT, was presented recently at AERA 2011. Here is the abstract for the paper, entitled Using Students’ Naïve Theories to Design Games for Middle-Grades Science:

This paper reports on one phase of a long-term research and development project that is creating video game modules for middle-school science classrooms. The games are intended to help teachers address common scientific misconceptions by providing students with opportunities to interact with visualizations of otherwise abstract or inaccessible concepts or phenomena that are the source of those misconceptions. The visualizations serve as metaphors for natural phenomena, and linking activities help teachers build connections between the visualizations and the targeted concepts. Findings presented here are derived from formative research conducted to inform the development of a game and associated classroom materials that address genetics and heredity. The paper discusses how teachers in our sample typically teach this material in seventh grade, student expressions of common misconceptions about genetics and heredity, and how an initial design for the game responds to and addresses those misconceptions. Students’ misconceptions were associated with the concepts of randomness of inheritance, gene expression, and natural selection.

Perhaps the most impressive aspect of the team’s approach to instructional handheld gaming design is the commitment to research-based efforts. Through direct research with middle school students, the team uncovered several misconceptions held by the students through a series of experiments. When showed a mixed race couple, students’ assumptions regarding the physical makeup of the couple’s children were based on misconceptions. Other experiments uncovered faulty assumptions based on genetic adaptations of beetles and the random characteristics of lotteries. With this research in hand, the team set out to tackle common misconceptions among students at this age and grade level. The remainder of the paper discusses results with prototypes of the resulting game modules.

It’s an excellent report of a work in progress. Research and design such as this will ultimately result in stronger and more effective educational video games.

Dartmouth’s Tiltfactor Researches and Designs Social Games

I’ve been conversing via e-mail with Dr. Mary Flanagan, the founder of Tiltfactor at Dartmouth, where she is the Sherman Fairchild Distinguished Professor in Digital Humanities. Dr. Flanagan and Tiltfactor are doing exciting work in educational gaming. One of the key areas many researchers think it’s strongest is the social arena. This is where “fuzzy” concepts that are so difficult to teach through reading and lecturing can be more effectively transmitted via gaming. Consequently, Tiltfactor focuses on social games, including health and educational initiatives. Here’s a paragraph from their website explaining the organization’s purpose:

Tiltfactor is the first academic center to focus on critical play–a method of using games and play to investigate issues and ideas. Our mission is to research and develop software and playful art that creates rewarding, compelling, and socially responsible interactions, with a focus on innovative game design for social change. We are interested in the processes through which designers imbue their games with moral, social, and political values, whether intentionally or inadvertently, and the corollary processes through which these values are interpreted by players. Our approach involves extensive cross-disciplinary work among the Humanities, Social Sciences, the Arts, and the Sciences.

The academic gaming lab is funded in part by the NEH, NSF, and Microsoft. The center has researched and developed a remarkable list of educational titles. These include, among many others:

• Pox: Save the People, a board game teaching the importance of vaccinations
• LAYOFF, a video game designed to educate players about the financial crisis
• The Adventures of Josie True, a web game devoted to teaching middle school girls math

It’s exciting to see strong academic centers involved in educational gaming efforts like Tiltfactor is, and I encourage other educators and researchers to examine their work. As with most government funded initiatives, such as Josie True, the end product is freely available to schools and teachers. The research potential from their many efforts is considerable, and a list of selected books and articles Dr. Flanagan has written is here. Last but not least, Tiltfactor blog posts can be found at grandtextauto.

Study: Information Asymmetry Results in Starcraft Spectator Appeal

Video game matches have hit the big time, drawing crowds to sports bars where fans can watch matches around the country live on TV. The Wall Street Journal reports today on the phenomenon, noting Blizzard’s Starcraft II was designed from the ground up to be spectator friendly.

The trend is even stronger in South Korea, where competitive video gaming could arguably be considered that country’s national pasttime. Spectators watch and cheer while commentators breathlessly announce onscreen happenings. It’s sort of like Monday Night Football, only virtual, in a video game.

Of more interest to us is the social science surrounding the issue. Here’s the key quote:

Two University of Washington graduate students recently published a research paper seeking to scientifically pinpoint “Starcraft’s” appeal as a spectator sport. The paper posits that “information asymmetry,” in which one party has more information than the other, is the “fundamental source of entertainment.”

For instance, spectators can see the activities of both players and know when one player is preparing an attack. Such asymmetry leads to “situations that result in suspense,” said the paper, including watching one player’s “invisible Dark Templar lying in wait for their opponent’s vulnerable workers.”

Reporter Amir Efrati considerately links the paper for us in the online edition of the story. Gifford Cheung and Jeff Huang, both grad students over at University of Washington’s Information School, presented the paper at CHI 2011. Their main contention, as Efrati reports, is that informational asymmetry is key to stoking up spectator interest in Starcraft II. Here is the abstract from their paper:

Video games are primarily designed for the players. However,video game spectating is also a popular activity, boosted by the rise of online video sites and major gamingtournaments. In this paper, we focus on the spectator, who is emerging as an important stakeholder in video games. Our study focuses on Starcraft, a popular real-time strategy game with millions of spectators and high level tournament play. We have collected over a hundred stories of the Starcraft spectator from online sources, aiming for as diverse a group as possible. We make three contributions using this data: i) we find nine personas in the data that tell us who the spectators are and why they spectate; ii) we strive to understand how different stakeholders, like commentators, players, crowds, and game designers, affect the spectator experience; and iii) we infer from the spectators’ expressions what makes the game entertaining to watch, forming a theory of distinct types of information asymmetry that create suspense for the spectator. One design implication derived from these findings is that, rather than presenting as much information to the spectator as possible, it is more important for the stakeholders to be able to decide how and when they uncover that information.

It’s a fascinating premise. We’ve seen the virtualization of most everything these days. Movies and music are downloads now, rather than physical items. Books and newspapers are read digitally rather than on paper. Software simulates vehicle aerodynamics and nuclear explosions. So, of course it stands to reason that spectator sports would likewise go virtual too, thanks to video games.

Be sure and visit Jeff Huang’s site, which has PDFs of all his papers.

How iPads are Better for Children than TV

A most intriguing post by Mike Elgan over at CultofMac is here, where he asserts that every child in the US needs an iPad. His arguments revolve around the benefits of an interactive device in the hands of a child as opposed to the one way mind feeding that a television set offers. Some quotes:

Everybody’s asking: Are iPads healthy for children?

I’m here to tell you: That’s the wrong question.

The right question is this: Is the iPad a healthy *replacement* for TV? And I believe the answer is a resounding yes.

The iPad is scary because it’s new. But most parents have already accepted a gigantic role for something truly in the lives of their children: television. The content kids see on their TV sets is mostly mind-numbing, soul-deadening, formulaic consumerist crap, punctuated by sophisticated ad campaigns designed to transform children into mindless consumers.

…

Kids spend more time watching TV than they do in class (1,500 hours on TV per year vs. 900 hours in school).

…

I could go on for pages. The bottom line is that TV is a massive, negative, toxic, unhealthy influence in the lives of American children. I think parents already know this.

The solution, Elgan states, is to offer children an interactive device with far less commercial content and mind numbing one way interaction to children.

That’s why fearing the iPad is such a colossal error. The iPad isn’t a new problem. The iPad is a new solution to an old problem.

By *replacing* TV time with iPad use, parents can dramatically improve the lives of their children.

From a parent’s perspective, the iPad is superior to a TV in every significant way:

* The iPad has far fewer, far less harmful ads than TV. It can even be rendered “commercial-free.” Imagine that.

* The iPad is interactive, for the most part, rather than passive. Instead of just staring motionless at TV, kids could be solving puzzles, actively playing games, typing, drawing and other activities.

Yes, even the simplest games on an iPad are far more interactive than any TV show. It’s a good article, and Elgan concludes by encouraging parents to buy iPads for their children as soon as possible. His points merit consideration, and the digerati are taking him seriously. Hopefully others will, too.

Educational App Q Racer Helps Master Terms

Educational game makers are faced with the challenge of inserting pedagogical content that is direct and appropriate. Typically the challenge is met by either making the actions in the game require thought processes that guide the player to useful conclusions, or seek to actionize traditional worksheets.

Apple’s iPod and iPad products immediately caught the attention of educators, and parents have discovered they serve dual purposes as productivity tools and gaming platforms, even for younger children.

I’ve been corresponding with Ben Tao over at Hug a Panda about their new educational app, Q Racer. Designed for kids, the app lets user avatars race against the computer’s characters as they master answers in various categories. Lists include fun items like celebrity quizzes and NFL teams, to more serious items found on tests like state capitals or inventors and their inventions. Avatars can be customized for boys or girls, and users can play their own music during the races. Here’s the trailer video for Q Racer:

Q Racer is available in the App Store at the moment for $.99. Learn more at Hug a Panda’s site, here.

Conspiracy Code Intensive Reading Video Game

Props to Scott McLeod for publicizing Florida Virtual School’s new video game course, Conspiracy Code Intensive Reading. This follows their first offering in the genre, Conspiracy Code American History.

The game seeks to bridge one of the key problems with educational video games. Video games are engaging with kids, and they stay glued to the screen while playing. However, pedagogical content is typically text and paper based. Conspiracy Code Intensive Reading seeks to engage players by making them read as part of the game. Students take the role of a young male or female secret agent and go about spy business in an immersive, 3D environment.

Check out the video below for more details.

Measuring the Wrong Thing: Home Computers and Academics

The New York Times ran an article by Randall Stross, a professor of business over at San Jose State, this summer about research on the impact of home computers on academic achievement in low socio-economic status households. Games and entertainment options were blamed for poor results.

The first paper discussed was by Ofer Malamud over at University of Chicago and Cristian Pop-Eleches at Columbia, who studied low income families in Romania receiving a voucher to assist in purchasing a home PC. The control group were families who applied for the voucher but did not receive it.

In a draft of an article that the Quarterly Journal of Economics will publish early next year, the professors report finding “strong evidence that children in households who won a voucher received significantly lower school grades in math, English and Romanian.” The principal positive effect on the students was improved computer skills.

At that time, most Romanian households were not yet connected to the Internet. But few children whose families obtained computers said they used the machines for homework. What they were used for — daily — was playing games.

Stross next discusses work by Jacob L. Vigdor and Helen F. Ladd over at Duke University, who performed similar research in the United States for the National Bureau of Economic Research. Their study noted after broadband became widely available in North Carolina, math and reading scores plummeted in low SES homes.

The Duke paper reports that the negative effect on test scores was not universal, but was largely confined to lower-income households, in which, the authors hypothesized, parental supervision might be spottier, giving students greater opportunity to use the computer for entertainment unrelated to homework and reducing the amount of time spent studying.

The North Carolina study suggests the disconcerting possibility that home computers and Internet access have such a negative effect only on some groups and end up widening achievement gaps between socioeconomic groups. The expansion of broadband service was associated with a pronounced drop in test scores for black students in both reading and math, but no effect on the math scores and little on the reading scores of other students. In the report, the authors do not speculate about what caused the disparities.

Last, the article touches on the final report (large pdf) from the Texas Center for Educational Research on the state’s one-to-one laptop pilot, which indicated modest improvement on some test scores in the experimental group over the control group.

THE one area where the students from lower-income families in the immersion program closed the gap with higher-income students was the same one identified in the Romanian study: computer skills.

Catherine Maloney, director of the Texas center, said the schools did their best to mandate that the computers would be used strictly for educational purposes. Most schools configured the machines to block e-mail, chat, games and Web sites reached by searching on objectionable key words. The key-word blocks worked fine for English-language sites but not for Spanish ones. “Kids were adept at getting around the blocks,” she said.

-+-

Unfortunately, all these studies are measuring the wrong thing. This is equivalent to giving a 1920s farmer a new radio, then measuring the increase or decrease in his crop yield. Why would you expect his crop yield to increase after he’s been given this new communication technology?

A better tool for increasing crop yield would be a tractor, which could plow more than his horse. But, the radio — the highest available technology at the time — would be considered a failure because it did not directly result in higher crop yield.

The farmer would have received timely news and weather reports which perhaps would have indirectly affected his yield. Also, the farmer and his family would have been exposed to an increase in entertainment, experiencing the latest from New York and LA. This cultural acclimation would be accompanied by a wider appreciation of world and national events, which would have had no effect on farming but would have benefited the farmer and his family nonetheless, albeit indirectly. Perhaps the farmer would learn something useful from an agricultural program, but again, looking for a direct increase in crop yield completely misses the point.

Ultimately, computers at home or in one-to-one programs should never be expected to increase academic scores, just as a farmer’s radio shouldn’t be expected to directly increase his crop yield. Home computer functions simply do not correlate well to traditional test taking. But computers do have value in several other areas, in the creative programs and games they run, and the communications capabilities they offer to students. But discerning whether an experimental group earns higher test scores than the control group is simply measuring the wrong thing.

References:
Stross, R. (2010, July 10). Computers at home: Educational hope vs. teenage reality. The New York Times, p. BU3.

Vigdor, J. L, & Ladd, H. F. (2010, June). Scaling the digital divide: Home computer technology and student achievement. National Bureau of Economic Research. [Online.] Available: http://www.nber.org/papers/w16078