Paper Reading #4: Not Doing But Thinking: The Role of Challenge in the Gaming Experience

Introduction

Not Doing But Thinking: The Role of Challenge in the Gaming Experience was written by Anna L. Cox, Paul Cairns, Pari Shah & Michael Carroll. Dr. Cox is a senior lecturer in Human-Computer Interaction at University College London and an Associate Chair for CHI 2013, CHI 2012, and CogSci 2012. Dr. Cairns is a senior lecturer in Human-Computer Interaction at The University of York interested in video games and modeling user interactions. Cox and Cairns coauthored Research Methods for Human-Computer Interaction, published by Cambridge University Press in 2008. Pari Shah graduated from University College London in 2008 with a degree in Psychology and Michael Carroll studied Computer Science at The University of York.

Summary

This paper investigates the role of challenge in a user’s experience of immersion through three studies. The concept of challenge is distilled into two modalities: pushing a gamer’s physical limits (twitch mechanics) and pushing a gamer’s cognitive limits (time constraints).The first experiment “manipulate[s] the number of interactions required to make progress in the game and thus the speed with which the gamer must interact with the game.” Their second and third experiments focus on making the gamer think faster by manipulating the level of time pressure under which the gamer must perform. They hypothesize that cognitive challenges have a greater effect on immersion and therefore expect higher levels of immersion to be reported in experiments two and three, in comparison to the first experiment.

The authors identify immersion as “a graded experience ranging from engagement, through engrossment to total immersion.” Total immersion is synonymous to being in a state of flow, during which all of a gamer’s mental faculties are focused on the task at hand (the game). Flow is achieved “as a result of an appropriate balance between the perceived level of challenge and the person’s skills.” This idea led the authors to consider the role of expertise, hypothesizing that immersion will decrease if the game is too challenging (resulting in a state of anxiety) or if the game isn’t challenging enough (resulting in a state of boredom).

Experiment one was conducted using a tower defense game and 40 players of varying levels of expertise. The game was calibrated to an appropriate rate of play, reduced approximately one third to create a low effort condition(LE), and increased approximately one third for the high effort condition(HE). Players were divided into 3 expertise groups: insufficient expertise (IX), low expertise (LX), and high expertise (HX) and immersion was measured using the Immersive Experience Questionnaire (IEQ) modified with 3 additional questions. Players were given a mandatory in-game tutorial, allowed to play for 8 minutes or until they ran out of lives, and immediately evaluated using the IEQ. Players in the HE condition performed an average of 60% more actions than LE players and players with more expertise consistently performed better, demonstrating the success of the experimental setup. The authors found that physical effort had no significant effect on player immersion and that LX players were equally immersed in both conditions, while HX players experienced decreased immersion when presented with the HE condition. This is hypothesized to be due to HX players viewing the challenge of the game to be cognitive and therefore found the added physical challenges of the HE condition unreasonable or frustrating. No interaction between expertise and level of challenge were observed (having more expertise didn’t decrease the number of actions performed). IX player data was discarded due to significant differences in performance, likely due to not reading the tutorial.

Experiment two tested the hypothesis that participants playing under time pressure will experience significantly higher immersion and challenge than those playing without time pressure. Testing was conducted using 22 players playing Bejeweled in a timed or un-timed mode for 15 minutes before completing the IEQ. The authors found that players playing the timed mode experienced a higher level of challenge as well as significantly more immersion. Effects of expertise were not measured in this experiment.

Experiment three addressed the hypothesis that expertise affects the level of cognitive challenge associated with a game, thereby affecting the level of immersion. The authors tested their hypothesis with 20 players, divided into expert or novice groups, playing Tetris at low difficulty (level 1) or high difficulty (level 6). Players played for 15 minutes with the low difficulty players not allowed to progress past level 2 and the high difficulty players allowed to continue until the game ended before resetting. All players played for a total of 15 minutes before completing their IEQ. Expert and novice players were equally immersed at high difficulty. At low difficulty, novice players experienced a slight increase in immersion and expert players experienced a significant drop. These results confirm that immersion is dependent on a balance between skill and challenge, but reveal that challenge has no effect on immersion when expertise isn’t taken into account.

Related Works

1. Immersion, Engagement, and Presence: A Method for Analyzing 3-D Video Games by Alison McMahan
2. Flow and Immersion in First-Person Shooters: Measuring the player’s gameplay experience by L. Nacke & C. A. Lindley
3. Ludic Engagement and Immersion as a Generic Paradigm for Human-Computer Interaction Design by C. A. Lindley
4. Revising Immersion: A Conceptual Model for the Analysis of Digital Game Involvement by Gordon Calleja
5. Patterns in Game Design by S. Björk & J. Holopainen
6. Video Games: Perspective, Point-of-View, and Immersion by L. N. Taylor
7. Sex Differences in Video Game Play: A Communication-Based Explanation by K. Lucas & J. L. Sherry
8. Video Game Designs by Girls and Boys: Variability and Consistency of Gender Differences by Y. B. Kafai
9. Heuristic Evaluation for Games: Usability Principles for Video Game Design by D. Pinelle, N. Wong & T. Stach
10. Explaining the Enjoyment of Playing Video Games: The Role of Competition by P. Vorderer, T. Hartmann & C. Klimmt

To begin, I will establish that the study of game experience is well established. Patterns in Game Design is a book on the paradigms of game design and Heuristic Evaluation for Games: Usability Principles for Video Game Design is a paper exploring rule-of-thumb evaluations aimed at improving game design. Both of these works focus on gaming experience from the point of a designer, explicitly emphasizing the role of challenge and the ultimate goal being the creation of an immersive experience. 

Additionally, the role of challenge is well established and studied within the context of gaming. Explaining the Enjoyment of Playing Video Games: The Role of Competition takes a psychological look at what makes games fun on a universal level. The authors argue, successfully, that the unifying feature of fun games is competition, namely, the desire to win. This competition comes either in the form of opposing players, or in the form of challenges imposed by the game itself. Further research into the effects of challenge on video games revealed important data that the authors of Not Doing But Thinking failed to take into account. Sex Differences in Video Game Play: A Communication-Based Explanation and Video Game Designs by Girls and Boys: Variability and Consistency of Gender Differences are two independent studies that discuss the variation in response to challenges within video games between men and women. Not Doing But Thinking makes claims on the effects of challenge on game immersion and experience without taking into account that it is well known that men and women have different cognitive responses to challenge and that these differences are evident in the way that they play games.

Next, I assert that the role of immersion in gaming is well studied. Immersion, Engagement, and Presence: A Method for Analyzing 3-D Video Games explores the effects of 3-D design on immersion, analyzing the relationship between immersion and artwork. Video Games: Perspective, Point-of-View, and Immersion is a similar study that focuses on the player’s perspective on a game world and their point-of-view within said world to discuss immersion. Revising Immersion: A Conceptual Model for the Analysis of Digital Game Involvement takes a systemic approach to exploring immersion, focusing on the various forms and levels of involvement that contribute to an immersive experience. Ludic Engagement and Immersion as a Generic Paradigm for Human-Computer Interaction Design identifies immersion as a critical goal for all human-computer interaction applications and explores its potential use in ludic systems. These studies all take a different approach to analyzing the effects and conditions of immersion and agree that immersion is critical to the gaming experience.

Having established that the study of game experience is nothing new and that the role of challenge and the importance of immersion are well studied topics, Not Doing But Thinking only remains novel in that it addresses immersion from a cognitive standpoint and uses quantitative measures in its analysis. This sets Not Doing But Thinking apart from the aforementioned studies, but not from Flow and Immersion in First-Person Shooters: Measuring the player’s gameplay experience. This paper explores immersion from sensory, imaginative, and challenge-based perspectives. Nacke & Lindley use a host of measurements ranging from psychophysiological indications of arousal to qualitative flow measurements to effectively analyze what factors contribute to an immersive experience. I found their study to be much more robust and conclusive, effectively rendering Not Doing But Thinking insignificant.

Evaluation

The authors use a combination of quantitative and qualitative measures to evaluate their experiments on both a systemic level and a component level. In experiment one, evaluation was done based on players’ scores in the tower defense game, the number of actions they performed while playing the game, a quantitative measure of the players’ expertise, and an Immersion Experience Questionnaire. These measures were effectively used to validate their experimental design (component-based evaluation) and conclude that physical effort had no significant effect on player immersion and there was no interaction between expertise and level of challenge. Experiment two used players’ Bejeweled scores and IEQ data to conclude that time constraints increase the level of immersion experienced. Experiment three used a qualitative assessment of players’ skill levels, their Tetris scores, and IEQs to conclude that the level of challenge only affects immersion when skill is taken into account.

Discussion

While the authors’ evaluation methods were excellent their attempt at novelty failed. There is at least one other study that measures immersion using quantitative data and that study does so much more successfully. This study had a good premise but ultimately concluded very little. I enjoyed the paper until I realized that their experiments yielded little data. What little enthusiasm I clung to after reading this paper was quickly dashed once I discovered Flow and Immersion in First-Person Shooters.