Abstract

Gamification is an umbrella term which denotes a series of instructional activities which use game elements in nongame settings. Under this definition, there are various categories of gamification-based applications for learning, including serious educational games (SEGS), serious games, and educational simulations. This is the first of a two-part series on how to build serious games for online learning. In this paper we will examine the use of SEGS in training and education through a literature review. The second article will be on how to create immersive games for training. Educational technology practices such as building high-engagement learning management systems, classroom game activities, and other forms of related curriculum development are all addressed. There is overwhelming evidence on the efficacy of SEGS and simulation games in higher education and training. Success factors and processes for building effective SEGS are reviewed in this first paper.

1. INTRODUCTION

Serious educational games (SEGS) are teaching methodologies which are based on the use of game elements in a nonentertainment context (Ciftci, 2018). Gamers engage in activities which demand high focus and decision-making, many of which are transferable to learning and other activities (Green, 2010). Playing games is related to many elements core to higher education, including teamwork, social connectedness, decision-making, planning, and resourcefulness (Galarneau, 2005). Further, SEGs often provide high-engagement learning and immediate feedback to the player which improves upon or can replace traditional formative and summative evaluation processes (Belotti, 2013).

Authentic alignment is defined as the coherent and explicit linking of educational practice to established educational theories with a view to improve the quality of outcomes of training (Macht, 2016). At this point in time, it is a core determinant of success in education. SEGS which are designed for the purpose of increasing alignment include simulation games and other forms of rehearsal of professional skills in settings where errors can be made in an attempt to master a given domain of knowledge. As such, SEGS are a tool which can be used to increase authentic alignment by permitting learners to focus on tasks specifically related to professional roles in a psychologically safe environment. Additionally, SEG design can include ongoing analytics which reflect the learner's progress which provide more detail and more timely outputs than traditional testing (Serrano-Laguna et al., 2017).

2. PREDICTORS OF SUCCESS IN HIGHER EDUCATION: TOWARD AUTHENTIC ALIGNMENT

A number of meta-analyses of the determinants of success in higher education have shown that course micro-organization is the strongest variable associated with strong alignment (Schneider and Preckel, 2017). High effect sizes were linked to social interaction, stimulating meaningful learning, assessment, presentation, and technology. Technology improvements did not scale, surprisingly, and saturated early. Thus, the idea that we would use game systems for the audiovisual impact or learning management tools they could provide seems unfounded. Teaching variables with the greatest effect size were those factors related to using the appropriate pedagogical methodology based on evidence for a given domain of instruction. Given that SEGS demand careful analysis of learner activities within the game system, they provide a great opportunity to microdissect each component of educational delivery to determine whether to include it within a game system.

Of interest also were the findings that self-regulated learning strategies, during which students control the pacing, depth, and engagement for learning, had a strong effect size. Class attendance had the greatest effect size of the student strategy variables, as SEGS demand high levels of participation. Notably, the authors comment that the more that students “engage in effort regulation – respond to challenging academic situations with persistence and effort,” the higher their achievement. This certainly points to the possible role of SEGS in providing consistently more difficult learning challenges by which students can not only benchmark their achievements but also engage in “titrated challenge.” Not only does this resonate with respect to the design of good learning games, but it is also consistent with Csikszentmihalhyi's “flow theory” (Csikszentmihalyi and Nakamura, 2014). Thus, SEGS not only satisfy the demand for authentic, ongoing self-regulation of learning but also provide a positive psychological experience.

The area of science, technology, engineering, and mathematics (STEM) education, in particular, demands that learners require support unique to these disciplines, which are the basis of many professional fields. Beginning as early as high school, girls who aspire to enter STEM professions recognize social barriers (Grossman and Porche, 2014), often fail to see themselves as STEM-capable (Aschbacher et al., 2010), and have difficulty identifying role models (Cheryan et al., 2011). As such, STEM enhanced training should focus on design as a classroom activity, where students have the opportunity to integrate this form of intelligence into their thought habits (Krajcik and Delen, 2017).

Interdisciplinary fields such as construction trades, where students need to have a full range of skills, from abstract planning to technical competence to soft elements such as communication, are particularly strong candidates for SEGS, given that these demand high strategy capability (Lizier et al., 2018). Problem-based projects in particular are extremely useful in teaching integrated skill sets demanding constant reflection and analysis.

Brain training is another area where continuous cognitive challenge is key to develop different forms of thinking. Successful brain training has not been evidenced outside of a small number of task-focused practices in fields such as stroke rehabilitation, so it is wiser to conceive of what we might call brain training as a form of continuous self-evaluation and challenge (Seitz, 2018). Game systems can be designed which are built upon this foundational activity loop, where increasingly more difficult challenges are encountered which provide direct, ongoing feedback to the learner. Similar to problems which present in video games, the ability to repeat a problem until it is solved is called resourcefulness and is identical in learning. Studies which show that video games improve resourcefulness and adaptability are directly transferable to higher education (Barr, 2017). If brain training is taking place, the evidence to date shows that it is nongeneralizable or transferable, but rather task-specific.

Lastly, new data on the “wandering mind” and its complement “default mode network deactivation” show that during learning there are several processes which SEGS can target to improve effectiveness. Mind-wandering shows that cognition is not tethered to the present moment, but that regions of the brain referred to as the default mode network (DMN) allow memory representations to form consciousness experience (Poerio et al., 2017). SEGS can be designed in the form of gameworlds, discussed below, where a player inhabits an alternate reality where many rule systems can be introduced, such as building a virtual business in a community or saving a dynamic population of simulated patients. In this rule-restricted gameworld, students can explore many aspects of learning from many perspectives and activities. Gameworlds have been used to describe epidemics using this modeling. In 2005, millions of players of the game World of Warcraft experienced an in-game epidemic which killed many avatars (symbolic players) off within a few hours (Lofgren and Fefferman, 2007). Players within gameworlds are therefore able to encounter simulations of real-world conditions and progressively learn about complex systems through direct experience.

On the other end of the spectrum, within a game system, one can provide highly focused learning activities which apparently deactivate the DMN (Howard-Jones, 2017). In one study, game-based learning deactivated mind-wandering in subjects compared to those using either self-study or quizzing. In this same study, the ventral striatum, the major learning reward center in the brain, was activated during game conditions but not by either study or quiz completion. Hence, game systems enable the student to focus on specific learning activities by inhibiting mind-wandering and simultaneously switch on the sense of reward that learners feel during engagement. Thus, SEGS appear to have the capacity to expand the learner's experience of content and then suddenly contract it to focus on the details within a problem. This alternate flow of open exploration followed by precise problem solving we refer to as “breathing.” Good SEGS seem to be able to breath, providing learners with numerous opportunities to increase alignment through the careful microdesign of learning that their design demands.

3. EMPIRICAL EVIDENCE FOR GAME-BASED LEARNING

We must make an important distinction here between learning games and game-based learning (GBL). GBL differs in that it substitutes for traditional instruction and is the core way in which learners acquire knowledge and skills. Learning games, such as “review Jeopardy,” of which there are dozens on the market, often available for free, may be used to supplement learning and make some tasks such as knowledge review more engaging. Frame games are simple games such as Bingo or Tic Tac Toe which can be integrated with questions, such that playing the game permits recall of facts (Duplaa et al., 2017). Kahoot!, a digital game for learning, is used to generate game play format and is now widely used in higher education (Dellos, 2015). These are not included as examples of GBL. GBL demands that curriculum delivery is dependent on the use of game systems and they usually integrate formative and summative assessment criteria, content curation, social interactions, leaderboards, badges, achievements, and other elements within a coherent system. A given GBL activity might be restricted to a small content area, but the learning is directly related to game performance. We cannot overstate the importance of this distinction.

3.1 Terminology

As we explore the evidence in this field, it is useful to provide a short glossary of terms (Table 1). This is a dynamic field and there are no specific dictionaries of terms; instead this is based upon significant exploration of the field by scholars and contemporary researchers (Kaufman and Sauve, 2010).

Ciftci (2018) reviews the major areas of interest in the field using text-mining methodology. Most research publications, accounting for 20% of the field, were in computer science, with other high-focus areas including educational research, psychology, engineering, health care services, environmental sciences, occupational health, rehabilitation, business economics, and psychiatry. The highest citation numbers were seen in systematic reviews such as this one, along with virtual reality (VR) studies, entrepreneurship education, and fields such as cognitive training in disorders such as schizophrenia. The USA is the current leader in the field, with the UK and Holland ranked 2nd and 3rd.

The leading meta-analysis of the field was published in 2018 (Lamb et al., 2018) where cognition, affect, and learning outcomes were all assessed when compared with serious games (SGs) and simulations. For example, in a review of 46 studies comparing SEG outcomes with traditional learning, the authors showed that effect sizes in learning for all three categories were no different from traditional learning but there were differences among them. In other words, consistent with earlier work on learning outcome studies, there is no demonstrable difference in course performance-related learning outcomes between SEGS and traditional instruction (Anetta et al., 2009; Fengfeng, 2008; Spires et al., 2011; Yang, 2012). Most of these studies were initially done with K-12 learners but the data seems to hold up well with adults. These data do not take into account the effects of simulation gaming in job performance rehearsal, which when assessed showed a strong effect size for cognition and affect. Affect, engagement, motivation, self-efficacy, cognition, skill development, and dimensionality were all assessed in Lamb's study. Across 2151 articles entered into the analysis, there was a trend to show that greater overall understanding of subject matter ranked under the categories of cognition and skill development had a strong effect size. Affect, indicating emotional connection to learning, was also clearly increased during SEG conditions. We can conclude from these data that SEGS permit learners to put basic theory into practice more effectively than traditional methods. However, there was unequivocal evidence across numerous studies that SEGS showed a higher engagement level than traditional instruction and that, presumably, leads to more robust participation in learning activities.

Descriptive overviews of the field (Jackson, 2016) show that learners are game-centric now, and this probably contributes to some of the positive responses to SEGS that have been noted. The average age of gamers is 37 years, with 97% of youth now playing video games. Video games are owned in 77% of households, and 68% of parents view video games as providing mental stimulation. The number of female gamers is 42%, and 65% of gamers play with other people rather than alone. Downloads are prolific, with 55% of games being played on mobile devices; over 2.6 million games are downloaded each year in Germany alone. Gamers have collectively spent 5.9 million years playing the popular role-playing game World of Warcraft, and the total time spent gaming in the US per day is 215 million hours. Clearly the data shows that video games are embedded in modern culture, and it seems rather out of place that we would not include them in the learning process.

Systematic literature reviews conducted and updated by other teams (Boyle et al., 2016) show that, using randomized control trials (RCTs) across different disciplines in seven strong studies, SEGS lead to better performance than control. Short- and long-term knowledge retention in anatomy courses, however, was higher with traditional instruction. Face-to-face teaching showed higher results than online gaming activity in other studies. However, when skill acquisition studies were analyzed (N = 10), both qualitative and RCT data showed a significant improvement over traditional methods. Only two strong studies have been published looking at higher cognitive tasks, such as city building or medical triage decision-making, and both of those showed a small effect size over a six-week game trial. These data show that SEGS have their greatest effect size in teaching learners how to apply knowledge rather than acquire it. This begets a deeper and more nuanced question about how cognitive and emotional factors come to influence learning in SEG conditions.

3.2 Psychological Needs Perspectives

The compelling evidence to date that SEGS promote higher affective outcomes in learning suggests that cognitive-emotional perspectives on gamification present a paradigm shift from a focus on learning outcomes to the improvement of the process of learning. Emotion and cognition are only minimally separable, and it appears that games can provide connection of emotion to memory, emotion and attention, and emotion and decision-making. Given that almost all our behavioral choices are now known to be emotional choices, we can now think about designing SEGS for emotion. Events such as earning a badge in a game will trigger either positive or negative valence. If another player earns a badge, there might be resentment or stress during competition, or happiness for the other player if it is team-based or altruistically motivated. So the first design goal appears to be to provide a system which might support artificial intelligence. Desired emotional outcomes then become the driving force in the choice of SEG design (Mullin and Sabherwal, 2018). Mullins and Sabherwal at the University of Arkansas discuss the consequences of events in the game, and thus rule mechanics are used to generate attributes which evoke emotional response. Agents are characters or simulation situations in the game, and objects are elements of the game such as the environment; all of these now drive the emotional experience within the rule set. Given that the data shows engagement as the primary differentiating outcome in game vs. traditional systems, these perspectives help unify a broad scope of data.

Van Roy and Zaman (2018) have developed a psychological need model based on self-determination theory. Among their conclusions based on a 15-week study, they note that badges encourage progression and provide reward, whereas summative evaluation fails to do so as its primary purpose. They also observed that when playing the SEG face to face, students felt inhibited and held back on solutions, fearing censure by fellow students in the team. This contrasts with studies of social interactions in online games, where new friendships are forged and teams of completely random strangers unite to achieve an in-game goal such as fighting an in-game creature (Ducheneaut and Moore, 2005; Nardi and Harris, 2006).

Another focus area is emerging on the role of SEGS in the education of nontraditional learners. Digital games within courses have the capacity to promote problem solving and critical skills (Turner et al., 2018). Building employment competencies, in particular, has been tested using game systems with high satisfaction self-reports (Snow, 2016). Digital games can be related to the desired job environment in the form of simulations and open gameworlds. They can increase perceived relevance and can be customized to incorporate a host of learning style preferences and multiple intelligences. The cookie-cutter approach to traditional teaching, even using flipped classrooms and other student-enabled learning, does not present a wide enough bandwidth of learning experiences to engage learners from diverse cultures and styles. By contrast, good game design can include major engagement pathways using player type categories of Peter Bartle, first developed by Erwin Andreasen and Brandon Downey, referred to as the Bartle Test of Gamer Psychology.

According to Bartle there are four major types of players who engage in multiuser dungeon games, gameworlds which consist of prolonged adventure narratives and gave rise to the massive multiplayer online role-playing games such as World of Warcraft, which still retains over 6 million paid subscribers a decade after its release. Achievers like to complete quests, whereas his second category, explorers, prefer to locate new experiences and locations in the gameworld. Socializers play mostly to connect with other players, and his last category, killers, like to compete. It is possible to think about a college course as having the same population of students who enter: some desire achievement, others enjoy exploration. Using game-thinking principles, then, it is possible to build courses that appeal to all four of Bartle's player types as an emotional design principle (see Figure 1).

FIG. 1: Bartle's player types

3.3 Technology Demands and Acceptance

Learner and instructor attitudes toward SEGS can be central determinants in the success or adoption of this methodology. In one study those students who played a lot of games enjoyed SEG-based instruction more than those who did not (Landers and Armstrong, 2017). The latter group received as much benefit from PowerPoint presentations as the game and preferred this format for delivery. However, on average, all players had a higher valence for gamified instruction, showing that it was preferred across groups. The level of comfort, however, was much lower for naïve gamers.

Toward the end of finding solutions to the questions related to how to select SEG technology, many models have been generated to provide a framework for decision-making in design. The Beaconing approach developed at the Department of Computer Science at the University of London suggests a taxonomy based on specific categories (Bourazeri et al., 2017). In this rubric they integrate learning objectives, pedagogical resources, narratives, game plots, and location-based activities. Game systems are then populated with activities driven by this conglomeration of player needs.

Game architecture refers to the coding of SEGS as either stand-alone applications or as enhancements/add-ons to learning management systems. SEG programmers need to consider both the pedagogical scenario for learning and the game scenario. These need to be blended in a coherent way on both conceptual and hard-coding levels. Programming games might use a 3rd party program such as Unity 3D, which is available for $35–$125 US per month as a license fee, which permits players to create three-dimensional gameworlds with a minimum of need for art design and coding (Wassila and Champagnat, 2017) (Figure 2).

FIG. 2: Third-party programs permit players to create three-dimensional gameworlds with a minimum of need for art design and coding

C++ and Java programming can be used to create real-time strategy games, alongside Pascal. For mobile devices, HTML5 is used for web-based device games, or Objective-C for iOS apps selected by Apple. Swift, C#, and Java are also mobile device languages of choice.

The gamification of learning management systems (LMS) is another path that faculty might take to create online autonomous, asynchronous game experiences. Surprisingly, very few reports are available to show how to leverage key features of an LMS for gameful experiences. Our team at Humber College in Ontario, Canada, has produced a simulation game for faculty use on Blackboard Learn. The key game features used included adaptive release, timed questions, digital badges, upvoting content, and timed release of content in discussion forums (Ihnat, 2018). These are simulation games where one cannot proceed to the next part of a multinode problem without completing the current phase, similar to quest lines in online gaming. Nodes are branching decision points in a linear quest or simulation line as shown below (Figure 3).

FIG. 3: In simulation games, one cannot proceed to the next part of a multinode problem without completing the current phase, similar to quest lines in online gaming. Nodes are branching decision points in a linear quest or simulation line.

One Blackboard Learn Exemplary college course publication by Machajewski (2017) showed the integration of an LMS with a number of technologies for a variety of game experiences within a single course. He used the MyGame mobile gamification app, Cengage Skills, Assessment Manager, Kahoot!, Amazon Alexa, Google Traveler, and Twitter alongside other apps to create a robust gaming experience. Although this lacks the coherent narrative structure of immersive simulation games for learning, this “arcade style” approach presents a variety of high-engagement learning activities under the general category of playful design.

Analog games, those where there is no dependence on computers, including card, board, and role-playing game variants, are in use across higher education and have a certain appeal. They are inexpensive to produce. Deb Fels at Ryerson University has been offering a fully gamified credit course for several years using manual tracking of player income, progress, and in-game purchases. Rob Bajko, Jaigris Hodson, Lori Schindel-Martin, and Karen LeGrow from Ryerson's Department of Professional Development and the School of Nursing have also used paper-based fully gamified course systems for undergraduates. In the early days of the field this author built out fully gamified courses using a photocopied rule book with supporting texts and teaching assistants to teach anatomy and physiology to midwifery and nursing students (MacDonald, 2015). There is no compelling reason to use technology-enabled SEG design other than scalability, the need for asynchronous delivery, and ease of player progress tracking. If these demands are unavoidable, then code needs to be written or third-party vendors need to provide a site license. Reach, a stand-alone gamification platform designed by the author and coder Klaus Rubba, is one such vendor offering which provides a full game experience without graphics linked to a LMS through SCORM (Shareable Content Reference Model). This necessitates entering into a contractual relationship with the service provider, a step which many educators are still reluctant to take. However, the overhead cost for production of even a simple game for one course can exceed $15,000 and run as high as $100,000. Over time we predict that vendor relationships and site licenses will become the norm in SEG design. Companies such Axonify, Bunchball, Game Effective, Reach, and Launchfire are all vendors who can provide training game platforms under license.

4. SERIOUS GAME DESIGN

Best practices for SEG design are maturing across the globe as various projects and publications yield data on how one should actually build a SEG. Is it enough to use a Jeopardy-style quiz mechanic for a given purpose, or are educational simulations with high immersion required? There are no clear answers to these questions yet, but there is a wealth of expert consideration and studies to enable us to think about design in a rational rather than an exposure-limited fashion. “Exposure-limited” refers to the fact that many trainers or faculty who design SEGS build them based on games they have played in the past, hence the plethora of simple “answer the quiz right” game mechanics used by startup gamification builders. As more academic rigor penetrates the field, game design should start to be based on using the right rule system and narrative for a given educational goal, rather than selected by default due to the instructional designer's limited experience playing games.

4.1 Pedagogical Design Criteria

When building a SEG there is a requirement that we identify the type of training to be delivered at the professional level, which ranges from trades education and apprenticeships to clinical decision-making in nursing care. Gardner's theory of multiple intelligences may be a useful launching point for making design decisions (Sajjadi et al., 2017). People exhibit multiple types of intelligence; it might be musical, strategic, or mathematical. These represent a number of learning dimensions which can be used to generated a player-centered design model. Specific video games such as Portal or Tetris, for example, access the visual-spatial dimension, whereas Kinect Sports or Just Dance involve the bodily kinesthetic intelligence dimension. Games such as Heavy Rain or The Room access logical-mathematical intelligence, whereas games such as Black and White and Fable lead the player into interpersonal dimensions. The game Deus Ex, for example, still cited as one of the top 10 video games of all time, has the player making moral choices during the open world game play which open up new adventures in the game. Fable has the player assume an avatar appearance which grows visibly nicer or uglier as they make moral decisions in the game, and this affects the kinds of further experiences they will encounter.

Achievement and progress benchmarking is another key consideration in design which has been investigated in some major studies on badging. Digital badges are achievements that players obtain as a one-time recognition of effort and/or competency. In one study conducted in Finland with over 1400 subjects in both game and control groups, digital badges increased job-related skill execution one year after the game was completed in training (Hamari, 2017). Recognition of achievement alongside continuous nonpunitive feedback is core to game success in most systems. Regular quizzes and assessments are all potentially punitive, and the instructor's role is that of benign dictator at evaluation time. The game system is divorced conceptually from the faculty member's leadership role in the course, since the rule set itself is the pedagogical agent. When a student fails an activity in the game, the game rules administer feedback, and multiple attempts at problem solving can take place in a psychologically safe atmosphere. This can, of course, be used to prime or prepare students for incoming formative or summative assessment.

Avatars are representations of a player, either on screen or within a larger narrative. Avatars contribute to engagement and motivation in games. Simulation games, those most often desired or used in SEGS such as business program case studies, usually do not use avatars. Hauge (2017) has shown that avatars provide such powerful motivational effects that they should be used in simulations. The avatar can represent actions of the player as they encounter other players in the game and can be used to transcend established social relationships to build collaborative teams for interprofessional education. Intrinsic motivation inventories including dimensions such as interest, perceived competence, effort expended, pressure/tension, and perceived choice, along with social presence measures such as empathy or negative feelings, have all been analyzed with respect to avatar development. The outcomes of this study did not clearly support the contention that including an avatar improved learning, so its role as a determinant in improvement of simulation learning is still open to question.

Other hidden aspects of design have to do with barriers to adoption by faculty. The core barriers to the use of gamified learning lie in four dimensions: lack of resources, student apathy, subject fit, and classroom dynamics (Sanchez-Mena and Marti-Parreno, 2017). There are four main drivers identified to date in implementation: motivation, entertainment, interactivity, and ease of use. How these drivers interact with barriers determines the success of launch for SEGS. The key finding is that faculty perceive SEGS as providing an element of risk, and if this risk outweighs the potential benefits of creating a gamified system, then there is little incentive to adopt the practice.

Analysis of drivers, then, is core to understanding how and why to implement SEGS in instruction. Studies in entrepreneurship training suggest that SGs improve innovation, leadership, strategic thinking, problem solving, business launch, and risk management. At the same time, they do not appear to show benefits in the area of accounting skills, regulation compliance, interpersonal soft skills, communication, self-control, and discipline (Almeida, 2017). In one study assessing the correlation between academic performance and game success in undergraduate accounting students, a positive association was shown (Malaquias et al., 2018).

Psychological enjoyment factors and learning mechanism relationships have been explored in environmental games using a novel environmental educational game enjoyment model (Fjaellingsdal and Klockner, 2017). The study observed students playing the game Fate of the World, a digital card-based global strategy game released in 2011. In the game, the player takes on the role of a CEO to implement worldwide policies to protect the environment. The authors analyzed 249 available game reviews, and after identification of 192 reviewers who rated it favorably, the player base showed that they had spent a total of 4064 hours playing the game (collective data, not per player). Challenge was key in this game, alongside other rater positive categories, including thought-provoking content about the environment, realism, and being generally well-designed. Negative reviews used descriptions such as unintuitive layout, too difficult, boring to look at, and, notably, lacking a sandbox. We will discuss sandbox design later; it refers to giving the player a creative space to play within.

4.2 Immersive Technology and Digital Neuropsychology

Video game use impacts the central nervous system in many ways, ranging from changes in hippocampal neuron density to alterations in dynamic functional connectivity. Internet gaming disorder (IGD) is an extreme example of functional changes resulting from video game play. Resting state functional connectivity of the dorsolateral prefrontal cortex has been shown to be a marker for IGD. Craving and overengagement despite negative life consequences is the hallmark of addiction. Diminished control of impulses seems to be generated by changes in resting state neuronal activity which weaken capacity to regulate game play (Han et al., 2018).

Physiologic changes in heart rate and facial actions during gaming are indicative of emotional responses, which can be assessed in the discovery of key game triggers (Bevilacqua et al., 2018). Boring and stressful moments in gameplay can be differentiated by these responses. Immersion refers, then, not only to sensory awareness of being “in” a game or simulation but also to the emotional participation in game events. We all know how we feel during a boring scene in a movie, and how immersed we feel when an exciting scene or actor appears on the screen. This switching from bored to engaged is tracked by increases in heart rate and facial expression as game conditions transition from boring to stressful. Immersion is therefore a primarily emotional/physiological process with no hedonic component which is necessarily attached; that is, we can dislike the content of the game but still maintain high emotional investment and immersion in the experience. In other words, high engagement does not indicate that learners are having a pleasurable or even helpful experience, it simply marks one element of immersion.

The relationship, therefore, between game engagement and VR and augmented reality (AR), what is referred to now as mixed reality (MR), bears some discussion. At their essence these are two entirely different questions, how MR affects learning vs. how SEGS affect function. There is another set of questions related to how combinations of SEG and MR design may interact to optimize instruction. In the design of MR environments, how can we promote user connectivity to form social learning? The issues related to gameworld design clearly overlap with MR research questions. Activity theory is a promising integrative framework for learning how to leverage MR based on what we know about game immersion (Jensen, 2017).

Problem-based learning has a natural home in SEG design in that the concept of exploration, attempting to solve cases, identifying gaps in knowledge, and using domains of knowledge as central markers for achievement rather than explicit learning objectives are compatible with both systems. In studies using construction management, particularly relevant to our discussion it was shown that VR collaboration can be realized by utilization of the virtual environment to mediate a problem-based process. In Jensen's study cited above, it was shown that spontaneous play can occur in MR collaborative settings, as though the technology itself engendered specific forms of behavior. Given that virtual universes tend to steer students toward specific actions and content, the same way that simulation cases do in problem-based learning, the principles which describe the latter seem to transfer to the MR setting. Hence, using MR as part of collaborative learning can be predicted to be more effective than solo experience if we are to accept that problem-based dynamics and methodologies are directly transferable.

The role of AR in learning is emerging, as evidenced by over 1200 papers in the field since 2011 (Sirakaya and Sirakaya, 2018). Studies cited in literature reviews show that AR can increase achievement, facilitate learning, enhance motivation, ensure long-term retention in some studies, and increase student participation in class. Other studies show that AR increases participation in class, helps shape positive attitudes, ensures cooperative learning, increases fun in learning, and decreases cognitive load. Very few qualitative studies on AR learning exist, despite a peak publication output of 26 papers in 2016. Biology and education, alongside engineering and medical training, dominate this field. Class sizes ranged from 31 to 100 students in most studies to date, primarily undergraduates and secondary school students; only 11% of studies focused on primary school.

Elementary school mathematics studies using AR to teach geometry show promising data that can be applied to the postsecondary setting. AR platforms tend to “freeze” and fail at times; this plagues all settings. Most teachers, like college faculty, are unfamiliar with how to use AR or generate their own AR content, but after training there is a higher level of adoption. In one study it was shown that faculty changes in beliefs about teaching and learning occur as they receive exposure to AR methodology (Onal et al., 2017). Technology acceptance is a core element of teacher identity and belief systems which influence teaching. This indicates that faculty who are comfortable with AR may come to view their professional role in a different way. This might parallel one contention in both problem-based and SEG-based learning that the role of the faculty member transforms from information provider to experience facilitator or, in the case of evolving simulation game narratives, author of an adventure.

5. COMPLEXITY IN SERIOUS GAME DESIGN

The use of SEGS in education relies upon several strong effect size considerations which we will now explore with a view to equipping the next generation of educators on how to use this in training:

1. Building games with high impact on motivation, attention, and learning outcomes
2. Player and gameplay characteristics
3. Motivation to play games
4. Gender and cultural determinants of player engagement
5. Simulation gaming and complexity

High-complexity games use gamification to create compelling experiences using multimedia and intricate rule sets designed to move the learner toward either high-fidelity immersiveness or enhanced agency. Agency refers to the feeling the player has that they are in control of what happens in the game. Agency is not fixed; it can vary with rule sets and skill. A beginning chess player feels low agency when matched against an experienced player. They have little control of the game. However, chess has an impressively rich rule set which lends itself to high agency with training and practice. Snakes and Ladders provides random events for the player, and thus agency is fixed and very low. Role-playing games such as Everquest have moderate agency; players can roam freely through a fantastical world, but if they get killed by a monster, their in-game progress is halted, and all of the treasures and weapons they have obtained are taken away with a death. By contrast, in World of Warcraft a player death just means they will have to run back to their corpse, reanimate, and continue. So player agency is dependent upon both rule set and skill.

In one study a total of 204 college students were divided into three groups: (a) one with an interactive digital game with rich multimedia elements such as animations, (b) one with a digital game without media upgrades, and (c) a paper and pencil game with the same desired outcomes (Chen, 2017). Learning outcomes were greater in the more complex game system in condition (a). There were no differences in attention or motivation across groups. This showed that analog games (paper and pencil, card games, and board games) scored favorably when compared with digitally enhanced games. However, short-term knowledge acquisition was higher in both enhanced and simple digital games, with the high-complexity game showing the strongest outcome.

Player and gameplay characteristics are therefore strong elements of design, but their impact varies across both game types and player types. This denotes that adaptivity is a central element of design; games need to meet players where they are most likely to benefit. Adaptive learning refers to instruction which responds to a learner's prior knowledge, goals, and needs (Vandewaetere and Clarebout, 2013). High granularity of design follows, during which analytics can be generated to show player success, areas of weakness, the need to provide “burst” instruction, and other measures to improve learning based on data generated during gameplay. Indeed, this level of microdata capture rarely exists in nongame instruction, and it may be one of the strongest reasons to use game systems for training. Player characteristics can be reduced to several broad categories: prior knowledge, learning style/cognitive style, gaming skills, personality, goal setting, and motivation (Vandewaetere, 2013). Motivation is a strong effect size factor and can be divided into the achievement component, social component, and immersion component.

Self-determination theory describes motivation as a combination of interest and enjoyment (intrinsic motivation), perceived competence, effort, value, felt pressure, and perceived choice (Deci and Ryan, 2000). These features can be used to build SEGS and guide their design. We can conceive of any game learning experience as a game state which is driven by prior player characteristics and “runtime” player behaviors related to what they actually do in the game. The design of SEGS, then, is a process of incorporating all of these elements into the final build rather than simply producing a game and having players adapt to the rule set. The rule set, in fact, should be adapted to the player using principles of user experience as a guide.

Motivations for online game players are of great help in designing SEGS. Highly successful multiplayer online games are played, on average, for 22 hours a week for players in their mid-20s (Yee, 2006). Ranking studies which broke down online play motivation fell under the three categories listed above. Achievement was related to in-game progress, accumulation of goods and status, along with optimizing play style for success and the ability to provoke or dominate other players in competition. Social motivation includes the opportunity to communicate in the game, making friends, giving support to other players, and team collaboration with group achievements. Immersive factors include exploration, discovery, storyline absorption, fantasy, appearance of objects in the game, and the ability to relax and avoid real-life problems. This last feature may seem incompatible with educational games, but in our experience it is central. Even when designing games to teach management of the frail elderly or emergency medical procedures, the element of escape and relaxation remain key. In fact, the learning in a good simulation games depends upon the player feeling motivated and rewarded for attempting a problem more than once in an atmosphere of psychological safety.

Gender and cultural differences in SEGS have been explored and there are distinct differences in acceptance and gameplay style that vary across both of these considerations (Lukosch et al., 2017). In some studies, planning ability was higher in female gamers compared to males in the same game conditions. Male players have shown stronger performance in visuo-spatial orientation, while female players have shown higher levels of anxiety in navigation through virtual environments (Lawton, 1996). As is the typical concern in such studies, personality factors are more determinant than gender factors in these outcomes. Of interest also are the data on how many females play games and what types of activities they prefer in games compared to male players. Statistics compiled from 2006 to 2018 show that females represented 38% of gamers in 2006 and now represent 45% (Statista, 2018). Worldwide video game sales amounted to $75 billion in the US in 2015 and will grow to $90 billion by 2020. Given that there are over 1.8 billion gamers in the world, this amounts to 25% of the global population. Video gamers between the ages of 18 and 35 number 27%; 26% of surveyed Americans are over 50 years of age. Action and shooting games account for the bulk of sales across both genders. The recent launch of Fortnite (EpicGames, 2018) and its adoption by female players with its display of mostly female avatars is of interest. This game involves killing other players off in a death match with various game conditions forcing players to encounter each other. The traditional notion that men like to kill things in video games and women prefer to build things or engage in fantasy does not appear to be accurate; it is probably reflective of gender stereotypes more than derived from actual industry data. Industry data changes all the time as well, but as more “millennial” females play games, this might simply reflect a change in the entertainment choices that are made as a result of exposure to the medium. Cultural differences in studies appear to be less conclusive; we cannot realistically compare Japanese to American players, or those from varying socioeconomic backgrounds at this point. It may be that online games show homogeneity, in that game rules demand specific in-game behaviors which have no specific cultural basis.

Simulation gaming is likely to be used to train, and so some discussion on how to build games with complexity is helpful. Simulations can be described in terms of complexity based on Klabbers typology (Klabbers, 2006). Algorithmic complexity refers to the amount of information necessary to describe a system, organizational complexity refers to the ability for organizations to self-organize, and organized complexity relates to the self-awareness of agents as they negotiate in groups of interest. The role of the simulationist is complexity reduction, to reveal aspects of reality to a learner in ways that are helpful for instruction. In short, simulation games consist of scenarios with content, roles, and play dynamics, and these must address fidelity as well. Is it enough to show a picture of a tumor to medical students to understand how it fits into a case study, or would a text description suffice, or would generating this in virtual reality be better? These are judgements at the present time, as very little is known about the role of fidelity in VR simulations. Is a picture better than a VR experience, and if so, why?

6. SUMMARY

In this review we attempt to demonstrate the elements which underlie SEG use and design and how we might begin to utilize this information in training. Some themes which run throughout this analysis can be summarized along with some recommendations for next steps forward.

• Serious educational games must be designed carefully to provide diverse learners with an appropriate level of challenge, exploration, social connection, and achievement.
• The existing literature shows that game-based learning affects motivation more than learning outcomes but that new discoveries in the field of neuroscience, such as functional magnetic resonance imaging (fMRI) studies during gameplay, imply that games can focus attention.
• Games for learning do not need to use enhanced technology but may benefit from it in cases where the technology, such as online access or VR, demonstrably improve learning outcomes.
• Games are played around the world by almost 25% of the global population, with 45% of these players being female. To ignore the obvious desire to play games in educational design is to disregard these compelling data.

How then might we take the next steps forward in education? The current landscape of SEG development in Canadian higher education may be instructive. There have been dozens of proposals, many of which have been successful to Ontario's ECampus program for funded game systems in both university and college sectors. Ryerson University's G. Raymond Chang School for Continuing Education has embraced serious games as a focus in their Digital Education Strategy team with several outputs, including role-playing and choose your own adventure video game systems. George Brown College, in collaboration with Baycrest Health Sciences and Centennial College, has produced a gamified internship program using an open world build for the training of gerontology workers. Baycrest Health Sciences, the leading research and treatment facility for dementia, has run internship programs using role-playing games and is now adapting game systems for teaching interprofessional development. Professor Deborah Fels at the Ted Rogers School of Management at Ryerson University runs a full semester course as a game in the field of multimedia. Internal studies by Professor Rob Bajko and Professor Jaigris Hodson (Ryerson and Royal Roads) have piloted full course game conversions in the field of professional communications in a course they called Social Media Celebrity. Professors Lori Schindel-Martin and Karen LeGrow have piloted family health nursing games based on reading peer-reviewed journals as one component within a longer 16-week course.

There are some myths and misconceptions we should address in closing. The first is that game designers should build SEGS. This is problematic chiefly due to the fact that game designers, as a profession, do not have expertise in teaching and learning. Fundamentally, SEGS are teaching and learning tools and they borrow technology and approaches from the entertainment industry. It seems more fitting, and is commonly the practice now, that instructional designers and faculty receive some kind of basic training in approachable entry points, such as simulation gaming or using games to review knowledge. Many of these games can be built using standard LMS features such as adaptive release or badging. Desire to Learn (D2L) is releasing a gamification module in the fall of 2018 which will certainly make game use in teaching more palatable for an instructor.

The second myth is that building serious games is a trivial process. The problem here, and it is significant, is that few faculty members have the experience, time, or in some cases the creative skills to build games for teaching. To expect that a professor of biochemistry will somehow take the time to build a good game to teach the subject is magical thinking. However, if instructional designers in colleges are trained in SEG design and basic theory, then the faculty can approach them for assistance in implementation.

The third myth is that SEGS are simply another pedagogical tool such as the flipped classroom and they have relevance as an option for faculty. Our data presented here shows compelling reasons to use SEGS instead of traditional teaching, defined as PowerPoint presentations/labs. The majority of teaching in colleges and universities is still instructor-driven lecture with some other activities such as case solving or discussion. Active learning is at the core of SEGS, and there is undeniable evidence that active learning is much more effective than passive learning in STEM programs in particular (Freeman et al., 2014). In an analysis of 225 studies reporting failure rates in STEM subjects, it was shown that active learning raises grades by half a letter and that passive learning has a 55% greater failure rate than those observed under active learning. SEGS are pure active learning and they represent a marked shift from instructor-led to game-led instruction.

What is the instructor's role in a SEG-based classroom in face-to-face instruction vs. online training? In early studies at Ryerson and Baycrest we let the students play the game and that took up the class time. Part of the game consisted of problem-based learning, where students had to identify a simulation problem and research its possible solutions, then bring those back to the group over a one-week period. Using WordPress blogs to drive the internship program game at Baycrest Health Sciences, students had to access cases on a discussion board and then bring their solutions to the board. In both cases, faculty members did not lecture but changed their own behavior to fit the game needs.

In face-to-face instruction the faculty member can now float from group to group as they take on simulation challenges, providing background information and acting as a consultant. In fact, in the George Brown College/Baycest gerontology certificate game, players can purchase consults or engage in mentorships with faculty as part of the game. In online games, the faculty member can review submissions to forums to determine if those are adequate and, on the basis of student submissions, provide coaching.

There is no reason to completely eliminate lectures in SEG-based training. In fact, you can award experience points for attending lectures and doing well on quizzes. Faculty members can use exams to test knowledge embedded in the game or from lectures, using participation marks to evaluate group and game contributions. Using SEGS as the basis for learning does not preclude the use of other methods; they simply remain as the central hub in the course which other activities feed into.

The one thing we would like to close with is again stating that simply using engagement games in a course is not the ideal use of SEGS. Well-designed SEGS should be able to replace conventional lecture-based training, and this poses a professional redefining for faculty. For many years faculty members have seen themselves as knowledge providers. SEG-based instruction converts this role to that of “experience provider.” It can be difficult for faculty to abandon the lectern and also the ability to control what happens during precious class time or during online design. This shift from leader to game-master is going to be gradual, and as more millennial students and faculty enter the ranks, the conversion to player/student-centered learning will be easier to envision. At the heart of SEG implementation is John Dewey's idea of exploration, and in this field we may finally realize the goal he set before us of creating a master of learning rather than a simple graduate. Terminal competencies are harder to define now, as fields emerge and change. Who would have thought that one could one day have a career as a serious game designer? Or a digital marketer, or a behavioral neuroscientist? Ideas, attitudes, and values shift over time, and the move toward play-informed learning is probably the most significant paradigm shift we will witness in education since problem-based learning emerged in 1966 and net 1.0 emerged in the late 90s. The journey is just starting, and there is enough data now to support this form of innovation across the education sector.

REFERENCES

Almeida, F. (2017), Experience with Entrepreneurship Learning Using Serious Games, Cypriot J. Educ. Stud., vol. 12, no. 2, pp. 69–80.

Anetta, L.A., Minogue, J., Holmes, S.Y., and Cheng, M.-T. (2009), Investigating the Impact of Video Games on High School Student's Engagement and Learning about Genetics, Comput. Educ., vol. 53, no. 1, pp. 74–85.

Aschbacher, P.R., Li, E., and Roth, E.J. (2010), Is Science Me? High School Students' Identities, Participation and Aspirations in Science, Engineering, and Medicine, J. Res. Sci., vol. 47, pp. 564–582.

Barr, M. (2017), Video Games Can Develop Graduate Skills in Higher Education Students: A Randomised Trial, Comput. Educ., vol. Oct., pp. 86–97.

Belotti, F., Kapralos, B., Lee, K., and Ger, P.M. (2013), Assessment in and of Serious Games, Adv. Human-Comput. Interaction, vol. 2013, Article 1.

Bevilacqua, F., Backlund, P., and Engstrom, H. (2018), Changes in Heart Rate and Facial Actions during a Gaming Session with Provoked Boredom and Stress, Entertain. Comput., vol. 24, pp. 10–20.

Bourazeri, A., Arnab, S., Heidmann, O., Coelho, A., and Morini, L. (2017), Taxonomy of a Gamified Lesson Path for STEM Education: The Beaconing Approach, Beaconing Project, Grant Agreement No. 687676.

Boyle, E.A., Hainey, T., Connolly, T.M., Gray, G., Earp, J., Ott, M., et al. (2016), An Update to the Systematic Literature Review of Empirical Evidence of the Impacts and Outcomes of Computer Games and Serious Games, Comput. Educ., vol. 94, pp. 178–192.

Chen, S.-W., Yang, C.-H., Huang, K.-S., and Fu, S.-L. (2017), Digital Games for Learning Energy Conservation: A Study of Impacts on Motivation, Attention and Learning Outcomes, Innov. Educ. Teaching Int., vol. 56 pp. 66–76.

Cheryan, S., Siy, J.O., Vichayapai, M., Drury, B.J., and Kim, S. (2011), Do Female and Male Role Models Who Embody STEM Stereotypes Hinder Women's Anticipated Success in STEM? Soc. Psychol. Personal. Sci., vol. 2, pp. 656–664.

Ciftci, S. (2018), Trends of Serious Game Research from 2007 to 2017: A Bibliometric Analysis, J. Educ. Train. Stud., vol. 6, no. 2, pp. 18–27.

Csikszentmihalyi, M. and Nakamura, J. (2014), Flow and the Foundations of Positive Psychology, Berlin: Springer, pp. 239–263.

Deci, E.L. and Ryan, R.M. (2000), The "What" and "Why" of Goal Pursuits: Human Needs and the Self-Determination of Behavior, Psychol. Inquiry, vol. 4, pp. 227–268.

Dellos, R. (2015), Kahoot! A Digital Game Resource for Learning, Int. J. Instruct. Tech. Distance Learning, vol. 12, no. 4, pp. 53–56.

Ducheneaut, N. and Moore, R.J. (2005), More than Just XP: Learning Social Skills in Massively Multiplayer Online Games, Interactive Tech. Smart Educ., vol. 2, pp. 89–100.

Duplaa, E., Kaufman, D., Sauve, L., Renaud, L., and Ireland, A. (2017), Can Mobile Digital Games Benefit Older Adults' Health? Mobile E-Health, pp. 115–146.

EpicGames (2018), Fortnite.

Fengfeng, K. (2008), A Case Study of Computer Gaming for Math: Engaged Learning from Gameplay? Comput. Educ., vol. 51, no. 4, pp. 1609–1620.

Fjaellingsdal, K.S. and Klockner, C. (2017), ENED-GEM: A Conceptual Framework Model for Psychological Enjoyment Factors and Learning Mechanisms in Educational Games about the Environmental, Front. Psychol., vol. 6, Article 1085.

Freeman, S., Eddy, S.L., McDonough, M., Smith, M.K., Okoroafor, N., Jordt, H., and Wenderoth, M.P. (2014), Active Learning Increases Student Performance in Science, Engineering and Mathematics, Proc. Natl. Acad. Sci., vol. 111, no. 23, pp. 8410–8415.

Galarneau, L.L. (2005), Spontaneous Communities of Learning: Learning Ecosystems in Massively Multiplayer Online Gaming Environments, SSRN, June.

Green, C., Pouget, A., and Bavelier, D., (2010), Improved Probabilistic Inference as a General Learning Mechanism with Action Video Games, Current Biol., vol. 20, no. 17, pp. 1573–1579.

Grossman, J.M. and Porche, M.V. (2014), Perceived Gender and Racial/Ethnic Barriers to STEM Success, Urban Educ., vol. 49, no. 6, pp. 698–727.

Hamari, J. (2017), Do Badges Increase User Activity? A Field Experiment on the Effects of Gamification, Comput. Human Behav., vol. 71, pp. 469–478.

Han, X., Wu, X., Wang, Y., Sun, Y., Ding, W., Cao, M., et al. (2018), Alterations of Resting-State Static and Dynamic Functional Connectivity of the Dorsolateral Prefrontal Cortex in Subjects with Internet Gaming Disorder, Front. Human Neurosci., vol. 12, p. 41.

Hauge, J.B., Barenbrock, A., and Thoben, K.-D. (2017), Avatars as Motivational Factors in Simulation Games, Faculty of Information Science at the University of Bremen, Bremen, Germany.

Howard-Jones, P. (2017), Gamification of Learning Deactivates the Default Mode Network, Front. Psychol., vol. 6, p. e9251.

Ihnat, M. (2018), Personal Communication – Director for E-Learning, Toronto, Ontario, Canada: Humber Center for Teaching and Learning.

Jackson, M. (2016), Gamification in Education: A Literature Review, Center for Faculty Excellence, West Point, NY: US Military Academy.

Jensen, C.G. (2017), Collaboration and Dialogue in Virtual Reality, J. Problem Based Learning, vol. 5, no. 1, pp. 85–110.

Kaufman, D. and Sauve, L. (2010), Educational Gameplay and Simulation Environments: Case Studies and Lessons Learned, Hershey, PA: Information Science Reference.

Klabbers, J.H. (2006), The Magic Circle: Principles of Gaming and Simulation, vol. 1, Rotterdam: Sense Publishers-Springer.

Krajcik, J. and Delen, I. (2017), How to Support Learners in Developing Usable and Lasting Knowledge of STEM, I. J. Ed. Math., Sci. Technol., vol. 5, no. 1, pp. 21–28.

Lamb, R., Anetta, L., Firestone, J., and Etopio, E. (2018), A Meta-Analysis with Examination of Moderators of Student Cognition, Affect and Learning Outcomes while Using Serious Educational Games, Serious Games and Simulations, Comput. Human Behav., vol. 80, pp. 158–167.

Landers, R.N. and Armstrong, M.B. (2017), Enhancing Instructional Outcomes with Gamification: An Empirical Test of the Technology-Enhanced Training Effectiveness Model, Comput. Human Behav., vol. 71, pp. 499–507.

Lawton, C.A. (1996), Gender Differences in Way-Finding Strategies: Relationship to Spatial Ability and Spatial Anxiety, Sex Roles, vol. 30, no. 11-12, pp. 765–779.

Lizier, J.T., Harre, M.S., Mitchell, M., DeDeo, S., Finn, C., Lindgren, K., et al. (2018), An Interview Based Study of Pioneering Experiences in Teaching and Learning Complex Systems in Higher Education, Phys. Educ., Cornell University Library, vol. 16.

Lofgren, E.T. and Fefferman, N. (2007), The Untapped Potential of Virtual Game Worlds to Shed Light on Real World Epidemics, Lancet, vol. 7, no. 9, pp. 625–629.

Lukosch, H., Kurapati, S., Groen, D., and Verbraek, A. (2017), Gender and Cultural Differences in Game-Based Learning Experiences, Electron. J. e-Learning, vol. 15, no. 4, pp. 310–319.

MacDonald, M. (2015), Playing Games at School, University Affairs. Retrieved from https://www.universityaffairs.ca/features/feature-article/playing-games-school/.

Machajewski, S. (2017), Gamification Strategies in a Hybrid Exemplary College Course, Int. J. Educ. Technol., vol. 4, no. 3, pp. 1–16.

Macht, S.A. (2016), Authentic Alignment – A New Framework of Entrepreneurship Education, Educ. Train., vol. 58, no. 9, pp. 926–944.

Malaquias, R.F., Malaquias, F.F., Borges Jr., D.M., and Zambra, P. (2018), The Use of a Serious Game and Academic Performance of Undergraduate Accounting Students: An Empirical Analysis, Turkish Online J. Distance Educ., vol. 19, no. 2, pp. 117–127.

Mullin, J.K. and Sabherwal, R. (2018), Beyond Enjoyment: A Cognitive-Emotional Perspective on Gamification, Proc. of the 51st Hawaii Intl. Conf. System Sciences, pp. 1237–1245.

Nardi, B. and Harris, J. (2006), Strangers and Friends: Collaborative Play in World of Warcraft, Proc. of the 2006 20th Anniversary ACM Conf. on Computer Supported Cooperative Work, Banff, Alberta, Canada, pp. 149–158.

Onal, N., Ibili, E., and Caliskan, E. (2017), Does Teaching Geometry with Augmented Reality Affect the Technology Acceptance of Elementary School Mathematics Teacher Candidates? J. Educ. Pract., vol. 8, no. 19, pp. 151–163.

Poerio, G.L., Sormaz, M., Hao-Ting, W., Margulies, D., Jefferies, E., and Smallwood, J. (2017), The Role of the Default Mode Network in Component Processes Underlying the Wandering Mind, Soc. Cognitive Affective Neurosci., vol. 12, no. 7, pp. 1047–1062.

Sajjadi, P., Vileghe, J., and De Troyer, O. (2017), Exploring the Relation between the Theory of Multiple Intelligences and Games for the Purpose of Player-Centered Game Design, Electron. J. e-Learning, vol. 15, pp. 320–334.

Sanchez-Mena, A. and Marti-Parreno, J. (2017), Drivers and Barriers to Adopting Gamification: Teacher's Perspectives, Electron. J. e-Learning, vol. 15, no. 5, pp. 434–443.

Schneider, M. and Preckel, F. (2017), Variables Associated with Achievement in Higher Education: A Systematic Review of Meta-Analysis, Psychol. Bull., vol. 143, no. 6, pp. 565–600.

Seitz, A.R. (2018), A New Framework of Design and Continuous Evaluation to Improve Brain Training, J. Cogn. Enhanc., vol. 2, pp. 78–87.

Serrano-Laguna, A., Martinez-Ortiz, I., Haag, J., Regan, D., Johnson, A., and Fernandez-Manjon, B. (2017), Applying Standards to Systematize Learning Analytics in Serious Games, Comput. Standards Interfaces, vol. 50, pp. 116–123.

Sirakaya, M. and Alsancak Sirakaya, D. (2018), Trends in Educational Augmented Reality Studies: A Systematic Review, Malaysian Online J. Educ. Technol., vol. 6, no. 2, pp. 60–74.

Snow, B. (2016), The Potential for Game-Based Learning to Improve Outcomes for Nontraditional Students, Muzzy Lane Software Report, funding from Bill and Melinda Gates Foundation.

Spires, H., Rowe, J.P., Mott, B.W., and Lester, J.C. (2011), Problem Solving and Game-Based Learning: Effects of Middle Grade Students' Hypothesis Testing Strategies on Learning Outcomes, J. Educ. Comput. Res., vol. 44, no. 4, pp. 453–472.

Statista (2018), Distribution of Computer and Video Gamers in the United States from 2006 to 2018, by Gender, The Statistics Portal, in Video Games and Gaming: Media Advertising.

Turner, P.E., Johnston, E., Kebritchi, M., Evans, S., and Heflich, D.A. (2018), Influence of Online Computer Games on the Academic Achievement of Nontraditional Undergraduate Students, Cogent Ed., In press.

Van Roy, R. and Zaman, B. (2018), Unravelling the Ambivalent Motivational Power of Gamification: A Basic Psychological Needs Perspective, Int. J. Human-Comput. Stud., In press.

Vandewaetere, M. (2013), Adaptivity in Educational Games: Including Player and Gameplay Characteristics, Int. J. Higher Educ., vol. 2, no. 2, pp. 106–114.

Vandewaetere, M. and Clarebout, G. (2013), Advanced Technologies for Personalized Learning, Instruction and Performance, in Handbook of Research on Educational Communications and Technology, J.M. Spector, M.D. Merrill, J. Elen, and M.J. Bishop, Eds., New York: Springer Academic.

Wassila, D. and Champagnat, R. (2017), Towards Architecture for Pedagogical and Game Scenarios Adaptation in Serious Games, IADIS Digital Library, pp. 63–70.

Yang, C.Y.-T. (2012), Building Virtual Cities, Inspiring Intelligent Citizens: Digital Games for Developing Student's Problem Solving and Learning Motivation, Comput. Educ., vol. 59, no. 2, pp. 365–377.

Yee, N. (2006), Motivations for Play in Online Games, Rapid Communication, Cyber Psychol. Behavior, vol. 9, no. 6, pp. 772–774.