EVALUATING TECHNOLOGIES FOR COMMUNICATING MATHEMATICAL AND SCIENTIFIC REASONING IN FULLY ONLINE ENGINEERING COURSES: A TECHNOLOGY CHOICE FRAMEWORK

Richard McInnes, * Joshua Cramp, & Claire Aitchison

Richard McInnes	Joshua Cramp	Claire Aitchison

Teaching Innovation Unit, University of South Australia, Adelaide SA 5000, Australia

*Address all correspondence to: Richard McInnes, Teaching Innovation Unit, The University of South Australia, Adelaide SA 5000, Australia, E-mail: richard.mcinnes@unisa.edu.au

1. INTRODUCTION

Designing effective online science, technology, engineering, and mathematics (STEM) degrees brings with it unique challenges since academics and students are expected to communicate complex mathematical and scientific formulas, graphs, and diagrams in learning management systems (LMSs) that do not easily accommodate these functions (Loch & McDonald, 2007). Communicating mathematical and scientific reasoning online has long been a problem, which has been recently exacerbated by the explosion in demand for virtual learning associated with the COVID-19 global pandemic, as well as student and institutional demands for flexibility and equity and the urgent need for secure, accessible, and functional online technologies. Furthermore, the move toward remotely invigilated online exams adds additional complexities to disciplines where students are required to digitally show fully worked solutions to problems as well as draw graphs and diagrams. Consequently, institutions are urgently seeking affordable technologies appropriate for supporting student learning and assessments that require nonlinear, non-text–based computational and graphical representation online.

Implementing a technological solution across a whole degree requires the navigation of competing institutional, technological, and teaching and learning intricacies. Technologies must be compatible with STEM-specific pedagogies (Winberg et al., 2019), while not detracting from the robustness of existing assessment practices (Winger et al., 2019). Moreover, effective technological solutions must walk a tightrope, balancing successful online learning design with end-user perspectives—that is, those of the student and teacher—on perceived usefulness and ease of use (Davis, 1989). Teaching STEM online requires innovative thinking that harmonizes these often competing expectations through the incorporation of suitable educational technologies. This paper demonstrates one process for achieving this by examining learning design considerations and institutional, disciplinary, student, and teacher requirements. Using a case study methodology (Yin, 2017), we demonstrate a process of decision making that enables the development of high-quality courses, positive student learning outcomes, and staff development opportunities.

2. LITERATURE

Teaching and learning in STEM disciplines is characterized by unique signature pedagogies that have been developed to maximize learning outcomes while imparting the professional values, attitudes, and dispositions required to be successful within the field (Shulman, 2005). These pedagogies emphasize cognitive and procedural fluency (Lafreniere, 2016) in disciplinary knowledge as well as professional habits of mind, such as system thinking, problem solving, design (Lucas & Hanson, 2016), pattern finding, visualizing, conjecturing (Cuoco et al., 1996), scepticism, objectivity, and curiosity (Çalik & Coll, 2012). Transitioning STEM teaching and learning from the classroom to online courses challenges traditional pedagogies, but doing so enables academics to manipulate their practices in order to achieve objectives not possible in a face-to-face setting (Bourne et al., 2005). However, teachers can only facilitate positive learning outcomes if students are able to engage effectively with the content and demonstrate their learning through authentic assessment within the online context.

Online STEM course design faces complexity intrinsic to the text-based nature of current LMSs, which results in less than optimal learning environments for mathematics-based disciplines (Loch & McDonald, 2007). Students and teachers require an interface that enables them to communicate STEM reasoning through creating and manipulating complex mathematical and scientific formulas, graphs, and diagrams online, preferably without the constraints of using a mouse and keyboard to express symbolic subjects (Urban, 2017). In addition, the predilection for complex online exams in mathematics-based disciplines (Alksnis et al., 2020), academic integrity concerns (Trenholm, 2007), and professional accreditation requirements brings further challenges since technological solutions must also be compatible with remote invigilation systems, which already present intrinsic pedagogical challenges (Cramp et al., 2019). For students completing formative and summative assessments, access to affordable and user-friendly technological solutions can be limited, with new technologies often being challenging to learn and slow to use (Akelbek & Akelbek, 2009). Linear syntax, such as LaTeX, requires significant effort and time to learn, a challenge with already full curricula (Loch et al., 2015; Winger et al., 2019). Comparatively, graphical equation editors, which are more suitable for beginners, can be slow to use and cause students to compress calculation steps (Loch et al., 2015; Winger et al., 2019). Compressing calculation can disrupt feedback cycles by eliminating the ability for teachers to view, mark, and therefore give feedback on students' techniques, strategy, accuracy, and efficiency in the application of procedural concepts. Furthermore, studies have found that when presented with equation editors, students prefer using handwritten and scanned responses (Misfeldt & Sanne, 2012). A common consequence for formative and summative assessments is the use of multiple-choice–style questions (Draskovic et al., 2016; Soares & Lopes, 2018). However, while multiple-choice questions can form valid and reliable assessments (Parkes & Zimmaro, 2016), they cannot be considered a valid replacement for fully worked solutions, diagrams, and graphing in summative assessments. Additionally, multiple-choice–style questions' focus on conceptual understanding means that they present a pedagogical concern, whereby teaching staff are unable to close the feedback loop on students' procedural fluency. Furthermore, it can be challenging to display non-text–based teaching and learning materials in the LMS in ways that maximize accessibility for students (Armano et al., 2018).

Digital inking technologies have recently shown promise in STEM teaching and learning as a way to easily and accurately represent mathematics and science online (Stephens, 2018), and the recent pivot to emergency remote teaching has triggered increased interest in their adoption (Dekkers & Hayes, 2020). In this paper, we refer to digital inking as the use of digital stylus-based technologies (for example, graphics or pen tablets, tablet PC and Mac devices, and smartpens) and computer software to produce digitized handwriting (Lafreniere, 2016). Prior contributions have highlighted increases in efficiency for instructors employing digital inking in the marking and production of multimedia (May, 2018), and students' preference for watching instructional videos utilizing digital inking (Maclaren, 2014). Furthermore, digital inking has advantages for online synchronous learning that enable instructors to facilitate interactions in real time (Iwundu, 2018), thus promoting collaborative problem solving. Digital inking technologies also exhibit high mathematical and cognitive fidelity—that is, the ability of the tool to accurately represent mathematic and scientific symbolic objects, expose students to procedural thinking, and as such allow teachers to evaluate and therefore give feedback on students' mathematical and scientific reasoning (Lafreniere, 2016).

Implementing a digital technology solution into course design and development is neither simple nor straightforward, with many competing institutional, technological, and teaching and learning factors. Employing an adaptable model such as analysis, design, development, implementation, and evaluation (ADDIE) (Molenda, 2003) can support the methodical and cyclical adoption of new educational technologies. Such systematic approaches promote consideration of end-user perspectives, including perceived usefulness and ease of use (Davis, 1989) and socio-economic barriers (Van Dijk, 2017). Likewise, the impacts of the technology on teaching, learning, and course design should be addressed to ensure that learning remains outcomes focused (Nuland et al., 2020) and equitable (Basham & Marino, 2013), and that the adoption of the technology for assessment does not affect valid, reliable assessment design (Brady et al., 2019).

3. METHOD

Cognizant of these complexities, we sought a viable solution for the efficient and accurate display and communication of non-text–based material in the LMS. In 2020, an Australian metropolitan university commenced the development of a new, fully online associate degree in engineering. Each online course (10-week unit of study) was developed in collaboration between academic subject matter experts and a team of academic developers and online educational designers over a 12-week development cycle. In total, 16 fully online courses were created over an 18-month period using an established quick course development model (McInnes et al., 2020).

A case study approach (Yin, 2017) was deemed the most appropriate way to investigate and articulate our method of technology appraisal and adoption for the particular teaching and learning challenges previously outlined. As indicated by Flyvbjerg (2006), in-depth case study research has particular resonance in scientific disciplines for the systematic development of exemplars otherwise rendered invisible through unrecorded practices. The aim of this case study was to interrogate practice and inform decisions in order to create a framework for future use. Guided by the ADDIE model, the research team analyzed our context and processes in light of the scholarly literature and were able to harness their cumulative experiences and knowledge as reflective practitioners systematically and objectively building “the collective process of knowledge accumulation” (Flyvbjerg, 2006, p. 227). This case study illuminates key aspects of teaching and learning scholarship by making pedagogical decisions transparent, replicable, and subject to critical scholarly debate (Boyer, 1990).

4. TECHNOLOGY TESTING AND SELECTION

To identify suitable technologies that could be used across a fully online engineering degree the team of academic developers and online educational designers, in consultation with academic subject matter experts, undertook the multi-step ADDIE process (Molenda, 2003). End-user analysis and an environmental scan were completed to establish the needs of the students and teaching staff and the scope of obtainable technology options, including those in use at the institution and documented in extant literature.

4.1 Student Technologies

For student use, the end-user analysis parameters required the technologies to be affordable, available domestically and internationally, intuitive to set up and use, useful across the student's whole degree, accessible, appropriate for formative and summative assessments, and compatible with university systems, including the LMS, virtual classroom, and online exam invigilation software. Several technology options were eliminated immediately, such as touch-screen and hybrid PCs, tablet devices, document cameras, and scanners for reasons including expense, non-ubiquitous availability, complexity, and inability to work with university systems—predominantly the exam remote invigilation system. After this preliminary round of investigation, a refined list of technologies was developed. Two styles of graphical equation editors were tested: WIRIS MathType/ChemType and Microsoft Equation Editor. Several styles of market-leading pen tablets were tested, including the Wacom Intuos and Wacom One software programs. The pen tablets were predominantly tested with Microsoft Inking across the Office 365 Suite (institutionally provided and supported software); however, a hybrid solution was also tried by using a pen tablet with the handwriting functionality in WIRIS. Finally, the LiveScribe Echo smartpen and Wacom Bamboo pen tablet were tested to digitize handwriting.

Utilizing the end-user analysis parameters, a set of technology testing criteria (Table 1) was created that the academic developers, online educational designers, and academic subject matter experts could use to evaluate each technology. Testing was conducted using sample assessment papers and solutions from previous iterations of the courses within the engineering degree in consultation with course teaching staff.

The results of the testing showed that the WIRIS and Microsoft Equation Editor were very similar in relation to the criteria—the key differences being the cost to the institution of using WIRIS (an annual license fee and the ongoing maintenance associated with integration into the LMS) and that the WIRIS editor could be directly integrated into the Moodle editor. Testers were impressed by both equation editors' ability to provide clear, well-formatted equations. However, the complexity of their use in synchronous classrooms for real-time collaboration would be difficult. Additionally, the equation editors presented limits on assessment types, constraining the use of graphing and freehand diagrams. Critically, the equation editors were slow to gain proficiency, requiring significant practice to memorize symbol organization in the editor. Once proficient, they were reasonably fast; however, on average, they were over twice as slow as traditional handwriting. Furthermore, only one line at a time could be written in the editor, meaning slower times to enter multi-stage calculations and complexities in their formatting. These findings reaffirmed results previously obtained in the literature (Akelbek & Akelbek, 2009; Loch et al., 2015; Winger et al., 2019).

The handwriting conversion functionality in WIRIS was more intuitive than that in the equation editor—being more similar to handwriting—thus, making this technique easier to learn such that proficiency can be quickly developed. However, the testers found that despite impressive character recognition, the system proved frustrating when characters and symbols were not recognized and noted that under the high-stakes pressure of an exam it could be easy to miss where handwriting was not rendered correctly—an important consideration since technologies used in online exams should not impose additional cognitive load (Cramp et al., 2019) or anxiety. As with the graphical equation editors, there were limitations on multi-line equation construction, echoing the findings from Winger et al. (2019).

The Smartpen technologies were quickly discounted due to their inability to be easily used in online exams (where they contravened hardware and physical paper restrictions) and virtual classrooms (where synchronous use was not possible). Despite their similarity to handwriting and intuitiveness, the high initial and ongoing cost and the complexity of their setup meant that they were considered unsuitable.

Comparatively, the pen tablets were operationally similar, being relatively intuitive, similar to handwriting (therefore, allowing solutions to questions to be quickly written), and easy to set up with both PCs and Macs. In addition, the immediacy of their input enabled real-time participation in virtual classrooms. The disadvantages included the cost to students (approximately A$80–100) and the potential for low readability while students gained proficiency in writing with a digital stylus. Furthermore, some technical hurdles were encountered when integrating the software and uploading documents during online exams, which were identified and solved (Cramp et al., 2019). Concerns were raised about the amount of additional time that could be required in exams and the clarity of student work. Measures were taken to support students through orientation, scaffolded practice, and application via course design to mitigate these concerns. The testing noted that while one of the pen tablets has increased sensitivity and additional customizable buttons, which had the potential to improve clarity and speed up writing notation, this was not significant enough to preference one product.

Based on the results of this extensive testing, it was determined that students would be required to obtain a wired Wacom pen tablet to be used with Microsoft Inking in the Microsoft Suite to digitally handwrite equations, diagrams, and graphs. Figure 1 illustrates a typical student technology setup and the resultant digital inking display. Wired technology was preferred due to academic integrity concerns associated with the use of wireless technology in online invigilated exams and the risk of the device running out of battery power part way through an assessment. These specific products, with minimum specifications and features, were recommended to students to mitigate concerns that variations in product functionality would influence or affect student performance (i.e., those that could afford better products would be unfairly advantaged). Support for mastering these technologies was scaffolded across the degree using videos and activities for familiarization such that students could gain proficiency before the pressure of a remotely invigilated online exam.

FIG. 1: Example of a typical student's Wacom pen tablet and laptop setup with a close-up view of the Microsoft Inking display in Microsoft Word

4.2 Teacher Technologies

For teachers, technologies are required to support the communication of mathematics and science online, including course design and development, the creation of multimedia, marking and feedback on student assessments, and the facilitation of synchronous classes. An ideal scenario is the adoption of a single technological solution that will address all of these applications. Technologies available in our case were glassboards, whiteboards, interactive whiteboards, equation editors, graphics tablets, tablet PCs, and document cameras. To determine the best fit of technologies for these applications, technology testing criteria (Table 2) were developed against which technologies could be evaluated.

Based on the testing criteria and existing literature (Dekkers & Hayes, 2020; Iwundu, 2018; Maclaren et al., 2017; May, 2018; Urban, 2017), it was determined that academics should use tablet PCs and digital inking, in this case utilizing existing on-campus Wacom Cintiq devices with Microsoft Inking software. Digital inking using tablet PCs was selected, in part because it was a single technology that could be used for creating multimedia, marking, giving feedback on student assessments, and facilitating synchronous classes, therefore minimizing technological complexities. Critically, for our purposes, digital inking enables high mathematical and cognitive fidelity while providing practical improvements to teaching without substantially altering signature STEM pedagogies (Maclaren et al., 2017). Throughout the investigation and testing, digital inking technologies demonstrated the potential to enhance the learning design for multimedia and synchronous classes. In synchronous classes, digital inking demonstrated the capability of increasing teacher presence and enabling active learning strategies (Iwundu, 2018). For multimedia, digital inking enabled presenters to conform to the many research-based principles that guide practice (Mayer, 2017; Seethaler et al., 2020). Figure 2 illustrates an example of a PowerPoint-based multimedia presentation utilizing digital inking. This example demonstrates the use of pre-setup questions, simultaneous narration of the reasoning underpinning content while digitally inking step by step the procedural fluency necessary for its creation, and color to highlight elements and provide cues to learners. Figure 2 shows the presenter view—when published for student viewing, extraneous menu bars and the surrounding frame are cropped out.

FIG. 2: Example of a slide from a multimedia presentation using digital inking

Another critical advantage of the tablet PC combined with digital inking technologies was the low barrier to usage, short learning curve, and the ability to work with existing software with which academics were already familiar, e.g., PowerPoint, Zoom, and Panopto (the institutionally supported video recording and hosting software). Seamless integration of existing technologies and rapid skill acquisition meant academics were able to work on the development of content and facilitation of classes with minimal training and without dedicated technical support. Furthermore, the ease of using these devices to directly annotate and give feedback on student submissions provided an intuitive and effective means for marking. Such ease of use contributes to the sustainability and scalability of these technologies across institutions and has likely more broadly aided the increasing uptake of these devices by academics (Dekkers & Hayes, 2020; Maclaren, 2014; May, 2018).

During the course development process COVID-19 restricted access to campus facilities, requiring a rapid solution to be deployed in order to ensure consistency of multimedia quality and allow access to technology familiar to course writers. After a condensed environmental scan and using lessons learned from the previous technology testing, the Wacom One 13 inch tablet was provided to staff working off campus. This technology was recommended due to the cost, portability, intuitiveness, continuity of experience from on-campus devices, and ease of use with the academics' existing hardware—including potential limitations of home devices. Figure 3 illustrates a typical academic technology setup.

FIG. 3: Example of a typical academic's Wacom pen tablet and laptop setup

For course design and development, another challenge was finding technologies to support the display of mathematical and scientific content in the text-based LMS, while maintaining accessibility standards and not impeding learning or assessment design. Based on the existing literature (Armano et al., 2018), it was evident that the most effective and accessible format for online content was using LaTeX and the MathJax Moodle filter, which was also implemented for the H5P software program. Therefore, the primary challenge was ensuring that the content was in the correct LaTeX format. In our case, due to the collaborative course development process, course content was provided by subject matter experts in a variety of forms, including LaTeX, typed equations using Microsoft Equation Editor, images, as well as handwritten and scanned files. Where content was not supplied using LaTeX, educational developers were forced to investigate technologies that would support the translation of content from its native format into LaTeX. After scoping several options, the development team identified the need for a range of tools, including MyScript Math in conjunction with inking to convert pen tablet digital inking, and MathPix to convert image files. The resulting LaTeX outputs could then be input to Moodle and H5P. For those without expertise in LaTeX, the use of a pen tablet and software to convert digital writing to LaTeX and digitizing handwritten content into LaTeX resulted in significant time efficiencies while maintaining the effective and accessible display of content.

5. TECHNOLOGY CHOICE FRAMEWORK FOR ASSESSING TECHNOLOGY SUITABILITY IN ONLINE STEM COURSES

When we embarked upon the process of identifying, evaluating, and selecting technologies to support online STEM learning and teaching there was a growing body of literature on teacher use of technology; however, there was sparse information on student technologies, other than equation editors, as previously reported. Through systematic research, we investigated, evaluated, and adopted digital inking technologies to address our needs. However, we are acutely aware of the pace of technological change and that context plays a critical role in technology adoption; therefore, while our decision to adopt digital inking was most appropriate for our context and cohort this is not necessarily a ubiquitous solution to the limitations of text-based LMSs. Having navigated hard choices to determine an appropriate technology for integration into fully online STEM courses, we have learned lessons about the teaching, learning, technological, and user requirements against which online STEM technologies can be evaluated. As such, by reflecting upon and analyzing our experiences we have created a technology choice framework (see Table 3) that may guide others through the process of technology investigation and adoption. The framework continues to be refined in relation to practice and peer input. It should be noted that other tools are available to support the selection of educational technologies (see, for example, Anstey & Watson, 2018); however, this framework originated from a STEM-specific experience and as such contributes a new angle to the body of scholarship in this space.

The framework supports the evaluation of technology suitability by categorizing its ability to enhance practice, minimally affect practice, or inhibit practice. The three-point scale is broadly based on the substitution, augmentation, modification, and redefinition (SAMR) model (Puentedura, 2010) of technology implementation; however, in this case, it was necessary to adapt the SAMR model to take into account technologies that do not improve teaching and learning or user experience. Despite the hierarchal nature of these classifications, it is important to consider the evaluation of technology from a contextual ecological perspective (Hamilton et al., 2016). The three-point scale in the framework is not meant to imply that a technology must meet a pre-determined threshold; rather, the framework enables contextualized evaluation of the comparative strengths and weaknesses of the technologies. Depending on teaching, learning, and institutional factors, it may be more appropriate to choose a tool that may be less technologically perfect but better fits teacher, student, or contextual needs. The criteria in the framework have been categorized into domains relating to learning and teaching, technological factors, and user experience. The criteria in each individual domain are revised versions of those used in our testing and evaluation process; however, we have made a concerted effort to combine student and teacher requirements into a single collated list, since ultimately it is unproductive to consider one group without also considering the impacts on the other group.

The technology choice framework works like a rubric to guide decision making by facilitating a comparison of the relative advantages of technology options. When using the framework, we recommend the following procedure:

1. Initially, each criterion should be assessed for its relevance to a particular context and the problem that is being addressed. For example, if an online course is delivered fully asynchronously, then the collaboration criterion (which is within the learning and teaching domain) could be discounted. Where necessary, additional context-specific criteria can be added at the end of the framework that address the specific needs of the tool.
2. Working systematically through each remaining criterion, the target technology should be assessed across a three-point scale (based on enhances, minimally affects, or inhibits practice). This process can be replicated for each of the technologies under consideration.
3. Completed frameworks provide a qualitative overview of the relative strengths and weaknesses of each technology, enabling objective comparisons between options.

Although this technology choice framework is valuable in assessing technology suitability for online STEM programs, it has limitations. In the post-COVID climate, it is important not to conflate fully online teaching and learning with emergency remote teaching since the technological solutions may not be applicable to both situations. The rapid pivot to remote learning and teaching brings a new set of contextual factors unaccounted for in this case study, such as the time and resources required to redesign courses to incorporate new technologies. Thus, although this framework may provide a broad outline of factors to consider, it is critical to reflect on these within the specific disciplinary, institutional, student, and environmental context at the time. Furthermore, the technologies and framework discussed are predicated on the premise that educators try to replicate successful signature pedagogies of face-to-face teaching in the online environment. On the other hand, it is useful to remember the Shulman (2005, p. 56) reflections that “[signature pedagogies] persist even when they begin to lose their utility”; therefore, perhaps for some contexts it may be worth investing resources to explore new STEM pedagogies (Deák et al., 2021). Regardless, it must be remembered that the technology choice framework is merely an instrument used to assist in assessing the suitability of technology for supporting online STEM teaching and learning; however, ultimately, successful student learning will be the outcome of a myriad of factors, of which the selection of appropriate technology is only one part.

6. CONCLUSIONS

This paper outlines some of the key considerations for academics and educational developers when adopting technologies to address the issues of online STEM learning and teaching in LMSs that are ill-suited to the task of communicating complex mathematical and scientific formulas, graphs, and diagrams. Determining an appropriate technology for integration into online STEM courses is complex, and whichever solution is selected will have advantages and limitations; however, despite the hard choices, this paper demonstrates that positive outcomes can be achieved by a well-researched, systematic, and broadly applicable approach. Until LMS design mitigates the limitations of text-based environments, academics and educational developers will continue to be challenged to assess and adopt technologies that meet their needs. The technology choice framework offers one solution by providing a tool that can facilitate the systematic criterion-referenced objective decision making necessary to evaluate and compare technological innovations. Given the scarcity of literature investigating the implementation of digital inking technologies for students studying STEM online, further contributions are essential in informing and guiding academics and educational developers working in this space. Studies that investigate student acceptance of these technologies and their experiences would make a valuable contribution.