Bill Heinrich

Jeno Rivera

Bill Heinrich holds a Ph.D. in Higher, Adult, and Lifelong Education from Michigan State University and a Director of Assessment in the Hub for Innovation in Learning and Technology at MSU. His research has a dual focus on organizational assessment practice and policy and undergraduate experiential learning. Recent work focuses on using multiple outcome perspectives to stretch the value proposition of experiential learning assessment. He has professional experience in student affairs, experiential education and research, learning outcomes assessment, and program evaluation. He has lead and contributed to college student programs ranging from residence education and programming, outdoor recreation programming, student conduct, and new student orientation.

Jeno Rivera is a lifelong learner immersed in the educational field originally as a K-12 educator, secondary teacher educator, community educator, and most recently as an administrator in a self-directed learning program. Jeno gets geeked out observing her students discovering a love for learning. @jenorivera

Abstract

This article focuses on how to assess reflective artifacts in ePortfolios from multiple perspectives for holistic and authentic learning and career development. We specifically identify the video-selfie, a series of short testimonials captured on video, as an artifact type because the assessor can observe multiple dimensions of learning likely to occur in an experiential learning process and important to integrated learning. The ePortfolio is a strong tool with the ability to capture and help students authentically describe rich, experiential, and high-impact learning gains across topics and environments. Using video reflections as an artifact in ePortfolios helps both learners and facilitators identify, document, and gain insight on cognitive, affective, and behavioral outcomes.

KEY WORDS: experiential learning, assessment, immersive

1. INTRODUCTION

This paper focuses on how to assess reflective artifacts in ePortfolios from multiple perspectives for holistic and authentic learning and career development. We specifically identify the video selfie, a series of short testimonials captured on video, as an artifact type because the assessor can observe multiple dimensions of learning likely to occur in an experiential learning process and important to integrated learning. The ePortfolio is a strong tool with the ability to capture and help students authentically describe rich, experiential, and high-impact learning gains across topics and environments (Batson, 2015). Using video reflections as an artifact in ePortfolios helps both learners and facilitators identify, document, and gain insight on cognitive, affective, and behavioral outcomes (Lambert, 2013; McKillop, 2007; Robin, 2009; Tao et al., 2013).

Well-designed uses of reflection in experiences are highly important to individual and group learning gains and are linked to the promotion of postgraduate career decision making (Brown, 2004), college and career success (Niehaus and Inkelas, 2015), holistic learning (Kolb and Kolb, 2005), critical thinking outcomes (Heinrich et al., 2015), and ethical development (Goralnik et al., 2012). Higher education practices include widespread implementation of learning outcomes and connections across disciplines through reflection (Kolb, 1984; Eyler and Giles, 1999; Jones and Abes, 2004; Kiely, 2005). Cross-discipline connections bring learners closer to achieving institution-level integrated learning outcomes. These outcomes are often embedded in experiences with characteristics of high-impact educational practices (Kuh and Schneider, 2008).

The iterative nature of high-impact environments and reflective practice leads to students′ prior, diverse experiences surfacing throughout learning activity (Brown, 2004). Awareness of prior learning allows students to acquire new perspectives and build on prior experiences and, in turn, discover new interests (Niehaus and Inkelas, 2015). This research addresses a specific gap in directly assessing individual learning in high-impact practices in higher education. Additionally, this method allows users to model learning expectations by linking assessment of both intended and embedded outcomes—those that are likely present, but unassessed.

2. HIGH-IMPACT AND EXPERIENTIAL LEARNING ASSESSMENT

2.1 A Method to Assess Intended and Embedded Learning Outcomes

High-impact and experiential learning assessment (HELA) describes a process to identify and assess learning outcomes that are embedded, but not specifically intended by program leaders. Learner reflections are the primary source of identifying and evaluating embedded learning outcomes. Embedded outcomes are known to exist as a by-product of high-impact educational practices (Kuh and Schneider, 2008). These outcomes generally fall in the range of transferable or boundary-spanning knowledge, skills, and abilities including, but not limited to, communication, affective awareness, identity development, or socioemotional growth. Outcomes intended in one experience may well be embedded in another.

2.2 Path and Process

Our process varied based on the concrete experiences of the students and the approach of the program leader, but relied on reflective learning processes captured by media artifacts in addition to traditional writing assignments. Assessments were planned to complement one another by identifying both intended and embedded outcomes. Partnerships between assessment providers and program leaders were foundational to the process.

The key personnel involved included the program leader, the assessment provider, and the learners. In our method, the term program leader referred to the individual or team that designed and/or delivered an educational experience for the purposes of some intended learning outcomes. Program leaders assessed the experience for intended outcomes such as a disciplinary lesson or leadership development, career awareness, or community engagement. The term assessment providers referred to the researcher, assessment team, or program evaluator who partnered with the program leaders planning to implement an experiential learning cycle. In all of our cases, the assessment providers were the research team responsible for identifying embedded outcomes.

The program leader and assessors performed a joint critical examination of the design to identify concrete experiences, reflective opportunity, sense making, and/or application (Kolb, 1984). The goal of the partnership was to gain clarity about the experience design, program leader, assessment provider, and learner expectations. The order of learning activity influenced outcomes—but just as importantly, the reflection process responded and aligned to the relative length and depth of the experiences.

Next, a shared assessment plan for intended and embedded learning outcomes was developed. Because program leaders and assessors intentionally approached learning from different avenues, assessors worked with our partners to clarify the assessment method for intended outcomes. Embedded outcomes were considered ahead of the course, but not planned for in assessment design, so anticipating outcomes during planning phases was helpful to streamline the actual assessment process later on. Both parties identified and preselected rubrics for embedded outcomes that were important to the student, program, and/or institution. Assessors found that using three rubrics that spanned learning domains (cognitive, affective, behavioral) was useful to identify diverse outcomes embedded in the experience.

Video self-reflections were used by learners to capture a series of reflective statements (four to five reflections over the duration of the experience). Reflections provided assessors an opportunity to identify multiple learning outcomes. The seriated format allowed learners to demonstrate changes without having to analyze or make conjectures about the reasons for the change in thinking, as in a summative artifact. In some cases, program leaders provided prompts that were not structured to create an intended outcome, but rather to solicit embedded outcomes. Assessors observed that a loosely structured prompt worked well to encourage learners to explore integration among previous and current experiences. Additionally, when prompts provided some parameters for learners, the individuals avoided filming a rambling video self-reflection.

File transfer, security, and logistics were addressed in planning phases, but continued to evolve given field placements. Consideration of bandwidth and upload time, file sharing between learners and multiple assessors, and public or private settings all improved the efficiency of the experience for both learners and assessors. While some ePortfolio and assignment platforms were optimized for video sharing and feedback, effective communication with learners about expectations for file formats ahead of the experience saved time during the assessment phase.

Once all the videos were transferred to a storage location for analysis, the main task was to format the videos for coding. One path that we utilized was to transcribe the videos; however, we observed some loss of emotional and affective qualities that occurred when viewing the videos. Another mode we utilized as an alternative to transcribing was uploading the videos into a coding software system that allowed segment coding of videos. This approach required specific software, faster upload/download speeds, and storage space. Assessors found that it was a more time consuming to code specific sections of the videos as compared to solely coding text.

Rubrics used as a coding tool increased the efficiency and reliability of the assessment process. Assessors worked together to code all artifacts focused on intercoder agreement through discussion of the presence or absence of a code. For each learning experience, a single codebook, including rubrics, was created for ongoing communication and feedback with program leaders and learners. Program leaders used additional perspectives on their content to reflect on the experience.

A feedback report to the learner provided individuals the opportunity to receive and integrate feedback from multiple perspectives to gain lessons from the high-impact practice and insight into their learning plans and career paths. Reporting and feedback was situated as a conversation, with time and space for both instructors and learners to consider the implications of comments, data, and possible pathways. Reports included both an itemized list of outcomes achieved and a weighted visual reports indicating the relative concentration of outcomes in the context of the experience.

3. RECOMMENDATIONS FOR IDENTIFYING EMBEDDED OUTCOMES IN ePortfolio ASSESSMENT

3.1 Cultivate High-Quality Artifacts

The presence of existing high-impact educational practices in a learning ecology is necessary to assessing outcomes. One of the main goals for the program leader is to support the learners in creating video artifacts to capture and document high-impact learning experiences. In our experience, when program leaders provided guidance through prompts or assignment objectives, learners better documented learning outcomes as compared to documenting with no parameters. Additionally, assessors observed that if learners were given a time parameter for each video artifact, it forced the learners to be succinct in their reflection. Time constraints tended to eliminate nonsequential rambling that may occur with an unlimited time frame. Finally, with today′s advances in technology, program leaders do not need to provide learners special equipment to create videos. High-quality videos can be created on most portable electronic devices (e.g., phones, cameras, tablets) that students possess. It is notable, however, that students in our studies were most comfortable using their own devices. Technical challenges such as lost audio occurred when students used equipment mandated by program leaders.

3.2 Use Rubrics to Aid Assessment

Using multiple rubrics to assess high-impact and experiential learning designs resulted in the identification of multiple learning domain outcomes. This was a major advantage of the HELA method. The perspective shift from a single learning outcome framework to multiple or boundary-spanning outcomes provided through multiple rubrics was passed on to learners through reporting and ePortfolios.

From a technical reporting standpoint, the use of some ePortfolio platforms or learning management systems may greatly aid in tagging video segments to rubric dimensions. Advances in video Web applications (i.e., YouTube, Vimeo) to allow content tagging for individual segments (coding instances) could help an assessor streamline the rubric coding process. By creating tags that align to rubric dimensions/levels, an assessor simply creates a codebook to apply to a video upload. Managing data this way can alleviate the need for more expensive qualitative coding software.

3.3 Coach Learners to Apply Learning

Learning outcomes emerging from high-impact experiences and intended and embedded learning processes provided a compelling narrative of disciplinary content and boundary-crossing communication skills. Translational scholarship, a form of storytelling about academic work, is a strong opportunity for learners to synthesize and integrate learning from multiple sources and experiences. The nature of the outcomes identified through this method may help a student select an appropriate audience for their story. Alternatively, the act of reflection can become the focus of a story.

3.4 Make Learning Visible

Individuals engaging in the HELA method can more easily amplify their individual brand in terms of desired values or trajectory. With advances in educational technologies, learners have an opportunity to share their learning outside of academe through badges and by creating an individual profile. Software badging platforms (Credly, Mozilla Open Badge, or Educate Empower) can amplify data and feedback for additional audiences. Further, these platforms have the capability to link to professional branding sites, such as LinkedIn, Wordpress, or Digication. Linking learning data to social media outlets could be a valuable asset for learners who are looking to broadcast non-credit-bearing achievements to specific audiences.

4. CONCLUSION

Learning through reflection in the context of ePortfolios creates space for learners to assemble and build on previous experiences, potentially deepening each subsequent reflective artifact. Instructors can take advantage of such opportunity by encouraging learners to review previous reflections prior to recording new reflective statements. The HELA method for assessing embedded learning contributes to learner and program gains by leveraging the depth and breadth of existing powerful learning experiences. HELA provides a feedback loop to both the learner and the program leader about the nature of embedded outcomes, otherwise unassessed. With multiple communication outlets available, learners can broadcast the value of their experiences to many audiences in customized ways. Through a rigorous application of this assessment method, learners are able to use data and feedback to inform next steps on a lifelong learning journey.