ACM Logo  An ACM Publication  |  CONTRIBUTE  |  FOLLOW    

Is My Classroom Flipped? Using Process Mining to Avoid Subjective Perception

By José Francisco dos Santos Neto, Sarajane Marques Peres, Paulo Correia, Marcelo Fantinato / December 2021

TYPE: HIGHER EDUCATION
Print Email
Comments Instapaper

The flipped classroom model is a type of blended learning [1, 2]. As its design suggests the study material is available prior to face-to-face interactions with the lecturer. When students access the study material in advance, they are well prepared to discuss the subject and collaborate during face-to-face class time. In addition, the flipped classroom relies on the principle of active learning, which places the student in charge of their own learning and requires metacognitive skills to self-regulate the process [1, 3, 4]. By empowering prior knowledge through relevant study material and by encouraging student engagement [5], the flipped classroom fulfills the requirements for meaningful learning [6, 7].

A constant concern associated with the flipped classroom method is how a lecturer can ensure the classroom is flipped and whether student activities conform to the task sequence proposed in the learning material. Simple actions lecturers can take to try verifying whether the method is working include: asking students if they have analyzed the study material in advance and how they have done it so, applying an assessment, observing the synergy between the students and the lesson content as well as between them and their colleagues in the face-to-face classes, or using basic statistics provided by the analytics features of learning management systems (LMS) to gain insights into student behavior. However, these strategies do not show actual evidence about the previous actions taken by the students and when applied alone they are not reliable enough to certify the desired flip. In addition, such simple steps do not answer questions about which specific items of study material each student explored or whether students have followed the lecturer’s guidance on how to access the items of study material.

So what is the solution? In this article, we show how we applied process mining in a course supported by Moodle, an LMS that provides native access to event logs, to assist in verifying the flip of the classroom.  

Process Mining

Process mining refers to the processing of event data generated from information systems that support the execution of different types of processes. Through process mining, the dynamic perspective of process science is incorporated into the context of data science. Thus, process mining strengthens ties and creates a compromise between process science and data science [8]. The three major goals of process mining are:

  • Automating the discovery of process models underlying event logs. These discovered process models are called descriptive models (or as-is models), as they reflect the actual process running in organizations.
  • Performing automatic conformance checking, by contrasting an event log with a reference process model to find deviations. The reference process model can be descriptive or normative, which is called the assumed model and is designed by a process analyst.
  • Suggesting improvements in the process design based on the analysis of event logs and identification of issues, such as bottlenecks and other unwanted behaviors, leading to the so-called to-be model; or making recommendations and predictions to optimize the processes at run time.

The basic piece of information used in process mining is the event data, which is recorded in logs by information systems that support the execution of processes. Different analysis strategies of these event logs help achieve the goals of process mining. The key characteristics of a classic event log are illustrated in Figure 1a. An excerpt from the event log used in the analysis discussed herein is shown in Figure 1b.

Process mining offers analytical possibilities that transcend its three major goals introduced above. For example, descriptive statistics can be applied to event log data. When it comes to flipped classrooms, process mining presents added value for the lecturer due to its inherent process-oriented analysis. The following sections present how valuable process mining can be for educational analysis purposes.


[click to enlarge]

Figure 1. (a) Key characteristics of a classic event log; (b) an excerpt from the event log used in our analysis. (Some events are apparently equal; however, they refer to different types of interaction which are not shown in this log excerpt).

Method

We conducted a process mining-based analysis of the flipped classroom problem in three stages: pre-processing the event log, analyzing the log’s events and cases, and discovering and analyzing as-is process models. The discovered process models reveal students’ behavior when using the LMS. An institutional Moodle learning platform-based LMS was used to organize the study period between face-to-face classes. The analysis was applied in the context of a face-to-face undergraduate course in natural sciences, taught at the University of S??o Paulo, Brazil, for 161 students.

The LMS used provides a log file with the activities performed by the students, lecturers, and lecturer assistants in the interaction with the system. The activities recorded in the log refer to any user activity, such as creating, updating, or accessing multimedia learning objects (e.g., texts, videos, and webpages). The event log used in our analysis has 111,439 events. Our analysis focused particularly on accessing learning objects related to items of study material since these data would allow us to analyze details of how students prepare for the face-to-face class, as required by the flipped classroom method. Thus, technical-context activities, such as creating and updating a learning object, were disregarded.

Two pre-processing tasks were carried out to map the original event log extracted from the LMS to the event log format required for process mining: (1) the LMS activities were classified by their pedagogical nature, and only those related to the preparation for the face-to-face classes were kept (called herein preparatory pedagogical activities), in order to reduce noise and maximize the consistency of inferences; and (2) each group of activities performed by the same student within a preparatory period, usually a week, for a particular face-to-face class was separated into a single case (called herein normative time cut). Figure 2 shows the fields of the original event log extracted from the LMS and the fields of the pre-processed event log for process mining. (The pre-processed event log for process mining is hereinafter referred to as simply “event log.”) Only three fields of the event log created for process mining do not originate from the LMS (Class, Flipped_classroom_scenario, and Normative_time_cut), and were created with the support of the lecturer. These three fields were used to filter the log events related to preparing each of the face-to-face classes.


[click to enlarge]

Figure 2. Fields in the original LMS event log and in the pre-processed event log created for process mining. In the pre-processed event log, Case_Id is composed of two attributes from the original LMS event log: User full name and normative time cut, which is calculated based on Time. Each case refers to a single user (student) in only one preparatory period for a particular face-to-face class. Arrow-connected fields refer to the mapping performed. The white fields in the original log were discarded during pre-processing and the white fields in the pre-processed log were created with the support of the lecturer.

The event log was pre-processed using Disco, a commercial process mining tool. In addition, Disco was used to discover and analyze process models and their elements (i.e., events and cases) obtained from the event log.

Results and Analysis

The event log refers to 16 weeks. The lecturer adopted the flipped classroom method, so that: (a) the study material to be discussed in the face-to-face class was made available at the LMS one week in advance (exceptionally, twice two weeks in advance due to the occurrence of holidays); (b) in each class, the lecturer explained to students what activities they should perform the following week; (c) during the week, the lecturer sent students an electronic message with preparatory instructions for the next class and other messages reminding students of the need to explore the study material; (d) three formal knowledge assessments about the lessons' content were applied throughout the course.

Based on the event log, the frequency of events over time was analyzed, which could show whether students remained committed to preparatory assignments, given the flipped classroom method. In addition, we carried out the automatic discovery of process models that represent the students’ behavior when using the LMS, aiming to qualify the students’ engagement. These discovered process models provide information that would allow analyzing whether the flipped classroom method induced students to assume the study behavior expected by the lecturer.

Analysis of event frequencies. Figure 3a and Figure 3b show the frequency of events in the LMS during the course. In Figure 3a, all events were considered. In Figure 3b, only the preparatory pedagogical activities were considered. The occurrence of events referring to each week assumes a better-defined behavior when only the preparatory pedagogical activities are considered. Figure 3c shows the frequency of events, also only for the preparatory pedagogical activities, but with colored vertical lines highlighting the days of face-to-face classes: red for normal classes and yellow for formal assessments. This figure portrays a clear increase in the volume of access to study material in the days prior to the face-to-face classes. Figure 3d depicts the use of LMS by students over time, represented by active cases. To obtain this view, the event log was analyzed from the perspective of the resource associated with the events, i.e., the user. This figure shows the number of students engaged in the study prior to Class 1 is low; however, this number increases significantly after Class 1, reaching almost the total number of students per day, and remaining high until the end of the course. Figures 3c and 3d together allow us to raise the hypothesis the engagement evidenced by the increase in events in the days prior to the face-to-face classes (as shown in Figure 3c) results from the participation of a number close to the total of students enrolled in the course (as shown in Figure 3d).

Other information can be obtained from the analysis of the event log. For the first three classes, the events in the log show students did not access all items of study material provided by the lecturer. However, this behavior changes from Class 4 onward, showing these students acquired the practice of early study over time. In addition, the event log shows a higher density of accumulated events per week to prepare for Classes 8 and 11. Class 8 had a higher number of items of study material, while for Class 11 the lecturer requested a greater commitment of students with the study of the material, as it was a more complex lesson.


[click to enlarge]

Figure 3. View of the frequency of events and active cases in the LMS during the course: (a) events referring to all activities in the LMS (Y-axis range: 0 to ~760 events/hour); (b) events referring to the preparatory pedagogical activities only (Y-axis range: 0 to ~710 events/hour); (c) the same as (b), but including colored vertical lines highlighting the days of face-to-face classes (Y-axis range: 0 to ~710 events/hour); (d) active cases, with a green rectangle highlighting the preparation for Class 1 (Y-axis range: 0 to ~157 active cases/hour). For (c) and (d): vertical red lines and red numbers highlight occurrences of normal classes; vertical yellow lines and black numbers highlight occurrences of formal assessments. For (a), (b), (c), and (d): X-axis range: 0 to 126 days (3,024 hours).

Analysis of process models. Although the previous analysis allowed to verify the students had accessed the study material prior to the face-to-face classes, process models underlying the event log provides even more valuable information to the analysis of the flipped classroom method. The analysis of process models allowed us to know the strategy used by the students to interact with the LMS learning objects. Thus, it was possible to verify which and how the items of study material were accessed over time, and by which and how many students.

We used two types of process models: descriptive, automatically discovered from the event log; and normative, developed by the lecturer to describe the students’ behavior as expected by him. The descriptive models represent the actual sequence of activities performed by the students in the LMS. The normative models, on the other hand, represent the lecturer’s perception of the ideal sequence of access to the items of study material as suggested to the students through the preparation instructions for each face-to-face class. Both types of process models were created for each class, i.e., per week. The activities and time frame for preparing for the next class were filtered using the Class, Flipped_classroom_scenario, and Normative_time_cut fields (see Figure 2). While the descriptive model is presented in graphic format (as provided by the discovery tool), the normative model is presented in structured English.

Figure 4 shows process models for Class 1. While Figure 4a shows the normative model, Figure 4b shows the descriptive model. (The descriptive process model is represented in the Directly-Follows Graph (DFG) notation, created with the Disco tool (using the academic license). In the models discovered by the tool (which is better seen in Figure 5), rectangles correspond to the preparatory pedagogical activities performed by the students at the LMS and recorded in the event log; arrows correspond to the transitions between two activities performed; numbers associated with rectangles and arrows show, respectively, how often an activity or transition was performed and hence appears in the event log. The more an activity was performed, the darker the color of the corresponding rectangle is; the more a transition occurred, the thicker the corresponding arrow is.


[click to enlarge]

Figure 4. Process models for Class 1: (a) sequence of activities to access the items of the LMS study material prescribed by the lecturer; (b) actual sequence of activities performed by the students in the LMS.

According to the discovered process model in Figure 4b, students accessed the LMS in the days prior to Class 1 but performed only one of the three available and suggested activities. Of 161 students, 131 accessed the activity related to Lesson: Click here to start – Opening activities (EC136), and none accessed the activities corresponding to Quiz: Take the “Ground Zero” quiz (EC019) or Questionnaire: Take the “News and headline analysis” questionnaire (EC078). The activity corresponding to Lesson: Click here to start – Opening activities (EC136) was accessed 6,788 times, representing an average of 51.8 times per each of the 131 active students. The access of this same activity more than once by the same student is represented in the model by the self-loop, performed 6,657 times. This self-loop hides details of finer-grained interactions performed by students on items of study material (e.g., lesson started, content visualized, lesson finished). These finer-grained interactions were condensed into the corresponding preparatory pedagogical activities. Figure 4 depicts a time frame when the students had not yet met the lecturer and hence neither been introduced to the flipped classroom method. However, even after Class 1, they did not access all items available in the study materials. The analysis of the discovered process models shows the flipped classroom occurred with little emphasis at the course beginning, even with the lecturer sending students weekly instructions.

Only for the preparation for Class 4, students accessed all items available in the study material, and this behavior persisted until the course ended. In fact, as highlighted in Figure 3d, the use of the LMS by students gradually increased until reaching the expected by the flipped classroom method. This incremental process benefited from the students’ gradual understanding of the guidance provided by the lecturer in the face-to-face classes. Figure 5 shows the students’ behavior to prepare for Class 6, fully adhering to the flipped classroom method. Figure 5b shows the descriptive process model in which the students performed all the proposed activities and in the expected order, since the most frequent path occurred exactly as the normative process model. The most frequent path (highlighted in yellow) was from the start to Quiz: Take the “Where are the errors in the concept map?” quiz (EC096) (followed by 124 students), then to Interactive Content: Watch the “Carlos Nobre at Roda Viva (Block 1)” interview (EC022) (followed by 121 students), then to Quiz: Take the quiz about the “Carlos Nobre at Roda Viva– Block 1” video (EC032) (followed by 155 students), and then to the end (followed by 115 students). However, there were other paths also taken by some students, e.g., three students started directly by Quiz: Take the quiz about the “Carlos Nobre at Roda Viva – Block1” video (EC032) (highlighted in blue).


[click to enlarge]

Figure 5. Process models for Class 6: (a) sequence of activities to access the items of the LMS study material prescribed by the lecturer; (b) actual sequence of activities performed by the students in the LMS, considering only the most frequent paths.

Figure 6 shows the process models for Class 5 when the first face-to-face formal assessment took place. The normative model in Figure 6a considers five items of study material, and all of them were accessed, when considering all students using the LMS (i.e., all the active cases). The descriptive model in Figure 6b depicts a summary view of the students’ behavior in the LMS, while the one in Figure 6c details the full view.


[click to enlarge]

Figure 6. Process models for Class 5: (a) sequence of activities to access the items of the LMS study material prescribed by the lecturer; (b) actual sequence of activities performed by the students in the LMS, considering only the most frequent transitions; (c) actual sequence of activities performed by the students in the LMS, considering all transitions.

The summary view of Figure 6b considers only the most frequent transitions in the event log. According to this summary view, most students followed the sequence of activities expected by the normative model of Figure 6a. Some elements for the lecturer’s reflection on the students’ behavior provided by this summary model are:

  • The transition from Assignment: Submit your timeline – Review task (EC077) to File: Watch the “How to prepare yourself to the Class 5 – Test 1” video (EC108) (highlighted in yellow) shows 16 students accessed the assignment activity and then returned to the instruction activity that explains how to prepare for the other activities. Is there a problem to understand how to perform the assignment activity?
  • Does the transition from Quiz: Take the “Where are the errors in the concept map?” quiz (EC075) to Assignment: Submit your timeline – Review task (EC077) (highlighted in blue) show the quiz activity may trigger reflections on the assignment activity?
  • Among the most frequent transitions, no student completed their preparation after performing the quiz activity, as expected. Is there any problem with the logic of the behavior expected by the lecturer considering the sequence of tasks proposed by him in the learning material for this case? Does the normative model deserve to be revised?

Process mining algorithms perform robust analysis on more complex process models, like the one in Figure 6c. Some issues raised from a simple visual analysis of this model are:

  • Some students did not start the preparation activities as expected by the normative model (highlighted in yellow). This behavior may have occurred due to two possibilities: (a) some students may have tried to bypass study stages, or (b) some students may have been looking for a particular study logic more suited to their needs.
  • There are a significant number of transitions from File: Read the “How should you create your timeline?” instructions (EC067) to Quiz: Take the “Where are the errors in the concept map?” quiz (EC075), as well as from Questionnaire: Take the “What predominated during the discussions about ‘UNIVERSE’?” questionnaire (EC010) to Quiz: Take the “Where are the errors in the concept map?” quiz (EC075), in both situations skipping Assignment: Submit your timeline – Review task (EC077) (highlighted in blue). This behavior may suggest that Quiz: Take the “Where are the errors in the concept map?” quiz (EC075) is also relevant for the completion of Assignment: Submit your timeline – Review task (EC077) and that the normative model should not be entirely linear.

Conclusions

In this article, we presented an overview of how process mining can assist lecturers to analyze the behavior of students submitted to the flipped classroom active learning method. Two types of process mining-enabled analysis showed the flipped classroom method was actually in use, besides revealing facts that allowed the responsible lecturer to evaluate how well they followed the method.

An analysis of the frequency of events in the log showed a higher number of accesses to the study material in the days immediately preceding the face-to-face classes, which served as preliminary evidence that the class had been flipped. The analysis of process models discovered from the event log, on the other hand, showed which items of study material were accessed by students and the order in which students organized their studies to access such items. Together, these two analyzes also showed, in the very beginning, student engagement with the proposed study method was gradual, although afterward they quickly achieved the expected behavior.

A compliance check was also carried out between normative models (created by the lecturer and used as guidance for students) and descriptive models (discovered from the students' activities logged by the LMS). This compliance check showed the lecturer's guidance was mostly followed during the course. However, the actual student behavior is more complex than the proposed linear normative model. These outcomes offer insights for the responsible lecturer to analyze the full suitability of the method application.

From the outcomes of the application of process mining, the question regarding the actual flip of the classroom became closer to being precisely answered. The outcomes provide evidence for the flip of the classroom, adding precision and reliability to lecturer analyses. Process mining enabled a novel perspective of verifying the classroom flipping, with a more granular and factual level of detail regarding the students’ activities. Thus, the application of process mining can significantly contribute to the follow-up work of lecturers who practice the flipped classroom method.

The comprehensive analysis presented in this article was only possible due to the power of process mining, which is currently not supported by any LMS. In fact, not even event log access is commonly provided by LMSs. By being restricted to the basic statistics provided by event logs or analytical features, lecturers are prevented from exploring more sophisticated educational layers that could help them to make didactic and pedagogical reflections. The use of the event log provided by Moodle in conjunction with a process mining tool opens opportunities to let lecturers be aware of the actual teaching-learning process established in the flipped classroom.

Like other data analysis approaches, process mining often needs to preprocess event logs before analyzing them. Although process mining tools are intuitive enough to be used by lecturers, event log pre-processing may be a hindrance, as it requires extra computational skills and time. Process mining could be more viable if LMSs implemented process mining features by design as an integrated data analysis tool. Better yet, LMSs should become process-aware information systems [9], making the benefits of process mining more readily available. Besides potentially benefiting lecturers, process mining may be a useful tool for education managers and their staff, as process analysis and management span the entire school scope.

The analyzes presented in our study are limited by the expressiveness of the process model notation provided by the Disco tool. Process mining can offer higher value-added outcomes if ad hoc, domain-oriented algorithms are applied. More refined algorithms can better clarify issues raised in our study and contribute to improving the ongoing teaching process. In addition, process mining can support real-time analysis so that corrective, customized actions can be taken while the course is being offered.

References

[1] Abeysekera, L. and Dawson, P. Motivation and cognitive load in the flipped classroom: Definition, rationale and a call for research. Higher Education Research & Development 34,1(2015), 1–14.

[2] Winger, A. Five ways to flip the online classroom on its head. eLearn Magazine 2019, 2 (2019).

[3] Steen-Uthein, A. T. and Foldnes, N. A qualitative investigation of student engagement in a flipped classroom. Teaching in Higher Education 23,3 (2017), 307–324.

[4] Yilmaz, R. M. and Baydas, O. An examination of undergraduates’ metacognitive strategies in pre-class asynchronous activity in a flipped classroom. Education Tech Research Dev 65 (2017), 1547–1567.

[5] Burke, A. S. and Fedorek, B. Does “flipping” promote engagement? A comparison of a traditional, online, and flipped class. Active Learning in Higher Education 18, 1 (2017), 11–24.

[6] Novak, J. D. Learning, Creating, and Using Knowledge: Concept Maps as Facilitative Tools in Schools and Corporations. Second edition. Routledge, New York,(2010.

[7] Viswanathan, V. K. Helping learners as they construct knowledge: how can instructors leverage research findings. eLearn Magazine 2020, 3 (2020).

[8] Aalst, W. van der. Process Mining: Data Science in Action. Second edition. Springer Verlag, 2016.

[9] Dumas, M., Aalst, W. van der, and Hofstede, A. H. M. ter. Process-Aware Information Systems: Bridging People and Software through Process Technology. Wiley, 2005.

About the Authors

José Francisco dos Santos Neto worked as a BI analyst, process analyst, and professional Scrum Master certified by scrum.org. He holds a bachelor’s degree in information systems from the University of São Paulo (USP), Brazil, and is currently studying for a master’s degree in Information Systems at USP. Currently, he is a member of the Process Mining @ USP research group and the Artificial Intelligence Group at USP, where he studies process mining applied to learning analytics. He is also a member of the Concept Map Research Group, also at USP, where he investigates the structural analysis of concept maps and their application as a learning and collaboration tool.

Sarajane Marques Peres is an associate professor at the University of São Paulo, Brazil. Ph.D. in electric engineering (2006) at the University of Campinas; Master of manufacturing engineering (1999) at the Federal University of Santa Catarina; bachelor’s in computer science (1996) at the State University of Maringá, Brazil. She worked as an assistant professor at the State University of Western Paraná (1998-2005) and at the State University of Maringá (2005-2007), Brazil. She co-wrote a data mining textbook, published in Portuguese, worked as the tutor of the PET Information Systems USP group, under the Tutorial Education Program of the Ministry of Education, Brazil (2010-2017), and worked as a guest researcher at the Vrije Universiteit Amsterdam (2018) and at the Utrecht University (2019), Netherlands. Currently, she is a member of the coordination committee of the Information System Master Program at the University of Sao Paulo, a member of the research board of the Advanced Institute for Artificial Intelligence AI2 (Brazil), and a collaborating researcher at the C4AI – Center for Artificial Intelligence (USP/IBM/Fapesp). Her main research interests are computational intelligence, data mining and machine learning applied to text mining, process mining, gesture analysis, and human-robot interaction.

Paulo Correia teaches and researches within the School of Arts, Science, and Humanities at the University of São Paulo. He has been involved in research on concept mapping applied to teaching and learning since 2005. His current research aims to understand how to optimize the use of concept mapping in considering human cognitive architecture. Paulo was the chairman of the Sixth International Conference on Concept Mapping (CMC2014) organized by USP and IHMC. He led the USP and Coursera partnership to launch the first MOOC dedicated to developing novices’ skills to learn and collaborate using concept maps in 2019. He became an associate professor in didactics in 2020.

Marcelo Fantinato is an associate professor at the University of São Paulo (USP), Brazil. He is currently a productivity research fellow at CNPq, Brazil. He was a guest researcher at Vrije Universiteit Amsterdam and Utrecht University in the Netherlands. He holds a Habilitation in Business Process Management from USP and a Ph.D. in Computer Science from the State University of Campinas, Brazil. He has professional experience in the software development industry and is Green Belt certified by Motorola’s Six Sigma Quality Improvement Program. He was chair of the Graduate Program in Information Systems at USP. He was the general chair of the Brazilian Congress on Software and of the Brazilian Symposium on Information Systems, as well as program chair of the Brazilian Symposium on Software Components, Architectures and Reuse. He is a member of the IEEE Computer Services Technical Committee. He represents USP at the European Research Center for Information Systems. He is an associate editor of the International Journal of Cooperative Information Systems. He is co-chair of the international symposia series on Computing on Companion Robots and Smart Toys at the Hawaii International Conference on System Science.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

© Copyright is held by the owner/author(s). Publication rights licensed to ACM. 1535-394X/2021/12-3495212 $15.00



Comments

  • There are no comments at this time.