Virtualization Increases Institutional Knowledge of Student Learning Activities

As educators, the following questions are always at the back of our minds: Are my students really learning? How are they learning? Should I improve my class? If so, how or what should I change? If I make these changes, what guarantees do I have that they will result in improvements? In a blended learning environment the events recorded by a learning management system (LMS) offer only a partial view of how students participate in a course. When a course uses active learning methodologies, including numerous practical activities, a significant part of the course events are not observed. If you cannot measure it, you cannot improve it.

By asking students to install a virtual appliance (a computer within their computer), schools can significantly widen the range of events observed in learning environments to include those occurring while students work on course activities both inside and outside of class. A pilot study at the University Carlos III of Madrid in Spain found that providing virtual machines to students increased the school's ability to observe student learning activities in blended learning courses that had previously gone unrecorded by our legacy LMS. The data collected offer a comprehensive account of how students interact in a particular learning environment, facilitating various initiatives to improve student learning and engagement. Virtualization gave the school the additional knowledge through observation of student learning activities needed to help answer our questions about their learning and continuing improvement of our courses.

Context

The study was conducted during two editions (2011–2012) of a 300 student, second-year Computer Engineering programming class that met twice a week for lectures. Classes followed an active learning methodology, in which students solve short exercises related to the course content. Students had to complete a set of activities both before class and in class. Previously, they had used a legacy LMS, which hosted a virtual course community with a discussion forum and related course material (figure 1).

Figure 1. Screenshot of the legacy LMS

The learning objectives for the class are:

• Design a software system using the C programming language
• Use several industry-category tools proficiently
• Work effectively in a team
• Learn autonomously

Additionally, instructors needed information about:

• The level of student engagement during the course
• Team dynamics
• Resources used
• The type of operations students performed while working on activities

These objectives required a solution capable of obtaining information beyond what is collected with traditional summative assessment or those based on the legacy LMS.

At the beginning of the course, students were offered the opportunity to install a virtual appliance, created in-house using open-source VirtualBox software, in their personal computers. This appliance was a complete and fully configured computer contained in a file and could be executed as a regular application. It provided the ecosystem to work in the course, containing all the required tools, links to the course material, and other resources.¹ Some learning activities and events in the appliance were recorded and relayed to a central server.² Figure 2 shows the virtual appliance while it is running.

Figure 2. Screenshot showing terms of use for the virtual machine

Video on the importance of observing all events in learning environment (1:08 minutes):

Lessons Learned and Benefits

The data from this study has provided comprehensive insights into how students interact within the class, and has yielded the following benefits:

• Greater flexibility: Instructors were able to see the level of participation in certain activities before the group met in a classroom. This information allowed for last-minute adjustments in the activities to include in the class session.
• Greater student and instructor engagement: The appliance featured a dashboard (figure 3) that visually summarized the information collected in the previous seven days. This provided opportunities for students to reflect on their activity within the course and compare their work with that of the rest of the class. Instructors could also compare the current information on student activity with other groups.

Figure 3. Dashboard for the course, September 2012 edition

• Enhanced LMS data: By hosting the processing and reporting stages outside the LMS, the study team was able to combine two heterogeneous set of information sources — the virtual appliance and the LMS. The data from the virtual appliance could be combined with the records obtained from the LMS to provide a more holistic perspective of classroom activity.
• Greater student-instructor interaction: Students increased their interaction with the instructors when receiving information about their current performance. The instructors informally observed that relaying activity information to students prompted visits during office hours specifically to discuss it. 
• Better monitoring: The achievement of active learning goals, such as the use of industrial-category tools in this case, could be monitored more effectively with this approach than with the LMS alone. With the feedback loop (see next item), instructors could quickly adjust activities to compensate for some students' failure to use the tools. For example, when instructors noticed that some students were not using the version control tool, they included its use in a hands-on test in the lab.
• Feedback loop: The institution can implement a feedback loop of data collected from the virtual machines and the LMS to institute continuous improvement of pedagogy and learning: collect data, create reports from the data, predict future performance, act to achieve improved performance, and refine the approach continuously to produce a culture of quality teaching and learning based on increased knowledge of student behavior. Instructors have a comprehensive view of the effort required for the activities as well as those aspects that may suggest an anomalous scenario, such as difficulties arising from incorrectly dimensioned activities, extra effort required because of a lack of appropriate resources, etc. With this data, instructors become aware of problematic situations sooner, to the point of deploying remediation actions while the course is running.

Increased Observation versus Privacy

We recognized the inherent trade-off between the benefits to the institution of increased observation of student learning activities and the risks to student privacy when students use virtual machines. Although the study assumed that students use the appliance only while working in course activities, students may be reluctant to install virtual appliances because in real life they use their computers for purposes unrelated to the university or their classes. We included the following steps to assure students of their privacy rights:

• Before downloading and installing the virtual appliance, students were given a detailed account of the type of information being recorded, for what purpose, and the rights they had to see, amend, or delete such information. The recorded information is stored in a shared folder accessible to the student, and they may request the removal of prior versions of such data through a contact person. The appliance is downloaded only when students agree to these terms of use.
• The virtual appliance does not interfere with or record other regular tasks carried out in the personal computer.
• To execute the appliance, students are required to install the so-called virtualizer (at no cost).
• At the end of the course, the virtual appliance can be cleanly uninstalled.

Lessons Learned/Challenges

Despite its potential to help transform the classroom as we know it, virtualization poses these inherent roadblocks:

• Minimum computing requirements: In the study, we assumed that students had a personal computer that had enough memory and processing power to download and execute the appliance, although this may not be the case in other contexts or schools. Using informal data, we concluded that this limitation is becoming less prevalent, because personal computers are increasing in computing power while virtualization has shown only a moderate increase in complexity.
• Distribution: The virtual appliance was perceived by students as "yet another application installed in my computer" for the purpose of working in a certain course. The size of the file to install in the personal computer (a few gigabytes) and the initial configuration steps made it challenging to ensure that all students had the appliance on their machines at the beginning of the course.
• Further configuration and support: The steps needed to configure virtual appliances may confuse some students and hinder user adoption. When deployed on a major scale, a school might need to provide additional technical support to resolve difficulties related to configuring the virtual machine. During the pilot study students posted a few requests for help in the forum, but we have seen a significant decrease in such requests as the virtualization software matures.
• Network constraints: Mass downloading of the large virtualization file from a school's servers may strain bandwidth. This challenge can be easily overcome by providing alternative media such as USB devices, DVDs, or peer-to-peer networks.
• Segregating events: In the study, all students received an identical copy of the virtual appliance. If events were simply recorded and relayed, it would be impossible to distinguish which events were produced by which student. To address this, students were required to exchange information related to the course using a version control application. In fact, this application became one of the industrial tools in which they acquired usage experience. The authentication step required when sending data to this application was used to segregate events by user when received.
• Costs: Although we don't have exact numbers, a rough estimate is that programming the virtual machine in-house took two person months.

Conclusions and Possible Future Uses

The virtual appliance offers students an environment ready to be used with a low adoption barrier. Our study also proved that virtualization holds the power to unleash a new approach to learning, especially in classes that require the use of specialized programs, such as concept map tools. A virtualized learning environment brings the potential of greater interactivity and flexibility and supercharges the capabilities of the traditional LMS in classrooms. This virtualization deployment can also be easily replicated on a large scale to encompass generic courses, and it can be customized at an institutional level for each degree or for a course within a degree.

With advances in virtualization technologies, virtual appliances in schools are becoming a commodity. We've found that the key to using this educational tool is a simple one: to maintain and offer students an environment that simplifies their work in the course. In return, the increased knowledge of their learning activities empowers the school to continuously improve the courses they take and their learning of the information taught in those courses. By measuring these elements, we can improve them.

Acknowledgment

Work partially funded by the eMadrid Excellence Network (S2009/TIC-1650).

1. Fully configured development tools included the compiler, debugger, and memory profiler.  Also included were manual pages for the programming environment, and additional development tools such as a version control system.
2. Students have a set of weekly activities that require the use of the development tools (compiler, debugger, editor, memory profiler). The events of those tools being used are recorded and sent to the server. Upon arrival, they are processed, and a summary is made available to the tutors (this phase is in development right now).