Degree Compass: A Course Recommendation System

As a regional institution at which more than 40 percent of the students are nontraditional learners and more than 50 percent are Pell recipients, Austin Peay State University (APSU) has many students who are unfamiliar with the subtleties of navigating their way through a degree program. Although each APSU student meets with an advisor each semester, the choices involved in constructing a successful curriculum are still difficult. We thus decided to create Degree Compass, a choice architecture powered by predictive analytics, to help students make advantageous and informed choices about their education.

The challenges that Degree Compass were designed to solve are widespread across higher education. With support from the Bill and Melinda Gates Foundation and Complete College America, Degree Compass has been replicated at three other Tennessee institutions in the past year. These replications have provided significant data to demonstrate the effectiveness of the architecture's predictive modeling techniques when applied across broader student populations and curricular structures.

Degree Compass Described

Inspired by recommendation systems implemented by companies such as Netflix, Amazon, and Pandora, Degree Compass successfully pairs current students with the courses that best fit their talents and program of study for upcoming semesters. The model combines hundreds of thousands of past students' grades with each particular student's transcript to make individualized recommendations.

In contrast to systems that recommend movies or books, Degree Compass does not depend on which classes are more popular than others. Instead, it uses predictive analytics techniques based on grade and enrollment data to rank courses according to factors that measure how well each course might help students progress through their own degree programs. From the courses that apply directly to the student's program of study, the system selects those courses that fit best with the course sequence in their degree and are the most central to the university curriculum as whole. That ranking is then overlaid with a model that predicts the courses in which each student will achieve the best grades. The system most strongly recommends a course that is necessary for a student to graduate, that is core to the university curriculum and the student's major, and that the student is expected to succeed in academically.

The system deliberately provides information facing in three directions:

• A student-facing interface presents course choices in an appealing and informative way.
• An advisor-facing interface provides information to advisors that lets them offer more nuanced advice to their students.
• An array of reports allow the institution to aggregate recommendation data and use it to optimize future class schedules; the data also provides early alert information on students who are at risk of performing poorly in their courses so that the institution can target support accordingly.

Refinements to the Implementation

Degree Compass has been providing information to students and APSU since its initial deployment in spring 2011. Since then, the interface has seen many important refinements.

The course recommendations for students (and their advisors) are now categorized according to the unmet degree requirements that they would satisfy. Students and advisors can now see at a glance the student's outstanding academic requirements and which courses would be most advantageous.

Many students pursuing degrees in higher education today are also juggling family or job responsibilities. Constructing a schedule that allows students to both meet these significant external obligations and make effective academic progress is a challenge using traditional methods. Degree Compass now provides search filters that let students and advisors find the courses that simultaneously best fit any practical constraints while also maximizing programmatic and academic success.

As noted, Degree Compass has been installed at three other institutions in Tennessee — one university and two community colleges — and is thus in use at schools with a broad cross-section of curricular structures and student populations. These schools are each at different stages of campus-wide implementation, but their data offer us an opportunity to further refine the predictive modeling as well as to increase the interface functionality to meet the needs of different institutional settings.

Also, Degree Compass was recently acquired as part of Desire2Learn's portfolio of predictive analytics and student success products. Degree Compass is now available as a commercial product for installation on a wide variety of campus or system settings.

New Challenges in Scaling Degree Compass

The initial experiences and challenges with creating Degree Compass are described in a section of the EDUCAUSE e-book, Game Changers.¹ The main initial challenge was to create a mathematical model to successfully estimate a student's future grades to an acceptable tolerance based on their transcript and grade legacy data. A secondary challenge was to design a sequencing system to order courses in a natural sequence, based on both a given major and the university curriculum as a whole. Once these models were designed and tested, the system then had to be taken to full scale, seamlessly interacting with APSU's CMS.

More recently, we have been extending Degree Compass from a single-institution system to one that can seamlessly interface with the technology in diverse institutional settings. This has required additional functionality to respond to the variety of needs that arise on different campuses. For example, Degree Compass now provides filters to allow students to identify the courses that best fit them that are available at specific branch campus locations or in a specific course delivery method.

We have also faced the technical challenges of working in different technology settings. These challenges have included working with a variety of degree audit systems and curricular structures. The solution required new interfaces to be developed so that Degree Compass could interact seamlessly with campus CMS and degree audit settings.

Having addressed both types of challenges, the Degree Compass system can now function in different environments and in a multiple institution setting.

Results

Although our initial predictive results at APSU were very encouraging, it was important that we establish that our modeling techniques could calibrate themselves to differing institutional settings and student populations.

Happily, the results from all three campuses replicate the ongoing grade prediction resolution achieved at APSU. Data from fall 2012 showed that the average predicted grades in the university settings were within .5914 of the awarded grades, and 89 percent of those who were predicted to pass the course indeed passed. In the community college setting, average predicted grades were within .6487 of the awarded grades, and 90 percent of students who were predicted to pass the course did so. These results confirm that the grade prediction engine successfully predicts grades in diverse settings, from a rural community college to an urban research 1 university.

The grade distributions across all the campuses, of all students, showed a picture in which a student had a 0.62 to 0.63 probability of getting an A or a B grade in their course. A more detailed analysis of the connections between the grades predicted by Degree Compass and actual earned grades showed that students at all four campuses who were predicted to earn a B or above had a significantly greater likelihood than this of actually getting that grade. Indeed, at each campus more than 90 percent of students who took a course in which they were predicted to get at least a B actually earned an A or a B grade. The analysis shows that this effect was evidenced at every school and at every course level from developmental classes through upper-division courses.

Further, more detailed analysis of the grade prediction model at APSU let us improve the resolution of the grade predictions. Also, rather than correctly predicting if a student will pass the class with 90 percent accuracy, the new analysis correctly distinguishes if the student will get either an ABC or a DF with 92 percent accuracy.

Of course, the motivation behind this work was not to predict grades, but rather to provide a choice architecture in which students and advisors could make more nuanced decisions about degree programs. To this end, our hope is that, when students follow Degree Compass advice, they will be more successful.

At APSU, the data is mature enough to establish the effect of following Degree Compass's advice. When ABC percentages were compared between fall 2010 (the semester before the system's introduction) and fall 2012, we saw an increase of 1.4 percent, which represents a 5.3 standard deviation shift. This very statistically significant shift was apparent across the student body, from freshmen to seniors. We saw similarly significant increases for several subpopulations, including African-American students (an increase of 2.1 percent, with 2.89 standard deviations) and Pell recipients (an increase of 3.9 percent, with 7.7 standard deviations).

Degree Compass was created as an attempt to use predictive analytics to improve student success and completion. It now seems clear that, across the institution spectrum and student population range, it has the promise to do just that.

1. Tristan Denley, "Austin Peay State University: Degree Compass," Game Changers: Education and Information Technologies, Diana G. Oblinger, ed. (Boulder, CO: EDUCAUSE, 2012).