Tomer Katz, Ex Libris
Student retention programs usually ignore metrics regarding the students’ use of course resources. But are they missing an important factor in student retention?
We’ve put this question to test and came back with some interesting results. But first, let’s take a look at what student retention is.
Simply put, retention refers to a student graduating from the institution where they are currently enrolled. This is an important factor in higher education for two main reasons: a school’s reputation is, in part, based on its graduation rates as a measure of success; and tuition revenue is lost whenever a student leaves the school, for whatever reason. Potential students may choose a college, for example, based on the assumption that high student retention indicates greater chances for academic success.
So what contributes to successful student retention?
We reached 88% predictive accuracy in identifying at-risk students, a 17% improvement over “rule of thumb” statistical analysis.
Traditional prediction models are not enough
Many retention strategies in use at colleges around the world are intended to be preventative, focusing on identifying students at risk of leaving school and intervening to prevent that eventuality. A lot of time and resources are dedicated to such efforts, primarily focused on students likely to withdraw for academic or financial reasons. Retention strategies involve probability models that incorporate data such as course grades, student admission information, self-reporting by students, and professor feedback.
What that amounts to are predictive “rules of thumb.” The traditional application of such models still tends to identify such at-risk students late in the game, when they are highly likely to drop out no matter what steps are taken. In addition, these models will also flag students who are not actually at risk. As a result, some retention teams are turning to data analytics to improve their results.
Setting a baseline & our methodology
In the context of leveraging data to improve student retention, Ex Libris and Curtin University collaborated in researching the relationship between student engagement with Leganto, the Ex Libris course resource list solution, and improving accurate identification of at-risk students.
Our initial statistical analysis revealed a simple and direct 71% correlation between past student course failure and the likelihood of failing again.
In order to incorporate Leganto into our analysis, we determined that student engagement with the resource list solution would be measured by weekly user mouse clicks within the system. The data had to be normalized by identifying appropriate benchmarks for each course, as some classes by their nature required less interaction with Leganto than others.
Our ultimate aim was to use the Leganto data, in combination with past performance data (grades, past at-risk scores, and the like), to train a machine-learning algorithm to detect students likely to have difficulty succeeding in a specific academic course. This strategy could then be applied more broadly to identify students at risk of dropping out of school, and do so at an earlier stage, when there is a greater chance of retaining them.
Improving prediction accuracy with machine learning and Leganto usage data
A machine can only learn from the data it is given. By including usage data from Leganto, alongside more traditional performance data, we aimed to achieve a more complete picture of student success indicators.
We used machine learning to test our predictive model, based on traditional performance data and a metric for student use of the Leganto resource list solution. The results were clear.
We reached 88% predictive accuracy in identifying at-risk students, a 17% improvement over “rule of thumb” statistical analysis.
Potential impact on student retention efforts
With improved precision in identifying at-risk students, retention personnel are less likely to invest in students who don’t need it or miss those who do. Put another way, a school’s investment of time and resources in retention activities will be 17% more effective.
In light of the Curtin University study, Ex Libris plans to enable libraries to provide Leganto usage data to enrich their institution’s student success or retention initiatives. If you would like to explore using Leganto usage data for student retention at your institution, please contact us.
Get the full breakdown, details, and statistics of the Ex Libris-Curtin University collaboration in our report, “Can the Leganto Solution Help Predict Student Success?”
March 9, 2020
