Plethora of data helps Lytics Lab to analyze MOOCs

Why do so many students start a Massive Open Online Course only to drop out? Why, and when, do they bypass certain elements of online classes? Why are they taking the classes to begin with?

Those and other questions prompted Emily Schneider, a doctoral student at the Graduate School of Education, to team up with two other Stanford graduate students to research student behavior in MOOCs. While the recent surge in these online courses has provided millions of students with access to lectures, online forums and other educational materials previously unavailable, it’s been difficult so far to gauge the learning that is occurring via the Internet.

Schneider and her colleagues — René Kizilcec in the Department of Communication and Chris Piech in the Department of Computer Science — looked at three MOOCs offered by Stanford faculty, and presented a paper on their research at a conference in Belgium in April. They identified the different types of students taking these classes, how they have different approaches to the courses and how the classes might better serve them.

 “There is an enormous amount of work to be done in this space in terms of developing and investigating good models for instructional and interface design and developing appropriate outcome measures and analytics,” Schneider said.

This work is part of a broad Learning Analytics initiative at Stanford, which includes graduate students, researchers, and professors from not only education but also computer science, communication and sociology. In addition to Schneider’s  project, there’s work under way on a dashboard to help instructors monitor student engagement; a study of peer assessment based on 63,000 peer grades in a MOOC on human-computer interaction; and development of predictors of student performance.

The “Lytics Lab,” which meets weekly under the auspices of the Office of the Vice Provost for Online Learning and the  GSE’s Learning Sciences and Technology Design program, is driven by the plethora of data resulting from Stanford’s early adoption of online learning. Data are collected when students complete assignments, take exams, watch videos, participate on class forums or do peer assessments. The data from these courses can be used to both improve these courses and to answer a multitude of questions about how humans learn and interact.

 “Learning analytics is all about patterns and prediction,” said Roy Pea, the education professor who worked with Schneider and other students to establish the Lytics Lab and serves as one of the program's two faculty directors. “It’s about algorithms for identifying patterns in data to infer a learner’s knowledge, their intentions and their interests,  and then predicting what should come next to advance their progress.”

Schneider’s group used learning analytics to better understand why so many students don’t complete MOOCs. To do so, they studied student behavior in three such courses offered by Stanford faculty: Computer Science 101, a high-school-level course; Algorithms: Design and Analysis, at the undergraduate level; and the graduate-level Probabilistic Graphical Models.

The study found that people take classes or stop for different reasons, and therefore referring globally to “dropouts” makes no sense in the online context. They identified four groups of participants: those who completed most assignments, those who audited, those who gradually disengaged and those who sporadically sampled. (Most students who sign up never actually show up, making their inclusion in the data problematic.) The point of all this is not simply to record who is doing what but to “provide educators, instructional designers and platform developers with insights for designing effective and potentially adaptive learning environments that best meet the needs of MOOC participants,” the researchers wrote.

For example, in all three computer science courses they analyzed, they found a high correlation between “completing learners” and participation on forum pages; the more students interacted with others on the forum page, the better they learned. This led the researchers to suggest that designers should consider building other community-oriented features, including regularly scheduled videos and discussions, to promote social behavior.

While many people take online courses for certification and skills acquisition, many more take them simply for intellectual stimulation. The completion rates for the three classes were 27 percent for the high-school-level class, 8 percent for the undergraduate-level course and 5 percent for the graduate-level class. But 74 percent of the undergraduate students and 80 percent of the enrollees in the graduate class sampled, meaning they may have dipped in and out according to time constraints and interest.

Finally, the researchers found substantial gender differences in the more advanced classes. Counting “active learners,” defined as those who did anything at all on the website (around half the original enrollees), 64 percent of the high-school-level class were men, and the percentage rose to 88 percent men for both the undergraduate-level and graduate-level courses.

“There are people coming to MOOCs from a vast range of backgrounds,” said Schneider. We want to optimize systems to best meet their needs.”

Schneider said that their next steps may include extending their study’s analysis to other courses; collaborating with MOOC researchers at other institutions to build on their work; beginning to develop an online evidence base on MOOC research; investigating the community aspect of MOOCs; and running experiments on team dynamics and interface design.

Schneider is also working with a colleague at MIT to organize what could well be the first research workshop on MOOCs, aka the moocshop, in July. For more information, see moocshop.org. SE

This story was adapted from an article by R. F. MacKay for the Office of the Vice Provost for Online Learning.