Measuring the success of the campus LMS

Here's a straw poll question, directed primarily at those whose responsibility includes the care, feeding, and promotion of the campus learning management system (LMS): what, currently, is your primary measure of a successful learning management system (LMS) project? Is it the percentage of faculty using it? I know there are probably a variety of ways you might calculate the success of your LMS. But my question is: what is the **primary** or most valuable way or metric you use? If you could use only one such measure, what would it be? Thanks Malcolm ---------- Malcolm Brown Director, EDUCAUSE Learning Initiative email: mbrown@educause.edu IM: fnchron (AIM) Voice: 575-448-1313 ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

1. How do you measure "success" of your LMS?.....or what metric do you use?
2. Do you also measure "uptime" or "availability"?
3. Do you run an "in-house" or "hosted" LMS?
4. What was your primary deciding factor in #3…..what made you decide to run in-house?.....what made you decide to run "hosted"?

We measure success based on the number of courses and enrollments. Erica

Fascinating that one would select statistics such as percentage of faculty adoption or raw numbers of courses and enrollments to measure the success of the LMS. Shouldn't we be looking for metrics of improvements in teaching and learning? I understand such metrics are more difficult to develop, but the investment in a CMS or any other technology tool could only be predicated on an improved learning outcome. Joel Backon Director of Academic Technology/History Choate Rosemary Hall 333 Christian St. Wallingford, CT 06492 203-697-2514 Sent from iPad2

Joel:

Perhaps. If one identifies improved communication between faculty and students as an improved learning outcome, then yes.

This depends entirely on how you define "success." If you think success is about being able to demonstrate adoption (and by doing so, suggest that the investment of money was worth it purely because people are choosing to use the system) than to approach this purely from a standpoint of how many people use the system might make sense. 

Message from trentbatson@me.com

Malcolm:  you posed such an open-ended question but maybe that's the secret to a good discussion.

Hi Malcolm, The percentage of faculty using it is probably the most important. BTW, this includes adjuncts. However, it's the one measure that Administration always asks about. This does not mean that we accept faculty usage as the only measure. We do look at student usage and the actual features that are used. We call these Bb Metrics and look at whether or not the class is a basic user (syllabus upload, announcements), substantive user (content uploaded, external links, etc.), and power user (discussion forums, grade center, blogs, journals, etc.). This gives us an idea of the depth of usage as well as amount of usage. Hope this helps, Sandie Sandra L. Miller, Ed.D. Director of Instruction & Research Technology William Paterson University 300 Pompton Road Wayne, NJ 07470 973.720.2530 millers@wpunj.edu Think before you print

1. The percentage of the student body using the system in any one semester is primary. (Most recently at about 95%.) The percentage of faculty is secondary. (Most recently about 85%.)

We can get the statistics on faculty and student participation in the system. We can't get the faculty to agree to let us even try to measure learning outcome differences.

A great topic. Given that Granite State College serves over 50% of courses online, the choice of LMS is critical. To answer the question, however, is difficult because it's like asking what would be the most significant criteria in evaluating the plumbing system of a 4 BR house. It has to respond positively and consistently to the demands it was intended to serve. Turn the valve(s) and it works. As residents don't necessarily "buy into" plumbing and choose to use it, our instructional system is what it is. This is our LMS. Here is your training, support system, teaching community, etc. To some extend, our view of the LMS we chose (Moodle 2.x, as a migration away from Bb) was that the major LMS are mostly commodities in terms of communication features and access. The value came in the longterm cost, tech support, adaptability to emerging tools/methods/media, ease of integration with Banner. Some of these values pay off immediately, some later on down the road in ways we can't know for sure. Same applies to a plumbing system, perhaps: using PEX instead of copper, choosing to "T" a connection in anticipation of adding another bathroom, etc. To answer the question, I'd say the most valuable metric would be usability. Do we spend an inordinate amount of time explaining to instructors and students how to use basic functions? Does it play nice with most browsers? Is the opensource development community behind it committed and responsive to user needs and demands? - Steve Covello -- Steve Covello Rich Media Specialist Granite State College 8 Old Suncook Road Concord, NH 03301 603-513-1346 Skype: steve.granitestate Scheduling: tungle.me/steve.granitestate

We can get the statistics on faculty and student participation in the system. We can't get the faculty to agree to let us even try to measure learning outcome differences.

In fact, I always question the push to measure the learning outcomes correlated to the use of technology in a course. One cannot really claim a change unless one has some accurate information on learning outcomes before incorporating the technology into the course. That is one measure our faculty, at least, do not wish for us to take. Then there is the problem of trying to claim that any difference in learning outcomes for any particular semester is "caused" by the use of the technology. 

The teaching and learning situation is much too complex to claim more than a correlation between the use of technology and the learning outcomes, whatever they may be. There are so many other variables: Is the instructor having a good year, or a bad one? Was he or she too focused on research, or an upcoming tenure review? Was the course scheduled at a bad time for the professor? Did he or she bother to consider changes in pedagogy related to the technology, or was the technology just "slapped onto" the course? How well prepared were the students in this particular course section for this material? Was the difference just due to the novelty of the technology? Was there adequate support for this technology application, or were the students (and possibly the faculty member) left to stumble through the process on their own?

It seems to me that if we want to make significant claims about the benefits of technology to learning outcomes, we need a baseline. Even with a baseline, we really need to take the same sorts of metrics over a period of several semesters of the same course to be able to "average out" so many of the variables that we can never control.

BTW - the foregoing was NOT an attempt to argue against measuring learning outcomes. I think we should do a great deal more of that, preferably across the curriculum, regardless of the technology application, or lack thereof. We should be trying to examine pedagogies, learning spaces, teaching and learning styles or modes, etc. We should be examining just as many of the variables we *can* control as possible, over time, searching for trends that might give a clue to cause and effect.

Nikki Reynolds

Hamilton College

Message from trentbatson@me.com

Hi, all -- just a clarification about "learning outcomes."  Learning outcomes have been created by most institutions -- a set of 6 or 8 or so of what students should have achieved by the time they graduate.  Based on those stated outcomes, on some campuses, rubrics have been created within colleges at a university, within major programs, within gen ed programs, and on down to the course level.  The rubric creates standards for what it means to reach a learning goal.  The disciplines decided on those standards.

-----Original message-----

From: Trent Batson <trentbatson@me.com>
To: INSTTECH@LISTSERV.EDUCAUSE.EDU
Sent: Tue, Feb 7, 2012 13:30:15 GMT+00:00
Subject: Re: [INSTTECH] Measuring the success of the campus LMS

Hi, all -- just a clarification about "learning outcomes."  Learning outcomes have been created by most institutions -- a set of 6 or 8 or so of what students should have achieved by the time they graduate.  Based on those stated outcomes, on some campuses, rubrics have been created within colleges at a university, within major programs, within gen ed programs, and on down to the course level.  The rubric creates standards for what it means to reach a learning goal.  The disciplines decided on those standards.

For each level -- first year, second year, etc., those standards will have criteria that become more challenging as the students develop.  

To show that students do satisfy the criteria, the students collect evidence fo their work -- in a first year experience, in undergraduate research, in an internship, in field studies, or in other hight impact practices and in regular courses.  As faculty evaluate the evidence each semester, they can then indicate in the institutional eportfolio backend that the student is making progress.  The learning outcomes process, then, is providing criteria-satisfying evidence of work that fits the standards and definitions in the rubric. 

Learning outcomes are being tracked constantly, in other words, and the institution can run reports for accreditation based on the progress of cohorts of students and any one time toward filling all the criteria in the rubrics.

The problem with an LMSs is that they don't perform this task.  Electronic Portfolios do.  If you want srong assessment, you need an electronic portfolio, not an LMS.  This is not learning because of  the technology but this is a measure of learning by whatever means using technology.  (eportfolios can improve learning, but that's a separate topic).  

I know this is a bit off the original question, or maybe it's really is at the heart of the question?

Best

Trent 

On Feb 6, 2012, at 9:59 PM, Backon, Joel wrote:

Nikki, thanks for the thoughtful response. You're right that isolating the appropriate metrics is a real challenge, particularly since the LMS is really a delivery system. For those delivering online courses, the LMS has inherent value as a delivery system. For F2F courses, there are other reasons for the LMS. Metrics are difficult because of the many ways teachers use the LMS, ranging from posting documents to interactive learning. How do I measure the success of our LMS when some of the 77% of teachers using it are posting syllabi and assignment sheets while others are actively using blogs, wikis, assessments, and multimedia content? At the end of the day, it's all about he teacher, not the tools.

Joel

Joel Backon 

Director of Academic Technology/History

Choate Rosemary Hall

333 Christian St.

Wallingford, CT 06492

203-697-2514

Sent from iPad2

On Feb 6, 2012, at 12:30 PM, "Nikki Reynolds" <nreynold@HAMILTON.EDU> wrote:

We can get the statistics on faculty and student participation in the system. We can't get the faculty to agree to let us even try to measure learning outcome differences.

In fact, I always question the push to measure the learning outcomes correlated to the use of technology in a course. One cannot really claim a change unless one has some accurate information on learning outcomes before incorporating the technology into the course. That is one measure our faculty, at least, do not wish for us to take. Then there is the problem of trying to claim that any difference in learning outcomes for any particular semester is "caused" by the use of the technology. 

The teaching and learning situation is much too complex to claim more than a correlation between the use of technology and the learning outcomes, whatever they may be. There are so many other variables: Is the instructor having a good year, or a bad one? Was he or she too focused on research, or an upcoming tenure review? Was the course scheduled at a bad time for the professor? Did he or she bother to consider changes in pedagogy related to the technology, or was the technology just "slapped onto" the course? How well prepared were the students in this particular course section for this material? Was the difference just due to the novelty of the technology? Was there adequate support for this technology application, or were the students (and possibly the faculty member) left to stumble through the process on their own?

It seems to me that if we want to make significant claims about the benefits of technology to learning outcomes, we need a baseline. Even with a baseline, we really need to take the same sorts of metrics over a period of several semesters of the same course to be able to "average out" so many of the variables that we can never control.

BTW - the foregoing was NOT an attempt to argue against measuring learning outcomes. I think we should do a great deal more of that, preferably across the curriculum, regardless of the technology application, or lack thereof. We should be trying to examine pedagogies, learning spaces, teaching and learning styles or modes, etc. We should be examining just as many of the variables we *can* control as possible, over time, searching for trends that might give a clue to cause and effect.

Nikki Reynolds

Hamilton College

And, Rob, the irony of using data that are (for example) tied to "page views" as an indicator of course interactivity or student engagement is that a poorly-designed course can end up looking great because students spend so much time searching for materials or having to drill down unnecessarily to reach the relevant tools or resources.

Susan


Susan M. Zvacek, Ph.D.
Senior Director, Teaching Excellence, Learning Technologies, and Faculty Development
Fort Hays State University
Hays, KS 67601
785-628-4194
smzvacek@fhsu.edu



From:        Rob Gibson <rgibson1@EMPORIA.EDU>
To:        INSTTECH@LISTSERV.EDUCAUSE.EDU
Date:        02/23/2013 11:19 PM
Subject:        Re: [INSTTECH] Measuring the success of the campus LMS
Sent by:        The EDUCAUSE Instructional Technologies Constituent Group Listserv <INSTTECH@LISTSERV.EDUCAUSE.EDU>



Malcolm posits a great question - one that I've attempted to reconcile for years. For reasons I've never been entirely clear, the LMS always seems to be under 'extra scrutiny' to justify its place, expense, and usage on campus. Never mind that the cumbersome ERPs would often be happily sidestepped if it were not for the fact that many campuses 'force' students through the portal in order to access the services they really want. Akin to a toll booth company telling everyone that their usage is very high - when the drivers simply want to get to their destination.

To help satisfy those questions, we conduct a student tech survey each year (based loosely on the ECAR annual survey) - and are starting a faculty survey this year as well. We've collected very useful qualitative information specific to the LMS. (Satisfaction, features most utilized, mobile access, etc.) We've been pleasantly surprised by the student resposes. The LMS is often the #1 most accessed and utilized instructional tool - even ahead of library databases. (Granted, this may be biased because the faculty are requiring the students access resources only available through the LMS. However, most students seem satisfied with the tool, nonetheless.)

The quantitative metrics pulled from the LMS itself (I'll not mention the brand) are useless. That a faculty member posts a syllabus should in no way be equated with a faculty member who relies on the tool exclusively. Yet, that's what the metrics often reflect. ("Shells opened and accessed.") A new LMS we're considering should allow us to extract more useful data from the system. (Fingers crossed.)

A sidebar note from something I heard at the ELI Annual Meeting that I found intriguing. A panelist from EDx indicated that she envisions a not-so-distant future of customizable learning environments based entirely on Big Data. If it comes to fruition, it will turn the LMS world inside out. Imagine a system that automatically adapts to, say, high-risk students or students who are returning adult learners. Talk about a game-changer. When that product comes out, I'm buying lots of stock...

RG




From: The EDUCAUSE Instructional Technologies Constituent Group Listserv [INSTTECH@LISTSERV.EDUCAUSE.EDU] on behalf of Phillip Long [longpd@MIT.EDU]
Sent: Saturday, February 23, 2013 9:47 PM
To: INSTTECH@LISTSERV.EDUCAUSE.EDU
Subject: [INSTTECH] Measuring the success of the campus LMS

Malcolm: to answer your 'success metric' for the LMS, sadly at my institution the answer is as you posited - faculty adoption rates. It has nothing to do with how they use it. And indeed it's a bogus measure because it's mandated that all academics must have a Bb course shell. There are exceptions permitted if the academic argues it and if the associate dean of T&L is open minded but the cultural norm is that the LMS is the corporate choice and thou shalt use it. Success circularly is in the nearly 100% adoption of this mandate…. Do you smell a problem here? Sign….

phil

(apologies for the delayed reply - it seems that for some reason this list no longer accepts posts to it from my UQ account only my MIT account - I don’t have the energy to correct this at the moment)

:: :: :: :: :: :: :: :: :: :: :: ::
Professor Phillip Long :: Ofc of the Deputy Vice Chancellor Academic :: The University of Queensland :: Brisbane, QLD 4072  Australia
ITEE

Director: Centre for Educational Innovation & Technology
http://ceit.uq.edu.au longpd@uq.edu.au




On 04/02/2012, at 4:45 AM, Backon, Joel wrote:

Fascinating that one would select statistics such as percentage of faculty adoption or raw numbers of courses and enrollments to measure the success of the LMS. Shouldn't we be looking for metrics of improvements in teaching and learning? I understand such metrics are more difficult to develop, but the investment in a CMS or any other technology tool could only be predicated on an improved learning outcome.


Joel Backon
Director of Academic Technology/History
Choate Rosemary Hall
333 Christian St.
Wallingford, CT 06492
203-697-2514
Sent from iPad2

"Perceived student learning" is as far as we've taken this question. However, we measure it with a single Likert question, so it has many shortcomings (e.g., reliability, face validity). About 2/3 of our students consistently agreed with the statement "ANGEL has improved my learning". We stopped asking that question several years ago, but our data is still available at: http://www.kumc.edu/ir/moe/ANGELLearningResponse.aspx We also measure Number of ANGEL Course and Group Logins: http://www.kumc.edu/ir/moe/AngelLearning.aspx Dave ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.