< Back to Main Site

EDUCAUSE review onlineEDUCAUSE review online

The Value of Teaching and Learning Technology: Beyond ROI

0 Comments

The Value of Teaching and Learning Technology: Beyond ROI
Measuring the value of an investment instead of dollar-for-dollar return provides a clearer picture of benefits and costs

Given the hundreds of millions of dollars spent each year on technology initiatives in higher education, questions about the value of these expenditures almost seem moot. However, virtually every CIO at every institution of higher education is asked to report the return on investment (ROI) of technology spending on his or her campus. Making such an accounting is a daunting challenge. Applying a strict business-school definition of ROI would require providing evidence of a dollar-for-dollar return on hardware, software, and infrastructure projects. The desired return in question, however, is not always so obvious. Should technology investments save money? Increase institutional capacity? Improve throughput?

In the face of such ambiguity, many CIOs throw up their hands in frustration. Bob Weir, vice president of Information Services at Northeastern University in Boston, asserted that technology ROI is “impossible to reliably calculate, compare, or claim” in higher education.1 This is so, he argued, because there is no universally recognized “coin of the realm” for assigning value to IT investments.

While the academic community has become accustomed to the “ROI question” as it relates to IT in the broad sense, the burgeoning scope and expense of instructional IT raises the question in new, more targeted ways. Virtually every university and college in the United States has implemented a course management system (such as Blackboard, Desire2Learn, Sakai, or WebCT). At the same time, most institutions have ramped up their support for instructors interested in developing online courses or multimedia enhancements for traditional courses. As a consequence, institutions now face the dilemma of storing and managing terabytes of teaching and learning content. Together with expenditures on e-portfolio solutions, online assessment and evaluation tools, and increasingly sophisticated classroom technologies, colleges and universities are spending a growing percentage of their overall IT budgets on what might be termed the teaching and learning infrastructure.

Just as it is difficult to demonstrate ROI for broad IT initiatives, it is difficult to show ROI for money and time spent building, implementing, and supporting a teaching and learning infrastructure. In this article, we propose an alternative method for accounting for the costs and benefits of investments made in teaching and learning technology (hereafter TLT). More specifically, we recommend that institutions focus on measuring the value on investment (VOI) of their instructional technology programs and initiatives. For reasons detailed below, we believe this approach is both more realistic and more helpful in terms of prioritizing and assigning scarce resources to maximize institutional effectiveness. A focus on VOI allows institutions to begin with the end in mind—that is, to design, build, and implement a teaching and learning infrastructure that demonstrably and consistently contributes to the realization of desired institutional outcomes. We believe that using the approach recommended here not only makes it possible to demonstrate the value of teaching and learning investments but also makes the value more obvious and easier to communicate to others.

VOI versus ROI

Perhaps because most colleges and universities have departments or schools of business, higher educationhas a powerful tendency to demand evidence of dollar-for-dollar returns for time and money expended on IT. While this seems a straightforward proposition, such efforts generally falter in the face of competing definitions of “return.” Indeed, the lack of consensus about how to measure ROI has led many to give up on precise, quantitative measures of the value of IT investments. Similar frustration in the corporate world prompted Gartner to challenge organizations to focus on the VOI of so-called soft initiatives that do not obviously or directly add to the bottom line.2 While value can be had from investing in organizational competencies, new methodologies and capabilities, and better organizational collaboration, the dollar-for-dollar return on such investments is difficult to measure.

Stated in the simplest terms, VOI differs from ROI in its focus on intermediate rather than final outcomes. A VOI approach to IT, for example, emphasizes the contributions of new hardware or software to institutional competencies. Targeted competencies are those shown by experience, evaluation, and research to contribute to an institution’s performance—its ability to produce key deliverables.

A VOI-driven resource allocation and evaluation process promotes a broader, more strategic view of project consideration and prioritization. ROI might offer greater precision, but it also creates a tendency to emphasize narrowly tactical IT.3 Although tempting, this approach is a mistake because ROI is much more difficult to demonstrate for strategic investments aimed at broad organizational competencies and capacities. Because these soft improvements are critical to the success of institutions of higher education, however, we believe that it is at least as important for organizations to account for VOI as to account for ROI.

A focus on value instead of return promotes more meaningful measurement of the benefits of investments in TLT, but measuring VOI is not without pitfalls. The most significant problem stems from the fact that VOI is not a direct measure of an investment’s impact on the bottom line. For institutions of higher learning, the most important measurable results are things like the number of students served, time-to-completion, graduation rates, and success of graduates (initial job placement, career performance, and so forth).

Adopting a VOI-centered approach to TLT is particularly appropriate given the difficulties associated with consistently measuring teaching and learning outcomes. For example, one possible indicator of student success is average GPA. However, with grade inflation and inconsistent grading standards across disciplines and institutions, this is not an entirely valid or reliable measure. Using exit exam scores, job placement statistics, or other measures would be similarly problematic. Each institution and each program within each institution has its own set of success criteria for students, a reality reflected in current accreditation standards. Investments in TLT do not always contribute to these things in an obviously direct way. Rather, these investments contribute to institutional capacity to produce these results more effectively and consistently.

It might not be obvious how a CMS, for example, contributes to an institution’s time-to-completion or graduation rates. While an ROI model would require demonstrating exactly how a CMS contributes to these things, VOI simply requires that we show how a CMS contributes to important institutional competencies (Gartner’s soft infrastructure). Appropriately used, a CMS can promote better teacher-student communication, enhance large lecture courses, make classes more flexible for students, and so forth. These things can, in turn, improve time-to-completion and graduation rates.

The more realistic (though messier) reality of VOI is illustrated in Figure 1. If we think of the inputs and outputs of a university, the inputs are dollars, faculty, and enrolling students. The outputs are graduates, research results, and general community improvement. In between, however, are thousands upon thousands of decisions about what to do with inputs to get the desired outputs. Focusing on VOI is synonymous with focusing on how to use inputs more effectively to promote better (both in quantity and quality) outputs. The middle of the diagram depicts this as a process of prioritizing the use of inputs to increase competencies and capacities in areas that contribute to desired outputs. When inputs are used to increase the ability of an institution to produce desired outputs, we can say that the inputs (investments) have been used to add value to the institution.


Click image for larger view.

Pursuing VOI at Brigham Young University

The late 1990s saw considerable administrative support for infusing technology into teaching and learning at Brigham Young University (BYU). Behind this support was a desire to make teaching and learning more effective, to provide learning opportunities for more students, and to save money. There were significant points of disagreement, however, about which goals were most important and how they should be pursued. From an instructional technology perspective, difficult questions were raised about resource allocation and project prioritization. Should we focus on online course development? Building a sophisticated learning management system? Buying and implementing a less sophisticated (but more practical) CMS?

University leaders decided it was important not just to answer these questions but also to create a framework for answering them consistently in the future. Several key stakeholders and a new teaching and learning with technology “czar” (the TLT Czar) were assigned to tackle these issues. The brief retrospective of this effort provided here might give the false impression that we had a clear idea of the steps we would take from the day the journey commenced. That is simply not the case. Rather, we began by focusing on what seemed to be the most important first question—What is it that we really value?—and then moved on from there. The process is represented in Figure 2.


Click image for larger view.

Identifying and Agreeing on Teaching and Learning Values

The first step in strategically mapping an institution’s teaching and learning infrastructure is to reconcile current activities and projects with the institution’s mission and objectives. While it might seem grandiose and pedantic to make such a connection, doing so is vital in establishing a successful VOI process.

Our process began with a year-long project to define the university’s distributed learning strategy, starting with revisiting the university’s founding documents. Through extensive conversations with stakeholders from across campus and with input from an outside higher education consulting group (Collegis Eduprise), we articulated a strategy for teaching and learning improvement at BYU.

The strategy consisted of five specific goals:

1. Develop and support faculty instead of courses, enabling faculty members to effectively integrate technology into their teaching and learning efforts.
2. Develop and refine distributed learning models to promote wider adoption of technology-mediated instruction.
3. Encourage and empower departments to take strategic advantage of available models, tools, and resources.
4. Unify production and delivery of instructional materials across campus.
5. Effectively manage teaching and learning resources.

These objectives have since been amended and fine-tuned. We incorporated the Sloan Consortium’s Elements of Quality in Online Education4 standards into our evaluation of a TLT initiative’s value, for example. Changes made collaboratively have contributed to the maintenance of a commonly agreed upon, unifying statement of TLT value.

Note that none of our TLT value objectives refers explicitly to university outputs (graduation rates, for example). Rather, the strategy emphasizes areas of opportunity where effective TLT integration can improve the university’s capacity to achieve its loftier goals.

Coordinating Support Organizations

Once articulated, these objectives became a common touchstone for organizations across campus that support teaching and learning. This in turn prompted a willingness to coordinate and collaborate in new and important ways. Under the direction of the TLT Czar, a loose federation of support organizations was established, called the Teaching & Learning Support Services (TLSS) Group. Group membership consisted of leaders or decision makers from the library, the IT office, independent study, copyright, and the Center for Instructional Design. At first, the group focused almost exclusively on learning about member organizations—what they did, how they did it, and why they did it. As areas of overlap or common purpose emerged, group meetings rapidly evolved into coordination and collaboration sessions.

The power of the TLSS was not simply open communication, however, but communication with a purpose. Each member group had bought in to the five objectives of the distributed learning strategy, so the communication (and subsequent collaboration) of the TLSS focused on better achieving these goals as an institution. Most importantly, the strategy provides a consistent, shared definition of value and serves as a foundation for a consistent framework for evaluating, approving, and prioritizing new TLT initiatives.

Creating an Environment for Collaboration and Coordination

Supported by a unified strategy statement and consistent goals, the TLSS organizations collaborated and coordinated in ways not possible a few years earlier. Strategic budget plans were shared and reviewed openly at group meetings; resources (budgets and personnel) were transferred to address institutional priorities; old programs were altered to foster better coordination; and new programs were created to take advantage of new organizational synergies.

One of the most significant accomplishments of the TLSS was the establishment of a unifying TLT infrastructure blueprint (see Figure 3). The schematic represents the group’s efforts to accomplish goals 4 and 5 in the distributed learning strategy: “Unify production and delivery of instructional materials across campus” and “Effectively manage teaching and learning resources.” The blueprint serves as a basis for rationalizing the teaching and learning systems the university adopts and supports. It also provides various groups a common framework for thinking about components of the infrastructure and how they affect other organizations. This significantly reduces the temptation to freelance and build or buy applications that meet local needs but tend to thwart enterprise interoperability and collaboration.


Click image for larger view.

This blueprint has the virtue of focusing on the functions necessary to extend and improve teaching and learning, not specific applications. For example, it is critical to manage curriculum at the university and program levels. Technology solutions can make this a more efficient and effective undertaking, certainly, but the key functional needs are the most important variables in the analysis, development or acquisition, and implementation process—not specific applications or tools.

Similarly, the blueprint identifies content management, content transformation and publication, course content delivery, and other functions that must be supported by the teaching and learning infrastructure. This non-application-specific approach keeps the focus on adding value to teaching and learning activities at the university and mitigates inevitable tendencies toward tunnel vision and vendor-induced myopia. The blueprint thus serves as a teaching and learning strategy reinforcement mechanism, constantly returning focus and attention to the value (defined by shared goals) of current and potential teaching and learning applications and tools.

Again, note that the common framework for TLT investments (as represented in BYU’s schematic) does not explicitly focus on the university’s key outcomes or results. The purpose of a common framework is instead to focus resources and efforts on improving the institution’s capacity to deliver these results. The VOI of a TLT expenditure is measured by its direct contribution to the institution’s statement of teaching and learning competencies.

Institutionalizing a VOI-Driven Process for TLT

These three steps (agreeing on common values, getting organizations on the same page, fostering better collaboration) were essential preconditions for implementing a VOI-driven process for identifying, prioritizing, and completing TLT projects that consistently add measurable value to the university. All of these efforts would have been for naught, however, if we had not also implemented a new process for evaluating, approving, prioritizing, implementing, and evaluating TLT projects. We recognized the necessity of creating a commonly adhered to, consistent framework for assessing the value of TLT efforts and expenditures at all stages of a product’s life-cycle.

In our experience, a disciplined, rigorous process is essential to the institutionalization of a VOI-driven approach to TLT decision making. At BYU, we instituted three specific support disciplines to create and sustain a culture of VOI: product management, priority management, and project management.

Product Management

Each application and tool in the TLT infrastructure is a product defined by function, not vendor. Each product has a natural life-cycle that must be carefully managed. Product management oversees all aspects of a product to ensure it meets immediate user needs as well as the university’s long-term goals and objectives. Product managers work closely with key stakeholders to evaluate the potential or current value of each product in the TLT portfolio. Product managers function most effectively when integrated into the organizations that implement the products they manage.

A product portfolio manager takes the larger view of several product managers’ products, the interactions of those products, and the value the portfolio adds to the university. Both product managers and product portfolio managers monitor the TLT environment, the effectiveness of the current configuration of products, and the value the products individually and collectively add to the university. Product managers initiate project proposals when existing products need to be upgraded or replaced or new products need to be acquired to fill gaps.

Product management is an essential support discipline because university executives responsible for TLT generally lack the time and often the product-specific expertise to monitor product spaces and make detailed recommendations about them. BYU leaders rely heavily on product managers to help them identify needs and solutions in the TLT infrastructure.

Priority Management

The number of good TLT ideas will always outstrip the resources available at any institution of higher education. Consequently, decisions must be made about the comparative value of competing TLT product proposals. The centerpiece of an effective VOI-driven process is a disciplined, value-focused priority management team (PMT). This team evaluates, approves, and prioritizes TLT project proposals. These decisions are based on a well-defined set of objective scoring anchors that allow the team to assign quantitative scores to projects based on the value they will add to the university. The scoring anchors are derived from the objectives of the distributed learning strategy, the TLT infrastructure schematic (Figure 3), and the Sloan standards. The PMT not only evaluates and approves work on specific TLT projects but also ranks and prioritizes projects to allocate resources to the most important projects first.

A PMT should consist of university and organizational leaders responsible for realizing institutional objectives affected by the product portfolio for which the team makes decisions. For example, the TLT PMT at BYU consists of the director and associate directors of the Center for Instructional Design and the TLT product manager from the Office of Information Technology. The associate academic vice presidents for Undergraduate Education and Faculty are also ex officio members of the PMT. These individuals provide executive input and endorsement of the TLT team’s decisions.

Project Management

Projects differ from products in that projects have clear start-and-stop dates and target specific, limited, product-related outcomes. For instance, a CMS is a product, but the effort to bring a CMS online in a specific hardware and software environment is a project.

Once TLT projects have been approved and prioritized, project management makes possible successful execution of initiatives in priority order. Project managers employ a set of principles, practices, and techniques that help project teams stay focused on priorities established by the PMT. In particular, project managers enforce decisions about project schedule, cost, and scope. At the task level, project managers carefully track the progress of each project and report regularly to the PMT. Project managers also optimize resource allocation and use and facilitate effective planning and collaboration for project teams.

Putting It All Together

In addition to the three specific support disciplines, a successful VOI-driven process also depends on executive-level sponsorship and participation with the PMT. Each product and project needs a sponsor—someone who has the authority to approve the product or project and to assign resources to it. Most importantly, the sponsor also confirms that a proposed product or project adds value by improving key university competencies and capacities.

The product or project sponsor may be a department chair, a vice president, or another institutional leader who champions the product or project in the context of all other competing demands for resources and attention. The TLT project prioritization framework provides the sponsor with a consistent, objective rationale for supporting specific initiatives. The sponsor anchors the product or project in the VOI-driven evaluation, approval, implementation, and evaluation process.

In addition to working closely with the organization supported or served by specific products, a product manager achieves success through his or her working relationship with a designated sponsor or sponsors. This is especially true for product portfolio managers. It is critical that these individuals partner effectively with the appropriate sponsor or sponsors having institutional stewardship for their product portfolios.

As product managers maintain a value-focused eye on each product in the portfolio, they continually consider how an established product can be optimized within the context of other university systems. Working closely with product sponsors, the product manager helps create and implement value-driven product strategies.

The product manager also works with product implementers, users, evaluators, vendors, and user communities to keep up with current local and market trends, emerging deficiencies, and opportunities. Based on these observations, product managers initiate proposals for product enhancements or replacements. As institutional requirements change (as defined by the sponsor), the product manager adjusts accordingly.

Product managers play a critical role in the VOI evaluation and reporting process by carefully analyzing the performance and financial aspects of each product, including end-user evaluations, profit and loss statements, and total cost of ownership. At BYU, product managers also work closely with TLT support organizations to conduct thorough evaluations of each product’s contributions to key institutional competencies (as delineated in the prioritization process). This process assesses the past, current, and projected future value of each product in the portfolio.5 Channeling this information back into the process permits measuring and refining the value-effectiveness of products.

Does It Really Work?

The process described above might seem too good to be true. After two years of implementing this process, however, we can declare that it does, in fact, work. An example is in order. Like many institutions, BYU has grappled with the ROI of our CMS. In the past, this question focused on the kinds of difficult-to-measure factors we have cited—does our CMS improve student GPA, job placement, exit-exam performance, and so forth? By changing the focus to VOI, we instead seek to measure the value of our CMS by measuring its contributions to our institutional capacity for effective teaching and learning.

Through surveys, focus groups, and interviews, faculty and students overwhelmingly report that our CMS enhances their teaching and learning activities. Using this data, we have been able to “count” the value of our CMS. In the context of our distributed learning strategy objectives, we can say that we have:

1. Developed and supported faculty members who are transforming teaching and learning by using a CMS.
2. Developed and refined our distributed learning models (although this remains a work in progress).
3. Encouraged and empowered academic departments to use technology strategically to address curricular challenges and opportunities.

Considering the Sloan Consortium’s Elements of Quality in Online Education, we can also objectively demonstrate that the implementation of a CMS at BYU has:

1. Improved learning effectiveness—a claim backed by rigorous studies of key hybrid courses where the CMS is used to free up instructor time to pursue higher-order learning objectives in the classroom and in one-on-one student consultations.
2. Improved student access to learning materials and learning opportunities.
3. Improved student and faculty satisfaction with the learning process.

We do not have space here for a detailed discussion of each of these assertions or the data collected to support them. Our focus on VOI, however, has allowed us to objectively evaluate the value added to the university by licensing and implementing a CMS. Through this exercise, we have determined that the value gained is worththe dollars and time required to get it. Among other things, our faculty and students report that the CMS improves their ability to communicate (particularly outside of regular class or office hours times) and to organize and manage their courses more effectively.6 Additionally, students are using the CMS to collaborate, share ideas, and complete group projects more consistently and effectively than in the past. Perhaps most significantly, growing numbers of faculty use the CMS to manage the didactic elements of their courses and low-stakes assessment (quizzes) so that they can spend in-class time on more intensive discussion, analysis, and synthesis. During the past four academic semesters, students at BYU have completed an average of more than 20,000 quizzes a day using the CMS. These and other kinds of activities are occurring on our campus at significantly higher rates because we have implemented a CMS.

A significant caveat is in order, however. While a VOI approach can help institutions avoid the pitfalls of vainly trying to measure ROI, value judgments must still be made. VOI does not promise a purely objective evaluation and decision-making process with regard to TLT. For example, we have debated (and likely will continue to debate) the dollar and time value of the benefits derived from our CMS. Are our CMS benefits worth $100,000? $500,000? $1,000,000? Another obvious factor is that wise administrators will seek to get as much value from their CMS (and other teaching and learning technologies) for as little money as possible. Consequently, we are constantly evaluating and considering alternatives to our current CMS that might return the same or similar value at a lower cost. In the end, though, a VOI approach has enabled us to make more rational, well-informed decisions about the real, measurable value of our CMS and the resources the university is willing to commit (now and in the future) to a product that produces that value.

Conclusion

We have outlined a replicable VOI process for making decisions about TLT in higher education. We readily admit, though, that no process or set of guidelines can yield a perfectly objective framework for evaluating the relative value of TLT projects. However, we believe BYU’s VOI process provides a much more consistent and objective framework than many TLT decision makers believed possible.

As you contemplate implementing a VOI process at your institution, we reiterate the importance of broad, open communication and patience. It will take time to get the right people involved in the conversation (at multiple levels both in the administration and the faculty). It is also critical to articulate and build consensus around a framework of goals against which products (and associated projects) can be evaluated and prioritized. We do not believe it is necessary to adopt the same structures and processes that we have at BYU. Nonetheless, you will need discipline, rigor, and consistency to define, implement, and sustain a VOI process that will enable decision makers to look at the same data through the same set of lenses and make widely agreed upon decisions about TLT.

Our experience tells us that with the right people involved and with a commitment to follow accepted objectives and processes, it is imminently possible for institutions of higher learning to consistently measure the relative value of potential and existing TLT products and projects. Doing so enforces a consistent focus on adding value to the core business of the university and mitigates against arbitrary and less-effective resource allocation.

Endnotes
1. R. Weir, “IT Investment Decisions that Defy Arithmetic,” EDUCAUSE Quarterly, Vol. 27, No. 1, 2004, pp. 10–13, <http://www.educause.edu/apps/eq/eqm04/eqm0412.asp>.
2. K. Harris, M. Caplan Grey, and C. Rozwell, “Changing the View of ROI to VOI—Value on Investment,” Gartner Research Note, SPA-14-7250, Nov. 14, 2001.
3. D. M. Norris, “Value on Investment in Higher Education” (Boulder, Colo.: EDUCAUSE Center for Analysis and Research, Research Bulletin, Issue 18, 2003). According to Norris, “tactical” investments also tend to anchor organizations in the status quo because they usually focus on incrementally improving the efficiency and probability of existing processes and practices.
4. J. Bourne and J. C. Moore, eds., Elements of Quality in Online Education: Practice and Direction (Needham, Mass.: The Sloan Consortium, 2003).
5. TLT support organizations also work with university leaders and the Office of University Assessment to evaluate the connection between these competencies and key university outputs.
6. Data from our CMS studies about increases or decreases in faculty time spent on teaching is inconclusive. Some faculty members report spending less time managing their course with a CMS while others report spending slightly more time. Still others report no change in the amount of time they spend teaching with a CMS. On average, the CMS does not appear to increase or decrease the amount of time faculty spend managing their courses. We anticipate that this will change over time as faculty members become more familiar with the CMS and use it more efficiently.
Jonathan D. Mott (jonmott@byu.edu) is Director of the Center for Instructional Design, and Garin Granata (garin_granata@byu.edu) is Enterprise Product Manager, Teaching & Learning, at Brigham Young University in Provo, Utah.

Jon Mott

Dr. Jon Mott serves as the Assistant to the Academic Vice President – Academic Technology at Brigham Young University where he provides strategic guidance on academic technology issues. He is an Adjunct Professor of Instructional Psychology & Technology and also teaches in the Masters in Public Policy Program. Jon has been developing and teacing online courses since 1995. He is the former Managing Director of the Center for Instructional Design (now the Center for Teaching and Learning) at BYU. He earned a B.A. in political science from BYU in 1992 and a Ph.D. in political science from the University of Oklahoma in 1998. He has worked as a legislative assistant in the Utah State Legislature and the United States House of Representatives. Jon and his wife Kim are the founders and co-developers of ThisNation.com. Jon and Kim live in Utah with their four children.

 

Garin Granata

Product Line Manager
Brigham Young University

 

Most Popular

Stay Up-to-Date

RSS Email Twitter

Share Your Work and Ideas

Issues coming up will focus on designing the future of higher ed, digital engagement, and new business models. Share your work and ideas with EDUCAUSE Review Online.

E-mail us >

Purchase

EDUCAUSE Members: $4.00
Non-Members: $4.00
Close
Close


Annual Conference
September 29–October 2
Register Now!

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

Digital Badges
Member recognition effort
Earn yours >

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2014 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.