< Back to Main Site

EDUCAUSE review onlineEDUCAUSE review online

Do-It-Yourself Metrics

0 Comments

Good Ideas
Do-It-Yourself Metrics
Metrics have arrived—which means we need a practical approach to implementing them

Something new is on the horizon, and depending on your role on campus, it might be storm clouds or a cleansing shower. Either way, no matter how hard you try to avoid it, sooner rather than later you will have to deal with metrics.

Metrics don’t have to cause fear and resistance. Metrics can, and should, be a powerful tool for improvement. Before we can intelligently discuss the proper use of metrics, however, we must come to a common understanding of what a metric is and how to create one.

What Is a Metric?

Most metrics are created, collected, and reported to satisfy a leader’s request. The leader’s role is to supply clarity and direction by providing the proper questions. Middle management’s role is to answer the questions. Metrics offer a means of providing the answers so that all involved can have faith in them.

Unfortunately, leaders often don’t know exactly what they want. Chances are you have played the Guessing Game with a leader, where the data you provided wasn’t what he needed, so he asked for different data, figuring he would know the right data when he saw it. Despite repeated failures, you continued to chase data as if all the effort invested in collecting the wrong data would eventually prove worth your perseverance. There is a better way.

Speaking a Common Language

Defining a metric requires a common language. Data, measures, information, and metrics are distinctly different for our purposes. Using an IT help desk as an example, we can demonstrate those differences:

  • Data: The simplest/lowest unit available. Data represent “raw numbers” and are of little to no use alone.
    • Number of trouble calls
    • Number of employees
  • Measures: A little deeper view that builds on the data. Measures are rarely useful alone.
    • Number of calls per hour
    • Number of cases closed by worker
  • Information: Usually a comparison. This level of abstraction serves as a useful indicator.
    • Number of calls for each hour compared to number of workers on a shift
    • Average length of time to close a case, grouped by type
  • Metric: Tells a complete story. It incorporates information (built of measures and data) to answer a question fully. Normally, a metric is conveyed through a graphical representation and explanatory prose.

To understand the relationships among the components that make up a metric, imagine a metric as an oak tree: it has a massive trunk, and the leaves and branches provide shade and comfort. Data are analogous to the leaves on the tree. Numerous and abundant, they are interesting to look at, easy to get, and serve a purpose; but by themselves they are not very useful and will not survive once removed from the branches.

The smallest and thinnest branches represent measures—they provide an essential connection between leaf and tree (between data and the metric), although not substantial or robust enough to create anything on their own. The thicker, inner branches, the limbs, are like information. Useful at times in themselves (for supporting tire swings or tree houses, for example), they die if taken away from the tree. Information without a connection to the trunk will fail to reach its potential.

The trunk of the tree, where you can determine its age and strength, represents the metric—a picture telling a complete story. The metric may consist of many pieces of information, derived from many measures and data. However, even the largest tree will wither and die without roots.

The roots of the tree represent the questions the metric is designed to answer. As with the oak, the roots define the type of tree it will become, where it will live, how strong it will be, and if it will survive a harsh environment. The roots are born of the original seed (need) and spread out, providing a life-giving foundation for the tree. Even if you cut down the tree, the roots will continue to spawn new growth. Unless the root question is no longer necessary (the tree is uprooted), you will continue to need data, measures, information, and metrics to feed the root need.

To aid in the process of properly building our metric tree from the roots up, rather than picking leaves (data) and branches (measures) off the ground trying to create a tree, we use an implementation guide. This straightforward template allows us to focus our energies on the root need.

Creating a Metric

To best use the concept of storytelling, metrics require a level of structure and rigor. The most effective and useful metrics are designed with the end in mind. Focusing your efforts up front ensures that you don’t waste time, money, or good will in collecting inappropriate data or in creating a flawed metric.

The implementation guide helps in the planning, documentation, and implementation of a metric. The guide takes basic components of a metric and lays them out for completion. It also holds the keys to building a successful metric. Continuing the mighty oak analogy, think of the implementation guide as a root stimulator. It consists of the following parts:

Executive-Level Summary: One to two paragraphs about the metric. Although it comes first in the guide you are putting together, we recommend capturing the executive summary content after you have completed the rest of the guide. It should include a definition, summary, and history of the metric (the question and the answer).

Purpose: The most important part of any metric. What question are you trying to answer? What is the root question? Why do you want to tell the story? What do you hope to achieve? This is so important that it is a go/no-go proposition. If the purpose is not clear, stop. If you cannot clearly define the root question and identify how you will use the metric, stop. An extra test is to identify how the metric will not be used. It helps if the person asking the question is open to the possibility that something other than a metric might satisfy the need more effectively or efficiently.

Success Key 1: If you don’t know the purpose of the metric, don’t collect data.

Success Key 2: If you don’t know why you’re collecting data or reporting a metric, stop.

Customer: Several possible, beginning with the “root” customer who provided the root question—the obvious recipient of the final metric. There are other customers for any metric, however. If you have people collecting data, it improves quality and accuracy when they also get to see the fruits of their labor. Don’t forget yourself—if your boss asked the question and you’re in charge of developing the metric, you should be interested in the answers, too.

Graphical Representation: A hard point to grasp for many. Rather than describe the data wanted, ask the customer to describe how they would like to view the story. Incorrectly focusing on the data instead of a metric focuses on the answer wanted instead of the question. Obviously, a focus on the answer biases the building of the metric and what it should explain. This leads to a tainted and limited view, which leads to chasing data. The graphic represents a guess at how to tell the story and at the charts or graphs to use. The graphic can be a trend, Pareto, benchmarking, bar, line, dashboard, or other type of representation.

Success Key 3: Don’t chase data: determine the question regardless of the answer.

Explanation: A prose version of the story the metric tells, explaining how to read the picture. Remember, this is only a guess. If you identified the root question well, the explanation of the answer should be evident.

Schedule: Large steps in the metric’s lifecycle. Will you start collecting data at the beginning of the school year, calendar year, or fiscal year? When will you make reports available? Finally, when will you stop collecting data? Or, asked another way, when will this metric cease to be useful?

Did we surprise you with that one? A metric has a purpose. Its original purpose can change or be overcome by events. A metric is not eternal, although your organization probably collects (and maybe reports) measures no longer used by anyone. The purpose has passed, but alas, the effort continues. Write an expected lifespan in this section. Explain how you’ll know when you can stop collecting and reporting this metric.

Measures: Time, finally, for the leaves and branches. Identify the specific data to be collected and used to develop the metric. Target the lowest-level view of the data. Nothing is eternal—the purpose, question, and data used to create the metric can, and most times should, change.

Collection: Time at last for the processes. Now you can document the processes and procedures used to actually collect the data. Be as detailed as possible—you are creating a guide for the collector to follow. Include the collector (person/role), source of the data, frequency of collecting or reporting of the data, and method of collection. Document the process for collecting the data.

It will help immeasurably if you can collect the data with as little human intervention as possible. Any time you can automate the collection process, do so. Not only do you reduce the risk of human error (inherent in anything humans do), you also minimize bias—intentional and unintentional. The less intervention, the less pain for your busy workforce. The less intervention, the less chance of human error in the collection.

Analysis: Now for the story. All the assumptions, constraints, and known flaws around the information go into the story. Document any formulas or mathematical equations needed. You might want to enlist a statistician. Many times the data will tell you how to proceed. If you’ve done the job well and identified the root questions, built a picture, and then worked on the data required, you can now allow the data to dictate—to a degree—what to do next.

Threshold and Target: The range of acceptability or expectation. Any results better than the threshold and below the target are acceptable. Any results below the threshold dictate further investigation to find out if the causes can be avoided or the processes improved. Any results above the target dictate further investigation to find out if the causes can be replicated or leveraged.

Lessons Learned: Time to get your money’s worth. Document your lessons learned, and plan to visit this section of the implementation guide periodically so that you don’t end up with a metric that outlives its usefulness, draining valuable resources past the need.

Proper Use of Metrics

Metrics offer a powerful tool for improvement. They can provide a vehicle for communication, insight for planning, and visibility for decision making. A well-thought-out metric can be a valuable asset for attaining goals and predicting the future. Metrics can help determine an organization’s health and whether its products and services align with the organizational mission and vision. In short, metrics can help leaders ensure they are doing the right things, the right way (preferably the first time).

With the power metrics provide comes responsibility. Let’s examine the risks. Metrics (and data to a greater degree) are extremely easy to misuse. Management could try to solve problems prematurely, making decisions without investigation. Someone can look at a metric and decide that it is the “whole truth.” Too often metrics are used to justify a personal agenda instead of answering a question. Misuse will create a culture of distrust, encourage workers to make the data inaccurate, and foster an atmosphere of secrecy and passive-aggressive behavior.

Success Key 4: It is not enough to ensure you don’t misuse data; you must convince everyone involved that you won’t misuse it.

The misuse of data can make all of your well-intentioned efforts worthless. Worse, it can make future communications, trust, and metrics difficult, if not impossible. Each step of the way, ensure that you are a good steward of the information you gather.

Metrics are only a tool, a means to an end, not the goal. Because metrics, like anything derived from information, contain a certain amount of error (variance), the only proper response to a metric is to investigate. Metrics in their fullest glory are still just indicators to help answer a question. You must be careful not to accept the metric without critical review. Before you act, before you make a decision based on a metric, investigate and ensure that it is telling you what you think it is. A metric provides additional information, in a structured format, but it is still just information—not a truth. You must investigate to find the truth and ensure that you make the right decisions.

Of course, some metrics won’t require further investigation, like a metric used to give you an additional level of comfort. Before making any important decisions, however, you should make sure that you are not only answering the question but answering it accurately and truthfully.

Success Key 5: The only valid response to data (or metrics) is to investigate!

Conclusion

We had three goals in writing this article. We wanted to:

  • Introduce the concept of metrics as a form of storytelling.
  • Promote the use of tools, specifically an implementation guide to provide structure and rigor.
  • Raise awareness of the benefits and pitfalls of metrics.

When discussing metrics, remember the mighty oak analogy; it’s an easy and powerful way to explain the differences and relationships that exist among the components that make up a metric. When done properly, a metric tells a complete story and answers one or more questions. Since the question is at the root of the metric, start there and never lose sight of it. (See the sidebar on OIT Metrics.) The goal isn’t to develop the perfect data set or metric; the goal is to answer the question.

Metrics are never an end in and of themselves—they offer a way to focus your investigation. While nothing can guarantee success, the five success keys combined with a fully prepared implementation guide will help you avoid the most dangerous pitfalls.

Martin Klubeck (klubeck.1@nd.edu) is Strategy and Planning Consultant, Planning and Programs; Michael Langthorne is Project Manager; and Don Padgett is Program Manager, Strategic Initiatives, in the Office of Information Technologies at the University of Notre Dame in Indiana.

Martin Klubeck

Martin Klubeck is a Strategy and Planning Consultant at the University of Notre Dame, where he plays a significant role in strategic planning, metrics, and organizational improvement efforts within the Office of Information Technologies and across campus. Recent endeavors include speaking at the 2005 Leadership Conference hosted by Notre Dame, teaching courses on strategic planning and metrics at the University, and performing personal coaching. He holds degrees in Instructional Systems Technology, Computer Science, and Human Resources Development. He is currently working on a Doctor of Philosophy (PhD) in Organization and Management. He has presented at numerous conferences and seminars, and taught courses on organization development throughout the United States. His first book, "Why Organizations Struggle So Hard To Improve So Little: Overcoming Organizational Immaturity" is now available from ABC-CLIO publishers and his new title, "Metrics: How To Improve Key Business Results" is available from Apress Publishers, Dec 2011 in both paperback and Kindle versions.

 

Michael Langthorne

Michael's career began in television news in 1977, and shifted in 1986 when he became the Associate Director of the Educational Media department at the University of Notre Dame. Over the past 25 years he has held a number of positions within at ND, but has been specializing since 2001 on project management and strategy & planning.

 

Don Padgett

Don Padgett is a Program Manager for strategic initiatives at the University of Notre Dame, where his key focus is supporting IT strategic planning, metrics and organizational development efforts. He has also served as project manager on the Central Data Center upgrade project.and co-lead in the design, roll out and implementation of the University's Campus Workstation Program (CWP). CWP with an annual budget of 1.9 million dollars provides over 5000 faculty and staff with renewable desktop workstations that allows them to keep pace with technology advancements. Don earned his degree in Education from the Ferris State University; prior to joining the University in 1996 he spent 20+ years in the IT industry holding a variety of positions including sales, sales management and District Product Manager for digital imaging and workflow

 

Most Popular

Stay Up-to-Date

RSS Email Twitter

Share Your Work and Ideas

Issues coming up will focus on designing the future of higher ed, digital engagement, and new business models. Share your work and ideas with EDUCAUSE Review Online.

E-mail us >

Purchase

Close
Close


Annual Conference
September 29–October 2
View Proceedings

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

Digital Badges
Member recognition effort
Earn yours >

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2014 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.