Big Data

What is “big data”? Who’s doing it? Why is the higher education community so interested in it?

According to Wikipedia, “‘Big data’ is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set.”

Currently, big data is most commonly used by businesses and will become the “key basis for competition,” according to the McKinsey Global Institute’s May 2011 report “Big Data: The Next Frontier for Innovation, Competition, and Productivity.” They point out five ways that big data can create value. And while this new abundance of data will help businesses make data-driven decisions and enhance target marketing of products to consumers by using data to identify pregnant women, for example (see “How Companies Learn Your Secrets,” New York Times, February 16, 2012), we are still in the early stages of data collection. While many are excited by the challenge of exploring and analyzing the abundance of collected data and applying it to improve daily interactions, others are leery of where this data explosion could lead. In “The Future of Big Data,” a recently published report by the Pew Research Center’s Internet & American Life Project, experts provide varying viewpoints on the use of big data and the future. Businesses are the most enthusiastic in their use of big data, but the federal government is also jumping on the data train. The Obama administration has issued plans to dive into the big data pool with a $200 million initiative spanning six federal departments in the hopes of “improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help solve some the Nation’s most pressing challenges.” If the federal government can manage such massive data collections and successfully promote this undertaking, this should motivate other areas of society to take up the challenge of using large data sets to improve various industries and the lives of their consumers.

Higher education is just beginning to explore how to leverage the massive amounts of data it collects to improve the student experience. In his article “The Rise of Big Data,” Louis Soares explains that giving students an opportunity to see their data “in useful ways…can allow students to become better managers of their own educational experiences and can also, perhaps, improve collective outcomes across all of higher education.” Students are starting to see that data and use it to their advantage to keep themselves on track academically, as discussed in an EDUCAUSE 2011 session, “Course Signals: A Student Success System/Stoplights for Student Success.” As we find ourselves in a vast wonderland of data gathering and analyzing in higher education in the next few years, it will be exciting to see how big data can meet the promise of reducing institutional costs, increasing college affordability, and improving student retention and completion.