This paper is the intellectual property of the author(s). It was presented at EDUCAUSE '99, an EDUCAUSE conference, and is part of that conference's online proceedings. See http://www.educause.edu/copyright.html for additional copyright information.

Assessing the Impact on Students of Online Materials in University Courses

In the Fall Term 1998, the University of Pittsburgh conducted a pilot project to study the instructional impact and support needs for a Web-based instructional management system. The system centered on the CourseInfo software package acquired from Blackboard, Inc. The pilot was the result of a nine-month campus-wide software selection process, and a four-month effort to design and develop training and support systems.

This paper describes the project, explains the assessment strategies designed for the pilot, and summarizes the results of formative and summative evaluation measures. The assessment includes data collected from students and faculty members in 20 courses (22 sections) with 1,850 student seats.

The University of Pittsburgh is a state-related public research university and a member of the Association of American Universities (AAU). Founded in 1787, Pitt offers graduate, undergraduate, professional, and continuing education programs through 19 schools and four regional campuses. The University typically serves about 28,000 FTE in the fall term, and employs nearly 4,000 full- and part-time faculty members. Pitt's regional campuses are located between 30 and 170 miles from the main campus in Oakland, Pittsburgh’s cultural and medical center. Among the notable sites on the Oakland campus is the 42-story Cathedral of Learning with its 28 nationality classrooms, each representing the heritage of one of the region's many ethnic groups.

The Center for Instructional Development & Distance Education (CIDDE) works directly with schools, departments, and individual faculty members to facilitate the academic goals of the University in the areas of instructional development and technology, media support, faculty development, and distance education. With this broad mission of instructional development and support, CIDDE pursues new initiatives that will optimally support the academic priorities of the institution. These initiatives fall into three general categories: the acquisition or development of tools and facilities, the development and delivery of instructional support services, and the provision of assistance in the development of academic programs and instructional materials.

During the summer of 1997, the Director of CIDDE met with 20 deans to identify their instructional technology and distance education needs and priorities. Based on these discussions she identified three common needs: (1) for computer-based conferencing, (2) for a method to easily create online materials, and (3) for a consistent "look and feel" to the University's online course offerings.

Based on this identified need, a CIDDE instructional designer organized and chaired a committee to evaluate emerging software tools designed to facilitate the development and use of the World Wide Web for instruction. The committee included faculty and staff from CIDDE, the centralized computing services groups, and the University Library System, who volunteered to review existing software, and to try out selected packages to evaluate their potential utility. The committee articulated detailed requirements for a Web-based instructional package, identified the packages that addressed those needs, and ranked and rated each of the packages. The most promising five packages were selected for detailed evaluation, and trial copies were either accessed online or obtained, installed and tested by faculty in actual classes for the Spring Term 1998.

In May 1999, after nine months deliberation, the committee recommended that the University acquire the CourseInfo instructional management system. The recommendation was presented, approved, and funded through the Software and Networked Information Working Group of the Executive Committee for Academic Computing. The work of the Web evaluation committee is summarized in a web page at URL http://www.pitt.edu/~washburn/ccs.html.

CourseInfo is an integrated software suite that enables faculty members to develop, manage, and use Web pages for instruction. CourseInfo acts as a "container" for organizing the instructional components of an online course. It offers tools to administer, grade, and record quizzes, to conduct synchronous chat sessions, and to hold asynchronous threaded discussion groups. It also provides user-friendly software to facilitate the instructor’s efforts to create new Web pages or incorporate existing pages, send e-mail, transfer files to and from students, and manage an online grade book. CourseInfo runs in both the Unix and NT Web server environments and is accessible using standard Web browsers and plug-ins.

The primary factors driving the selection of CourseInfo included its ease of use and its adherence to standards such as SQL and the emerging Instructional Management System metadata model (the EDUCAUSE IMS Project). CourseInfo was the only package previewed that enabled inexperience faculty members to create effective materials with no knowledge of HTML, yet still allow more advanced users to leverage their knowledge of HTML and Web-based authoring tools to produce sophisticated results.

Project implementation was a cooperative effort between CIDDE and the centralized computer service units (the Computer Services and System Development unit and the Network Services unit), coordinated through a series of regular planning meetings and advised by several faculty committees. This section explains some of the important factors in the project implementation plan.

The University acquired multiple licenses of the CourseInfo package and implemented three servers in order to create a stable, secure and efficient environment. First, CIDDE implemented a test version of CourseInfo under Apache on a low-end Pentium processor running Linux. The test version was used to experiment with new releases and patches of the software. A second "development" server was implemented on a small Sun processor running Solaris and Apache. Faculty members used this server to develop their Web pages in a secure, confidential environment. The centralized Network Services group managed the third server, a high-end Sun processor also running Solaris and Apache. This "production" server was maintained seven days a week, 24 hours per day, and supported the online courses, faculty, and students.

In response to concerns raised by several faculty committees about the sufficiency of remote access to the Pitt network, the Network Services group increased the dial-in modem pool by 40% to over 800 dial-in ports. In addition, to handle the anticipated increase in student and faculty problems that might occur when implementing Web-based instruction, the Computer Services and Systems Development group expanded the existing single-shift help desk to a 7 ´ 24 environment. The group also took proactive steps to prevent problems by ensuring that all requisite plug-ins were available in the public student computing labs and providing in-class orientations to students at the beginning of the term.

Another potential issue was the anticipated increased desire by instructors to use computer projection technology in the classroom to demonstrate components of CourseInfo to their students. In response to this issue, the Classroom Management Team escalated the rate of classroom renovations to install computer projection devices in more classrooms. CIDDE’s Instructional Media Services area also increased their inventory of portable microcomputers and projection units for mobile classroom presentations, and hired and trained additional student AV operators.

To increase the likelihood of the project’s success, administrators in the Provost’s office gave two important directives. First, they made a strong commitment to training faculty in the use of CourseInfo, and second, they limited the number of courses that would be offered in the pilot term. For the Fall Term 1998 (99-1) pilot, 20 faculty members developed 20 courses (22 different course sections) serving 1850 students. The goals of the pilot were to evaluate CourseInfo and to determine the access and support needs of implementing Web-enhanced instruction.

Training for the pilot term was supported under the auspices of a newly proposed Summer Instructional Development Institute (SIDI). SIDI was initially conceived to provide intensive training and development opportunities for faculty over the summer. For its inaugural season, the Institute focused on training two groups of 12 faculty members in the use of CourseInfo. The first group was selected by the chairpersons of five science departments identified by the dean of the College of Arts and Sciences. This selection specifically targeted instructors of large, predominately freshmen courses with a goal of evaluating the effectiveness of Web-based course components in improving the experiences of students in large lecture settings (greater than 100 students per class). The second group was selected by the deans of schools and campuses that had expressed strong interest in the project. The faculty members involved were from the Faculty of Arts and Sciences (the Biology, Chemistry, Psychology, Geology, and Physics departments), the School of Education, the School of Nursing, the School of Pharmacy, the School of Health and Rehabilitative Sciences, and from the Johnstown and Bradford regional campuses.

The Institute covered basic course design topics and featured hands-on instruction related to the mechanics of using CourseInfo and other related software tools. The courses were designed and taught in collaboration with the centralized computing services and library groups.

After analyzing CourseInfo and its anticipated training needs, several key decisions were made. First, training for CourseInfo would be implemented using CourseInfo. That is, faculty members would be enrolled in a CourseInfo "meta course" that would (1) serve as an exemplar of desired CourseInfo organizational structure and concepts, (2) serve as a repository of training and reference material, and (3) provide faculty with the opportunity to gain experience with CourseInfo from the students’ perspective.

The second decision was to focus CourseInfo training more on the instructional design requirements of creating a Web course than on the mechanics of using the software package. Training would therefore promote the use of a straightforward instructional design model, originally developed at Pitt’s Learning Research and Development Center and later tailored to meet the needs of the University’s paper-based distance education program. This approach was possible because of the relatively easy task of training instructors in the use of CourseInfo, due to its friendly user interface.

The final decision was to provide faculty course developers with a recommended structure in the form of a "template" course. In this approach, CourseInfo is viewed as a "container" of instructional materials. These materials are contained in documents and folders under appropriate subdivisions within a CourseInfo course site. Native CourseInfo ("out of the box") gives instructors a blank slate to contain any type of document in any structure or order. CIDDE instructional designers developed the "Pitt CourseInfo Template" to provide a more-specific structure to CourseInfo courses. This structure is intended to provide consistency across different courses, allowing students familiar with one CourseInfo section to easily adapt to a second.

The Pitt CourseInfo Template specifies and explains a suggested structure for the "Course Information" and "Course Documents" components of CourseInfo. The Course Information component is considered analogous to a course syllabus and defined to include the following sections (documents):

Whereas the Course Information component of CourseInfo contains information about the course, the Course Documents component contains the course materials themselves. The template suggests that the course materials be subdivided into manageable units of work. These units, or modules, could be associated with chapters of a text, weeks of the term, or concepts of the course. Each module would be implemented via a separate folder within the Course Document component of CourseInfo. Each folder is defined to include the following documents:

The student would complete the course by working through each module in turn until all modules where mastered.

CourseInfo training was designed around the process of implementing a Web-based course using the Pitt CourseInfo Template. For the Summer Institute, a five-day modular training program was developed that covered this process and addressed other related tools and issues. This five-day sequence was later divided into more discreet components. A one-day core course was a prerequisite for participation in CourseInfo. The core course, titled "Course Design And Development Using CourseInfo," addressed the following topics:

In addition to the required core training, several supplemental training modules were developed and offered to faculty. Participation in these modules was optional, base on each faculty member’s goals and interests. The supplemental modules included:

Upon the completion of the Summer Institute, the pilot program was launched in the Fall Term 1998. Data from subsequent terms reflect significant growth in the faculty’s adoption of CourseInfo (See Table 1, page *).

Evaluation of the pilot program was a crucial component of the project plan. To create an evaluation strategy, two instructional designers and a research design consultant formed a project assessment team. The team designed assessment goals and strategies based on the need to gain feedback from both students and faculty on issues related to access, support, and the instructional value of using CourseInfo. The evaluation team reviewed The Flashlight Project’s student inventory items as a basis of reference, drawing upon this pool of questions to develop an instrument to meet specific evaluation goals.

Because the primary goal of formative evaluation was to identify any serious problems that students might encounter with accessing and using CourseInfo, early intervention was important. Therefore, in mid-October, CIDDE staff visited a representative sample of nine of the 22 course sections using CourseInfo and administered one-page paper surveys to all of the students attending class. Of the nine courses included in this evaluation, several were high enrollment, introductory science courses (the focus of the first Summer Institute). The sample also included undergraduate and graduate courses, low enrollment courses, and those offered on the Pitt’s main campus and regional campuses. Out of the 1850 students using CourseInfo during the pilot term, 485 students were surveyed for the formative evaluation.

These data indicated that most students (92%) had already accessed the CourseInfo pages for their courses (by the sixth week of the term), and that more than half of them (59%) had not previously used a Web-based tool. The students indicated that the initial in-class explanations they received from the University computer support area or from their instructor on how to use CourseInfo were adequate (95%). Suggestions for improvements included providing handouts and conducting the training sessions in computer labs. Students reported very few difficulties using CourseInfo (87% of students surveyed did not contact the 24-hour help desk with questions/problems), and those who did have difficulty were able to obtain the needed assistance. Data from the help desk indicated that most student questions focused on how to configure their personal computers for dial-in access or how to establish computing accounts.

One of the ten questions in the formative survey addressed the instructional value of CourseInfo, resulting in 69% of the students reporting that CourseInfo materials had helped their learning to a moderate or high degree. Additional positive student comments as to what CourseInfo components were helpful included "Lecture notes available online," "Checking my grades," "Assignments available online," and "Taking quizzes and practice exams." Only 4% of the students identified problems or concerns about using CourseInfo, specifying problems such as "access from home using a modem" and insufficient "computer lab hours."

One of the main goals of this initiative was using instructional technology to enhance teaching and learning. Consequently, the summative evaluation focused on the instructional value of CourseInfo. The primary evaluation objective was to identify which components were helpful to students’ learning and why.

Evaluation measures used during the pilot program included surveys, anecdotal data, help desk reports, and Web server statistics. Similar to the formative evaluation procedures, staff administered paper surveys to students in classrooms during the last two weeks of the term. Of the 1850 students enrolled in these courses, 796 surveys were returned.

The data indicated that students were accessing their course Web pages primarily through University student labs (436 responses), residence halls (125 responses), and via modem (263 responses). See Table 2, page * for Web server access statistics.

Students were asked to identify which components were helpful (of those used in their courses) and why. A content analysis of students’ open-ended responses was completed, and the following summarizes these data. It should be noted that not all CourseInfo components were used in every course. For example, some courses did not use group work tools at all. The number of students’ open-ended comments for each item is reported in Table 3 through Table 9.

Three of the most highly used components were also the ones that students rated as most helpful: Lecture Notes, Quizzes/Practice Exams, and Course Announcements (see Table 3, Page *). Other components rated as helpful included: Assignments, External Links, and E-mail. Student responses were very low for the remaining components (Group Work, Students Drop Box, Online Discussions, Surveys and Online Chats), probably because instructors used them only minimally during the pilot term.

Lecture Notes in CourseInfo took a variety of forms, including basic lecture outlines, detailed instructor notes, and PowerPoint slides. Students listed these as a major asset to their learning. The majority of students responding to this question indicated that notes enabled them to better organize information before class, allowing them to spend more time listening to the lecture. Students also reported that the notes helped them clarify and organize the information they received in class. Many students reported using the lecture notes for study and review of class information and preparing for exams (See Table 4, page *).

As expected, students felt that Quizzes and Practice Exams helped them to prepare for their in-class supervised exams and focus their study efforts (see Table 5 page *). Although these responses may reflect the availability of quizzes and practice exams, per se, and not the accessibility of them through CourseInfo, the features of immediate scoring and feedback through CourseInfo were clearly valuable to students.

Predictably, students commented that Course Announcements were helpful for keeping them informed of course-related events, such as assignment due dates, information about up-coming tests, and any changes in the course procedures. Course Announcements was the most highly used component in CourseInfo (reported by faculty as used "extensively"), perhaps due to the ease of application by course instructors. Also, students unable to attend classes noted that Course Announcements helped them to keep informed (see Table 6, page *).

Students responded positively concerning the ease of access to assignments through CourseInfo (see Table 7 page *). The instructional value of the practice gained from these assignments rated high as well, with students commenting that they had a better understanding of what was happening in class by completing these assignments. As with Quizzes and Practice Exams, it should be noted that the students’ responses could be addressing the assignments, and not necessarily the added value of accessing them online. Many students explicitly commented on the usefulness of having these assignments online, however, as this procedure helped them to stay informed of due dates and follow a schedule for preparing and submitting assignments.

Nearly two-thirds of the students commented on the ease of access that CourseInfo provided to additional and helpful information through External Links (see Table 8, page *). About one-third of students responding to this question specifically described that these links were helpful in completing homework and assignments.

Students were positive in their responses about e-mail communication (see Table 9 page *). Over two-thirds of the students responded that e-mail through CourseInfo enabled them to communicate more easily with their instructors--asking questions and conversing about the course. Almost 20% commented that the ease of communication through e-mail was helpful, while others reported that it helped in contacting other students and to keep informed.

During the pilot term, instructors did not significantly use the Online Discussion, Group Work, Surveys, or Online Chat components of CourseInfo. Several instructors reported using these components only "minimally" and, therefore, these data will not be reported.

Students were asked to compare the course they were currently taking, using CourseInfo, with other courses not using CourseInfo. In this comparison, students responded (either agreeing or disagreeing) that with CourseInfo they were more likely to perform 13 different instructionally-related activities. Of those students for whom these questions were applicable, Table 10 presents these data. The degree to which students reported using these activities varied from the highest rating of over 87% ("refer to course syllabus material") to a low of 58% ("work on assignments with other students.")

The two highest ratings were related to the access of online materials. Over 87% responded that they were more likely to "refer to course syllabus material," and 85% agreed that they were more likely to "access other online materials related to the content of this course" when compared with other courses not using CourseInfo. The next highest ratings related more to instructional activities and indicated that with CourseInfo, students were more likely to "review lecture notes to gain clarification," (85%), "complete assignments on time" (nearly 81%), "seek clarification when I did not understand something" (77%), and "actively participate in the course" (76%).

Other preferred activities included receiving feedback on quizzes and exams quickly, communicating with the instructor, spending more time studying for the course, discussing ideas from this course with other students, discussing ideas from this course with the instructor, and working on assignments with other students

Finally, students were also asked to agree or disagree with several statements as to the impact that certain online enhancements had on their learning. The highest positive response rate was to having CourseInfo used in other courses with over 90% agreeing. Almost 86% of the students agreed that online quizzes were helpful to their learning, and 81% agreed that the online environment contributed to their learning in the course (see Table 11, page *).

This section summarizes some of the lessons learned from the evaluations and experiences gained during the pilot and subsequent terms. The lessons are gleaned from an analysis of the formal formative and summative evaluation measures, Web server statistics, help desk reports, and anecdotal data collected from students, faculty, and support staff.

First, using online enhancements increased the time that students were involved in learning tasks, such as accessing other online materials related to the content of the course, reviewing lecture notes to gain clarification, and completing assignments on time. Time on-task is one factor that has been found to contribute to increased student learning. Both students and faculty also valued the ability to use quizzes and practice exams, with immediate scoring and feedback. This was especially important in high enrollment courses where logistics and scoring time can be barriers to their use.

Students reported that they liked online materials and that the online environment contributed to their learning. Students clearly want these enhancements in their courses. Providing outlines of lecture notes prior to a class allows students to organize information before class and spend more time listening in class. Students reported that procedures as simple as providing lecture outlines have a major positive impact on their learning. It should be noted that few faculty reported any decrease in attendance as a result of using CourseInfo.

During this pilot term, most instructors used more content-delivery components of CourseInfo (e.g., Course Announcements, Lecture Notes, and Course Assignments) than the interactive components (e.g., online discussions, chats, and group work), possibly because the former required less additional instructional design and development effort. However, as observed during terms subsequent to the pilot, faculty tend to develop online enhancements to their courses in stages, beginning with providing course materials online, then developing more interactive components, such as discussion and online quizzes. It is important to assist faculty to recognize the value of increasing the use of interactive components. As faculty become more familiar with these benefits, they may be more likely to design them into their courses. A recommendation for further study is to identify what changes, if any, are occurring with in-class teaching when online components are used.

On the basis of the pilot and subsequent terms, the key ingredients of the success of the project include the ability to engage faculty, provide training and support, and plan and coordinate the project activities.

In summary, the pilot and subsequent terms demonstrate that, given a well-supported tool that enables them to use the Web for on-campus course enhancement, without requiring extensive technical knowledge, faculty will respond positively by implementing the tool, as illustrated by the dramatic growth in the use of CourseInfo at Pitt (See Table 1).

The following table is based on an analysis of hits (Web page visits) taken from the Web server log file generated by the production server for the Fall Term, 1998.