2003 Summer Focus Session

Building a Culture for Transformative Assessment
June 20, 2003
Seattle, Washington

Proceedings

• General Sessions
• Breakout Sessions

Participants

While open to individuals, this focus session was designed to help institutional teams in particular that have been tasked to plan and/or implement a major change for their program or institution to transform teaching and learning with technology. The invitation was extended to teams from institutions that

• are involved in significant institutional improvement of teaching and learning with technology;
• use assessment in a variety of ways to guide and support that effort;
• share a commitment to the purposeful application of assessment and dissemination of assessment results to support deep change.

The focus session concentrated on the use of data and systematic assessment to advance such efforts. Ten institutional teams ranging in size from two to 11 members were in attendance as well as nine individuals from different organizations. A diverse audience was in attendance: more than 30% were faculty; 25% were from university administration; 17% were instructional technology staff; 17% were assessment specialists; and the rest were divided among faculty development, other information technology staff, learning/instructional designers, librarians, and students.

Program

This focus session was jointly sponsored by the National Learning Infrastructure Initiative (NLII), the Flashlight Program of the TLT Group, the Coalition for Networked Information (CNI), Center for Teaching, Learning and Technology at Washington State University (WSU), and theAmerican Association for Higher Education (AAHE).

The focus session was designed as part of the Transformative Assessment Project (TAP) to elicit new ideas about assessment practices and systems that can transform teaching and learning and to help institutions of higher education put these ideas into action. We envision transformative assessment systems as institution-wide assessment strategies based on institutional goals and integrated across all levels-the course, the program, and the institution-to systematically transform teaching and learning with technology.

This focus session is one of several venues the NLII, the Flashlight Program, CNI, WSU, and AAHE are collaboratively developing to discuss and apply these ideas.

During the daylong focus session, participants alternated attendance at general session presentations, followed by small group discussions and institutional team member meetings. Participants explored the processes and practices associated with transformative assessment as an initial step in planning and carrying out a transformative assessment project at their institutions.

Meeting Purpose

During the focus session, participants worked with facilitators and each other to begin developing plans for using data to advance the improvement of teaching and learning with technology. Based on previous work from TAP, specifically, a set of transformative assessment rubrics, we believe that several elements must be addressed for assessment to advance an ambitious agenda for improvement. These elements are organized around the following four dimensions.

Assessment Purpose

The assessment plan aligns with other institutional plans and promotes a collaboration of administration, faculty, students, and community.

Data Acquisition and Analysis

Data from multiple and diverse sources illuminate students' learning, learning processes, and learning purposes, particularly learning as those aspects of learning extend beyond course-specific outcomes.

Application to Transformation

The assessment findings are used to systematically inform and reshape teaching and learning practice to improve effectiveness, efficiency, and/or value and specifically to promote an operational "culture of evidence" (for example, influence promotion and tenure decisions).

Dissemination

Results are reported internally and externally with plans for expanding the collaboration for transformation.

As part of this session, we used the rubric and the transformative assessment conceptual framework to begin to

• refine our understanding of transformative assessment;
• develop strategies for using assessment to aid a major technology-enabled educational improvement at each of our institutions;
• explore the elements of a healthy and effective assessment culture;
• identify the opportunities and challenges to creating an effective assessment culture; and
• provide an opportunity to begin inter-institutional sharing of experiences and insights about developing and implementing their transformative assessment activities and explore continued sharing through a community of practice.

Session Work Products

As part of this focus session (including pre-meeting preparations), participants had an opportunity to:

1. Complete a self-assessment (to get a clear picture of their individual institution's assessment culture)
2. Begin to define a working structure for their individual assessment projects (including nature of project team and representation needed for stakeholders)
3. Develop a half- to full-page proposal describing their individual assessment projects, primarily for team recruitment (proposal will include justification/incentive for team members' participation)
4. Draft an agenda for their individual project team's first meeting
5. Design a high-level project plan (6–12 months out; work products, general timeline, high-level milestones) and plan for next steps once they returned to their home institutions.

Questions We Tackled

1. What is a helpful conceptual framework for effectively exploring the issues of building a culture of evidence and assessing the transformation of teaching and learning with technology?
2. What do we mean by "transforming teaching and learning with technology?"
3. What are the barriers to transforming teaching and learning with technology?
4. What is "transformative assessment?"
5. How can transformation be defined for each institution to describe the nature of the desired change in teaching and learning?
6. How should we go about identifying and involving the stakeholders in the change?
7. What are some (anticipated and unanticipated) barriers to change?
8. What is a "culture of evidence and assessment?"
9. How is alignment of planning involved in transformative assessment?
10. How might assessment become an integral part of the learning process, at the individual level (students, faculty, and staff) as well as the organizational level (instead of an add-on used strictly as a carrot or stick)?
11. With regard to the rubric, what does it mean in a practical sense to be at the administrative, progressive, or transformative category for "application of findings" and "dissemination?"
12. What can be built into assessment project plans that would nurture a culture of evidence, and how does that culture need to evolve over the course of a project?
13. How does the national research agenda on assessment currently inform and facilitate institutional activities in this area?
14. How could the national research agenda inform and facilitate institutional responses and activities?
15. How can we influence the national research agenda with the work we are doing in this session?

Next Steps

Focus session participants and other interested parties will have an opportunity to participate in an online community of practice. The TAP team will be hosting a series of follow-up discussions on the new academy virtual community of practice site. In addition, the transformative assessment conceptual framework and the TAP rubric will be refined based on feedback received at the focus session and during community discussions. TAP team members will collect information about transformative assessment projects that attendees and other interested parties are undertaking and conduct a needs assessment. A series of experimental online workshops may be designed to support these projects, depending on the needs identified. Finally, the transformative assessment key themes page will continue to be updated to reflect the new resources created and gathered for this topic for the focus session and by the community of practice.