When policy makers are asked to identify the biggest issues in higher education, they often cite college costs and high tuitions or access for underserved students. But although those issues are important, resolving them will make little difference if college and universities don’t deliver high-quality education. Federal and state governments recognize this, as evidenced by the growing number of programs for student assessment, institutional accountability, and performance measurement. Unfortunately, however, such programs often fail to improve quality.
Today, a small but growing number of institutions both in America and abroad are achieving significant quality improvements through academic audits. Institutions in Great Britain first began doing audits about 1990, and then the concept was adapted by institutions in Sweden, New Zealand, Australia, and, under my leadership, Hong Kong. Deans, provosts, or external oversight bodies have usually initiated and conducted such audits.
The objective of an academic audit is to elicit thoughtful conversations about how to produce tangible improvements in education quality without having to spend more money. Like program reviews and accreditation visits, audits include a self-study by the unit being reviewed and a site visit by peers from outside the department or institution. Unlike those other forms of evaluation, however, audits do not focus on the ratio of faculty members to students, disciplinary coverage, research prowess, and other matters relating to resources and prestige. Nor do they second-guess the decisions faculty members make about curriculums or the quality of classroom performance.
Instead, academic auditors evaluate what are coming to be called “education-quality processes” -- the key faculty activities required to produce, assure, and regularly improve the quality of teaching and learning. By so doing, they sidestep problems that usually arise when outsiders try to gauge teaching performance or assess student learning -- like intrusive classroom visits, trouble interviewing enough students and faculty members to get a clear picture of what’s going on, and even threats to academic freedom.
Rather than trying to micromanage teaching, an audit asks how professors organize their work and the kinds of data they use to make decisions, as well as how faculty members can use the resources available to them and work collegially to do better. An academic audit relies on the following simple and widely held tenet of academic life: “Professors want to provide quality education, and they will do so when supported by good processes.” Good education-quality processes systematize a department or other unit’s approach to quality, instead of leaving it mainly to unmonitored individual initiative.
A case in point is the University of Missouri System, which asked me to help create a pilot program to audit a department on each of its four campuses. According to a faculty member who participated: “We all learned a lot. People on both sides of the table were thinking outside their disciplines. We identified some real problems. For example, the curriculum is haphazard’chaotic’ as one department member put it. And we generated some good, tangible, ideas for improvement.” Those ideas included developing a comprehensive statement of learning goals for each course and program, informed by interviews with alumni and employers as well as consultation between students and faculty members. Also, faculty members now meet regularly to discuss what works and what doesn’t -- and to build enthusiasm for experimenting with new teaching methods.
Other benefits result when outsiders look at a department’s methods. By evaluating someone else’s efforts, the people who perform the audits often gain new ideas about how to improve quality in their own departments. Noted Stephen W. Lehmkuhle, the vice president of academic affairs at Missouri, “Sharing experiences among faculty from all four campuses as well as working together on this unique audit ... builds bridges in many ways ... , some related to the audit ... , others just come up as they talk.”
Effective audits require “structured conversation,” both within the department while preparing its self-study and then with the site-visit team. Conversations are important because that is how ideas become actions, and progress occurs, in academe. Structure is key because it focuses people’s attention on the five key areas that are important for improving quality.
First, participants in audits must identify the underlying purpose of the education offered at their institution, and determine the desired learning outcomes. What should a student who successfully completes the course or program know and be able to do? How will the course or program build on the student’s prior knowledge and capability? How will it contribute to the student’s future employment opportunities, capacity to make social contributions, and quality of life?
Second, the audit should determine whether the curriculum achieves the education’s purpose. What will be taught, in what order, and from what perspective? How will it contribute to the desired learning outcomes? What course materials will be used? How will these materials relate to other parts of the student’s program?
Most professors think education quality revolves around simply getting the right course content. That certainly is necessary, but it is far from sufficient. As a third step, audit participants must consider the design and organization of the teaching and learning process. What methods will professors use to introduce the material to the student, to answer questions and provide interpretation, to stimulate involvement, and to offer feedback on student work? What new roles and responsibilities will faculty members need to assume? What other resources will be required, and how will they be used?
The audit should also deal with how to assess student learning. What will be the measures and indicators, and will they provide information on multiple dimensions of student performance? Will they compare performance at the beginning and end of the term to get the value-added dimension? How will the long-term outcomes of the educational experience be determined? Will baseline and trend information be available? Who will be responsible for assessment and how will the results be used?
The assessment that I’m describing differs from most state-mandated programs because departments with good education-quality processes seek to improve assessment to further their own purposes, not to satisfy some external authority. As Peter Ewell, a senior associate at the National Center for Higher Education Management Systems, noted in Change magazine (January/February 2003): “Assessment ... will be adopted and internalized only when it becomes useful to faculty members as part of their everyday practice and reinforced in widespread and meaningful ways by the institutions they inhabit.”
Finally, audits must focus on whether faculty members are delivering content as they intend, are using the most appropriate teaching methods consistently, and are performing assessments and measuring results effectively.
At best, each department will move from having no organized activities at all, beyond firefighting and informal efforts, to a “learning organization” in terms of education-quality processes. In that scenario, the focus on quality will be embedded in the departmental culture, and the idea of regular improvement in all five areas will be a well-accepted way of life.
Yet all these steps will require planning, diligence, and reinforcement. Most of all, professors need to resist competing demands on their time and the dulling effects of routine. They must be self-disciplined and actively collaborate with colleagues in the work team and department. Researchers at the National Center for Postsecondary Improvement at Stanford University have identified seven common-sense principles and practices that can help a department improve its processes:
Define quality in terms of outcomes. The quality of student learning, not teaching per se, is what ultimately matters. Learning should pertain to what is or will become important for the students enrolled in the program -- not some “ideal” student. Exemplary departments determine their students’ needs and then work to meet them.
Base decisions on facts. Departmental teams should collect data on student preparation, learning styles, and, where relevant, probable requirements for employment. Team members might obtain feedback from past students and their current employers. They should analyze the data carefully in light of disciplinary standards and their own professional experience, and then incorporate the findings in the design of curriculums, learning processes, and assessment methods.
Focus on teaching, learning, and assessment. Departments should carefully analyze how professors teach, how students learn, and how they all approach assessment. Departments should consult the literature on pedagogy in their academic disciplines and collect data on what works and what doesn’t. They should stress active learning, exploit information technology, and not hesitate to experiment with new teaching and learning methods. Colleagues should be quick to adopt successful innovations, which should become part of the department’s modus operandi and form the baseline for future experimentation and improvement.
Strive for coherence in curriculums and educational activities. Departments should view learning through the lens of the student’s entire educational experience. Courses should build upon one another to provide the desired depth and breadth. This also applies to the typical student’s “portfolio” of class sizes and learning approaches. For example, a mix of large lectures and small seminars may produce better learning than a succession of medium-size classes that consume the same amount of time.
Work collaboratively to achieve mutual involvement and support. Professors should demonstrate collegiality in teaching, just as they do in research. Departments should encourage faculty members to work together, hold one another accountable, and bring a broad array of talent to bear on difficult problems. Such teamwork can make the department a “learning organization” with respect not only to disciplinary content, but also to teaching and education.
Identify and learn from best practices. Audit reports should be shared widely to exchange information and motivate improvement. Departments should seek out examples of good practice and adapt the best to their own circumstances. They should compare good versus average or poor-performing methods and students, assess the causes of the differences, and seek ways to minimize the variation.
Make continuous improvement a priority. Departments should strive to improve the quality of teaching and learning on a regular basis. While most professors will continue to place strong emphasis on research, they should spend enough discretionary time on quality processes to keep the improvement ball rolling. Personnel committees should make the results of such work, along with teaching and research performance, a criterion for promotion and tenure.
Today’s audits differ markedly from the original British version, which was criticized as lacking in academic content and stressing whether formal quality committees existed and maintained complete minutes rather than whether people were effectively grappling with the issues. The postsecondary-improvement center’s articulation of these principles and practices has helped overcome that problem, and Britain’s Quality Assurance Agency for Higher Education renewed its commitment to audits last summer.
Audits at many institutions should grow significantly in the future, as they can spur improvement and accountability in flexible and inexpensive ways. Professors can discuss education-quality processes without the defensiveness associated with direct quality evaluations. Furthermore, it is impossible to fake an audit -- if one has not focused seriously on quality processes it’s impossible to discuss them coherently. And because audit conversations don’t require expertise in any particular discipline, professors from all fields can learn and spread exemplary practices by serving on audit teams.
Those who do are pleased with the experience, as are the professors in the departments being audited. Everyone wins: the faculty members and administrators who participate in the audit, the institution, and, most importantly, the students.
William F. Massy is a professor emeritus of education and business administration at Stanford University and president of the Jackson Hole Higher Education Group, which provides consulting services to colleges and universities on education-quality processes. He is the author of Honoring the Trust: Quality and Cost Containment in Higher Education (Anker Publishing Company, 2003).
http://chronicle.com Section: The Chronicle Review Volume 49, Issue 41, Page B16