|
William F. Massy
The Hong Kong University Grants Committee and
Stanford University, U.S.A.
Hong Kong's University Grants Committee (UGC) will initiate a second-round
Teaching and Learning Quality Process Review (TLQPR) during 2001. The
TLQPR, a type of academic audit, was described at INQAAHE 4 and 5 (Massy
and French, 1997, 1999). The second round follows the recommendation
of the Center for Higher Education Policy Studies (CHEPS) at the University
of Twente, NL, that the first-round TLQPR was the "right process
at the right time for Hong Kong" and that "the UGC should
keep up the beat" for quality by promptly initiating a second round.
Academic Audit
Academic audit is one of three main types of higher education quality evaluation
in use today around the world. (The other two are accreditation and subject-level
assessment.) David Dill describes it this way:
In contrast to accreditation, program review, or student assessment initiatives,
[academic] audits look deeply into the heart of the academic enterprise.
They test whether institutions and their faculties in fact honor their public
responsibility to monitor academic standards and improve student learning.
(Dill, 2000, p. 35)
In addition to Hong Kong, audit is used in New Zeeland, Sweden, The Netherlands,
and as an element of Denmark's and the UK's comprehensive evaluation programs.
In the United States, the Western Association of Schools and Colleges (WASC,
one of the country's seven regional accreditation agencies) is experimenting
with audit. The Education Commission of the States (ECS, an association of governors
concerned about education) is working with the author on a proposal to pilot
test the audit methodology in U. S. public universities.
The United Kingdom, New Zeeland, Sweden, and Hong Kong have completed their
first cycle of academic audits, and each experience has been formally evaluated.
(Coopers & Lybrand, 1993; Meade and Woodhouse, 2000; Brennan, et al., 1999;
Stensaker, 1999a,b; and Nilsson and Wahlen, 2000.) Working for the U. S. National
Center for Postsecondary Improvement (NCPI), Massy (1999) reviewed the Swedish
program and contrasted it with Denmark's assessment approach.) According to
Dill (2000, p. 36), the reviewers generally agree that academic audits have:
Made improving teaching and learning an institutional priority. Facilitated active discussion and cooperation within academic units on
means for improving teaching and learning. Helped clarify responsibility for improving teaching and learning at the
academic unit, faculty [i.e., school], and institutional level. Provided information on best practices within and across institutions.
Moreover, audit focuses on "education quality work" (EQW, to be defined
below), which is emerging as the key element of institutional quality programs.
External agencies can evaluate EQW more easily than education quality itself.
Panel selection and training appear easier, cycle times can be shorter, and
institutional diversity more easily respected than in other forms of evaluation.
Education Quality Work
"Education Quality Work" (EQW) means the activities of faculty,
academic leaders, and oversight bodies that are aimed at improving and
assuring education quality. EQW should provide what higher education quality
pioneer Frans van Vught termed "...a framework for quality management in
higher education...drawn from insights in Deming's approach [and that of the Baldrige
and ISO-9000], but grounded in the context of academic operations."(van
Vught, 1994, p. 13). It should empower and stimulate faculty to continuously
improve teaching and learning, and help academic leaders and others to discharge
their oversight responsibilities without micromanagement.
EQW should not be confused with teaching itself. For example, finding the most
appropriate curricular content and working to improve teaching and learning
processes fall under the rubric of EQW, but delivering the content is part of
teaching. Working to improve assessment methods falls under EQW, but the actual
assessment of students does not. Peer evaluation of teaching would be EQW, but
not the act of teaching itself.
The EQW idea is not foreign to higher education. Curriculum committees are
familiar parts of the academic landscape, for example, and most professors regularly
engage in course development and make decisions about teaching and assessment
methods. Moreover, student course evaluations now provide a centerpiece for
many institutions' "quality assurance" programs.
What makes "EQW" different is the breadth, depth, and degree of organization
of the work. For example, curricular decisions could be informed by more serious
research into student needs and wants. Systematic experimentation and the benchmarking
of design alternatives could improve teaching and learning processes and assessment
methods. Quality assurance could be improved by regular peer evaluations of
teaching. As business found in the 1970s and 1980, quality work needs to be
elevated to a "key results area."
Domains of EQW
The UGC specified four domains of education quality work in the first round
of TLQPRs. (A fifth area of review, resource allocation in support of education,
is not a domain of EQW.) The first two domains deal with what is called "design
quality."
- Design of curricula. What is to be taught, in what order, and from
what perspective? What are the goals of the course or program, what are their
key quality indicators, and how do they relate to student needs? What resources
and resource materials will be used as content vehicles? How does the design
relate to other courses the student will take as part of his or her program?
- Design of teaching and learning processes. How will teaching and
learning be organized? What methods will be used for first exposure to material,
for answering questions and providing interpretation, for stimulating student
interaction with the material, and for providing feedback on student work?
What roles and responsibilities will the faculty need to assume. What other
resources will be required?
The third domain addresses the design and use of student assessment measures.
- Student assessment. What measures and indicators will be used to
assess student learning? Will they assess value added, or only performance
at the end of the program? How will the long-term outcomes of the educational
experience be determined? Will baseline and trend information be available?
Who will be responsible for assessment?
The fourth domain covers what I call "implementation quality assurance."
- Implementation quality assurance. How will staff organize to carry
out the designs effectively, day in and day out, regardless of distractions?
How can they assure themselves and others that content is delivered as intended,
that teaching and learning processes are being implemented consistently, and
that assessments are performed as planned and their results used effectively?
What processes will the department and institution use to assure implementation
quality?
Principles of Good Practice To Be Used in The Second-Round TLQPR
Hong Kong's second-round TLQPR will incorporate seven principles of good practice
in education quality work that the author developed as part of his work for
NCPI. The principles are responsive to the CHEPS Committee's recommendation
that the second round provide more specificity about the audit criteria. The
principles apply separately to each of four EQW domains.
The seven quality principles are as follows:
- Define education quality in terms of outcomes.
- Focus on the process of teaching and learning.
- Strive for coherence in curricula and educational process.
- Work collaboratively to achieve mutual involvement and support.
- Base decisions on facts wherever possible.
- Minimize controllable quality variation.
- Make continuous quality improvement a top priority.
The principles were distilled from the business quality literature and then
translated to the vernacular of higher education. I want to emphasize, however,
that their adoption came after my colleagues and I had developed a working knowledge
of higher education quality processes -- i.e., the working knowledge emerged from
academic experience, not from business. All seven constructs were used extensively
in Hong Kong's first-round TLQPR, even though the "principles" had
not yet been formalized.
The site visit protocol now in development will question respondents about
departmental EQW as it applies to combinations of the domains and principles.
For example, "Have staff consciously considered what students need in order
to be successful in their chosen field and acquire meaningful values and social
skills? How effectively has this knowledge been integrated into the curriculum?"
Such questions can elicit the dialog needed to facilitate quality work, and
to understand. The first-round experience shows that panels can quickly determine
whether staff have been pursuing quality work seriously. If so, the discussion
will contain many examples -- not only about the issues posed by the questions,
but how the respondent reached his or her conclusions about them. If the respondent
hasn't thought much about EQW, however, the answers will be vague, superficial,
and steeped in the conventional wisdom of the discipline. In other words, our
queries will include these implicit questions: "What do the respondents
know about the issues?" Can they discuss them intelligently? Have they
incorporated conscious judgments about them into their teaching?"
I don't anticipate that the panels will follow the protocol in a mechanical
fashion. Indeed, the specific questions provide no more than examples of what
might be considered. The best review sessions are informal -- participants can
take the discussion anywhere that seems relevant at the time. The panel leader
should keep track of coverage and pose questions to plug emergent gaps, but
he or she ought not to impose a rigid discussion order. On the other hand, the
panel will have an agreed structure for organizing the discussion and the TLQPR
report.
Capability maturity models
At this writing the TLQPR Consultative Committee is considering my suggestion
that we experiment with so-called "capability maturity models" for
assessing the penetration of education quality work into departmental and institutional
routines. (The result should be known at the time of the conference.) The model
has been used in systems analysis and software development since about 1985:
according to Cusick (1997, p.2), to "bring data and structure to a very
difficult task -- process improvement." I have been working to adapt it to
higher education.
Figure 1 presents the model as it would be applied at the departmental level.
A department's journey up the EQW learning curve begins at Level 0, where quality
work concepts have yet to challenge traditional methods. It achieves Maturity
Level 1 when a significant number of professors begin experimenting with quality
work. These professors work as individuals and use ad hoc methods, but their
experiments with EQW principles will have begun to produce positive feedback
from students and colleagues. Advancing to Level 2 requires organized rather
than individual effort. Quality work initiatives will be planned and tracked
by work groups or task forces, and the methods will be consciously rooted in
principle rather than being ad hoc. Moreover, the groups will be developing
metrics for gauging quality work and norms for what should be done. Fully mature
departments (Level 3) will have embedded education quality work as a central
part of their academic culture. Their members will share common expectations
for EQW and enforce them through informal routine interactions. Professors will
continuously improve their quality work and share this knowledge across and
outside the institution.
Research at Carnegie-Mellon University provides a model for gauging the capability
maturity of a whole organization. This model defines five levels of improvement,
which I have mapped into EQW as follows:
Level 0: No performance. There is essentially no EQW; traditional methods
dominate throughout the university.
Level I: Ad hoc experimentation & firefighting. A few departments
have begun to experiment with EQW principles as exemplified by Level 1, above,
but their efforts have not attracted much attention or support. Most quality
work is motivated by immediate problems that have to be solved, rather than
being viewed as a necessary and continuing academic responsibility.
Level II: Emergent programs. Some units appear to have reached level
2 of the departmental maturity model and more are experimenting at Level 1.
The university's educational quality office is beginning to assist faculty
with quality work. EQW is becoming visible across the institution and is being
discussed by school and university quality committees.
Level III: Rollout and tracking of programs. Nearly all departments have
reached Level 1, many appear to be operating at level 2, and a few are striving
for Level 3. EQW has become a core offering of the university's education
quality office. School and university quality assurance committees have begun
to track departmental progress on EQW and they are working to propagate exemplary
practice across the university.
Level IV: Key results area. EQW has become a key results area for the
university and for most if not all schools. The university has institutionalized
the tracking of EQW and includes school and departmental progress as a performance
criterion in budgeting. It provides sufficient funding to its education quality
office to make assistance on quality work readily available throughout the
institution.
Level V: Continuous improvement. EQW has become a central element of the
university's academic culture and the continuous improvement of education
has reached parity with research. Participation in quality work has become
an important criterion in faculty promotion and salary setting.
It should come as no surprise that most colleges and universities around the
world operate at Levels 0 and I. Hong Kong raised the level with its first-round
TLQPR and hopes to do so again in the second round.
| 
