|
William F Massy President,
The Jackson Hole Higher Education Group, Inc.
Research Professor: Stanford University & Nigel J French
Secretary-General, University Grants Committee of Hong Kong
Background
The University Grants Committee of Hong Kong (UGC) is a non-statutory advisory body whose members comprise distinguished overseas academics, prominent local professionals and business people, and senior locally-based academics. Its main responsibility is to advise the Government of the Hong Kong Special Administrative Region of China (HKSAR) on the academic development and funding of Hong Kong's publicly funded institutions of higher education. It also plays a vital role in assuring the quality of higher education provision in the tertiary education institutions under its aegis.
As part of the UGC's on-going activities to assure quality and value for money in the Hong Kong higher education sector, the Committee embarked in 1995 on a programme of teaching and learning quality process reviews. This followed the implementation of a performance-based funding model and a research assessment exercise in 1993-94.
A paper describing the background to, and methodology of, the "Teaching and Learning Quality Process Reviews" (TLQPRs) was presented at the International Conference on Quality Assurance and Evaluation in Higher Education held in Beijing, China on 7 - 9 May 1996. 2 A further paper presenting the outcomes of the first round of TLQPRs of seven UGC-funded tertiary education institutions, and describing the lessons learned from the programme, how they were expected to be followed up by the institutions and the UGC, and the changes that the UGC planned to introduce in the next round was presented at the 1997 conference of the International Network for Quality Assurance Agencies in Higher Education - South Africa, 24-28 May, 1997 (INQAAHE 1997). 3
Towards the end of the first round of TLQPRs, in April 1997, a seminar was organised by the UGC and the TLQPR Consultative Committee to provide an opportunity for the institutions to share experiences and examples of best practice identified during the course of the Reviews. During discussions with the Consultative Committee about the arrangements for the seminar, a further suggestion was raised of undertaking a more formal evaluation, via an outside consultant(s), of the impact of the TLQPRs. Specifically it was suggested that the following questions should be addressed:
What were the original aims of TLQPR? Have these aims changed over the course of the initiative? If so, why? Have these aims been clearly communicated to the UGC-funded higher education institutions? Have the institutions interpreted TLQPR in the same way? To what extent has TLQPR affected institutional teaching and learning processes? Have the TLQPRs resulted in any substantial impact on teaching policy and practice in the various institutions? Are there any outcomes that might be considered as unintended in regard to the stated aims of TLQPR?
The UGC accepted in principle that such an evaluation should be undertaken, but decided that, in view of the Committee's and the institutions' other activities and commitments, the earliest that it could be initiated would be in late 1998. This timing was also felt to be more appropriate in view of the UGC's expectation that the institutions would themselves undertake a review of progress with the implementation of measures in response to the areas for improvement identified in their TLQPR reports. These reviews were expected to be undertaken so that a report could be made to the UGC on progress about two years after the review visit to each institution.
In January 1998 the UGC decided to initiate the proposed external evaluation of the TLQPR process and agreed that an academic consultant should be appointed to undertake the task under the direction of a small Steering Group comprising three UGC members and the Convenor of the Consultative Committee. The UGC at the same time decided that the review should also assess the effectiveness of projects supported by two types of grants awarded centrally by the UGC to promote teaching and learning quality initiatives in the institutions. 4 The review was organised by the Center for Higher Education Policy Studies (CHEPS) at the University of Twente in the Netherlands.
The UGC has received a preliminary oral report from the review team.
This paper describes the review process and the consultants' preliminary findings, 5 and then discusses the way forward.We remain convinced, as we concluded in our paper of May 1997, that "the TLQPR process is worthy of replication, both in Hong Kong and in other countries.” We also are convinced that "evaluating evaluation,” the theme of this conference, is an idea whose time has come. We found the evaluation process and the consultants' preliminary conclusions to be very useful, and we look forward to their written report.
The External Review Methodology
After reviewing the report of the Consultative Group and considering its own agenda, the UGC determined that the aims of the external review would be as follows:
to determine the extent to which TLQPRs have achieved their stated aims:
to focus attention on teaching and learning (as the primary mission of higher education institutions); to assist institutions in their efforts to improve teaching and learning quality; and to enable the UGC and the institutions to discharge their obligation to be accountable for quality;
to recommend any improvements; and to investigate the unintended effects of the TLQPRs. The Committee envisaged that the review would involve several stages:
(A) Documentary Review
The consultant would examine relevant documentation such as UGC papers, submissions from institutions, sub-group reports, TLQPR reports, institutional responses to their TLQPR reports, the Report on the TLQPR Seminar, the Massy-French paper 6, institutions' progress reports in response to their TLQPR reports, institutions' reports on their use of Teaching Development Grants and any further briefing materials prepared by the UGC/UGC Secretariat.
(B) Meeting with the UGC Steering Group
The consultant would meet with the Steering Group to pose questions, seek advice and exchange views, particularly on the TLQPRs.
(C) Visits to institutions and Meetings with key groups
The consultant would visit the institutions to meet key personnel including Heads of Institutions, selected Faculty and Departmental staff and students and would also meet with the TLQPR Consultative Committee.
(D) Analysis and verification of information
The consultant would analyze all information gathered and verify the information with the UGC Secretariat and/or the institutions.
(E) Preparation of review report
The consultant would prepare a report evaluating the effectiveness of the TLQPRs and the various centrally funded projects in support of teaching and learning. The report would identify the strengths and weaknesses of the TLQPRs and suggest improvements.
(F) Publication of review report
The final report would be discussed and endorsed by the UGC before being published and sent to institutions for their reference. The UGC invited suggestions from the institutions through the Consultative Committee for suitable experts to undertake the review. In April 1998, the UGC accepted a recommendation from the Steering Group, based on the suggestions from institutions and advice from other sources, CHEPS was best placed to undertake the review and should be approached with a view to their forming a team to do so.
The UGC Secretariat subsequently approached CHEPS and formally invited the Center to undertake the review and to submit a proposal for a consultancy accordingly. The CHEPS proposal was accepted by the UGC on the advice of the Steering Group in August 1998 and the consultancy was formally awarded late in 1998. 7
Type and Purpose of the Review
Lee Harvey has described the various approaches to evaluating evaluation along two dimensions. 8 The first dimension categorizes the kind of analysis and evidence used in the review:
Opinionated or theoretical Analysis based on available (anecdotal) evidence Analysis based on systematic data collection
The second dimension addresses the review's sponsorship with an eye to determining independence:
The external review of Hong Kong's TLQPR clearly falls in Harvey's highest analysis and evidence category: "analysis based on systematic data collection”. It spans the second and third categories on the independence dimension. While the evaluation was funded by the UGC, it was conducted by an independent agency whose experience and independence are above reproach. The UGC sought and by all indications received a fully independent evaluation.
Harvey also provides a taxonomy of the purposes of external evaluation: 9
Feasibility study or evaluation of pilots Modification of existing process Evaluations of effectiveness or ability to deliver the 'underlying rationale' Fundamental review of impact on the sector
The Hong Kong review spans the second and third categories. CHEPS evaluated the TLQPR's ability to deliver its 'underlying rationale'. As indicated below, they answered this question in the affirmative and recommended that the process be continued. The team then went on to recommend modifications of the existing process for use in subsequent rounds.
Preliminary Findings
The consultants' major conclusion is that the TLQPR was "the right instrument at the right time” for Hong Kong. While the reviews represented a learning process for all concerned, and thus certain things might have been done differently, the consultants felt that the exercise's most important goals had been achieved.
The first broad goal, to improve the focus on teaching and learning, turned out to be an easy win. There is no doubt that the TLQPR heightened the consciousness of institutional leaders and staff with respect to teaching, and to at least some extent blunted the effects of the UGC's Research Assessment Exercises in driving attention toward research. However, respondents interviewed by the consultants cited some confusion about the UGC's "sometimes conflicting aims” in respect of teaching and research.
The second broad goal, to improve teaching and learning quality, also appears to have been achieved. The consultants observed positive changes in the institutionalisation of quality management, especially among institutions that had not stressed such processes before the TLQPR. This was helped by the expectation, based on signals provided by the UGC, that the TLQPR would be repeated in some form. There also were improvements in quality processes themselves, but these would have been greater if there had been more inter-institutional dissemination.
The consultants felt that the third broad goal, to exercise accountability for quality, was not fully discharged by publication of the reports. While more detail will be forthcoming, we believe this observation stems from the broad language used and the avoidance of any semblance of quantitative rating or ranking. The consultants remarked that achievement of the TLQPR's formative goals might well have been compromised had a more accountability-oriented approach been taken in this first round.
The consultants offered a number of recommendations about how future TLQPRs might be improved.
Continue the signal. Some kind of quality review will be required as long as teaching remains the primary goal of tertiary education in Hong Kong. While some changes might be made eventually, the TLQPR should be repeated at least once in more or less its present form. Continue attention to quality processes. The second evaluation round should focus on processes, not on subject-oriented teaching quality assessments. Narrower focus. The next round might focus on progress since the first round, and it should place particular stress on whether institutions, departments, and staff are developing and using learning outcome measures. Clearer about goals. Experience from the first TLQPR should be used to sharpen the statement of goals for the second round, and the revised statement should be widely disseminated. Local ownership of the process. The next TLQPR round should be chaired by a Hong Kong UGC member. Foreign leadership was appropriate during the first round, but now it's time to embed the process firmly in the local academic environment. Process design decisions. Given the diversity of the Hong Kong tertiary education sector, the next TLQPR round should focus on each institution's internal agenda to the extent possible.
The consultants also provided commentary about the structural elements of the TLQPR process as it might be applied in a second round.
Self-Evaluation. The first-round TLQPR asked institutions to prepare a self-evaluation document. The process of reflection triggered within institutions during the self-evaluation was described as being very valuable. However, the submissions could be streamlined and stricter length guidelines should be provided to avoid a "self-evaluation arms race" Review Panel. The visits were conducted by an intact panel of eighteen people: eight from the UGC, two foreign experts, and eight members of the Consultative Committee. The panel can be smaller in a second round (because the process will be more familiar), its membership can vary from visit to visit, and it should continue to include external quality assurance experts. The chair should be a local person with knowledge of quality assurance, someone who is viewed as independent and accepted by the whole tertiary system. Site Visit. The first-round process consisted of plenary sessions with institutional leaders and quality assurance committees and visits to faculties, departments, and educational development units. No significant changes were suggested. Respondents viewed the unit-level visits as being particularly valuable and some suggested that they be lengthened. Panel's Report. The reports included both general observationsand, to provide specific examples, disguised unit-level descriptions of exemplary and deficient practice. In general, the reports "touched on very significant points in all universities they covered a lot of the right stuff.” However, certain respondents said that some conclusions may have gone beyond the evidence by citing anecdotes as general truths, that some may have gone beyond process to address teaching quality itself, and that some may have encouraged a compliance orientation. The review group had no immediate recommendation on whether unit-level descriptions should be included in the next round, deleted entirely, or given to the institutions for private consideration.
The consultants cited a number of positive impacts of the TLQPR. "Lots of things happened at the university level”: for example, stimulation of quality processes adoption by institutions without strong prior experience in this area; more explicit university-level processes everywhere; greater involvement in quality management by senior institutional leaders, and more active quality roles for educational development units. The Consultative Committee's active involvement and followup was cited as a "very welcome result” with important long-term potentiality for the quality of tertiary education in Hong Kong.
In the area of accountability, the UGC imposed a penalty via a reduction in one institution's research student numbers for failing to recognize and be prepared to correct quality process deficiencies. The consultants confirmed the UGC's conclusion that the penalty produced the desired corrective action. The manner by which the TLQPR could and did "inform funding” was not sufficiently publicised, however. The penalty assessed by the UGC was not widely known, even by the management of the other institutions. In short, the consultants recommended that the method by which future TLQPRs inform funding should be made more transparent.
In summing up their preliminary conclusions, the consultants agreed that the UGC might wish to coordinate its quality assurance reviews sometime in the future, but urged that the TLQPR should be repeated in identifiable form for at least another round. They felt that a three year cycle is too short, that a five- to six-year cycle might be about right. The next TLQPR round should be organised by the UGC in its capacity as funding body, and it should be given stature comparable to the UGC's Research Assessment Exercises. The UGC also should work with the Consultative Committee to disseminate good educational quality practices.
Discussion and The Way Forward
The two of us were pleased with the consultants' preliminary report. Their message confirmed that the TLQPR has made a difference. The institutions' educational quality processes, and hopefully educational quality itself, are better than they would have been without it. While the initial TLQPR was by no means perfect, it was judged to be sensible given the conditions and knowledge base of the time. The recommendations about a further TLQPR round confirm the basic approach while offering valuable suggestions for change. We were impressed with the consultants' insight and deeply appreciate their efforts.
The most basic issue raised by the consultants is the tension between the TLQPR's formative and summative goals: between the quest for improvement and the need for accountability. We were not surprised to hear that some respondents were confused about the relative priorities. Indeed, we believe a degree of confusion is inevitable. Efforts to polarise an educational quality assurance exercise as either formative or summative are misplaced: any good exercise embraces both goals. Formative programmes with no normative consequences may be helpful, but in the end educational quality will take a back seat to areas where such consequences are apparent in most cases, research. At the same time, successful quality assurance requires wholehearted participation at all levels: "quality cannot be inspected in at the end.” We think the best way to channel the tension positively is to retain flexibility - which may well involve ambiguity before the fact - and then apply a rule of reason when confronted with specific situations. The process should be transparent so that people can observe the facts as they unfold and draw their own conclusions.
The consultants concluded that the first TLQPR round emphasised improvement over accountability, and that this was appropriate. We agree. We also agree that accountability should have a higher priority in subsequent rounds. In this context, we note that the UGC's single TLQPR-based penalty was not for quality process deficiency per se, but for refusing to commit to improvements once the deficiencies had been identified. The UGC had no desire to penalise or reward accidents of history (e.g. an institution's particular development path), but it was determined to shape the future. The consultants confirmed that, when the chips were down, the TLQPR did achieve this goal. We also note that future quality process deficiencies can no longer be regarded as "accidents of history.”
The UGC's decision to focus on quality processes rather than attempt a subject-based review of teaching and learning quality has been affirmed. The decision was based partly on the UGC's experience of undertaking subject-based reviews 10 and the difficulty of performing them effectively, especially in a place as small as Hong Kong where collegiality among local academics in the same discipline is high and foreign expertise is expensive. Perhaps more important was the diversity of Hong Kong's tertiary education sector. What constitutes "good teaching" at a research university may well differ from the appropriate standard at a liberal arts college or an institution specialising in applied work. (The UGC is grappling with an equivalent question in its Research Assessment Exercise 11.)What is invariant across institutions is the importance of self-conscious attention to educational quality assurance and improvement: that is, to quality processes. We believe that such attention is a necessary condition for optimising educational quality. The sufficient conditions include good staff and adequate resources. However, we reject the "invisible hand" approach to teaching quality, which states that good staff and adequate resources automatically produce good results.
The recommendation that the TLQPR be continued at least one more round comes at a time when the institutions and the UGC itself are suffering from "review fatigue.” The last five years have seen the TLQPR, two rounds of the Research Assessment Exercise, reviews for institutional self-accreditation at four of the institutions, a review of post-graduate education and the just-concluded Management Reviews. 12 The UGC has recently formed a sub-group to consider how best to rationalise the various quality assurance review processes. The sub-group has not yet reached any firm conclusions, but has agreed that the ensemble of processes should economise institutional and UGC effort, while continuing to steer the sector in desired directions, should inform funding where appropriate, and should maintain the UGC's accountability obligations. The two of us share the consultants' desire to "keep up the beat" with respect to educational quality processes. As member of the subgroup and UGC Secretary-General, we will work to achieve this result either through a free-standing TLQPR or as an identifiable part of a wider-ranging meta-review of institutions' quality assurance processes.
Rationalisation will get easier as quality processes mature and more responsibility can be devolved to the institutions. Implementing the consultants' recommendation that the next TLQPR place more emphasis on learning outcome measures would be an important step. The effective measurement of learning outcomes lies at the core of educational quality assurance and improvement, and the development of such measures should be a very high priority for every programme in every institution. Once the measures are developed and validated to a reasonable operational standard, the TLQPRs could begin by viewing trends and, where possible, inter-institutional comparisons. Institutional visits would audit the figures and explore what institutions and operating units are doing to continuously improve performance on the measures and the measures themselves. Such reviews would be more rigorous and also less intrusive and time-consuming than today's mainly subjective approach.
To endorse the development by programmes and institutions of appropriate learning outcome measures does not mean we have concluded that producing a few global performance measures leading to league tables is possible or desirable. We cited the diversity of Hong Kong's tertiary sector previously, and from this it follows that "one size fits all” performance measures should be viewed with suspicion. The UGC has addressed this issue in a preliminary way, and has commissioned a study of worldwide practice with respect to performance indicators and their utilisation for funding and public accountability. We expect the study to provide additional impetus for prioritising the development of indicators at the programme and institutional level and for reviewing progress in the next TLQPR. Publicising data from well-grounded programme-level performance indicators would provide public accountability and boost the effectiveness of market forces.
In time, the institutions themselves may be able to shoulder much of the responsibility now carried by the TLQPRs. This would be consistent with the worldwide trend toward deregulation and local empowerment. In addition to the elements of quality assurance and improvement already cited, such devolvement would require that the institutions create their own TLQPR processes. Some are already doing so. Going even further, one of us has begun advocating a kind of "academic internal audit unit” to effect formative teaching and learning quality process review on a regular basis. 13 Whatever the mechanisms, deployment of fully effective internal processes will simplify future TLQPRs. To use an analogy, the TLQPR's task will be to verify and marginally boost the velocity of a wheel that is already spinning rapidly. The UGC, and we suspect the institutions, would welcome such an outcome.
|
