The tasks of the evaluation team were:

1. to undertake an independent evaluation of the TLQPRs of the seven UGC-funded tertiary institutions in order to determine the extent to which the TLQPR process had achieved its stated aims of (a) focusing attention on teaching and learning (as the primary mission of higher education institutions), (b) assisting institutions in their efforts to improve teaching and learning quality, and (c) enabling the UGC and the institutions to discharge their obligation to be accountable for quality;

2. to recommend any improvements on the TLQPR process; and

3. to carry out a broad-brush review of the effectiveness of the Teaching Development Grants (TDGs) and the projects supported by the UGC Central Allocation Vote (CAV) which are aimed at supporting initiatives to improve teaching and learning, and to recommend any improvements.

The full evaluation team visited Hong Kong from 11-16 April 1999. During this second visit interviews were held with representatives of each of the institutions (at institutional, faculty and departmental levels, including the institution's members of the Consultative Committee). In addition, more dedicated interviews were undertaken on the following issues: (i) role and position of the university's teaching and learning support centre, (ii) students' participation in quality of teaching processes, (iii) the university's members of the TLQPR Review Panel, (iv) the university's experience with the Central Allocation Vote and the Teaching Development Grants. Overall, we met with just over a hundred staff in the institutions.

We also met the former and current convenors of the Consultative Committee.

At the UGC, we met with members of the Secretariat, the Chair and members of the Quality Assurance Review Sub-Group, representatives of the TLQPR Steering Group, and the Chair of the Teaching Development Sub-Group.

Semi-structured interviews were used to ensure coverage of all relevant aspects, leaving room for improvisation and further probing. The interviews were structured around the following three main issues:

1. to find out the effects of the first round of TLQPR in the institutions (facts as well as perceptions and opinions);

2. to find out strengths and weaknesses of the present TLQPR system;

3. to discuss suggestions on how the TLQPR system could be improved.

The full evaluation team was involved in all the interviews at the system level as well as the interviews during the visit to the first of the seven institutions. For the interviews in the other six institutions we split into two teams. We held daily briefings and debriefings to share experiences and to prepare for the next day's interviews.

Perhaps the most important differences in context concern the purposes of quality assurance. These are frequently described as being a mixture of 'accountability' and 'improvement' although the reality is rather more complex.

• The most commonly stated purpose is accountability. This appears to work in two principal ways. The first is by providing independent assurance to a principal stakeholder and funder - i.e. the state - that the quality of education is sufficient to justify the continued provision of public funds. This is an assurance of threshold quality, sometimes linked to a process of accreditation. The second way of achieving accountability is by providing comparative information to a much wider group of stakeholders - including crucially the customers or clients of higher education - so as to influence choices and decisions. This is much more a market approach with the quality assurance process working to provide information to enable the market to operate rationally.

• The second commonly stated purpose of quality assurance is to improve the quality of educational provision. But there are differences in the way this is intended to be achieved. The first, and most common, way is through the quality assurance process itself - the self-evaluation, the 'mirror' provided by external peers, the published national reports. A second but related approach is through the direct stimulation of innovation and educational development in individual institutions. A third approach is through the separate support for research and development relevant to quality of education.

• The third purpose of quality assurance is to stimulate competitiveness. By linking assessment to rewards of funding or reputation, by influencing the markets for students and for staff, quality assurance is thought to increase competition between institutions and between departments, providing incentives and encouraging responsiveness.

• A fourth purpose of quality assurance is to impose a quality check on new institutions, sometimes linked to a process of accreditation.

• A fifth and related purpose is to use quality assurance to assign institutional status in diversified systems.

• A sixth purpose concerns the transfer of authority from the state to institutions. It is frequently seen as the replacement of government control over the inputs to higher education by government supervision through assessment of processes and outputs of higher education. Although formally this is intended to represent a devolution of state power to institutions, the new accountability demands are often experienced as 'government centralism in a new disguise' (Goedegebuure et al, 1993³).

• A seventh purpose of quality assurance is to make international comparisons of the quality of education. This reflects the emphasis given to higher education as a determinant of national economic success.

The stated aims of the TLQPRs were:

• to focus attention on teaching and learning as the primary mission of Hong Kong's tertiary institutions;

• to assist institutions in their efforts to improve the quality of teaching and learning;

• to enable the UGC and the institutions to discharge their obligation to maintain accountability for the quality of teaching and learning.

In our interviews and review of relevant materials the following points about the site visits emerged.

• Each university provided information and prepared for visits of all academic departments/units, even though only a maximum of 12 per institution was to be visited by the Review Panel. A number of heads of academic units reported that the effects of preparing for a review, even though they were not selected for review, was salutary and led to productive discussions of teaching within their departments that have had lasting effects. Several departments also reported finding the framework of questions provided by the Review Panel of genuine assistance in fostering faculty discussion about means of improving teaching and learning. Based on our interviews, this across the board impact at the unit level appeared to be an unanticipated benefit of the design of the reviews.

• The unit visits were also seen by university administrators to be worthwhile. The university management received feedback about variations in quality assurance processes at the unit level of which it had frequently been unaware. At one university the administration reported that several units had actively volunteered to be visited because they wished to improve their processes. The University of Hong Kong reported at the TLQPR Seminar (8 April 1997) that the identification of good practices in the 'Unit-level Observations' permitted the university to recommend these examples to other faculties and departments as a means of assisting their development of quality processes.

• A number of units reported that the visits created an administrative burden and were time consuming, primarily it appeared because of uncertainty as to the focus of the review, and particularly because it was time consuming to develop the documentation required.

• Several department heads interviewed expressed dissatisfactions with the length of unit visits. They mentioned that the attention to process was necessarily superficial and often biased toward documentary evidence given the short amount of time panel members had to visit with each unit. They suggested lengthening the unit visits to permit more time for discussion, participation, and observation at the unit level by review panel members, so that they could become better informed of the informal as well as formal processes of quality assurance being employed. Similar recommendations were offered by one of the presenters at the TLQPR Seminar (8 April 1997).

• Those interviewed were generally impressed with the competence and fairness demonstrated by the Review Panel members during their visits. But several respondents also remarked that not all members of the Review Panel appeared to be clear as to what they were trying to accomplish. Members of the Review Panel themselves complimented their fellow panel members on their objectivity, and particularly in their skill in conducting unit visits and developing unit reports. The Massy and French (1997) paper on the TLQPR suggested as an improvement the development of a more extensive protocol of questions to guide Panel members in their visit interviews. They also suggested that this standard list of questions could be supplemented with institution-specific questions developed through analysis of the institutional self-assessment documents.

• Several respondents from the Review Panel suggested that the affiliations of some members of the Panel might have compromised their ability to play an active role in Panel deliberations. For example, while complementing the objectivity of the institutional members of the Panel when reviewing their home institution, one interviewee noted that these Review Panel members appeared to have said little during the panel discussions of their own institution. Several Panel members also said that university administrators on the Review Panel appeared constrained from making comments that might be construed as critical of their administrative colleagues at the institution being reviewed. These observations may not be representative, but they raise potentially useful questions as to the make-up of the Review Panel.

• Several members of the Review Panel, as well as the Massy/French (1997) paper, noted that the opening plenary sessions were least valuable to the Review Panel and that more visit time might be set aside for unit visits.

Two other concerns about reports were made to us by members of institutions. The first of these concerns the relationship between the oral feedback given at the end of a visit and the content of the written report. There is often a tension in this. A lack of time and an understandable wish to be positive in oral feedback can sometimes lead to discrepancies between the two kinds of report. On the other hand, in our experience elsewhere, institutions usually wish to receive some kind of oral report at the end of a visit. Our own view is that oral reports are desirable but are best kept short and the author of the written report must use them as the basis for the latter. The written report should not contradict what was said during the oral feedback. Of course, points can be elaborated and explained in much greater detail in the written report.

The other frequently expressed concern related to unit reports. The brevity and anonymity of these made them misleading. There was an inevitable tendency to attempt to guess the units to which they referred. The shortness of unit visits also cast doubts on the accuracy and validity of these aspects of the reports. Views differed about the best solution to the problem. Some would prefer to see more substantial and named unit reports. Others would prefer to drop them altogether. We would go for the latter solution because TLQPR focuses on the university as a whole although, as we have noted elsewhere, the inclusion of unit visits remains, in our view, a strong feature of the TLQPR process.

We undertook our own analysis of the TLQPR reports. Our conclusions broadly support the views of institutions. We note the following points:

1. The standard wording used in the reports includes formulations such as the 'panel believes' or the 'panel suggests' and the 'panel encourages'; although the terminology of 'recommendations' is not used, there is little doubt that this is implied - indeed, it might be better to be more explicit about it; however, quite often there appear to be no reasons given for why a particular 'recommendation' is being made.

2. The review reports are written in a style that supports the view that the TLQPR Panel did attempt to take account of institutional contexts and approaches to quality assurance. Thus, for example, it supported the creation within Lingnan College, as a small specialist institution, of quality processes based on what is termed as the 'Lingnan culture' of a collegial environment with informal processes which work well due to the close interaction between staff, students and administration.

3. However, there is also evidence in the reports of an implicit set of assumptions being used by the panel, in addition to those outlined in Annex A. These unstated characteristics include quality processes being institution-wide, coherent, systematic, and centrally monitored. Reports contain many references to the desirability of such characteristics, which are presented as being axiomatic and are not explained or justified in the body of the reports themselves and which may easily lead to a uniform quality framework, which the Review Panel wished to avoid.

   In our discussions with members of the UGC related principles underlying the practice of quality assurance were introduced and presented as UGC policy. Some relevant assumptions underlying the TLQPR process were also presented in a paper given in Beijing China in May, 1996 by Prof. Kenneth Young. We would suggest that it would be beneficial to the UGC and to the development of quality assurance processes in the institutions if these underlying principles were more clearly developed and publicly articulated. In the UK, for example, the former HEQC developed guidelines on quality assurance, which included general principles, and the new Quality Assurance Agency is developing codes of practice related to quality assurance activities. We also noted that in the recent Management Audits conducted by the UGC, the materials included "Principles of Good Management Practice."

   We would support the view expressed in the UGC TLQPR materials that institutions themselves must make the decisions about both the teaching and learning activities to be employed and the methods for improving these activities and assuring that they are appropriate and well executed. But if the UGC has assumptions and expectations about administrative and faculty responsibilities and behaviour, or other matters, related to quality assurance processes that are common across the institutions, we believe it important to clearly articulate those shared assumptions.

4. There were nearly 40 'recommendations' arising from the reviews of the seven institutions. They are written from the perspective of how the institution might improve its teaching and learning processes. Accountability issues are raised primarily in terms of internal accountability (mainly to institutional leadership) rather than to the UGC or external stakeholders.

5. Each of the review reports contains a list of commendations made by the TLQPR panel. These include: oversight of responsibility for quality issues by appropriate committees, mandatory student evaluation questionnaires, teaching awards, establishment of teaching development/educational technology centres, internally initiated unannounced class visitations, student-staff consultative committees, the broadening of annual review processes to include teaching. These also seem to confirm the panel's view that quality is best assured by formalised institution-wide processes driven by institutional leaders.

Teaching Development Grants (TDGs)

The effects of TDGs were discussed in broad terms in every institution we visited. In three universities, additional interviews were organised to find out about TDGs in more detail.

During our visits we found that, in general, TDGs were welcomed by the institutions. They appreciated the UGC's willingness to fund projects dedicated to innovation in teaching and learning as part of its effort to strengthen the quality of teaching and learning. The fact that the UGC, since 1994/95, has set aside a notable amount of money for this purpose enhanced the credibility of UGC policy in this respect.

The distributed part of TDG (an earmarked budget on top of the block grant, based on student numbers) allowed the senior management of universities to finance both teaching and learning support infrastructure (e.g. teaching development units) as well as to reward individual initiatives. In general the budgets seemed to us to be appropriately allocated by the institutions. Several institutions developed internal competitive bids for this part of the TDG.

The institutions' progress reports about TLQPR showed numerous examples of projects related to the TLQPR recommendations and financed by TDGs; other projects were drawn to our attention during the interviews. Progress of the projects was carefully monitored by the institution's management and reported to UGC regularly.

The bidding part of the TDGs, which increased from one-third in 1995-1998 to two-thirds of the TDG budget in 1998-2001, was somewhat more controversial. In terms of rewards and reputation, inter-university competitive bidding may be a powerful instrument, but both the criteria and the procedures followed by the UGC were subject to criticism.

Some universities noted that the priority criteria and the way they were supposed to be used by the Teaching Development Sub-Group (TDSG) of the UGC in the upcoming round were not clearly communicated. This hampered priority setting within the institutions. (We heard several examples of conflicting interpretations between the TDSG and university staff responsible for the preparation of the university's bid.) The minimal size of 1 million HK$ for a project was seen as too high for smaller institutions and the emphasis on inter-university projects was seen as an unnecessary prerequisite, and the criteria used for this remain unclear.

In general, doubts were expressed about the efficiency and effectiveness of the procedures followed for the competitive bidding, including the report structure. Some institutions were concerned about the demotivating effect on units of well formulated projects that were turned down apparently as a result of lack of clarity of the central bidding procedures. Some universities, however, mentioned that they were willing to support such projects on a lower scale by using part of the distributed part of the TDG budget.

The Consultative Committee's initiative in organising a system-wide conference at which results from TDG projects (both from the distributed part and the bidding part) were communicated was widely applauded.

Based on the information provided by the UGC and our visits to the institutions, we conclude that the distributed part of the TDGs was timely and welcomed. It introduced or greatly enhanced universities' ability to support initiatives to enhance the quality of teaching and learning, without the UGC becoming prescriptive or getting involved in the micro management of the institutions. As such it acts as a complementary instrument to the TLQPRs.

The competitive bidding part of TDG was criticised more widely with respect to priority criteria, minimal size of the bids as well as with respect of transparency and efficiency in the allocation process. It is our impression that TDSG's efforts to make the procedure transparent and objective have not been completely successful. A suggestion was made by one of the institutions to strengthen the relationship between the outcomes of TLQPR and the balance between the distributive and the competitive part of TDG. We will come back to these suggestions in the conclusions and recommendations of this report.

Central Allocation Vote (CAV)

Based on the documents and other information provided by the UGC we learned that the UGC uses CAV for promoting inter-institutional collaboration and for supporting new and unforeseen developments during a triennium. In the present triennium (1998-2001) CAV covers both research and teaching and learning initiatives, such as the development of areas of excellence, joint research schemes with the UK, Germany, France and the National Natural Science Foundation of mainland China, and the establishment of co-operative research centres, as well as the support of institutional restructuring. In the past triennium (1995-1998), part of the CAV budget was used to increase the bidding part of the TDG budget.

In contrast with our findings with respect to the TDG, we found that the interviewees within the institutions had none or only limited knowledge of the use of the Central Allocation Vote. The necessity of having some budget set aside at the system level to stimulate the efficiency of the system by encouraging and facilitating co-operation between institutions was not questioned. However, greater transparency of such initiatives was required.

Academic quality assurance embraces more than reviews of individual institutions. In a number of countries institutional or subject reviews are followed by on-going discussions and contacts between quality assurance agencies and institutions, by system-wide dissemination of best practice in the form of quality assurance workshops and meetings among university faculty and administrators, by related evaluation and research studies, and by developmental activities designed to enhance the quality of teaching and learning. In Hong Kong, related activities included the TLQPR progress reports by the universities and the December 1998 workshop on teaching strategies and innovation sponsored by the Consultative Committee. In our evaluation, the following relevant findings were identified:

• a number of universities and academic units involved in the review felt that the Review Panel could have made more specific recommendations for improvement;

• several of the department heads interviewed had hoped the reviews would lead to subject-related suggestions for improving teaching and learning quality;

• several respondents felt that the UGC responses to their progress reports were somewhat perfunctory;

• a number of respondents commented positively on the December 1998 workshop sponsored by the Consultative Committee, and suggested this had been very useful. It was unclear to us, however, whether this type of activity would be continued by the Consultative Committee, and indeed, the future activities and agenda of the Committee were not clearly defined;

• while the Teaching Development Grants have been of some help to the institutions in implementing university-based quality assurance processes, particularly in the development of centres for learning and teaching, there appears to be little connection between UGC funding of teaching development and the insights into the university system gleaned from the TLQPR process. In our review of the TLQPR Reports, for example, we noted the following issues were common to a number of universities: the design of external examiner systems; use of student centred/problem-based teaching and learning; means of better involving teaching-learning centres in academic quality assurance work; means of obtaining feedback from external stakeholders; and means of obtaining student feedback on instruction. It is possible that collective action on these types of issues among the universities in addition to individual actions within each university might be an effective means of improving the quality of teaching and learning.

Strengths

1. TLQPR has placed attention to improvement of teaching and learning firmly on both UGC and institutional agendas.

2. It has involved institutional staff at both unit and institutional levels.

3. Its intention has been to focus on processes and to be non-prescriptive.

4. The Consultative Committee has emerged as a body with potentially long-term importance.

5. TLQPR has recognised the primary responsibility of institutions themselves over quality matters.

6. Teaching Development Grants (TDGs) have helped to support the development of university-wide quality assurance processes.

7. Size of the higher education system and close proximity of institutions should make dissemination and sharing of good practices easier and efficient.

Weaknesses

1. There has been uncertainty within institutions about the underlying rationale and purposes of TLQPR.

2. For some institutions, there was ambiguity about the requirements of the self-assessment and preparation for the review visit.

3. For all institutions, there was a high workload in a context of pressures from other reviews; the workload was particularly high because of the involvement of academic units in the review process.

4. Reports:

   • a language of assessment crept into reports

   • substantive issues, which seemed go beyond the focus on quality processes, crept into reports

   • sometimes insufficient evidence and rationale was provided to support the conclusions and recommendations

   • the function of the unit reports was not always understood correctly.

5. There has so far been insufficient dissemination and follow-up across higher education in Hong Kong to achieve the considerable potential of TLQPR for improvement and the sharing of good practice.

6. There is an absence of clear consequences arising from the results of TLQPR, i.e. little by way of rewards/sanctions compared with the results of RAE.

In this section we set out our conclusions and recommendations. These are framed by our remit to:

1. undertake an independent evaluation of the TLQPRs of the seven UGC-funded tertiary institutions in order to determine the extent to which the TLQPR processes have achieved their stated aims of (a) focusing attention on teaching and learning (as the primary mission of higher education institutions), (b) assisting institutions in their efforts to improve teaching and learning quality, and (c) enabling the UGC and the institutions to discharge their obligation to be accountable for quality;

2. recommend any improvements on the TLQPR process; and

3. carry out a broad-brush review of the effectiveness of the Teaching Development Grants (TDGs) and the projects supported by the UGC Central Allocation Vote (CAV) which are aimed at supporting initiatives to improve teaching and learning, and to

There were also achievements towards the second goal - to assist higher education institutions in their efforts to improve teaching and learning quality assurance processes. In some institutions, we found institutionalisation of quality management procedures where these had not previously existed in a formalised way. We also found a number of examples of innovation to existing quality processes. Sustainment of these present efforts and any new initiatives would be helped by the signal that there would be a second round of external reviews of teaching and learning in about 5 or 6 years after the first one (i.e. from 2001 onwards, or in the 2002 - 2004 triennium).

Given our remit, we cannot judge whether increased quality assurance processes led to increased quality of learning outcomes. Some interviewees suggested that a focus on universities' processes to assure quality of learning outcomes might be useful in a second round.

The second goal of TLQPR would be greatly assisted by system-wide learning processes with respect to quality assurance of teaching and learning. The dissemination of good practices across universities has not had much effect to date. Maybe we found this because our visit followed too soon after the dissemination activities that have already taken place, i.e. the publication of the progress reports by the universities one and half to two years after the TLQPR, the April 1997 seminar, and the Consultative Committee workshop to showcase good practices in December 1998.

First, continue the 'campaign for quality'

2. A second round of some form of external review is needed as a signal for the higher education community. The importance of addressing the quality of teaching and learning processes has grown with the growth of student numbers and the concomitant diversification of the student body, as well as the massification of teaching interactions.

3. To offset the continuous pressure of the RAE - even if the RAE criteria will be broadened to include scholarship of teaching - the UGC needs to show clearly that it is serious about what was the first goal in the first round of TLQPR, i.e. that education is the primary mission of higher education institutions. To make this message as forcefully as possible, a separate review of teaching and learning, not integrated with other reviews, would seem to be the best means. In reviews integrating management and education, quality of teaching and learning might become of secondary importance to the review process and thus to the higher education institutions. Moreover, management reviews do not easily reach down to the 'shop-floor' level where the most important contributions to quality education are made. In reviews integrating education and research aspects, international experience shows that the latter is dearer to academics' hearts and therefore would tend to take prime importance.

4. We realise that continuing separate reviews for education, management and research puts a heavy burden on the Hong Kong higher education community and on the UGC members especially. Accordingly, we advocate a lower frequency of reviews of quality of teaching and learning than the present three-year cycle for the RAE. A five- or six-year interval has been mentioned as desirable.

5. Another way of reducing the burden of external reviews while maintaining the benefits of separate reviews for education, management and research, might be to co-ordinate the preparatory work for the higher education institutions. Some benefits might be found especially with respect to the two process-oriented reviews of teaching and learning and management.

6. Moreover, reckoning that a single repetition of a separate review of quality of teaching and learning would result in a sufficient degree of internalisation of a quality culture, and that a third round would probably have much less added value, integration of the various reviews should be on the UGC agenda in the long run.

Our second main recommendation would be to continue to give attention to quality assurance processes with regard to teaching and learning. The reasons for UGC choosing a process review rather than evaluation of teaching in all subjects or departments remain - formulated inter alia in several papers by W.F. Massy and N.J. French - as valid as they were for the first round.

The focus of the next round of TLQPR should continue to be on teaching and learning quality assurance processes, but might pursue a narrower agenda. This would help limit the burden on the higher education community and on the UGC. It would also give an opportunity to reinforce the 'signal' already given out by introducing external reviews of the education function. Suggestions for a narrower focus, which could vary between institutions (see 3.3.4), might be for example:

• to review the quality assurance policies and processes implemented since the first review, with particular emphasis on the perceived effectiveness of those policies or processes from the point of view of students and of faculty members at the unit level;

• to review the procedures and measures to ascertain student learning that are being utilised by each university and each academic unit as a means of determining that they are in fact offering quality teaching. An added advantage of such a focus would be to shift attention from (formal) processes per se to their effectiveness in stimulating quality education;

• to review the 'value added' from the unit perspective of institution-wide quality assurance procedures.

The third main area of our recommendations concerns clarity about the aims and objectives of an external review.

1. The UGC should provide more clarity about the balance between the different aims and communication about them to the academics within the higher education institutions could be improved.

2. With regard to the first goal of TLQPR - to focus attention in the higher education community on education - we have already made clear in earlier parts of our report that this should be the basis of any approach for a second round of TLQPR.

3. If the UGC should, however, wish to emphasise the second goal - supporting higher education institutions to improve the quality of teaching and learning - an approach which takes the diversity of the Hong Kong higher education system into account might be adopted. Realising that the different higher education institutions have diverse traditions and processes of ensuring the quality of teaching and learning, UGC could consult with the individual universities about what, in their eyes, are the areas that warrant special attention from an external review panel⁶. In this way, a tailor-made external review could be provided for each institution. We assume that the tailor-made elements would be in addition to a common core that would be addressed in all higher education institutions. With this option, it is of utmost importance to establish a very focused approach to the common core.

4. If, on the other hand, the third goal - the accountability function - should become the primary aim of a second round, then the external review panel's role would become more like an auditor's role, i.e. to check that all higher education institutions have satisfactory and operational quality processes in place. A narrower focus could then be established by selecting certain crucial quality processes, e.g. those most closely connected with student learning outcomes. In the logic of audits, a lighter touch for the process could also be achieved by reducing the number of unit visits to a small selection of 'audit trails'.

As a fourth area of recommendations we think that, in order to attain internalisation of a quality culture in the higher education community, it is important to maximise the local ownership of the process. The TLQPR Panel was successful in its 'campaign for quality', but better embedding in the Hong Kong higher education community is desirable for the future. The main issue for decisions in this respect is the design of the procedure for a second round. There is enough time to develop a reasonably detailed and stable procedure for a second round; the learning process should be separated from the actual review process.

Many decisions around the design of the procedure depend on future choices by the UGC on the balance between the different aims and objectives for a second round. From the point of view of internalisation of a quality culture, a major factor in all design decisions should be the acceptability of the design in the Hong Kong higher education community. Whereas, therefore, external expertise (including international experiences) may be very useful, involvement of local experts and/or consultations with the Hong Kong higher education institutions ought to continue to be a major element of the design process.

We now turn in more detail to each of the elements of the design of a second round of external quality process reviews.

1. Preparation and self-evaluation

   The reflection at both the institutional and unit levels associated with the internal preparation was universally seen in the higher education institutions as one of the most useful parts of the TLQPRs. The emphasis on self-evaluation should be maintained.

   There is however, compared to the first round of TLQPR, room for streamlining, so that there will be good preparation in the higher education institutions but not 'over-preparation'.

   Professional advice about the internal preparation is an option to be studied further. The obvious parallel was made with the role of external consultants in the management reviews undertaken by the UGC, which was generally seen as a positive experience.Our caveat is that to remain effective within the higher education institutions, the issues of ownership and involvement of local staff should be taken into account carefully when deciding to involve external consultants.

2. Review panel

   As we stated before, the TLQPR Panel chair and members were very successful in designing and implementing the first round of TLQPR. Based on the experiences of this we can make the following recommendations:

   • the Review Panel should be smaller in a second round. The exact size depends on the goals of visits to the higher education institutions. Obviously, an audit panel can be smaller than a panel that operates in a consultative mode for a large number of units;

   • the panel should be composed of local higher education peers, complemented by foreign higher education experts;

   • the Panel's chair should be

     1. independent,

     2. expert on higher education and higher education quality management, and

     3. accepted by the Hong Kong higher education system;

   • the Panel need not have (completely) the same composition from one university to the other;

   • a professional secretary (experienced in higher education and higher education quality management in Hong Kong) could be considered to support the Panel and to write the Panel's report. The relative efficiency to the UGC in 'outsourcing' or 'insourcing' this function should be part of such considerations;

   • training for Panel members should be introduced so as to minimise dependence on the continuity of Panel members from one university to the other. This should include training in questioning skills;

   • an established protocol of questions to guide Panel Members during visits should be considered.

3. Site visits

   The site visits of the TLQPR Review Panel to the higher education institutions were, on the whole, valued positively. The principle of the inclusion of unit visits was valued but the visits were too short to be of great value to the units concerned. The unit visits appeared to have functioned successfully to bring the 'campaign for quality' to a wide audience. We would, therefore, recommend that this practice be continued in a second round but sufficient time should be allowed to ensure their usefulness.

   The length of the visits was perceived to be very short, even for the limited purpose of getting insight into quality processes. In a second round each meeting should last sufficiently long enough to establish effective communication between the reviewers and the interviewees. Visit length might vary according to size and complexity of the institution.

4. Panel's report and analytic framework

   The reports' recommendations were sometimes seen, however, to go beyond the evidence, or to reflect facts and interpretations that might not have reflected the general situation in the university or the view of the majority of the Panel members. Sometimes the Panel raised substantive issues, which seemed go beyond the focus on quality processes. In a second round of TLQPR, the Panel should avoid this.

   The Panel's analytic framework in the form of the questions/areas to be addressed received broad support. Yet it had a more unifying or compliance-inducing effect on the higher education institutions than intended by the Review Panel and the UGC.

   The treatment of unit level reports should be reconsidered in a second round. Opinions among our interviewees varied as to whether the unit reports, which in the first round appeared anonymously and as an appendix to the TLQPR report, should be deleted altogether, or should mention names of units. In our view, the reconsideration should take the aim of reporting into account in the context of the overall aims and objectives of the whole process. In an audit-type exercise, few units would be visited and then only to check the veracity and actual operations of university-level policy statements; publication of unit-level reports would, therefore, not be expected. In an improvement-oriented mode, emphasis should be given to detailed operational feedback to the institutions and units visited.

5. Follow-up by UGC

   The phrase that TLQPR outcomes would 'inform funding' was well known in the higher education institutions and was a source of confusion. More clarity is needed about the link between reviews and UGC funding decisions.

   Rewards for innovations in teaching and learning, i.e. through the TDG and CAV grants, have had positive effects. While a full evaluation of the TDGs was not within our remit, the money appeared to be well spent.

   In section 2.9 on TDGs, we mentioned the balance between the tranches of TDG distributed on student numbers and on competitive bidding, respectively. An additional way to reward higher education institutions (which prove through TLQPR to have good processes in place for the allocation of TDG money and for monitoring the effective use of this money) would be to build on trust, complemented with monitoring of the use of the TDGs en bloc and ex post, rather than institutions continuously reporting on each project.

   The publicity about TLQPR was not optimal; in a second round the UGC should develop a more active role in informing the public and press.

   Sustained positive reinforcement of the signal that teaching and learning are the main functions of higher education institutions, and especially positive reinforcement of innovation of teaching and learning in between two rounds of TLQPR, is of utmost importance. TDGs, mentioned above, play an important role in this respect.

   In the light of support for institutional quality assurance activities, more investment should be made in on-going, informed quality enhancement and dissemination activities, in addition to the TLQPR process and the TDGs. Consideration should be given to creating a separate unit for such activity under the direction of the Heads of Universities Committee (HUCOM) and/or under the Consultative Committee. This unit might engage in the following types of activities:

   • continued encouragement of dissemination of best practice with respect to quality assurance among the Hong Kong universities;

   • encouraging the development of subject-level review processes with an emphasis on continuous improvement of teaching and learning within each university. UGC funds might be designated for (co-operative) institutional activities in this area;

   • developing subject-level reports on the teaching and learning developments that could serve as the basis for workshops for relevant faculty (cf. system-wide subject reports published in the Netherlands, UK, and Denmark);

   • a system-wide study of the strengths and weaknesses of the external examiner system with recommendations on how this system might be improved to enhance teaching and learning, especially in relation to the introduction of credit-unit based curricula;

   • developing a regular system-wide survey of student learning activities that could serve as an indicator of the level of student involvement in learning in each university. Similar surveys are being developed in the US as a means of tracking the quality of undergraduate teaching and learning.

6. Consultative Committee

   The emergence of the Consultative Committee as a responsible and respected group representing the universities in the area of quality assurance was a serendipitous and valuable consequence of the TLQPR. UGC should continue to facilitate the Consultative Committee, for this reason.

7. Organisational locus of a second round of external quality process reviews

   To signal the importance attached to teaching and learning by the UGC and to offset the research push of the RAE, we would recommend that a second external review of teaching and learning is given symbolic recognition by remaining directly under the aegis of the UGC.

   In other, equally important tasks, on-going support should be given to the teaching and learning functions of Hong Kong higher education institutions, especially the inter-university learning about good practices. The Consultative Committee or another appropriate collective of the higher education institutions could take centre stage with the support of the UGC.