Team Members

The University Grants Committee of Hong Kong, as part of its on-going activities to assure quality and value for money in the higher education sector, embarked on a programme of Teaching and Learning Quality Process Reviews (TLQPRs) during the period September 1995 to April 1997. The goals of the reviews were:

• to focus attention on teaching and learning as the primary mission of Hong Kong's tertiary institutions;

• to assist institutions in their efforts to improve the quality of teaching and learning; and

• to enable the UGC and the institutions to discharge their obligation to maintain accountability for the quality of teaching and learning.

Before embarking on a next round of TLQPRs, the UGC on recommendation of the Consultative Committee (which acts as a representative of the universities) commissioned the Center for Higher Education Policy Studies (CHEPS) of the University of Twente in the Netherlands to undertake an independent evaluation.

B The evaluation

CHEPS constituted an evaluation team with experience in higher education quality processes internationally to carry out the evaluation. The tasks of the evaluation team were:

• to undertake an independent evaluation of the TLQPRs of the seven UGC-funded tertiary institutions in order to determine the extent to which the TLQPR process has achieved its stated goals;

• to recommend any improvements on the TLQPR process; and

• to carry out a broad-brush review of the effectiveness of the Teaching Development Grants (TDGs) and the projects supported by the UGC Central Allocation Vote (CAV) which are aimed at supporting initiatives to improve teaching and learning, and to recommend any improvements.

The full evaluation team visited Hong Kong from 11-16 April 1999. During this visit, interviews were held with representatives of institutions at institutional, faculty and departmental levels, including institutional members of the Consultative Committee and the TLQPR Review Panel. The evaluation team met with just over a hundred staff in the institutions. In addition, the team met the former and current convenors of the TLQPR Consultative Committee, members of the UGC Secretariat, Chair and members of the Quality Assurance Review Sub-Group, Representatives of the TLQPR Steering Group, and the Chair of the Teaching Development Sub-Group.

Interviews were semi-structured around the following three main issues:

• to find out the effects of the first round of TLQPR in the institutions;

• to find out strengths and weaknesses of the present TLQPR system;

• to discuss suggestions on how the system could be improved.

• TLQPR has placed attention to improvement of teaching and learning firmly on both UGC and institutional agendas.

• It has involved institutional staff at both unit and institutional levels.

• Its intention has been to focus on processes and to be non-prescriptive.

• The Consultative Committee has emerged as a body with potentially long-term importance.

• TLQPR has recognised the primary responsibility of institutions themselves over quality matters.

• Teaching Development Grants (TDGs) have helped to support the development of university wide quality assurance processes.

• Size of the Hong Kong higher education system and close proximity of institutions should make dissemination and sharing of good practices easier and efficient.

• There has been uncertainty within institutions about the underlying rationale and purposes of TLQPR.

• For some institutions, there was ambiguity about the requirements of the self-assessment and preparation for the review visit.

• For all institutions, there was a high workload in a context of pressures from other reviews; the workload was particularly high because of the involvement of academic units in the review process.

• Reports:
• a language of assessment crept into reports

• substantive issues, which seemed go beyond the focus on quality processes, crept into reports

• sometimes insufficient evidence and rationale was provided to support the conclusions and recommendations

• the function of the unit reports was not always understood correctly.

• There has so far been some, but insufficient dissemination and follow-up across higher education in Hong Kong to achieve the considerable potential of TLQPR for improvement and the sharing of good practice.

• There is an absence of clear consequences arising from the results of TLQPR, i.e. little by way of rewards/sanctions compared with the results of RAE.

• Hong Kong higher education institutions have diverse historical backgrounds and missions. As a consequence, the appreciation of TLQPR was different in each institution.

• The effects of TLQPR in each institution were positive but different. The main divide seemed to be between institutions that had experience of external quality assurance by the Council for National Academic Awards and the Hong Kong Council for Academic Accreditation and institutions for which formal external review of teaching was a mainly new experience. In the latter, the TLQPR often led to the creation of formal quality assurance processes. In the former, effects were mainly found in innovations to existing processes.

• Other factors reinforced the effects of TLQPR. Teaching Development Grants (TDGs) were the most important of these but other examples, i.e. the appointment of new senior managers with special responsibility for quality management, acted as a parallel force.

• The open attitude of the TLQPR Panel, together with the Consultative Committee and other university representatives, in evolving the design of the quality review process during the course of the TLQPRs showed a commitment to learning and adapting TLQPR in the light of experience. However, this also led to a degree of confusion in the institutions about the objectives, the consequences (especially funding), and the requirements of TLQPR.

• The overall major conclusion is that 'TLQPR was the right instrument at the right time', because the review was a positive stimulus to institutional attention to teaching.

• There were clear achievements with respect to the first goal of TLQPR - to focus on teaching and learning as the primary mission of higher education institutions. The signal given by TLQPR that teaching and learning were as important as research was seen by many as the prime impact. However, as the RAE is tied to substantial funding and TLQPR is not there was conflicting pressure between the RAE and TLQPR in the institutions, especially at the 'shop-floor' level, and RAE appears still to be the dominant force.

• With respect to the second goal of TLQPR - to assist higher education institutions in their efforts to improve teaching and learning quality assurance processes - there appeared to be institutionalisation of quality management procedures in some institutions, especially those that did not have such procedures in place before. A number of examples of innovation in existing quality procedures were also found. Sustainment of present efforts and new initiatives would be helped by the signal that there would be a second round of external reviews of teaching and learning processes.

• Achievement of the second goal of TLQPR would be greatly enhanced by the dissemination of good practices across universities. Activities in this respect have been minimal to date, although the TLQPR Review Seminar (April 1997) and the Conference on innovative teaching approaches (December 1998) are important beginnings for such continued activities.

• The TLQPR has certainly contributed to achieving the third goal - accountability of the UGC and the higher education institutions to society - firstly, through the review process itself and secondly through the publication of the reports and the institutional progress reports. However, in the eyes of the higher education community, the press coverage of the reports was not seen as a balanced reflection of the process.

• Continue the campaign for quality. The higher education community needs a signal that a second round of external review of teaching and learning will be undertaken. It should not be integrated, yet, with the other review processes of the UGC. Continuing the separate reviews for education, research and management would put a heavy burden on the higher education community and the UGC. Therefore, reviews of the quality of teaching and learning should be at a lower frequency than the present three-year cycle of the RAE. A five- or six-year interval has been mentioned as desirable. The burden of external reviews, while maintaining the benefits of separate reviews, might be reduced by greater co-ordination.

• The focus of the next round of reviews should continue to be on teaching and learning quality assurance processes but might pursue a narrower agenda and could vary between institutions. Examples are (i) a review of the quality processes implemented since the last review, and (ii) a review of the measures of student learning that are being utilised by the institutions.

• There should be clarity about the aims and objectives of the next round of external review and this should be communicated effectively.

• In order to attain internalisation of a quality culture in the higher education community, it is important to maximise the local ownership of the process.

Preparation and self-evaluation: internal preparation was seen as one of the most useful parts of the TLQPRs. The emphasis on self-evaluation should be maintained. There is, however, room for streamlining and professional advice about internal preparation is an option to be studied further.

Review Panel: should be smaller but will depend on the goals of the visits. It should be composed of local higher education peers complemented by foreign higher education experts. The chair needs to be independent, have experience of higher education quality management and be accepted by the Hong Kong higher education institutions. The composition of the panel need not be the same for all institutions. A professional secretary should be appointed to support the panel and to write the panel's report. Training for panel members should be introduced.

Site visits: were on the whole valued positively, in particular the unit visits. The length of the visits was perceived to be very short, even for the purposes of getting insight into quality processes. In a second round, visits should allow sufficient time to establish effective communication between the reviewers and the interviewees. This is likely to require an extension of time devoted to meetings during the visits and a possible extension of the overall visit length, at least for larger institutions.

Panel's report: the TLQPR reports touched on significant points in all institutions and on the whole were seen to reflect the situation in the institutions accurately. However, conclusions sometimes went beyond the evidence provided in the report and issues were raised which seemed to be outside the focus of the TLQPR process. The treatment of unit reports should be reconsidered in the context of the whole procedures, aims and objectives.

The analytic framework received broad support.

Impacts and follow-up by higher education institutions: all institutions appeared to have initiated, enhanced or stimulated changes and improvements with respect to quality processes following the TLQPR. Many of the initiatives seemed to be steps towards a quality culture and/or towards institutionalising quality management.

Follow-up by the UGC:clarity is wanted about the link between reviews and UGC funding decisions. Rewards for innovations, i.e. through TDG and CAV grants, have had positive effects and variants on the processes by which such grants are made to take account of the TLQPR process should be considered.

The UGC should develop a more active role in informing the public and press in a second round.

Sustained positive reinforcement of the signal that teaching and learning are the main function of an institution is of utmost importance. TDGs play an important role in this respect.

More investment should be made in quality enhancement and dissemination activities. Consideration should be given to creating a separate unit for such activities under the direction of the Heads of Universities Committee (HUCOM) and/or the Consultative Committee.

Consultative Committee: the UGC should continue to facilitate the Consultative Committee as its emergence as a responsible and respected group representing the universities was a serendipitous and valuable consequence of TLQPR.

Organisational locus of a second round of external quality process reviews: in order to signal that teaching and learning are important and to offset the research push of the RAE, review of teaching and learning should be given symbolic recognition by remaining directly under the aegis of the UGC.

1.1.1 The University Grants Committee of Hong Kong (UGC) is a non-statutory advisory body whose members are appointed by the Chief Executive of the Hong Kong Special Administrative Region (SAR) of the People's Republic of China. Its members comprise distinguished overseas academics, prominent local professionals and business people, and senior locally based academics. It is responsible for advising the government on the academic development and funding of Hong Kong's institutions of higher education. It also plays a central role in assuring the quality of higher education provision in the tertiary education institutions under its aegis.

1.1.2 As part of its ongoing activities to assure quality and value for money in the higher education sector, the UGC embarked in 1995 on a programme of Teaching and Learning Quality Process Reviews (TLQPRs). The reviews were undertaken, during an 18 month period from September 1995 to April 1997, by a specially constituted TLQPR Panel consisting of eight UGC members, eight members from the UGC-funded institutions and two overseas experts on higher education quality assurance. One of the members acted as the chair of the Panel. The TLQPR Panel made site-visits to the UGC-funded institutions under scrutiny and reported its findings to the UGC. The framework used by the UGC for the TLQPRs is attached as Appendix 1 to this report.

1.1.3 Two other initiatives of the UGC may have influenced the institutions' appreciation of TLQPR: (a) a 3-yearly Research Assessment Exercise (RAE) based on peer assessment of research performance, the next round of which in 1999 will be the third one; and (b) a first round of Management Reviews focussing on the quality of the institutions' management processes.

1.1.4 At the start of the TLQPR process UGC had seven institutions under its aegis: City University of Hong Kong (CityU), Hong Kong Baptist University (HKBU), Lingnan College (LC), Hong Kong Polytechnic University (PolyU), The Chinese University of Hong Kong (CUHK), The Hong Kong University of Science and Technology (HKUST), and The University of Hong Kong (HKU). In 1996 The Hong Kong Institute of Education (HKIEd) was designated to the UGC, but to date a TLQPR of this institute had not been undertaken to date.

1.1.5 Before embarking on a next round of TLQPRs, the UGC, and the Consultative Committee (which acts as a representative of the universities) agreed that an independent evaluation of the first round of TLQPRs would be desirable. The UGC commissioned the Center for Higher Education Policy Studies (CHEPS) of the University of Twente in the Netherlands to undertake this independent evaluation.

1.2 Aims and Methods of the Evaluation

1.2.1 CHEPS constituted an evaluation team of members with experience in higher education quality processes to carry out the evaluation. The members were: Don Westerheijden and Adrian Verkleij (Center for Higher Education Policy Studies (CHEPS), University of Twente, The Netherlands); John Brennan (QSC, Centre for Higher Education Research and Information, The Open University, UK), and David Dill (University of North Carolina at Chapel Hill, USA). Tarla Shah (QSC, Centre for Higher Education Research and Information, The Open University, UK) acted as academic secretary to the evaluation team.

1.2.2 The tasks of the evaluation team were:

1. to undertake an independent evaluation of the TLQPRs of the seven UGC-funded tertiary institutions in order to determine the extent to which the TLQPR process had achieved its stated aims of (a) focusing attention on teaching and learning (as the primary mission of higher education institutions), (b) assisting institutions in their efforts to improve teaching and learning quality, and (c) enabling the UGC and the institutions to discharge their obligation to be accountable for quality;
2. to recommend any improvements on the TLQPR process; and
3. to carry out a broad-brush review of the effectiveness of the Teaching Development Grants (TDGs) and the projects supported by the UGC Central Allocation Vote (CAV) which are aimed at supporting initiatives to improve teaching and learning, and to recommend any improvements.

1.2.4 During an explorative three-day first visit to Hong Kong in January 1999 by three members of the evaluation team, discussions took place with the UGC Secretariat, the Consultative Committee and the TLQPR Steering Group. In addition the evaluation team visited two universities (PolyU and CUHK) to get acquainted with the local context. This first visit led to the proposal for the interview scheme for the full evaluation team visit which was discussed with the Consultative Committee and the UGC Secretariat.

1.2.5 The full evaluation team visited Hong Kong from 11-16 April 1999. During this second visit interviews were held with representatives of each of the institutions (at institutional, faculty and departmental levels, including the institution's members of the Consultative Committee). In addition, more dedicated interviews were undertaken on the following issues: (i) role and position of the university's teaching and learning support centre, (ii) students' participation in quality of teaching processes, (iii) the university's members of the TLQPR Review Panel, (iv) the university's experience with the Central Allocation Vote and the Teaching Development Grants. Overall, we met with just over a hundred staff in the institutions.

We also met the former and current convenors of the Consultative Committee.

At the UGC, we met with members of the Secretariat, the Chair and members of the Quality Assurance Review Sub-Group, representatives of the TLQPR Steering Group, and the Chair of the Teaching Development Sub-Group.

1.2.6 Semi-structured interviews were used to ensure coverage of all relevant aspects, leaving room for improvisation and further probing. The interviews were structured around the following three main issues:

1. to find out the effects of the first round of TLQPR in the institutions (facts as well as perceptions and opinions);
2. to find out strengths and weaknesses of the present TLQPR system;
3. to discuss suggestions on how the TLQPR system could be improved.

1.3 The Report

This Report contains our findings and recommendations. Our recommendations are based on (i) the results of our fact-finding, (ii) our conclusions of strengths and weaknesses, and (iii) the support found for suggestions for improvement to the TLQPR process.
 
 

A draft of the report was sent to the UGC Steering Group, the Consultative Committee and the UGC Teaching Development Sub-Group for factual corrections and comments. We gratefully acknowledge their responses and have adjusted the draft in places. However, the Report is the sole responsibility of the evaluation team.

The term 'universities' is used in this report in a generic sense to refer to all institutions which were reviewed.

1.4 Acknowledgement

We would like to thank all the people involved in this evaluation for their open and frank discussions and for their willingness to share their experiences and ideas with us.

General background

2.1.1 The recent past in Hong Kong's higher education has been marked by expansion on a scale that has been great even by current international levels. Student numbers have expanded by 50 per cent during the first half of the 1990s only to an age participation rate of 18 per cent. There have been 3500 new staff appointments. There has also been growth in research, reflecting a greater emphasis being placed on it by the UGC. There has been growth in private provision of higher education.

2.1.2 As elsewhere, with growth has come greater diversity. In Hong Kong, this has included an increase in postgraduate work, greater emphasis on continuing professional education (much of which is outside the universities), greater diversity of programmes and in the student mix. To this educational diversity needs to be added the cultural diversity which characterises Hong Kong and which, in the universities, finds staff whose educational backgrounds range principally from American, Australian, British and Chinese traditions in somewhat different mixes in different institutions.

2.1.3 Hong Kong's higher education has traditionally been well resourced and this has been maintained during the recent expansion. Indeed, the unit of resource has actually risen during the period of expansion.

2.1.4 The period of expansion is now at an end. No significant further growth is predicted for undergraduate education for the next few years although some further expansion of postgraduate and continuing professional education is expected to occur, much of it privately funded. During this period of relatively steady-state, the UGC appears to be placing its emphasis on three things: (i) the achievement of greater efficiency in the system now that the period of rapid expansion is over, (ii) how to maintain flexibility in the higher education system - seen as more difficult in the absence of expansion, e.g. problems of under-recruiting subjects, and (iii) improving the English language competence of students.

2.1.5 For the future, the UGC emphasises the following: (i) the development of 'areas of excellence' within the universities, (ii) attention to multi-lingual and multi-cultural issues, and (iii) the development of a greater regional role for Hong Kong's higher education, within China and in South East Asia more widely.

2.1.6 Underlying all of this are some clear values underpinning higher education policy in Hong Kong. These can be summarised as (i) a continued emphasis on a 'hands off' approach by government and the UGC (funding continues to be essentially block grant); (ii) an emphasis on quality - the RAE, TLQPR, Management Reviews and the Teaching Development Grants can be cited as evidence of this, the expressed concerns about English, and the UGC preference for under-recruitment to maintain standards in those subjects where demand for places from well-qualified applicants is falling; and (iii) a commitment to 'campus education' for undergraduates - there is a large hostel building programme underway.

Specific features

2.1.7 During our visit a number of additional contextual features were observed that may have an influence on the quality of teaching and learning in the Hong Kong University sector over the next decade. The possible effects of this changing context may therefore need to be considered in the evolution of UGC policies regarding academic quality assurance.

2.1.8 A more specific aspect to the diversity referred to in 2.1 above concerns the diversity of traditions in Hong Kong's higher education institutions as to how quality is best maintained and improved. Some institutions have traditionally placed the greatest emphasis on the professionalism and excellence of their academic staff and afforded individuals the maximum autonomy to achieve high quality and standards within their own disciplinary and research traditions. Other institutions have developed a more collective institutional culture of concern about quality matters which has placed greater emphasis on institution-wide processes, both as a means of control but also as an opportunity for institutional learning. Some institutions are well-used to external scrutiny (in particular through their history of validation and accreditation by the Council for National Academic Awards (CNAA) and the Hong Kong Council for Academic Accreditation (HKCAA)) and have found ways of integrating the results and processes of external review within their internal review and quality assurance procedures. Others are having to come to terms with the very principle of external scrutiny, apart from traditional forms such as external examiners.

2.1.9 In other words, the institutional starting points for the introduction of a process such as TLQPR vary substantially. It might be expected, therefore, that TLQPR would face a different initial reception and have different kinds of effects according to the tradition and context provided by the individual institution. This poses a challenge for the design of an external quality system which aims to be equally effective in all institutions. Where the primary purpose of external quality assurance is support for the institution's own quality assurance and improvement processes, this might imply a system marked by flexibility and negotiation. If the primary purpose is external accountability, a more standardised system may be required.

2.1.10 The quality of teaching and learning in the university sector was historically maintained within a traditional framework that included the norms of the single subject honours degree and the use of external examiners to maintain the validity and reliability of degrees and examinations. In addition, the inclusion of research support in institutional base funding effectively meant that university faculty were under less direct pressure to engage in research and publication than was true in systems which awarded research funds competitively.

2.1.11 Over the last decade each of these contextual factors, which arguably helped maintain the quality of teaching and student learning, has been altered. All of the universities are now adopting the credit/unit form of organising instruction featuring much more complex forms of curricula and with a necessary reliance on continuous forms of assessment. The traditional role of the external examiner is therefore being adjusted. The adoption of the Research Assessment Exercise has, in the observations of a number of those interviewed by us, substantially changed the internal culture of many of the universities, reinforcing a general trend in higher education toward greater focus of faculty effort on research and publication. Furthermore, unlike the TLQPR, which is episodic, the pressure for research and publication is continuous and the RAE - with its tie to funding - occurs every three years. Most of the institutions that were previously within the orbit of the CNAA/HKCAA have now received university designation, which means they are no longer subject to the validation procedures of those agencies. In this new and changing context the quality of teaching and learning has become increasingly dependent upon the quality assurance procedures implemented by the universities themselves and, partially as a result of the forces just noted, many of these institutional-based quality assurance systems are themselves newly implemented and under continuing development.

2.1.12 Of course, institutions differ in many other ways that have implications for quality issues. Size is one, as are internal organisational structures and management practices. The balance between research and teaching is another, as is the disciplinary mix of the institution's educational and research activities. We do not believe that it is sensible to categorise institutions into general types but rather to attempt to recognise the unique mix of characteristics which defines a particular institution.

2.1.13 We would endorse the basic view of the UGC that institutionally based mechanisms are likely to be, over time, the most effective means of safeguarding the quality of teaching and learning. However, the environment of higher education in Hong Kong, as briefly sketched above, is both dynamic and continually changing. Therefore, we believe that attention to assuring the quality of teaching and learning must also be continuous and academic quality assurance procedures must be constantly improved to accommodate the ever-changing nature of higher education institutions and academic work.

International developments in the assessment of quality in higher education

2.1.14 Although accreditation of higher education has been firmly established in North America for most of the century, it is only within the last ten years that accreditation, and systems of quality assurance more generally, have become widely established in most parts of the world. Thus, the International Network of Quality Assurance Agencies in Higher Education, founded in 1991, now boasts nearly 100 members. The HKCAA is a founder member of the INQAAHE and the UGC has been a member since 1994.

2.1.15 There have been several international surveys of quality assurance systems carried out by bodies such as the EU, CEPES/UNESCO and OECD and both similarities and differences observed in their structures and functioning. The similarities have sometimes been described as a 'general model' of quality assurance in higher education, comprising (i) a co-ordinating agency at the national level, (ii) self-evaluation at the institutional level, (iii) external evaluation by expert 'peers', (iv) published reports, and (v) the absence of a direct link between quality assurance and funding of higher education (van Vught and Westerheijden, 1993¹). Most systems of accreditation and evaluation fit the main elements of this model.

2.1.16 However, as well as similarities there are also important differences between the various national systems of quality assurance: differences in context, in methods used and, as is becoming increasingly clear, in impact upon higher education (Brennan and Shah²).

2.1.17 Perhaps the most important differences in context concern the purposes of quality assurance. These are frequently described as being a mixture of 'accountability' and 'improvement' although the reality is rather more complex.

• The most commonly stated purpose is accountability. This appears to work in two principal ways. The first is by providing independent assurance to a principal stakeholder and funder - i.e. the state - that the quality of education is sufficient to justify the continued provision of public funds. This is an assurance of threshold quality, sometimes linked to a process of accreditation. The second way of achieving accountability is by providing comparative information to a much wider group of stakeholders - including crucially the customers or clients of higher education - so as to influence choices and decisions. This is much more a market approach with the quality assurance process working to provide information to enable the market to operate rationally.

• The second commonly stated purpose of quality assurance is to improve the quality of educational provision. But there are differences in the way this is intended to be achieved. The first, and most common, way is through the quality assurance process itself - the self-evaluation, the 'mirror' provided by external peers, the published national reports. A second but related approach is through the direct stimulation of innovation and educational development in individual institutions. A third approach is through the separate support for research and development relevant to quality of education.

• The third purpose of quality assurance is to stimulate competitiveness. By linking assessment to rewards of funding or reputation, by influencing the markets for students and for staff, quality assurance is thought to increase competition between institutions and between departments, providing incentives and encouraging responsiveness.

• A fourth purpose of quality assurance is to impose a quality check on new institutions, sometimes linked to a process of accreditation.

• A fifth and related purpose is to use quality assurance to assign institutional status in diversified systems.

• A sixth purpose concerns the transfer of authority from the state to institutions. It is frequently seen as the replacement of government control over the inputs to higher education by government supervision through assessment of processes and outputs of higher education. Although formally this is intended to represent a devolution of state power to institutions, the new accountability demands are often experienced as 'government centralism in a new disguise' (Goedegebuure et al, 1993³).

• A seventh purpose of quality assurance is to make international comparisons of the quality of education. This reflects the emphasis given to higher education as a determinant of national economic success.

2.1.19 There are important differences in the methods used by quality assurance agencies in different countries. The balance between institutional and programme level evaluation varies with relatively few systems having fully-developed mechanisms at both levels. There are also significant differences in what is meant by self-evaluation, in how peers are selected and trained, in the focus and conduct of visits, in the extent and the mechanisms through which student opinion is gained, and in the content and purposes of published reports. These differences partly reflect national traditions and circumstances but they also reflect the ownership and purposes of quality assurance. Systems which place a strong emphasis on accountability to the state tend to be more standardised, more quantitative and subjected to greater bureaucratic controls than do systems which place greater emphasis on improvement and which tend to allow greater autonomy to academic peers and greater variation in methods to meet differing institutional needs.

2.1.20 The differences mentioned above point to considerable international debate about the purposes, methods and impact of quality assurance. There is certainly no one best way of going about the assurance of quality. The uniqueness of individual higher education systems and institutions also needs to be taken into account.

2.1.21 Quality assurance developments in Hong Kong are part of these world-wide discussions and trends in higher education. The methods used for TLQPR build on growing international experience and indicate a choice for an approach that represents a certain balance between the different purposes, especially accountability and improvement. This in turn reflects the role of the various parties in the Hong Kong higher education system, in particular the role of the UGC and its responsibility for high quality education.

2.2 Aims of TLQPR

2.2.1 The stated aims of the TLQPRs were:

• to focus attention on teaching and learning as the primary mission of Hong Kong's tertiary institutions;

• to assist institutions in their efforts to improve the quality of teaching and learning;
• to enable the UGC and the institutions to discharge their obligation to maintain accountability for the quality of teaching and learning.

2.2.3 There was no dissent about the legitimacy of the second aim, although there existed differences between institutions in the extent to which people felt that TLQPR was achieving it. The differences partly related to institutional origins and characteristics. TLQPR had clearly 'kick-started' teaching quality activities in the pre-1994 universities, which had not been subject to the external review requirements of the CNAA/HKCAA. For the more recently designated universities, there was much less consensus about whether TLQPR added much that was new. For all institutions, however, there was an appreciation that the existence of TLQPR bolstered institutional commitment to teaching quality issues and provided a useful encouragement to progress internal quality initiatives.

2.2.4 Perhaps unsurprisingly, it was the third aim that was the source of most confusion. Although there was widespread recognition within institutions about the need for accountability in general terms, the statement by UGC of an intention to use TLQPR to 'inform funding' was widely quoted. There was a lack of clarity about whether and how this aim was being or would be implemented.

2.2.5 In our view TLQPR has gone a considerable way towards achieving its main aims. The signal to higher education about the primary importance of teaching and learning has been effective and generally welcomed. Initiatives to improve the quality of teaching and learning are occurring in all institutions. It would of course be wrong to give TLQPR the credit for all of them but it seems certainly to be the case that TLQPR has been a significant source of initiatives on teaching quality taken at leadership levels within institutions. In our view, the existence of TLQPR and the publication on the web of its reports provide a useful strengthening of public accountability in the Hong Kong higher education system.

2.2.6 Insofar as we have reservations about the aims of TLQPR, they concern the clarity of its broader purposes and rationale. As staff in one institution put it, the 'why' question. The stated aims are understood but the underlying reasons for TLQPR being done at all are not everywhere appreciated. There remains suspicion of a hidden but underlying accountability agenda. This would not necessarily be opposed within institutions. Some of the people we interviewed would welcome a quality assurance system whose outcomes had real consequences for institutions: rewards and punishments. Others would not. But all would welcome greater clarity in this respect with regard to any future rounds of TLQPR.

2.2.7 In considering the aims of any new public policy initiative such as TLQPR, it is always worth being mindful of the possibility of there being consequences unforeseen by the original proponents of the policy. Unintended consequences can be either positive or negative; examples of both are provided below but here we would like to draw attention to some of the important ones. One positive one is the emergence of the Consultative Committee as a possible long-term player on the Hong Kong higher education scene. Other less positive consequences, though hardly surprising ones, have been the workload requirements on universities (in a context of a more general growth in the number and demands of external review processes), and the way in which the Hong Kong media have 'over-reported' and perhaps tended to distort the conclusions of TLQPR reports. In addition, and as TLQPR or some successor scheme is developed, there may be significant consequences for the internal cultures of higher education institutions, in particular the relationship between the views and interests of institutional leaders and the views and interests of academic departments. This will partly be a function of the quality assurance methods and approach used. See the discussion in section 2.8 of this report.

2.2.8 Although it was not our purpose to give a general overview of the effects of the first round of TLQPR, we could conclude that TLQPR put quality of teaching and learning higher on the universities' agenda by giving support to institutional quality processes and by stimulating the initiation of new ones. It offered the opportunity for institutions to examine the appropriateness and adequacy of their policies and guidelines on quality. Several times it was reported that ad hoc initiatives taken to prepare the self-evaluation and the review were now functioning on a permanent basis, that the position of the university's teaching and learning support unit (or equivalent) was strengthened, that more attention was given to student evaluations. It led to an increased dialogue and a greater willingness to exchange ideas and open up practice to external observation. It improved recognition of good teaching by introducing teaching awards, teaching fellowships, teaching development grants, and the need for training of teachers.

2.3 The Introduction and Preparation of TLQPR

2.3.1 The introduction of TLQPRs did not come as a surprise for the institutions. In 1993 the UGC had recommended, and the Government had agreed, that UGC should undertake periodic process reviews of all the UGC funded institutions. From that time on several discussions and conferences were arranged (in which institutions were involved) to discuss the development of an approach that was suitable for the higher education system in Hong Kong. Several approaches were discussed, varying from subject assessments to audits. In the end Teaching and Learning Quality Process Reviews (TLQPRs) were agreed. Although university members were involved in these discussions, almost all universities reported that they felt unsure about what to expect from and how to prepare for TLQPR. During the course of the first round of TLQPR this feeling diminished, due to the informal sharing of experiences between the institutions which had already been visited and the institutions which were preparing for a visit. The Hong Kong based members of the TLQPR Review Panel greatly contributed to the sharing of experiences.

2.3.2 The review process started with an initial half-day visit by the TLQPR Review Panel to the institutions. Panel members, as well as the institutions, reported to us that these introductory meetings were very useful. However, the meetings were mainly for the purposes of explaining the aims of the review process and to persuade faculty members of the necessity or inevitability of TLQPR. At least one issue was reported to remain unclear, which was whether, and if so how, the outcomes of TLQPR were to 'inform funding⁵'.

2.2.3 However, institutions felt ill-informed about how to interpret the guidelines given by the UGC for the preparation of the 20-page self-evaluation document, which would serve as the basic information for the TLQPR Panel and the site-visits.

2.4 The Analytic Framework

2.4.1 The analytic framework (see Appendix 1) used in the reviews had seven main elements: TLQPR goals; TLQPR methods; Report preparation; TLQPR dimensions; Teaching and Learning Processes; Quality improvement and assurance; Sample approaches. Teaching and learning processes were divided into five sub-processes: curriculum design, pedagogic design, implementation quality, outcomes assessment, resource provision. Quality improvement and assurance had four principal elements: quality programme framework, direct quality programme activities, quality programme support, values and incentives.

2.4.2 The analytic framework was supplemented by a matrix, which showed how the four quality assurance methods could be related to the five teaching and learning processes. The matrix was not intended to be prescriptive but to illuminate the various elements of the framework.

2.4.3 By and large, the analytic framework appears to have been accepted within the institutions as providing a reasonable basis for the questions raised during the review process. Although the framework evolved somewhat during the cycle of reviews, this did not seem to create serious problems for institutions. There seemed to be greater clarity about what was going to be involved in the review process within institutions which were reviewed later on in the cycle. There appears to have been somewhat less satisfaction with the matrix which some people found to be rather prescriptive, implying a 'best way' to address quality matters.

2.4.4 We felt that the framework provided a fair, although broad, basis for the reviews. However, we have some sympathy for the views expressed in some institutions that a particular approach to quality underpinned the framework and, in particular, the way it was interpreted by the review team and expanded in the matrix. This point is explored in section 2.7, which considers the TLQPR reports. But, in addition, the very breadth of the framework may have been a weakness in the review process in some institutions. This relates to the recommendation made elsewhere in our report that a rather sharper focus should be considered for any subsequent rounds of TLQPR. If TLQPR is going to be effective in supporting the development of effective institutional level quality assurance procedures, it may need to reflect more closely the diversity of approaches being taken within institutions. For example, at least one major institution is looking to focus its quality processes on learning outcomes. The TLQPR framework is sufficiently broad to accommodate different institutional approaches but there may be a case for applying it in a more selective way in future rounds.

2.5 The Self-Evaluation

2.5.1 Those interviewed in the institutions found the self-evaluation element of the TLQPR process very useful, but the reasons for this appreciation and the added value of the self-evaluation differed from university to university. In general, the framework developed by the TLQPR Panel offered, according to the universities, the opportunity to analyse and evaluate the comprehensiveness of their - formal and informal - quality assurance processes. Those institutions which had experience of the procedures of CNAA and HKCAA, however, sometimes reported limited added value or complained about a repetition of procedures, or of introducing a new terminology to describe familiar practices. Some of the old universities disputed the relevance of information about (formal) processes only and expressed a preference for assessment of outcomes.

2.5.2 Although the framework used was quite helpful for analysis, the self-evaluations submitted to the TLQPR Panel were often perceived to be unclear or incomplete and additional material was often requested from institutions. The idea that existing documents could be adequate was optimistic. During our discussions several TLQPR Panel members reported that the deluge of paper nearly drowned them. This was mostly due to the decision taken by some institutions and departments to send all material available 'to impress the panel'. Others mentioned 'a documents arms race' based on what they saw other institutions do. On the whole, some Review Panel members concluded that the written information base of TLQPR was weak with respect to its analytical and evaluative content.

2.5.3 Some institutions commented on the parallel of the role of external consultants who advised and supported the UGC Management Review Steering Group and Panels in preparing the documentation for the on-going management reviews, and recommended similar procedures for a next round of TLQPR. We support the idea that preparations for a next round of TLQPR need streamlining. Using external consultants is an option to be studied further, but this option should be balanced to the need of strengthening faculty members' ownership of and involvement in quality assurance and quality improvement procedures.

2.6 The Site Visit

2.6.1 The site visit of the TLQPR Review Panel consisted of a one and a half day visit to each institution. The site visits each followed the same format, with introductory plenary sessions with university administrators, half-day visits by members of the review panel to representative departments/faculties, an executive session for the panel, and an exit plenary session with university administrators. Members of the Review Panel reported to us that the visits to the unit levels were most informative in learning about the processes for assuring the quality of teaching and learning in a particular institution. Over the course of the reviews, the panel placed more stress on the unit visits and reduced somewhat the plenary sessions.

2.6.2 In our interviews and review of relevant materials the following points about the site visits emerged.

• Each university provided information and prepared for visits of all academic departments/units, even though only a maximum of 12 per institution was to be visited by the Review Panel. A number of heads of academic units reported that the effects of preparing for a review, even though they were not selected for review, was salutary and led to productive discussions of teaching within their departments that have had lasting effects. Several departments also reported finding the framework of questions provided by the Review Panel of genuine assistance in fostering faculty discussion about means of improving teaching and learning. Based on our interviews, this across the board impact at the unit level appeared to be an unanticipated benefit of the design of the reviews.
• The unit visits were also seen by university administrators to be worthwhile. The university management received feedback about variations in quality assurance processes at the unit level of which it had frequently been unaware. At one university the administration reported that several units had actively volunteered to be visited because they wished to improve their processes. The University of Hong Kong reported at the TLQPR Seminar (8 April 1997) that the identification of good practices in the 'Unit-level Observations' permitted the university to recommend these examples to other faculties and departments as a means of assisting their development of quality processes.
• A number of units reported that the visits created an administrative burden and were time consuming, primarily it appeared because of uncertainty as to the focus of the review, and particularly because it was time consuming to develop the documentation required.
• Several department heads interviewed expressed dissatisfactions with the length of unit visits. They mentioned that the attention to process was necessarily superficial and often biased toward documentary evidence given the short amount of time panel members had to visit with each unit. They suggested lengthening the unit visits to permit more time for discussion, participation, and observation at the unit level by review panel members, so that they could become better informed of the informal as well as formal processes of quality assurance being employed. Similar recommendations were offered by one of the presenters at the TLQPR Seminar (8 April 1997).
• Those interviewed were generally impressed with the competence and fairness demonstrated by the Review Panel members during their visits. But several respondents also remarked that not all members of the Review Panel appeared to be clear as to what they were trying to accomplish. Members of the Review Panel themselves complimented their fellow panel members on their objectivity, and particularly in their skill in conducting unit visits and developing unit reports. The Massy and French (1997) paper on the TLQPR suggested as an improvement the development of a more extensive protocol of questions to guide Panel members in their visit interviews. They also suggested that this standard list of questions could be supplemented with institution-specific questions developed through analysis of the institutional self-assessment documents.
• Several respondents from the Review Panel suggested that the affiliations of some members of the Panel might have compromised their ability to play an active role in Panel deliberations. For example, while complementing the objectivity of the institutional members of the Panel when reviewing their home institution, one interviewee noted that these Review Panel members appeared to have said little during the panel discussions of their own institution. Several Panel members also said that university administrators on the Review Panel appeared constrained from making comments that might be construed as critical of their administrative colleagues at the institution being reviewed. These observations may not be representative, but they raise potentially useful questions as to the make-up of the Review Panel.
• Several members of the Review Panel, as well as the Massy/French (1997) paper, noted that the opening plenary sessions were least valuable to the Review Panel and that more visit time might be set aside for unit visits.

2.7 The TLQPR Reports

2.7.1 Reports of TLQPRs were short and followed a common structure within which a section on 'Areas for Improvement' was normally the most substantial. In addition, reports contained a number of annexes: the TLQPR framework, the institution's self-assessment document, and the Review Panel's observations on individual academic units (anonymised) in the institution.

2.7.2 Institutions found the reports to be broadly accurate although it was pointed out in some institutions that the brevity of the review visits inevitably meant that the panel failed to fully understand some features of the institution. We gained the impression that while they would respond if they felt it was necessary, institutions tended not to challenge minor inaccuracies contained in reports, reflecting the impression that the reports did not have real consequences outside of the institution.

2.7.3 Institutions also felt that the reports sometimes tended to exceed the brief of the reviews in a number of important ways. We support this view. The first way in which the brief appears to have been exceeded, perhaps inevitably, is in the use of a 'language' of assessment in the reports. To commend or to advise on improvements is implicitly to assess. This problem was recognised by the TLQPR steering committee and attempts were made to address it in the later reviews. It seems to us that the attempts were not entirely successful and could not be expected to be so. The second way the brief was sometimes exceeded lay in the inclusion of substantive evaluations of aspects of academic programmes and practices in reports. These could take the form of commendations or criticisms, but either way they seemed to go beyond the stated purposes of the reviews. A third way in which the brief was exceeded, related to the first one, lay in the seeming existence of an implicit approach to quality assurance against which institutions were apparently being assessed. We return to this point below.

2.7.4 A related point to the above concerns the extent to which the Review Panel's conclusions were substantiated within the reports. No doubt because of the Panel's commendable wish to keep reports short, recommendations for improvement frequently come across as 'assertions' made by panel members (sometimes by individual panel members in the views of some within institutions) and are not backed up by convincing evidence or argumentation. They may of course make sense in the light of discussions 'on the day' but if these are not recorded the force and acceptability of the conclusions reached are inevitably lessened.

2.7.5 Two other concerns about reports were made to us by members of institutions. The first of these concerns the relationship between the oral feedback given at the end of a visit and the content of the written report. There is often a tension in this. A lack of time and an understandable wish to be positive in oral feedback can sometimes lead to discrepancies between the two kinds of report. On the other hand, in our experience elsewhere, institutions usually wish to receive some kind of oral report at the end of a visit. Our own view is that oral reports are desirable but are best kept short and the author of the written report must use them as the basis for the latter. The written report should not contradict what was said during the oral feedback. Of course, points can be elaborated and explained in much greater detail in the written report.

The other frequently expressed concern related to unit reports. The brevity and anonymity of these made them misleading. There was an inevitable tendency to attempt to guess the units to which they referred. The shortness of unit visits also cast doubts on the accuracy and validity of these aspects of the reports. Views differed about the best solution to the problem. Some would prefer to see more substantial and named unit reports. Others would prefer to drop them altogether. We would go for the latter solution because TLQPR focuses on the university as a whole although, as we have noted elsewhere, the inclusion of unit visits remains, in our view, a strong feature of the TLQPR process.

2.7.6 We undertook our own analysis of the TLQPR reports. Our conclusions broadly support the views of institutions. We note the following points:

(i) The standard wording used in the reports includes formulations such as the 'panel believes' or the 'panel suggests' and the 'panel encourages'; although the terminology of 'recommendations' is not used, there is little doubt that this is implied - indeed, it might be better to be more explicit about it; however, quite often there appear to be no reasons given for why a particular 'recommendation' is being made.

(ii) The review reports are written in a style that supports the view that the TLQPR Panel did attempt to take account of institutional contexts and approaches to quality assurance. Thus, for example, it supported the creation within Lingnan College, as a small specialist institution, of quality processes based on what is termed as the 'Lingnan culture' of a collegial environment with informal processes which work well due to the close interaction between staff, students and administration.

(iii) However, there is also evidence in the reports of an implicit set of assumptions being used by the panel, in addition to those outlined in Annex A. These unstated characteristics include quality processes being institution-wide, coherent, systematic, and centrally monitored. Reports contain many references to the desirability of such characteristics, which are presented as being axiomatic and are not explained or justified in the body of the reports themselves and which may easily lead to a uniform quality framework, which the Review Panel wished to avoid.

In our discussions with members of the UGC related principles underlying the practice of quality assurance were introduced and presented as UGC policy. Some relevant assumptions underlying the TLQPR process were also presented in a paper given in Beijing China in May, 1996 by Prof. Kenneth Young. We would suggest that it would be beneficial to the UGC and to the development of quality assurance processes in the institutions if these underlying principles were more clearly developed and publicly articulated. In the UK, for example, the former HEQC developed guidelines on quality assurance, which included general principles, and the new Quality Assurance Agency is developing codes of practice related to quality assurance activities. We also noted that in the recent Management Audits conducted by the UGC, the materials included "Principles of Good Management Practice."

We would support the view expressed in the UGC TLQPR materials that institutions themselves must make the decisions about both the teaching and learning activities to be employed and the methods for improving these activities and assuring that they are appropriate and well executed. But if the UGC has assumptions and expectations about administrative and faculty responsibilities and behaviour, or other matters, related to quality assurance processes that are common across the institutions, we believe it important to clearly articulate those shared assumptions.

(iv) There were nearly 40 'recommendations' arising from the reviews of the seven institutions. They are written from the perspective of how the institution might improve its teaching and learning processes. Accountability issues are raised primarily in terms of internal accountability (mainly to institutional leadership) rather than to the UGC or external stakeholders.

(v) Each of the review reports contains a list of commendations made by the TLQPR panel. These include: oversight of responsibility for quality issues by appropriate committees, mandatory student evaluation questionnaires, teaching awards, establishment of teaching development/educational technology centres, internally initiated unannounced class visitations, student-staff consultative committees, the broadening of annual review processes to include teaching. These also seem to confirm the panel's view that quality is best assured by formalised institution-wide processes driven by institutional leaders.

2.7.7 The reports of TLQPR are potentially valuable documents but, in our view, there is a case for clarifying their purpose. Thus, if part of the purpose is assessment, we believe that it would be better to make it explicit and unambiguous and not the basis for informal textual analysis across Hong Kong higher education. Similarly, if the TLQPR panel has views about the relative effectiveness of different approaches to the assurance of quality in higher education, we would recommend that these be made explicit and discussed openly with institutions. Finally, the potential value of the reports in disseminating good practice is considerable, both within and between institutions. It can probably be achieved more effectively if separated from any assessment function concerned with a particular institution.

2.8 The TLQPR Review Panel

2.8.1 Based on interviews with TLQPR Panel members, including the chair, and discussions with the institutions, we observed that the TLQPR Panel appeared to have functioned well.

2.8.2 In general, the Panel members saw their workload as being very high, not only because of the time-consuming visits to the universities, which were generally seen as useful and interesting, but especially because Panel members had read so much documentation to prepare for the site-visits.

2.8.3 The institutions seemed to accept the authority of the Panel and sometimes found the exchange of ideas useful, although in some cases complaints were made about questions raised during the unit-visits, which went beyond the Panel's brief to review processes.

2.8.4 It was clear to almost everybody we interviewed that the chair of the TLQPR Panel combined different roles, all of which were essential for the design and implementation of TLQPRs. He had central roles in the design of the system, the explication of the system during the primary visits, and the organisation of the site-visits. He also acted as the Panel's major speaker during the feedback sessions at the end of the site visits and he directed and wrote most of the TLQPR Reports. Institutions and Panel members complimented his expertise and recognised him for the enthusiastic and persuasive way in which he carried out his work and most of the success of this first round was contributed to him personably.

The downside of this however is that the review process becomes dependant of one person.

2.9 Teaching Development Grants and Central Allocation Vote

2.9.1 In relation to the evaluation of TLQPR, the UGC also asked us to undertake a broad-brush evaluation of the Teaching Development Grants (TDGs) and the Central Allocation Vote (CAV). We operationalised this request by focussing on the way TDG and CAV grants have contributed to the enhancement of teaching and learning in institutions. We, therefore, integrated these aspects in our interviews. In addition we organised some interviews about the effects of CAV and TDGs in three institutions. The outcomes of the interviews were augmented by interview with the Chair of the Teaching Development Sub-Group. A summary of the purposes of CAV and TDGs is provided in Appendix 2.

2.9.2 Teaching Development Grants (TDGs)

The effects of TDGs were discussed in broad terms in every institution we visited. In three universities, additional interviews were organised to find out about TDGs in more detail.

During our visits we found that, in general, TDGs were welcomed by the institutions. They appreciated the UGC's willingness to fund projects dedicated to innovation in teaching and learning as part of its effort to strengthen the quality of teaching and learning. The fact that the UGC, since 1994/95, has set aside a notable amount of money for this purpose enhanced the credibility of UGC policy in this respect.

The distributed part of TDG (an earmarked budget on top of the block grant, based on student numbers) allowed the senior management of universities to finance both teaching and learning support infrastructure (e.g. teaching development units) as well as to reward individual initiatives. In general the budgets seemed to us to be appropriately allocated by the institutions. Several institutions developed internal competitive bids for this part of the TDG.

The institutions' progress reports about TLQPR showed numerous examples of projects related to the TLQPR recommendations and financed by TDGs; other projects were drawn to our attention during the interviews. Progress of the projects was carefully monitored by the institution's management and reported to UGC regularly.

The bidding part of the TDGs, which increased from one-third in 1995-1998 to two-thirds of the TDG budget in 1998-2001, was somewhat more controversial. In terms of rewards and reputation, inter-university competitive bidding may be a powerful instrument, but both the criteria and the procedures followed by the UGC were subject to criticism.

Some universities noted that the priority criteria and the way they were supposed to be used by the Teaching Development Sub-Group (TDSG) of the UGC in the upcoming round were not clearly communicated. This hampered priority setting within the institutions. (We heard several examples of conflicting interpretations between the TDSG and university staff responsible for the preparation of the university's bid.) The minimal size of 1 million HK$ for a project was seen as too high for smaller institutions and the emphasis on inter-university projects was seen as an unnecessary prerequisite, and the criteria used for this remain unclear.

In general, doubts were expressed about the efficiency and effectiveness of the procedures followed for the competitive bidding, including the report structure. Some institutions were concerned about the demotivating effect on units of well formulated projects that were turned down apparently as a result of lack of clarity of the central bidding procedures. Some universities, however, mentioned that they were willing to support such projects on a lower scale by using part of the distributed part of the TDG budget.

The Consultative Committee's initiative in organising a system-wide conference at which results from TDG projects (both from the distributed part and the bidding part) were communicated was widely applauded.

Based on the information provided by the UGC and our visits to the institutions, we conclude that the distributed part of the TDGs was timely and welcomed. It introduced or greatly enhanced universities' ability to support initiatives to enhance the quality of teaching and learning, without the UGC becoming prescriptive or getting involved in the micro management of the institutions. As such it acts as a complementary instrument to the TLQPRs.

The competitive bidding part of TDG was criticised more widely with respect to priority criteria, minimal size of the bids as well as with respect of transparency and efficiency in the allocation process. It is our impression that TDSG's efforts to make the procedure transparent and objective have not been completely successful. A suggestion was made by one of the institutions to strengthen the relationship between the outcomes of TLQPR and the balance between the distributive and the competitive part of TDG. We will come back to these suggestions in the conclusions and recommendations of this report.

2.9.3 Central Allocation Vote (CAV)

Based on the documents and other information provided by the UGC we learned that the UGC uses CAV for promoting inter-institutional collaboration and for supporting new and unforeseen developments during a triennium. In the present triennium (1998-2001) CAV covers both research and teaching and learning initiatives, such as the development of areas of excellence, joint research schemes with the UK, Germany, France and the National Natural Science Foundation of mainland China, and the establishment of co-operative research centres, as well as the support of institutional restructuring. In the past triennium (1995-1998), part of the CAV budget was used to increase the bidding part of the TDG budget.

In contrast with our findings with respect to the TDG, we found that the interviewees within the institutions had none or only limited knowledge of the use of the Central Allocation Vote. The necessity of having some budget set aside at the system level to stimulate the efficiency of the system by encouraging and facilitating co-operation between institutions was not questioned. However, greater transparency of such initiatives was required.

2.10 Quality Enhancement and Dissemination

2.10.1 Academic quality assurance embraces more than reviews of individual institutions. In a number of countries institutional or subject reviews are followed by on-going discussions and contacts between quality assurance agencies and institutions, by system-wide dissemination of best practice in the form of quality assurance workshops and meetings among university faculty and administrators, by related evaluation and research studies, and by developmental activities designed to enhance the quality of teaching and learning. In Hong Kong, related activities included the TLQPR progress reports by the universities and the December 1998 workshop on teaching strategies and innovation sponsored by the Consultative Committee. In our evaluation, the following relevant findings were identified:

• a number of universities and academic units involved in the review felt that the Review Panel could have made more specific recommendations for improvement;

• several of the department heads interviewed had hoped the reviews would lead to subject-related suggestions for improving teaching and learning quality;

• several respondents felt that the UGC responses to their progress reports were somewhat perfunctory;

• a number of respondents commented positively on the December 1998 workshop sponsored by the Consultative Committee, and suggested this had been very useful. It was unclear to us, however, whether this type of activity would be continued by the Consultative Committee, and indeed, the future activities and agenda of the Committee were not clearly defined;

• while the Teaching Development Grants have been of some help to the institutions in implementing university-based quality assurance processes, particularly in the development of centres for learning and teaching, there appears to be little connection between UGC funding of teaching development and the insights into the university system gleaned from the TLQPR process. In our review of the TLQPR Reports, for example, we noted the following issues were common to a number of universities: the design of external examiner systems; use of student centred/problem-based teaching and learning; means of better involving teaching-learning centres in academic quality assurance work; means of obtaining feedback from external stakeholders; and means of obtaining student feedback on instruction. It is possible that collective action on these types of issues among the universities in addition to individual actions within each university might be an effective means of improving the quality of teaching and learning.

2.11 Strengths and Weaknesses
 

2.11.1 The strengths and weaknesses of TLQPR in terms of its original intentions are summarised below.

2.11.2 Strengths

(i) TLQPR has placed attention to improvement of teaching and learning firmly on both UGC and institutional agendas.

(ii) It has involved institutional staff at both unit and institutional levels.

(iii) Its intention has been to focus on processes and to be non-prescriptive.

(iv) The Consultative Committee has emerged as a body with potentially long-term importance.

(v) TLQPR has recognised the primary responsibility of institutions themselves over quality matters.

(vi) Teaching Development Grants (TDGs) have helped to support the development of university-wide quality assurance processes.

(vii) Size of the higher education system and close proximity of institutions should make dissemination and sharing of good practices easier and efficient.

2.11.3 Weaknesses

(i) There has been uncertainty within institutions about the underlying rationale and purposes of TLQPR.

(ii) For some institutions, there was ambiguity about the requirements of the self-assessment and preparation for the review visit.

(iii) For all institutions, there was a high workload in a context of pressures from other reviews; the workload was particularly high because of the involvement of academic units in the review process.

(iv) Reports:

• a language of assessment crept into reports

• substantive issues, which seemed go beyond the focus on quality processes, crept into reports

• sometimes insufficient evidence and rationale was provided to support the conclusions and recommendations

• the function of the unit reports was not always understood correctly.

(vi) There is an absence of clear consequences arising from the results of TLQPR, i.e. little by way of rewards/sanctions compared with the results of RAE.
 
 
 
 

3.1 In this section we set out our conclusions and recommendations. These are framed by our remit to:

(i) undertake an independent evaluation of the TLQPRs of the seven UGC-funded tertiary institutions in order to determine the extent to which the TLQPR processes have achieved their stated aims of (a) focusing attention on teaching and learning (as the primary mission of higher education institutions), (b) assisting institutions in their efforts to improve teaching and learning quality, and (c) enabling the UGC and the institutions to discharge their obligation to be accountable for quality;

2. recommend any improvements on the TLQPR process; and

3.2 Major Conclusions

3.2.1 On the whole, we conclude that 'TLQPR was the right instrument at the right time', because the review was a positive stimulus to institutional attention to teaching. The further remarks that we make must be viewed in this, basically positive, light. We can sustain this conclusion in more detail by looking at the extent to which each of the aims of TLQPR was achieved.

3.2.2 There were clear achievements towards the first goal of TLQPR - commitment to teaching and learning as, in the words of the UGC, "the primary mission" in higher education institutions. The signal given by the introduction of a review process for quality of teaching and learning, that teaching and learning are as important for Hong Kong higher education institutions as research, was seen by many as the prime impact. Regardless of the form of the review process, it went some way towards shifting the balance of attention to teaching and learning in the higher education community; the focus had previously been on research as a result of the introduction of the Research Assessment Exercise (RAE). The fact that RAE (i) is repeated every triennium, and (ii) is tied to substantial funding, implies continued and strong pressure to remain attentive to research outputs. On the 'shop-floor' level especially, we often found conflicting pressures between RAE and TLQPR. Sometimes, especially in universities with a historically strong research mission, this was expressed as resistance to the introduction of formal quality assurance processes.

3.2.3 There were also achievements towards the second goal - to assist higher education institutions in their efforts to improve teaching and learning quality assurance processes. In some institutions, we found institutionalisation of quality management procedures where these had not previously existed in a formalised way. We also found a number of examples of innovation to existing quality processes. Sustainment of these present efforts and any new initiatives would be helped by the signal that there would be a second round of external reviews of teaching and learning in about 5 or 6 years after the first one (i.e. from 2001 onwards, or in the 2002 - 2004 triennium).

Given our remit, we cannot judge whether increased quality assurance processes led to increased quality of learning outcomes. Some interviewees suggested that a focus on universities' processes to assure quality of learning outcomes might be useful in a second round.

The second goal of TLQPR would be greatly assisted by system-wide learning processes with respect to quality assurance of teaching and learning. The dissemination of good practices across universities has not had much effect to date. Maybe we found this because our visit followed too soon after the dissemination activities that have already taken place, i.e. the publication of the progress reports by the universities one and half to two years after the TLQPR, the April 1997 seminar, and the Consultative Committee workshop to showcase good practices in December 1998.

3.2.4 With regard to the third goal - the discharge of the UGC and higher education institutions to maintain accountability - we found that the TLQPR process and the reports, plus the institutions' progress reports, clearly demonstrate that the higher education institutions in Hong Kong are accounting for their quality improvement processes. We could not gauge to what extent the relevant audiences in Hong Kong's society were satisfied with the current ways in which accountability is maintained - even less if they will remain so in future. In the eyes of the higher education community in Hong Kong, the way the press had reported about the TLQPR reports on individual institutions was not seen as a balanced reflection of the process.

3.3 Major Recommendations

3.3.1 Based on the major conclusions summarised above, and on the findings stated earlier in our report, our main recommendations are as follows.

3.3.2 First, continue the 'campaign for quality'

(i) A second round of some form of external review is needed as a signal for the higher education community. The importance of addressing the quality of teaching and learning processes has grown with the growth of student numbers and the concomitant diversification of the student body, as well as the massification of teaching interactions.
(ii) To offset the continuous pressure of the RAE - even if the RAE criteria will be broadened to include scholarship of teaching - the UGC needs to show clearly that it is serious about what was the first goal in the first round of TLQPR, i.e. that education is the primary mission of higher education institutions. To make this message as forcefully as possible, a separate review of teaching and learning, not integrated with other reviews, would seem to be the best means. In reviews integrating management and education, quality of teaching and learning might become of secondary importance to the review process and thus to the higher education institutions. Moreover, management reviews do not easily reach down to the 'shop-floor' level where the most important contributions to quality education are made. In reviews integrating education and research aspects, international experience shows that the latter is dearer to academics' hearts and therefore would tend to take prime importance.

(iii) We realise that continuing separate reviews for education, management and research puts a heavy burden on the Hong Kong higher education community and on the UGC members especially. Accordingly, we advocate a lower frequency of reviews of quality of teaching and learning than the present three-year cycle for the RAE. A five- or six-year interval has been mentioned as desirable.

(iv) Another way of reducing the burden of external reviews while maintaining the benefits of separate reviews for education, management and research, might be to co-ordinate the preparatory work for the higher education institutions. Some benefits might be found especially with respect to the two process-oriented reviews of teaching and learning and management.

(v) Moreover, reckoning that a single repetition of a separate review of quality of teaching and learning would result in a sufficient degree of internalisation of a quality culture, and that a third round would probably have much less added value, integration of the various reviews should be on the UGC agenda in the long run.

3.3.3 Our second main recommendation would be to continue to give attention to quality assurance processes with regard to teaching and learning. The reasons for UGC choosing a process review rather than evaluation of teaching in all subjects or departments remain - formulated inter alia in several papers by W.F. Massy and N.J. French - as valid as they were for the first round.

The focus of the next round of TLQPR should continue to be on teaching and learning quality assurance processes, but might pursue a narrower agenda. This would help limit the burden on the higher education community and on the UGC. It would also give an opportunity to reinforce the 'signal' already given out by introducing external reviews of the education function. Suggestions for a narrower focus, which could vary between institutions (see 3.3.4), might be for example:

• to review the quality assurance policies and processes implemented since the first review, with particular emphasis on the perceived effectiveness of those policies or processes from the point of view of students and of faculty members at the unit level;

• to review the procedures and measures to ascertain student learning that are being utilised by each university and each academic unit as a means of determining that they are in fact offering quality teaching. An added advantage of such a focus would be to shift attention from (formal) processes per se to their effectiveness in stimulating quality education;

• to review the 'value added' from the unit perspective of institution-wide quality assurance procedures.

3.3.4 The third main area of our recommendations concerns clarity about the aims and objectives of an external review.
(i) The UGC should provide more clarity about the balance between the different aims and communication about them to the academics within the higher education institutions could be improved.

(ii) With regard to the first goal of TLQPR - to focus attention in the higher education community on education - we have already made clear in earlier parts of our report that this should be the basis of any approach for a second round of TLQPR.

(iii) If the UGC should, however, wish to emphasise the second goal - supporting higher education institutions to improve the quality of teaching and learning - an approach which takes the diversity of the Hong Kong higher education system into account might be adopted. Realising that the different higher education institutions have diverse traditions and processes of ensuring the quality of teaching and learning, UGC could consult with the individual universities about what, in their eyes, are the areas that warrant special attention from an external review panel⁶. In this way, a tailor-made external review could be provided for each institution. We assume that the tailor-made elements would be in addition to a common core that would be addressed in all higher education institutions. With this option, it is of utmost importance to establish a very focused approach to the common core.

(iv) If, on the other hand, the third goal - the accountability function - should become the primary aim of a second round, then the external review panel's role would become more like an auditor's role, i.e. to check that all higher education institutions have satisfactory and operational quality processes in place. A narrower focus could then be established by selecting certain crucial quality processes, e.g. those most closely connected with student learning outcomes. In the logic of audits, a lighter touch for the process could also be achieved by reducing the number of unit visits to a small selection of 'audit trails'.

3.3.5 As a fourth area of recommendations we think that, in order to attain internalisation of a quality culture in the higher education community, it is important to maximise the local ownership of the process. The TLQPR Panel was successful in its 'campaign for quality', but better embedding in the Hong Kong higher education community is desirable for the future. The main issue for decisions in this respect is the design of the procedure for a second round. There is enough time to develop a reasonably detailed and stable procedure for a second round; the learning process should be separated from the actual review process.

Many decisions around the design of the procedure depend on future choices by the UGC on the balance between the different aims and objectives for a second round. From the point of view of internalisation of a quality culture, a major factor in all design decisions should be the acceptability of the design in the Hong Kong higher education community. Whereas, therefore, external expertise (including international experiences) may be very useful, involvement of local experts and/or consultations with the Hong Kong higher education institutions ought to continue to be a major element of the design process.

3.3.6 We now turn in more detail to each of the elements of the design of a second round of external quality process reviews.

(i) Preparation and self-evaluation

The reflection at both the institutional and unit levels associated with the internal preparation was universally seen in the higher education institutions as one of the most useful parts of the TLQPRs. The emphasis on self-evaluation should be maintained.

There is however, compared to the first round of TLQPR, room for streamlining, so that there will be good preparation in the higher education institutions but not 'over-preparation'.

Professional advice about the internal preparation is an option to be studied further. The obvious parallel was made with the role of external consultants in the management reviews undertaken by the UGC, which was generally seen as a positive experience. Our caveat is that to remain effective within the higher education institutions, the issues of ownership and involvement of local staff should be taken into account carefully when deciding to involve external consultants.

(ii) Review panel

As we stated before, the TLQPR Panel chair and members were very successful in designing and implementing the first round of TLQPR. Based on the experiences of this we can make the following recommendations:

• the Review Panel should be smaller in a second round. The exact size depends on the goals of visits to the higher education institutions. Obviously, an audit panel can be smaller than a panel that operates in a consultative mode for a large number of units;

• the panel should be composed of local higher education peers, complemented by foreign higher education experts;

• the Panel's chair should be (i) independent, (ii) expert on higher education and higher education quality management, and (iii) accepted by the Hong Kong higher education system;

• the Panel need not have (completely) the same composition from one university to the other;

• a professional secretary (experienced in higher education and higher education quality management in Hong Kong) could be considered to support the Panel and to write the Panel's report. The relative efficiency to the UGC in 'outsourcing' or 'insourcing' this function should be part of such considerations;

• training for Panel members should be introduced so as to minimise dependence on the continuity of Panel members from one university to the other. This should include training in questioning skills;
• an established protocol of questions to guide Panel Members during visits should be considered.

The site visits of the TLQPR Review Panel to the higher education institutions were, on the whole, valued positively. The principle of the inclusion of unit visits was valued but the visits were too short to be of great value to the units concerned. The unit visits appeared to have functioned successfully to bring the 'campaign for quality' to a wide audience. We would, therefore, recommend that this practice be continued in a second round but sufficient time should be allowed to ensure their usefulness.

The length of the visits was perceived to be very short, even for the limited purpose of getting insight into quality processes. In a second round each meeting should last sufficiently long enough to establish effective communication between the reviewers and the interviewees. Visit length might vary according to size and complexity of the institution.

(iv) Panel's report and analytic framework

The reports' recommendations were sometimes seen, however, to go beyond the evidence, or to reflect facts and interpretations that might not have reflected the general situation in the university or the view of the majority of the Panel members. Sometimes the Panel raised substantive issues, which seemed go beyond the focus on quality processes. In a second round of TLQPR, the Panel should avoid this.

The Panel's analytic framework in the form of the questions/areas to be addressed received broad support. Yet it had a more unifying or compliance-inducing effect on the higher education institutions than intended by the Review Panel and the UGC.

The treatment of unit level reports should be reconsidered in a second round. Opinions among our interviewees varied as to whether the unit reports, which in the first round appeared anonymously and as an appendix to the TLQPR report, should be deleted altogether, or should mention names of units. In our view, the reconsideration should take the aim of reporting into account in the context of the overall aims and objectives of the whole process. In an audit-type exercise, few units would be visited and then only to check the veracity and actual operations of university-level policy statements; publication of unit-level reports would, therefore, not be expected. In an improvement-oriented mode, emphasis should be given to detailed operational feedback to the institutions and units visited.

(v) Follow-up by UGC

The phrase that TLQPR outcomes would 'inform funding' was well known in the higher education institutions and was a source of confusion. More clarity is needed about the link between reviews and UGC funding decisions.

Rewards for innovations in teaching and learning, i.e. through the TDG and CAV grants, have had positive effects. While a full evaluation of the TDGs was not within our remit, the money appeared to be well spent.

In section 2.9 on TDGs, we mentioned the balance between the tranches of TDG distributed on student numbers and on competitive bidding, respectively. An additional way to reward higher education institutions (which prove through TLQPR to have good processes in place for the allocation of TDG money and for monitoring the effective use of this money) would be to build on trust, complemented with monitoring of the use of the TDGs en bloc and ex post, rather than institutions continuously reporting on each project.

The publicity about TLQPR was not optimal; in a second round the UGC should develop a more active role in informing the public and press.

Sustained positive reinforcement of the signal that teaching and learning are the main functions of higher education institutions, and especially positive reinforcement of innovation of teaching and learning in between two rounds of TLQPR, is of utmost importance. TDGs, mentioned above, play an important role in this respect.

In the light of support for institutional quality assurance activities, more investment should be made in on-going, informed quality enhancement and dissemination activities, in addition to the TLQPR process and the TDGs. Consideration should be given to creating a separate unit for such activity under the direction of the Heads of Universities Committee (HUCOM) and/or under the Consultative Committee. This unit might engage in the following types of activities:

• continued encouragement of dissemination of best practice with respect to quality assurance among the Hong Kong universities;

• encouraging the development of subject-level review processes with an emphasis on continuous improvement of teaching and learning within each university. UGC funds might be designated for (co-operative) institutional activities in this area;

• developing subject-level reports on the teaching and learning developments that could serve as the basis for workshops for relevant faculty (cf. system-wide subject reports published in the Netherlands, UK, and Denmark);

• a system-wide study of the strengths and weaknesses of the external examiner system with recommendations on how this system might be improved to enhance teaching and learning, especially in relation to the introduction of credit-unit based curricula;

• developing a regular system-wide survey of student learning activities that could serve as an indicator of the level of student involvement in learning in each university. Similar surveys are being developed in the US as a means of tracking the quality of undergraduate teaching and learning.

The emergence of the Consultative Committee as a responsible and respected group representing the universities in the area of quality assurance was a serendipitous and valuable consequence of the TLQPR. UGC should continue to facilitate the Consultative Committee, for this reason.

(vii) Organisational locus of a second round of external quality process reviews

To signal the importance attached to teaching and learning by the UGC and to offset the research push of the RAE, we would recommend that a second external review of teaching and learning is given symbolic recognition by remaining directly under the aegis of the UGC.

In other, equally important tasks, on-going support should be given to the teaching and learning functions of Hong Kong higher education institutions, especially the inter-university learning about good practices. The Consultative Committee or another appropriate collective of the higher education institutions could take centre stage with the support of the UGC.
 
 
 
 

• TLQPR Goals
• Teaching and Learning Processes
• TLQPR Methods
• Quality Improvement and Assurance
• Report Preparation
• Sample Approaches
• TLQPR Dimensions