Home > UGC Activities > Research > Research Assessment Exercise > Documents on Previous RAEs > Research Assessment Exercise 1999 Consultation Document (10.7.1998)

Research Assessment Exercise 1999 Consultation Document

Introduction

  1. This consultation document seeks views on how the University Grants Committee (UGC) should assess the quality of research in the UGC-funded institutions. It describes the original purpose and the present and future objectives of the Research Assessment Exercise (RAE). It also sets out some proposed modifications to the design and implementation of the next RAE. This document should be read by senior management and all academic staff of the UGC-funded institutions.

Purpose of the RAE

  1. In 1993, the UGC began to move away from a historical-based model for the assessment of the public recurrent funding requirements of the UGC-funded institutions to a more performance-based funding model. Following detailed study aided by an expert consultant, the UGC adopted a zero-based model which relates the level of funding allocations both to the tasks that each institution is expected to accomplish during the funding period, and to the quality of its recent performance.

  2. In principle the funding of each institution is the sum of funding for its two principal activities, teaching ('T') and research ('R'), each carrying an implied overhead of administrative or support activity. It is the UGC's intention that public funds in support of research should be allocated preferentially to those institutions which demonstrate that they can use them well. There is therefore a need to assess research performance in some way so that it can be reflected in the funding level.

  3. The RAE is thus part of the UGC's performance-based funding assessment process. It aims to measure the output and quality of research of the UGC-funded institutions by cost centre as the basis for allocating some of the research portion of the institutional recurrent grant for the next triennium in a publicly accountable way.

History

  1. The first RAE was conducted in 1993, the second RAE in 1996. The first RAE was designed and implemented with the help of external consultants. A quality threshold which was not overly stringent was used. The outcome of the exercise was a factor, expressed in terms of full-time equivalent active researchers (FTE-AR) by cost centre, applied as a multiplier to an assessed unit cost of research activity by discipline. The second RAE was essentially designed locally -- external referees were also involved, in significant numbers, but they all functioned as normal panel members. A conscious decision was taken to adopt a higher quality threshold. The UGC subsequently concluded that the results of the second RAE revealed that there had been an overall and significant improvement in both the range and quality of research work being carried out by colleagues in Hong Kong.

  2. In retrospect, the RAE has been effective as a means of -

    1. informing funding;

    2. symbolising public accountability; and

    3. inducing improvement in research by continually raising the quality threshold for assessment.

  3. Nevertheless, there have been criticisms of certain aspects of the RAE process and, as with any other such exercise, it should be possible to seek continuous improvements to make the RAE more effective, more transparent and more widely accepted.

  4. Several suggestions for improvements to the design and conduct of the RAE were received following RAE 1996. Among them, the major concerns were :

    1. the definition of research output was felt to be too narrow, resulting in non-traditional types of research output and other scholarly activities not being submitted or accepted for assessment or, where accepted, not adequately assessed; and

    2. there was felt to be a lack of transparency in the conduct of the RAE.

  5. Against this background, after careful consideration within the UGC, it is now proposed that the design and conduct of the next RAE should incorporate various improvements. Draft 1999 Guidance Notes for the next RAE are annexed to this consultation document. The main changes from previous RAEs are highlighted in this document below.

  6. Since the RAE is expected to remain an important measure for determining the distribution of a significant part of the 'R' portion of the institutional recurrent grants, planning for the next RAE needs to tie in with the funding cycle of the next triennium 2001-2004. It is therefore intended that the next RAE will be undertaken in mid-late 1999, see paragraph 30 below.

General Principles

  1. The UGC considers that the next RAE should be conducted on basically similar lines to RAE 1996, but with improvements in several areas. The RAE will continue to rate cost centres, not individual staff members. It will determine the 'research indices' of all cost centres in each UGC-funded institution which will be aggregated to obtain an overall research index of that institution. However, the term 'active researcher' will no longer be used.

  2. The UGC noted that as the RAE results in the last two RAEs directly informed funding, academic staff, academic units within institutions and institutional management perceived an overly strong incentive for research, even to the detriment of teaching. The UGC wishes to stress that research is not an isolated activity; research should support and illuminate teaching and learning. The RAE results should therefore be a major input but not the sole input, for determining the 'R' portion of institutional funding (cf paragraph 3 above). Prior to the conduct of an RAE, the UGC proposes that each institution should map out its research strategy, in relation to its role and mission. The numerical research index of an institution resulting from the RAE, taken together with that institution's research strategy, will help the UGC to form a judgement about the appropriate level of the institution's recurrent funding for research, but it will not be the sole deciding factor.

  3. The 'R' allocations resulting from the RAE will be made at the institutional level, not at the faculty or departmental level. While the RAE will necessarily produce a retrospective view, each institution will be expected to allocate its research funds according to future research potential. Since future institutional investment may or may not follow past productivity patterns, it is expected that institutions need not follow the RAE results in their internal allocation decisions.

Other Proposed Changes

Definition of Research Output

  1. In response to the impression that the definition of research output was too narrow, in January 1998 the UGC-funded institutions were invited to submit samples of non-traditional research output items which they believed should be accepted for assessment in the next RAE.

  2. The UGC has carefully examined the samples received from the institutions and found that most of them actually fall within the criteria adopted for RAE 1996. The misunderstanding, for example, that only journal papers are accepted, seems to be widespread. In fact, in RAE 1996 research output was clearly and broadly defined in paragraphs 25 to 28 of the 1996 Guidance Notes as follows :

    "25. All output items submitted for assessment must be research-related, publicly accessible by academic peers, and contain a sufficient element of innovation.
    26. The following are considered to be (submittable items) :
    1. any publication, artifact etc., provided it was -

      1. published (or for non-traditional works, such as drama, appearing) within the assessment period; or

      2. not yet published, but officially accepted for publication (without need for further amendments of any kind) in that period; in this case, the letter of acceptance need to be attached; or

    2. "non-traditional output" that may or may not be in publishable form, e.g. concert performance, video tape, computer software programme, buildings, or any creative work that can be evaluated for merit and an assessment obtained on either a public or confidential basis (RAE panels to decide on the basis of the criteria in (a) above).

    27. On the other hand, proprietary research that does not result in output that is accessible to the public and the profession is not accepted as an output for this purpose.
    28. It is stressed that output items are not restricted to papers in journals."
  • Nevertheless, to enhance the understanding of the definition of research output, and to enlarge slightly the scope of the RAE to recognise other scholarship closely akin to research, the UGC proposes to adopt the Carnegie Foundation's definitions of research and research-related scholarly activities, in four broad categories :

    1. scholarship of discovery;
    2. scholarship of integration;
    3. scholarship of application; and
    4. scholarship of teaching.

    The four kinds of scholarship are explained in greater detail in Appendix A of the draft 1999 Guidance Notes attached.

General Criteria

  1. The UGC considers that the definition of research output in RAE 1996 is still sound and valid, but has advised that, before the third RAE is conducted, more efforts should be made to promote understanding of its scope and purpose within the institutions and the RAE subject panels. In general, items which meet all of the following criteria can be submitted for assessment in the RAE :

    1. the output contains an element of innovation;

    2. the output and the process involved contribute to scholarship;

    3. the output is publicly accessible; and

    4. the output is of interest to peers and is generalizable.

  2. Items included in the RAE submission will then be subject to quality assessment, and only those meeting the necessary quality requirements will attract any score.

  3. Regardless of the nature of the research, it has been suggested that the UGC should, through the RAE, build in an element to reflect the level of interaction between an institution and the industry/business sector in research & development activities, as a means to foster greater collaboration between the institutions and the business/industry sector. The UGC believes that research output meeting the criteria in paragraph 17 should be accepted for assessment and scored according to the quality threshold. They would also normally be recognised for funding by the UGC. If the research activities and output arising from the interaction between an institution and the industry/business sector do not meet all the criteria set out in paragraph 17, it is open to question whether such activities and output should be funded by the UGC, and whether in such cases the institutions concerned should rather obtain the necessary funding from other appropriate sources (e.g. the Industrial Support Fund), or seek full-cost recovery, including on-costs, from the appropriate clients. It is also open to question whether such research activities and output, if they are not to be funded by the UGC, should be assessed by the RAE in the first place.

  4. The UGC notes that elsewhere, for example in the US and the UK, state/ federal/national educational funding by and large would not take such research output into account. Nonetheless, the UGC would welcome views on this aspect in deciding on the scope of the 1999 RAE in the Hong Kong context.

Quality Threshold

  1. The UGC proposes that the definition of the quality threshold for assessing research output should largely be the same as RAE 1996, as follows :

      "Quality of output equates to an attainable level of excellence appropriate to the discipline in Hong Kong, and showing some evidence of international excellence."

  2. Although the definition of the quality threshold largely remains the same, the UGC believes that the actual threshold will have risen in step with the advancement of research in the local and international environment since RAE 1996 for which the cut-off date for research output was 31 December 1995.

Transparency

  1. Concerns have been expressed about the transparency of the RAE results, and transparency of the criteria and processes of assessment.

  2. As indicated earlier, the primary purpose of the RAE is to assess the research output performance of UGC-funded institutions by cost centre; it is not intended to evaluate the performance of individual staff within the institutions. The subject panels will only produce total scores for each cost centre in each institution.

  3. In RAE 1993 and RAE 1996, the results of each institution were forwarded to the institution concerned only and were not made known to other institutions or the public. This was to avoid misinterpretation by casual comparison of the results of different institutions since each institution has its distinct specified role and mission, and different institutions are at different stages of development. Nevertheless, to give a rough indication of the level of attainment of a particular cost centre across the whole sector, in RAE 1996 each institution was also given, alongside the research index for each of its cost centres, an average and median research index across the whole sector. This was not done, however, in respect of cost centres which were only found in fewer than three institutions.

  4. Against this background, comments are invited on whether and if so, what, when, to whom and how much data should be made public after the completion of the RAE.

  5. With regard to the desire for more transparency in the criteria for and process of assessment, the UGC agrees that :

    1. there should be more thorough consultation with all stakeholders to promote a clearer understanding of the RAE framework and mode of operation of the RAE;

    2. the essential documents, i.e. the RAE Guidance Notes and the Guidance Notes for Panels, when finalised, will be mounted on the UGC Web for public information;

    3. all RAE subject panels will receive training before the actual assessment process begins and be invited to describe in writing for dissemination the standards and criteria used, and a description of how these have been applied; and

    4. the RAE panels should have more overseas and independent referees, including professional members as appropriate.

RAE Subject Panels

  1. Comments are also invited with regard to the number, size and composition of subject panels, and the mapping of cost centres to the subject panels.

Other Modifications to the RAE Guidance Notes

  1. In the light of experience and feedback on the RAE 1996, the UGC has proposed several minor modifications to the Guidance Notes for the next RAE. These are summarised as follows :

    1. Eligibility

      The criteria for eligibility will remain largely the same (ie academic staff in grades A - I plus M for the purposes of the Common Data Collection Format (CDCF), but in view of the growing trend for institutions to employ senior research staff or research fellows in staff grade 'M' as defined in the CDCF, only staff members holding a paid appointment for a continuous period of 24 months or more should be admitted for assessment.

    2. Definition of output

      Staff submitting non-traditional research output items will be required to submit a one-page explanation of why they should be accepted for assessment.

    3. New researcher

      The definition of 'New Researcher' has been simplified to include eligible staff who have been academic staff for less than 3 years.

    4. Co-authorship

      The philosophy of and arrangements for assessing co-authorship in RAE 1996 remain valid, but staff members submitting such items must explain the nature and degree of their contribution.

    5. Data required

      The number of copies of papers required for processing by the UGC Secretariat will be reduced.

  2. Comments are invited on whether the above proposed changes are appropriate and/or whether there are other amendments that should be made.

Timetable for the Next RAE

  1. In order to tie in with the recurrent funding assessment for the triennium 2001-2004, the UGC proposes to conduct the third RAE in 1999. Subject to the conclusion of the consultation process before the end of 1998, it is planned that RAE submissions will be called for in March 1999, with the assessment process conducted in the autumn of 1999. The proposed census date for RAE 1999 is 31 December 1998.

Summary of Comments Sought

  1. Comments from individuals, academic units and institutions on the following specific recommendations/aspects are welcome :

    1. as a matter of principle, the RAE results should be a major input, but not the sole input, for determining the 'R' portion of institutional funding; and the numerical research index resulting from the RAE taken together with the institution's research strategy, which should be mapped out prior to the RAE according to the institution's role and missions, should help the UGC to form a judgement on the institution's appropriate level of recurrent funding for research (paragraphs 11 - 13);

    2. slight enlargement of the scope of the RAE to recognize other scholarship akin to research, adopted from the Carnegie Foundation's definition of research and research-related scholarly activities (paragraphs 14 - 16);

    3. whether research items to be submitted for assessment should meet the four criteria as set out in paragraph 17;

    4. whether the UGC should rate and fund research output arising from interaction between institutions and the business and industry sectors which do not meet all the criteria set out in paragraph 17 (paragraphs 19 - 20);

    5. the definition of quality threshold (paragraph 21);

    6. the measures to increase the transparency of the RAE process, including whether and if so, what, when, to whom and how much data should be made public after the completion of the RAE (paragraphs 22 - 26);

    7. the number, size and composition of subject panels, and the mapping of cost centres to the subject panels (paragraph 27); and

    8. other modifications to the RAE Guidance Notes (paragraphs 28 - 29).

    Other comments on the draft Guidance Notes for RAE 1999, and on any other aspects of research assessment are also welcome.

  2. To enable the UGC to address major comments arising from the recommendations set out in this consultation document at its coming meetings in August 1998 and January 1999, initial comments in writing on those recommendations set out in paragraph 31, particularly those in (a) - (e), are invited to reach the UGC Secretariat by post (to Suite 202, 2/F., Hutchison House, 10 Harcourt Road, Central, Hong Kong), by fax (2845 1596), or by e-mail 'ugc@ugc.edu.hk' by 3 August 1998. Further comments after 3 August 1998 will still be welcome, but probably they will only be able to be considered by the UGC after its August 1998 meeting.

UGC Secretariat
July 1998