Home > UGC Activities > Research > Research Assessment Exercise > Documents on Previous RAEs > University Grants Committee Research Assessment Exercise 1999 (10.7.1998)

University Grants Committee Research Assessment Exercise 1999

Guidance Notes*

[* The contents of this draft Guidance Notes are subject to comments to be received on the Research Assessment Exercise 1999 Consultation Document.]
(printed on 8 July 1998)

I. INTRODUCTION

Background

  1. The third Research Assessment Exercise (RAE) will be undertaken by the University Grants Committee (UGC) in Autumn 1999. Like the second RAE conducted in 1996, the purpose of the exercise is to assess the research output performance of the UGC-funded institutions by cost centre, to be used as one of the bases for allocating some of the research portion of the institutional recurrent grant for the triennium 2001-2004.

  2. The UGC again wishes to emphasize that the research assessment exercise does not imply an interest in research to the possible detriment of teaching quality. The Committee recognizes that both teaching and research are important and are inter-related elements in the development of higher education. Indeed, a much larger portion of the recurrent grants allocated to institutions is and should be attributed to teaching. However, it is necessary for the Committee to adopt different approaches to look at teaching and research in view of the different nature of these activities.

Contents of this document

  1. This document consists of the Guidance Notes, a set of blank forms numbered Tables 1 to 3, and Appendices A to E. A diskette containing these papers in Word 6.0 or Excel 5.0 format is also provided to each institution to facilitate the generation of sufficient copies for internal distribution.

Distribution of this document

  1. Each institution should issue this document to every member of its academic staff, and to every member of its administrative staff responsible for institutional governance or involved in research policy and support, in order that the principles, aims and methodology of the RAE are fully appreciated. A copy of this Guidance Notes is also accessible from the UGC Website at .

Principles and Philosophy for RAE 1999

  1. The results of the last two RAEs directly informed funding, in that they were factored into the determination of a significant part of the 'R' portion of institutional recurrent grants. The UGC notes that this may have unduly influenced institutions' commitment to research versus teaching, and therefore wishes to clarify the principles and philosophy for RAE 1999.

  2. The UGC considers that research is not an isolated activity; rather it should support and illuminate teaching and learning. Moreover, in recognition of the wider scope of research and scholarly/creative activity intended to be assessed, but evidently not, or only partially, assessed in previous RAEs, the UGC wishes to redefine 'research' as well as to enlarge the assessment to cover other forms of scholarship inherent in the process and/or outcome of research.

  3. The aim in RAE 1999 will be to identify and measure, on the basis of outputs, the following kinds of scholarship :

    1. scholarship of discovery;
    2. scholarship of integration;
    3. scholarship of application; and
    4. scholarship of teaching.

    A brief definition of these four kinds of scholarship, adapted from the Carnegie Foundation's Special Report entitled 'Scholarship Reconsidered', 1990, is at Appendix A.

  4. Prior to the conduct of RAE 1999, each institution will be expected to map out its research strategy, in relation to its role and mission. Upon completion of the RAE, the resultant numerical research index will help the UGC to form a judgement about an institution's recurrent funding for research. The research index will be viewed in the context of the institution's declared research strategy and in any case will not be the sole deciding factor.

  5. The UGC-funded institutions' principal functions are to teach and, to varying degrees, to undertake research. In principle, the funding of each institution should be the sum of funding for the two principal activities, teaching ('T') and research ('R'). In respect of research, it should be recognized that for the purpose of research funding assessment, there are some research activities which should be funded by the UGC, and there are others which should not, in which case the institutions concerned should obtain the necessary funding from the appropriate sources or seek full-cost recovery including on-costs, where appropriate, from the clients.

  6. It is envisaged that RAE 1999 will proceed along lines similar to RAE 1996, with improvements in several areas. The RAE will continue to rate cost centres, not individual staff members. It will determine the "research indices" of all cost centres in a UGC-funded institution, which will then be aggregated to obtain an overall research index for that institution. However, each UGC-funded institution has its distinct role and mission, and it is to be expected that the amount and type of research activity will differ among institutions as a result. There is no expectation by the UGC that all academic staff need to be active in research.

  7. The principles and objective of the third RAE are essentially the same as the second RAE, and therefore the general scheme of the second RAE will again be largely applicable for the purposes of the third exercise, i.e.:

    1. assessment will be based on research outputs;

    2. research outputs by cost centre will be assessed by different subject panels; and

    3. the subject panels will be expected to determine whether each output is up to a threshold standard which they will determine.

  8. Although the third RAE will again look at individual records, it is not intended to be an assessment of individuals' research performance. It will rather be an assessment of the institutions' research activity by cost centre. The subject panels will only produce a total score for each cost centre. If the score(s) of an individual researcher or researchers can be inferred from the cost centre's final research index (especially in the case of very small cost centres), the institutions are advised not to use the inferred information for internal evaluation of the performance of the researchers concerned, because staff appraisal must involve dimensions other than research, even research widely defined; even for research alone, methodologies that are appropriate for assessment in the aggregate for funding purposes may not be appropriate for individual assessments for the purposes of personnel decisions.

Quality

  1. In the second RAE, all subject panels worked to the same threshold definition although they exercised their judgement with due regard for the nature and culture in different disciplines. While the threshold definition for RAE 1999 will remain largely the same as in RAE 1996, in RAE 1999 the panels will be expected to set a threshold standard at a level higher than the previous exercise. The definition of the threshold is given in paragraph 42 below.

General Scheme

  1. The general scheme of the third RAE will be to determine the research index of a cost centre, i.e. the percentage of full-time equivalent researchers in each cost centre whose research work is judged to have reached the quality threshold. The index, p, is determined by the following formula:

    • p = 100% x A / T

      where

      T = the total number of academic staff (in fte) in a cost centre who meet the criteria stated in paragraphs 18 and 19 regardless of the source of funding and whether they submit research output items for assessment;

      and

      A = the total number among these who are judged by the Panel to have reached the quality threshold, including fractional counts.

  2. These Guidance Notes therefore explain in detail the way in which assessments will be made, and the kind of information required and how it will be presented to the RAE panels for assessment. The following areas are covered by the Guidance Notes:

    1. the definition of cost centres (Section II);

    2. the convention for counting the total number of academic staff in each cost centre to arrive at the denominator T (Section III);

    3. the submission of data for each member of staff included (Section IV);

    4. the evaluation guidelines for the panels in its judgment of the submissions to arrive at the numerator A (Section V); and

    5. the handling of the results (Section VI).


II. COST CENTRES

  1. To bring departments onto a common classification system, institutions are asked to map their departments and research units onto a common list of 'Cost Centres' according to the list in Appendix B (not attached. Subject to comment). The mapping should be the same as that used in the UGC's Common Data Collection Format (CDCF) returns. The Cost Centre forms the basis of the data we are now seeking.

  2. For each cost centre, an appropriate representative from the institution, e.g., Department Head or Faculty Dean, may choose to submit in the form of Table 1 a one-page summary giving a factual description of the research activities in the cost centre. The descriptive summary will not be assessed, but will provide a context for the panel's deliberations.


III. NUMBER OF ACADEMIC STAFF IN EACH COST CENTRE

  1. All academic staff who meet both of the following criteria will be counted in the total T, and may submit information to be assessed:

    1. they are holding a paid appointment at the institution concerned for a continuous period of twenty-four months or more covering a particular census date (see paragraph 22), whether or not the period of continuous appointment is principally before or after the census date; and

    2. they do not hold concurrent paid positions at other institutions.

  2. For this purpose, academic staff include staff grades of 'Professor' to 'Assistant Lecturer' plus 'Senior Research Staff'. These correspond to Staff Categories 'A' to 'I' and 'M' as defined (see Appendix C) for the purposes of the Common Data Collection Format (CDCF).

  3. Institutions are required to submit in Table 2 the list of all academic staff who meet the above criteria regardless of whether they intend to submit items for assessment or not. The institutions' Table 2 returns will need to be submitted to the UGC Secretariat by 1 February 1999.

  4. Part-time staff, other than those remunerated on an hourly rate, should also be assessed. They may be expressed in full-time equivalent terms. Each staff member reported, whether he/she is full-time staff or part-time staff expressed in full time equivalent terms, will be counted against the cost centres to which he/she contributes and expressed as a fraction (employment fraction) in the cost centre. To illustrate, a full-time staff member counted against one single cost centre will be expressed as a whole unit '1'. A part-time staff member equivalent to 0.5 full time staff member contributing solely and equally to cost centres A and B will be expressed as 0.25 in cost centre A and 0.25 in cost centre B.

  5. The census date for the reported data is 31 December 1998, which is the middle of the academic year 1998-99.


IV. SUBMISSION OF DATA FOR EACH MEMBER OF STAFF

Who to submit

  1. Institutions are invited to submit in respect of every member of staff listed in Table 2 that the institution feels has output for assessment material in accordance with Table 3 by 15 March 1999.

  2. The UGC recognizes that research is only one aspect of an institution's activities, and understands that there may be valid reasons why some valuable and respected members of staff are not, for the period in question, active in research. For example, some staff could be heavily involved in course development, or in institutional administration. Therefore each institution is free to decide, in consultation with the individual, not to make a submission, and no stigma should be attached to any individual taking such a decision.

What to submit

  1. The material that the institution is invited to submit in respect of each eligible staff member will be

    1. up to five (see paragraph 32) best research output items within the period of assessment, defined in paragraphs 27 to 28 below; and

    2. if considered appropriate, a separate submission of one exceptional research output item that meets the criteria in paragraph 27, but does not fit into category (a) above.

Assessment period

  1. For the purpose of paragraph 25(a) above, the assessment period will be the three-year period from 1 January 1996 to 31 December 1998.

Definition of output

  1. All output items submitted for assessment must meet all of the following criteria:-

    1. it contains an element of innovation;

    2. the output and the process involved contribute to scholarship;

    3. it is publicly accessible; and

    4. it is of interest to peers and is generalizable.

    [In this context, it should be noted that not all research activities (and hence not all the output items to be submitted for assessment) are or should be funded by the UGC (cf paragraph 9 above). Please refer to paragraphs 19 and 20 of the Consultation Document.]

  2. The following are considered to be items falling within category in paragraph 25 above :

    1. any publication, artifact etc., provided it was -

      1. published (or for non-traditional works, such as drama, appearing) within the assessment period; or

      2. not yet published, but officially accepted for publication (without need for further amendments of any kind) in that period; in this case, a letter of acceptance will need to be attached; or

    2. 'non-traditional output' that may or may not be in publishable form, e.g. concert performance, video tape, computer software programme, buildings, or any creative work that can be evaluated for merit and an assessment obtained on either a public or confidential basis (RAE panels to decide on the basis of the criteria in (a) above, see also paragraph 36.

  3. On the other hand, proprietary research that does not result in output that is accessible to the public and the profession is not accepted as an output for this purpose.

  4. For the avoidance of doubt, the UGC wishes to stress that assessable output items are not restricted to papers in journals.

New Researchers

  1. New researchers, including senior professionals who have just joined academia, present a special case, since they may not have had time to produce significant or publishable outputs according to the RAE definition. Therefore, all staff who first took up an academic appointment (anywhere) only within the last three years i.e. on or after 1 January 1995, will be given special consideration by the RAE panels. Apart from the kind of output defined at paragraphs 28 and 34, recent doctoral dissertation research may be included by these new appointees in the lists of research outputs for assessment.

Quality criterion

  1. Although a maximum of five output items could be submitted, individual staff may wish to submit fewer items, even only one item. A researcher can be judged to be fully active on the basis of one single tem of high quality. The UGC wishes to emphasize that the focus of the RAE is not on quantity, but on the quality of output. It is believed that in the majority of cases, a clear decision should be able to be made on the basis of three items.

  2. Research output items produced under joint-authorship are also accepted. An explanation should be given if the number of co-authors is unusually large in the context of the discipline concerned. In addition, where a research output item is submitted for assessment by more than one member of academic staff within the same cost centre, the institution should invite each academic staff member submitting the item for assessment to provide an explanation (maximum one page) attached to the item to indicate the nature and degree of his/her involvement. This is essential since the exercise serves to score the cost centre as a whole. Any possible double-counting should be avoided.

Exceptional item

  1. As regards paragraph 25(b) above, individual staff may submit an exceptional or additional item if considered appropriate. This is to cater for special situations where, for example, output items within the definition of paragraph 25(a) could not be produced during the assessment period. It is accepted that works of great impact could take more than three years to complete and they tend to be infrequent. In submitting an exceptional items for assessment, the individual concerned should provide a one-page justification together with adequate evidence to explain why it should be accepted for assessment.

Data required

  1. For each item falling within category (a) of paragraph 25 above, the data required to be submitted include one copy of the full set of the output items to be assessed (all such copies will be returned after the assessment exercise); and in addition 5 copies :

    1. in the case of a journal paper, the first page and the abstract pages (output items will be judged on their own merit and will not be judged simply on their category or venue of publication);

    2. in the case of a book, a short description of the book, the title page and table of contents and, where appropriate, a book review;

    3. in other cases, sufficient information for the item to be assessed.

  2. For a submission in category (b) of paragraph 25 above, a description limited to one A4 page should be provided.

  3. Data not required

  4. Attention will be focused on quality rather than quantity. Moreover the quality of each item will be judged on its own merits and not solely in terms of its category (e.g. journal paper is not necessarily of higher or lower merit vis-à-vis book chapter, or refereed vis-à-vis unrefereed) or venue of publication.

Sample submission

  1. A sample submission is shown in Appendices D1 to D3 (not attached, refer to Guidance Notes RAE 1996).

Further information

  1. If an RAE Panel requires any further information, it will approach the institution(s) concerned through the UGC Secretariat for assistance.


V. EVALUATION GUIDELINES

Panels

  1. Since the RAE covers the whole range of cost centres, with different types of academic research outputs, the cost centres are grouped into groups, each group to be assessed by a separate RAE panel. The groupings of the cost centres and a list of the RAE panels are shown in Appendix E (not attached. Subject to comment).

  2. Each panel will consist of local and overseas academics in the relevant disciplines and, where appropriate, also professionally qualified people from business, government and the arts. They will be appointed on an ad personam basis and will be expected not to represent the interests of their own institutions. Thus standards will ultimately be set and judgment made by academic peers and not by the UGC.

Threshold standard

  1. In view of the emphasis on quality, the operational definition of the threshold standard in the third RAE will be strict. The definition is:

    • "Quality of output equates to an attainable level of excellence appropriate to the discipline in Hong Kong, and showing some evidence of international excellence."

  2. The UGC will strive to ensure broad comparability across disciplines, but it will be up to each Panel, with its subject expertise and knowledge of local situations, to translate this general definition into more precise benchmarks appropriate to each discipline or groups of discipline. Panels are also expected to interpret 'international excellence' with due regard to the nature of those subjects that may, by their nature, necessarily have a strong local or regional focus. In the case of publications or other outputs of a local nature, the panel should assess whether the item represents a contribution to the work of the international research community in terms of its intellectual content, as well as rigour of process and methodology.

  3. To minimize any possible divergence in judgement, all RAE subject panels should make reference to the following amplifications to the definition in paragraph 42 above:

Attainable:

Reference is made to 'attainable' levels, not 'attained' levels. Thus, for a hypothetical subject in which reasonable norms common elsewhere are much higher than the standard shown by all academics in Hong Kong, then it is the former ('attainable') rather than the latter ('attained') which should prevail as the benchmark.

International excellence:

This should not be equated with publication outside of Hong Kong or the region; rather it is intended that evaluation should be made with reference to the best international norms in the mainstream of that discipline or sub-discipline. It is possible that in some particular disciplines, such norms are set by publications in Hong Kong or the region.

International vs local:

An effort should be made to distinguish between

  1. a publication that is local because it addresses local issues, and

  2. a publication that is local because it does not meet the standards of rigour and scholarship expected internationally in the mainstream of that discipline. In the former case, the item should not be discounted; in the latter, it should be.

  1. The RAE panels will view the submission as a whole, and will not score each item individually. In particular, panels will be asked not to define thresholds for each item or give fractions on n/5. Individual RAE panels will attempt to draw up their own thresholds, calibrate with one and other, and consider common working procedures as soon as they are formed.
  2. It is expected that the panels will consider, first of all, a binary cut, i.e. whether the output of each individual concerned does or does not meet the threshold standard as defined.
  3. However, it is recognized that it may prove difficult in some cases to adopt a binary cut, and it will be up to individual panels to consider whether a fractional score (i.e. lower than 1) should be assigned. In assigning fractional scores, RAE panels are expected to give simple grading. The panels will be instructed not to adopt a mechanical approach during the assessment.
  4. Output items will be judged on their own merits and will not be judged simply on their category or venues of publication. In many disciplines, an output item appearing in a quality venue, e.g. an article in a prestigious journal, or a musical composition performed in an internationally acclaimed concert hall, may be presumed to be of high quality, and there is no need for the panel to study the item in detail. However, panels should recognize that there could be quality output items in venues that may not be prestigious. In these cases and when in doubt, the panel (or designated member(s)) will study the item in question and not judge it automatically according to the venue.

VI. HANDLING OF RESULTS

  1. The exercise is ultimately only intended to assess the research output of a cost centre, not individuals, so it will not generate individually identifiable scores. Only the total scores of a cost centre will be produced. Subject to the agreement of all the UGC-funded institutions, the results of the RAE of individual institutions will be conveyed only to the Head of the institution concerned, together with the sector-wide average and median indices for the relevant cost centres (subject to comment).
  2. The UGC fully recognizes that different institutions have different missions, discipline profiles and histories and the purpose of the exercise is neither to identify and compare the number of 'star' researchers nor to measure the research output with a view to giving a grading to institutions for research performance.
  3. The results will help the UGC form a judgement, in the light of other relevant factors, to inform funding in respect of the research portion of the recurrent grants for institutions in the next triennium 2001-2004.

Timetable for the submission of data for RAE 1999

  1. To summarize, institutions are requested to submit the following Tables and supporting documents in accordance with the dates shown below:
Date Table
1 February 1999 - Table 2 for each cost centre
15 March 1999 - Table 1 for each cost centre
- Table 3 for each individual cited in Table 2 who wishes to make a submission

Form of submission

  1. The form of submission of Tables 1 to 3 and any other data submitted in accordance with these Guidance Notes should be by way of hard copies forwarded to the UGC Secretariat. However, Table 2 should be submitted with a soft copy.



UGC Secretariat
8 July, 1998



Appendix A

Scholarship as defined by the Carnegie Foundation*

The Carnegie Foundation considers that there is a more inclusive view of what it means to be a scholar -- a recognition that knowledge is acquired through research, synthesis, practice, and teaching. While scholarship means engaging in original research, scholarship now has a broader and capacious meaning. Beyond the age-old "teaching Vs research" debate, there are four separate, yet overlapping functions: They are the scholarship of discovery; the scholarship of integration; the scholarship of application; and the scholarship of teaching.

Scholarship of Discovery

The scholarship of discovery, at its best, contributes not only to the stock of human knowledge but also to the intellectual climate of an institution. It is a scholarly investigation, closest to the literal meaning of what academics speak of "research", that confronts the unknown and creates new knowledge. Not just the outcomes, but the process, and especially the passion, give meaning to the effort.

Scholarship of Integration

It is a serious, disciplined work that seeks to interpret, draw together and bring new insight to bear on original research. The scholarship is closely related to discovery. Such work is increasingly important as traditional disciplinary categories prove confining, forcing new topologies of knowledge. This scholarship also means interpretation, fitting one's own research -- or the research of others -- into larger intellectual patterns. A variety of scholarly trends -- interdisciplinary, interpretive, integrative, are examples of scholarship of integration.

Scholarship of Application

It is a dynamic process of creating new intellectual understandings arising out of theory and practice. The term itself may be misleading if it suggests that knowledge is first "discovered" and then "applied". The process is in fact more dynamic; new intellectual understanding can arise out of vital interaction between theory and practice, and one renews the other.

Scholarship of Teaching

It is a process that transforms and extends knowledge while transmitting an intelligible account of knowledge to the learners



Appendix C

STAFF GRADES, Modes and Funding Sources


Academic Grades
Academic, Senior Academic, Junior
A. Professor E. Principal Lecturer (P), junior
B. Reader F. Senior Lecturer (P)
C. Senior Lecturer (U) G. Lecturer (U)
D. Principal Lecturer (P), senior H. Lecturer (P)
I. Assistant Lecturer
J. Instructor
K. Demonstrator/Tutor/Teaching Assistant
L. Other, including language assistant, field work supervisor etc
Research Staff (Staff who spend essentially all their time on research)
M. Senior Research staff ["leaders", N. Junior Research staff ("followers"; usually Post Doctoral) usually Graduate]
Non-Academic Grades
O. Admin, Senior P. Admin, Junior (including secretarial)
Q. Technical, Senior R. Technical, Junior
S. other, including "Mod 1"
Staff Mode Source of Staff salary funding
F: Full-time W: wholly from General Funds (WfGF)
P: Part-time P: partly from General Funds (PfGF)
N: not from General Funds (NfGF)