Research Assessment Exercise 1996 - Guidance Notes (issued on 29 April 1996)
I. INTRODUCTION
Background
The second Research Assessment Exercise (RAE) will be undertaken by the University Grants Committee (UGC) in October/November 1996. Like the first RAE conducted in early 1994, the purpose of the exercise is to assess the research output performance of the UGC-funded institutions by cost centre, to be used as the basis for allocating some of the research portion of the institutional recurrent grant for the triennium 1998-2001.
The UGC also wishes to emphasise that the research assessment exercise does not imply an interest in research to the possible detriment of teaching quality. The Committee recognises that both teaching and research are important and are inter-related elements in the development of higher education. Indeed, a much larger portion of the recurrent grants allocated to institutions is and should be attributed to teaching. However, it is necessary for the Committee to adopt different approaches to look at teaching and research in view of the different nature of these activities.
Contents of this document
This document consists of the Guidance Notes, a set of blank forms numbered Tables 1 to 3, and Appendices A to D. A diskette containing these papers in Word 6.0 or Excel 5.0 format is also provided to each institution to facilitate the generation of sufficient copies for internal distribution.
Distribution of this document
Each institution should issue this document to every member of its academic staff, and to every member of its administrative staff responsible for institutional governance or involved in research policy and support, in order that the principles, aims and methodology of the RAE are fully appreciated.
Principles and Philosophy for RAE 1999
The UGC-funded institutions are there to teach and to undertake research. In principle, the funding of each institution should be the sum of funding for the two principal activities, teaching and research. In respect of research, the allocation of funds should be based on the percentage of staff who are active in research and the quality of their research. However, each UGC-funded institution has its own role and mission, and it is expected that the amount and type of research activity will differ among institutions. Likewise, the percentage of academic staff who are active researchers is expected to vary as well. Not all academic staff need to be active researchers.
In the first RAE, eight RAE panels were formed to undertake the exercise. Each RAE panel consisted primarily of local academics from the UGC-funded institutions together with one UK process expert. The key task for each panel was to determine the number of "active researchers" out of the nominated researchers from each cost centre, on the basis of the quality of the best three research outputs from each nominated researcher.
Upon completion of the assessment of the research outputs of nominated researchers, the RAE panels determined, for each cost centre of the seven institutions, the number and proportion of Research Full-time Equivalent Active Researchers (RFTE-ARs). The number of RFTE-ARs was the number (including fractions) judged to be research active, weighted by the employment fraction in that cost centre. The proportion was determined by dividing the number of RFTE-ARs by the total number of academic staff in that cost centre (not by the number of those nominated), again weighted by the employment fraction in that cost centre. The results of the exercise were used as the basis for allocating some of the research portion of the institutional recurrent grant for the triennium 1995-98.
The principle and objective of the second RAE is essentially the same as the first RAE, and therefore the general scheme of the first RAE is largely applicable for the purpose of the second exercise, i.e.:
assessment will be based on research outputs;
research outputs by cost centre will be assessed by different subject panels; and
the subject panels will determine whether the output is up to a threshold standard which they would determine.
Similar to the first RAE, the results of the second RAE will inform how recurrent funding for supporting research activities should be allocated in the 1998-2001 triennium.
Although the second RAE will look at individual records, it is not an assessment of individuals' research performance. It is an assessment of the institutions' research activity by cost centre. As the criteria adopted by the RAE panels are not designed for staff appraisal purposes, the institutions' administrations are not expected to use identical or similar criteria for individual staff appraisal, especially since staff appraisal must involve dimensions other than research. The results of individuals' performance in the RAE exercise will not be published or released to the institutions.
Quality
In the first RAE, each RAE panel was asked to set its own threshold standard, against which the output of each researcher was measured; this threshold was, on the whole, not stringent in terms of quality. It is intended that in the second RAE, greater emphasis should be given to the quality of the research. Accordingly, the Panels will set the threshold standard at a higher level than in the last exercise. The definition of the elevated threshold is given in paragraph 40 below.
General Scheme
The general scheme of the second RAE will be to determine the percentage of full-time equivalent active researchers in each cost centre. The percentage, p, is determined by the following formula:
p = 100% x A / T
in which
T = the total number of academic staff (in fte) in the cost centre who meet the criteria stated in paragraphs 16 and 17 regardless of the source of funding and whether they submit research output items for assessment; and
A = the total number among these who are judged by the Panel to be research active, including fractional counts.
These Guidance Notes therefore explain in detail the way in which assessments will be made, and the kind of information required and how it will be presented to the RAE panels for assessment. The following areas are covered by the guidance notes:
the definition of cost centres (Section II);
the convention for counting the total number of academic staff in each cost centre to arrive at the denominator T (Section III);
the submission of data for each member of staff included (Section IV);
the evaluation guidelines for the panels in its judgment of the submissions to arrive at the numerator A (Section V); and
the handling of the results (Section VI).
II. COST CENTRES
To bring departments onto a common classification system, we ask institutions to map their departments and research units onto a common list of "Cost Centres" according to the list in Appendix A. The mapping should be the same as that used in the UGC's Common Data Collection Format (CDCF) returns. The Cost Centre forms the basis of the data we are now seeking.
For each cost centre, an appropriate representative from the institution e.g. Department Head, faculty Dean, etc. may choose to submit in the form of Table 1 a one-page summary giving a factual description of the research activities in the cost centre. The descriptive summary will not be assessed, but will provide a context for the panel's deliberations.
III. NUMBER OF ACADEMIC STAFF IN EACH COST CENTRE
All academic staff who meet both of the following criteria will be counted in the total T, and may submit information to be assessed:
they are holding a paid appointment at the institution concerned for a continuous period of eighteen months or more covering a particular census date (see paragraph 20), whether or not the period of continuous appointment is principally before or after the census date); and
they do not hold concurrent paid positions at other institutions.
For this purpose, academic staff include staff grades of "Professor" to "Assistant Lecturer" plus "Senior Research Staff". These correspond to Staff Categories "A" to "I" and "M" as defined (see Appendix B) for the Common Data Collection Format (CDCF).
Institutions are required to submit in Table 2 the list of all academic staff who meet the above criteria regardless of whether they intend to submit items for assessment or not. This table will need to be submitted to the UGC Secretariat by 31 May 1996.
Part-time staff, other than those remunerated on an hourly rate, should also be assessed. They may be expressed in full-time equivalent terms. Each staff member reported, whether he/she is full-time staff or part-time staff expressed in full time equivalent terms, will be counted against the cost centres to which he/she contributes and expressed as a fraction (employment fraction) in the cost centre. To illustrate, a full-time staff member counted against one single cost centre will be expressed as a whole unit "1". A part-time staff member equivalent to ½ full time staff member contributing solely and equally to cost centres A and B will be expressed as ¼ in cost centre A and ¼ in cost centre B.
The census date for the reported data is 31 December 1995, which is the middle of the academic year 1995-96.
IV. SUBMISSION OF DATA FOR EACH MEMBER OF STAFF
Who to submit
Every member of staff listed in Table 2 is invited to submit material for assessment in accordance with Table 3 by 19 July 1996.
The UGC recognizes that research is only one aspect of an institution's activities, and understands that there may be valid reasons why some valuable and respected members of staff are not, for the period in question, active in research. For example, some staff could be heavily involved in course development, or in institutional administration. Therefore it should be free for each individual, in consultation with the department/unit, to decide not to make a submission, and no stigma should be attached.
What to submit
The material that the institution is invited to submit in respect of each eligible staff member will be
up to five (see paragraph 32) best research output items within the period of assessment, defined in paragraphs 27 to 28 below; and
if considered appropriate, a separate submission of one exceptional research output item that meets the criteria in paragraph 27, but does not fit into category (a) above.
Assessment period
The material that each member of staff is invited to submit will be:
up to five (see paragraph 30) best research output items within the period of assessment, defined in paragraphs 25 to 27 below; and
if considered appropriate, a separate submission of one exceptional research output item that meets the criteria in paragraph 25, but does not fit into category (a) above.
Assessment period
For the purpose of paragraph 23(a) above, the assessment period will be the four-year period from 1 January 1992 to 31 December 1995.
Definition of output
All output items submitted for assessment must be research-related, publicly accessible by academic peers and contain a sufficient element of innovation.
The following are considered to be items falling within category (a) in paragraph 23 above :
any publication, artifact etc., provided it was
published (or for non-traditional works, such as drama, appearing) within the assessment period; or
not yet published, but officially accepted for publication (without need for further amendments of any kind) in that period; in this case, the letter of acceptance need to be attached; or
"non-traditional output" that may or may not be in publishable form, e.g. concert performance, video tape, computer software programme, buildings, or any creative work that can be evaluated for merit and an assessment obtained on either a public or confidential basis (RAE panels to decide on the basis of the criteria in (a) above).
On the other hand, proprietary research that does not result in output that is accessible to the public and the profession is not accepted as an output for this purpose.
It is stressed that output items are not restricted to papers in journals.
New Researchers
New researchers, including senior professionals who have just joined academia, present a special case, since they may not have had time to produce significant or publishable outputs according to the RAE definition. Therefore, all academic staff appointed within the last three years, i.e. on or after 1 July 1993, and who have had less than three years of post-qualification experience as an academic at the time of appointment will be given special consideration by the respective RAE panels. Apart from the kind of output defined at paragraphs 26 and 32, recent doctoral dissertation research may be included in the list of research outputs of these new appointees.
Quality criterion
Although a maximum of five output items could be submitted, individual staff may wish to submit fewer items, even only one item. A researcher can be judged to be fully active on the basis of one single item of high quality. The UGC wishes to emphasise that the focus of the RAE is not on quantity, but on the quality of output. It is believed that in the majority of cases, a clear decision can be made on the basis of three items.
Research output items produced under joint-authorship are also accepted. An explanation should be given if the number of co-authors is unusually large in the context of the discipline concerned. In addition, where a research output item is submitted for assessment by more than one member of academic staff within the same cost centre, the institution should invite each academic staff member submitting the item for assessment to provide an explanation (maximum one page) attached to the item to indicate the nature and degree of his/her involvement.
Exceptional item
As regards paragraph 23(b) above, individual staff may submit an exceptional or additional item related to research if considered appropriate. This is to cater for special situations where, for example, output items within the definition of paragraph 23(a) could not be produced during the assessment period. It is accepted that works of great impact could take more than four years to complete and they tend to be infrequent. In submitting an exceptional research-related items for assessment, the academic staff should provide a one-page justification together with adequate evidence to explain why it should be accepted for assessment.
Data required
For each item falling within category (a) of paragraph 23 above, the data required to be submitted include one copy of the full set of the output items to be assessed (all such copies will be returned after the assessment exercise); and in addition 10 copies of
in the case of a journal paper, the first and the abstract and references pages (output item will be judged on their own merit and will not be judged simply on their category or venue of publication);
in the case of a book, a short description of the book, the title page and table of contents and, where appropriate, a book review;
in other cases, sufficient information for the item to be assessed.
For a submission in category (b) of paragraph 23 above, a description limited to one A4 page is required
Data not required
Attention will be focused on quality rather than quantity. Moreover the quality of each item will be judged on its own merits and not solely in terms of its category (e.g. journal paper is not necessarily of higher or lower merit vis-à-vis book chapter, or refereed vis-à-vis unrefereed). Therefore, in contrast to the last RAE, the following are not required for the purpose of the second RAE, either on the part of individuals or on the part of cost centres:
a total count of output items; and
a designation of the category to which each output item belongs.
Sample submission
A sample submission is shown in Appendices C1 to C3.
Further information
If an RAE Panel requires any further information, it will approach the institution(s) concerned through the UGC Secretariat for assistance.
V. EVALUATION GUIDELINES
Panels
Since the RAE covers the whole range of cost centres, with different types of academic research outputs, the cost centres are grouped into eight groups, each group to be assessed by a separate RAE panel. The groupings of the cost centres and a list of the RAE panels are shown in Appendix D.
Each panel will consist of academics in the relevant disciplines from the local higher education institutions and, where appropriate, also professionally qualified people from business, government and the arts. They will be appointed on an ad personam basis and will be expected not to represent the interests of their own institutions. Each panel may also include, on a need basis, overseas members to provide subject expertise and/or international calibration for the exercise. Thus standards will ultimately be set and judgment made by academic peers and not by the UGC.
Threshold standard
In view of the intended emphasis on quality, the operational definition of the threshold standard in the second RAE will be strict. The definition is:
"Quality of output equates to an attainable level of excellence appropriate to the discipline in Hong Kong, and possibly showing some evidence of international excellence."
The UGC will strive to ensure broad comparability across disciplines, but it will be up to each Panel, with its subject expertise and knowledge of local situations, to translate this general definition into more precise benchmarks appropriate to each discipline or groups of discipline. Panels are also expected to interpret "international excellence" with due regard to the nature of those subjects that may, by their nature, necessarily have a strong local or regional focus.
The RAE panels will view the submission as a whole, and will not score each item individually. In particular, panels will be asked not to define thresholds for each item or give fractions on n/5. Individual RAE panels will attempt to draw up their own thresholds, calibrate with one and other, and consider common working procedures as soon as they are formed.
It is expected that the panels will consider, first of all, a binary cut, i.e. whether the academic staff concerned is up to the level of Hong Kong excellence.
Given this mechanism, it is expected that the percentage of staff being assessed as research active is likely to be significantly lower than the percentage assessed in the first RAE.
However, it is recognised that it may prove difficult in some cases to adopt a binary cut, and it will be up to individual panels to consider whether a fractional score should be assigned. In assigning fractional scores, RAE panels are expected to give simple grading.
Output items will be judged on their own merits and will not be judged simply on their category or venues of publication. In many disciplines, an output item appearing in a quality venue, e.g. an article in a prestigious journal, or a musical composition performed in an internationally acclaimed concert hall, may be presumed to be of high quality, and there is no need for the panel to study the item in detail. However, panels should recognise that there could be quality output items in venues that may not be prestigious. In these cases and when in doubt, the panel (or designated member(s)) will study the item in question and not judge it automatically according to the venue.
VI. HANDLING OF RESULTS
All information relating to the identity and scores of individual staff in the exercise will be kept confidential. The results of the second RAE of individual institutions will be conveyed only to the Head of the institution concerned. The results will only contain the number of active researchers and the proportional RFTE-AR by cost centre. There will not be any reference to the identity and score of individual staff.
The UGC will not publish any "league tables" regarding the research performance of institutions, as the exercise is designed only to provide objective data to inform the funding decisions of the UGC. The UGC fully recognises that different institutions have different missions, discipline profiles and histories and the purpose of the exercise is neither to identify and compare the number of "star" researchers nor to measure the research output with a view to giving a grading to institutions regarding research performance.
The results on active researchers in the institutions will inform funding in respect of the research portion of the recurrent grants for institutions in the next triennium 1998-2001.
Timetable for the submission of data for RAE 1996
| Date | Table |
|---|---|
| 31 May 1996 | Table 2 for each cost centre |
| 19 July 1996 | - Table 1 for each cost centre
- Table 3 for each individual cited in Table 2 who wishes to make a submission |
Form of submission
The form of submission of Tables 1 to 3 and any other data submitted in accordance with these Guidance Notes should be by way of hard copies forwarded to the UGC Secretariat.
UGC Secretariat
April 1996