Research Assessment Exercise 1999 - Guidelines for Panel Members
Purpose
The document University Grants Committee Research Assessment Exercise (RAE) 1999 Guidance Notes, dated 22 January 1999, lays down the basic philosophy and principles of the Research Assessment Exercise (RAE), as well as guidance to institutions and to those who had to submit research output items for assessment. These supplementary guidelines are addressed to Panel Members, who are charged with evaluating the submissions.
Membership
All Panel Members are appointed in their personal capacity, and should not seek to represent their institutions in the Panel's deliberations.
Eligibility
Eligibility of academics for assessment, i.e., for inclusion in the "denominator", is a purely administrative matter not involving academic judgement. Queries about any case should be brought to the attention of the UGC Secretariat for resolution well ahead of the panel meetings.
Categories of scholarship
The aim in the RAE 1999 is to identify and measure, on the basis of the submitted output, contributions to the following kinds of scholarship:
- scholarship of discovery (Dis);
- scholarship of integration (Int);
- scholarship of application (App); and
- scholarship of teaching (Tea).
A brief definition of these four kinds of scholarship, adapted from the Carnegie Foundation's Special Report entitled 'Scholarship Reconsidered', 1990, is at Appendix A.
Material for assessment
The assessment should be based purely on the material submitted. Personal knowledge on the part of Panel Members (e.g., about other research output items that have not been submitted) should not enter the deliberation, since inclusion of such factors could lead to unintentional bias. If absolutely necessary, additional information may be sought, but this should be limited to information about the items already submitted for assessment, and requests should be made through the Secretariat.
Assignment of cases to Panel Members for assessment
The Panel Convenor will assign cases to Panel Members for initial assessment. The latter will be responsible for a detailed reading,and for putting forward a recommendation to the full Panel or sub-group(for Panels which are divided into sub-groups) for final decision. To ensure fairness and consistency, each case must be assessed in detail by at least two members.
To ensure that non-traditional output items (i.e.,those in categories Int, App and Tea) receive adequate attention, Panels may consider setting up a sub-group with suitable membership (including members drawn from outside academia, where appropriate) to evaluate such items separately, and to make recommendations regarding their assessment to the full panel. Each Panel will need to consider its own situation having regard to the nature of the subjects as well as the submissions and work out procedures for the operation of the sub-group. Alternatively, panels may refer doubtful cases to the panel members with expertise in these categories of scholarship on their panels for advice as and when necessary.
To the extent possible, Panel Members should not be assigned the responsibility for assessing submissions from his/her own institution, to avoid embarrassment.
Panel Members need not be excused while cases from their own institutions are discussed, unless for personal reasons they choose to do so. However, they must excuse themselves when their own cases are discussed, and when items which they have co-authored are discussed.
Basis of evaluation
The general scheme of the RAE 1999 is to determine the research index of a cost centre, i.e., the percentage of full-time equivalent (fte) researchers in each cost centre whose research work is judged to have reached or surpassed the quality threshold. The index,p, is determined by the following formula:
p = 100% x A / T
where
T = the total number of academic staff (in fte) in a cost centre who meet the criteria stated in the Guidance Notes regardless of the source of funding and whether they submit research output items for assessment; and
A = the total number among these who are judged by the Panel to have reached or surpassed the quality threshold, including fractional counts.
A list of the cost centres under the 12 RAE subject panels is at Appendix B. The task of each Panel is to assign a 'score' to each individual (on a scale of 0 to 1), whose sum (weighted by the fte fraction where appropriate) will be the numerator A. However, as explained below, the record will be kept in such a way that the 'score' of any individual cannot be identified.
In accordance with the Guidance Notes for RAE 1999, each institution has submitted, on the basis of a self-assessment,a Research Strategy Statement to reflect its research philosophy, vision and priorities in relation to its role and mission and stage of development. Also, the Table 1 of each institution's submissions has stated its research priorities and distribution of research activities amongst the four categories of scholarship in respect of each cost centre. The declared research strategy, together with the information contained in Table 1, will provide the context for the UGC in viewing the research index of the institution as a whole upon completion of the RAE. While such information would not have any direct relevance to the Panel assessment process, Panel Members may wish to note the information for reference.
Split between cost centres
As in the previous exercises, there would be cases where staff members (fte) are split between cost centres. Panels may wish to refer to the general guidelines adopted in the 1996 RAE at Appendix C in dealing with these cases.
Co-authorship
There may be cases in which co-authored items are submitted for evaluation. These cases require special attention and care when assessing the intellectual contribution of the cost centre concerned. Guidelines are provided at Appendix D for reference.
Threshold standard
The threshold standard is defined in Paragraph 51 of the Guidance Notes as:
"Quality of output equates to an attainable level of excellence appropriate to the discipline in Hong Kong, and showing some evidence of international excellence."
As evidence has suggested that the quality of research has substantially improved, Panels should interpret this description of the threshold stringently. This definition should be applied in the manner as described in para. 22.
To minimize any possible divergence in judgement, all RAE subject panels will be asked to make reference to the following amplifications to the definition of threshold standard:
| Attainable | Reference is made to 'attainable' levels, not 'attained' levels. Thus, for a hypothetical subject in which reasonable norms common elsewhere are much higher than the standard shown by all academics in Hong Kong, then it is the former ('attainable') rather than the latter ('attained') which should prevail as the benchmark. |
| International excellence | This should not be equated with output items published outside of Hong Kong or the region; rather it is intended that evaluation should be made with reference to the best international norms in the mainstream of that discipline or sub-discipline. It is possible that in some particular disciplines, such norms are set by output items published in Hong Kong or the region. |
| International vs local | A distinction should be made between
|
The Panels are expected to give recognition to those items which have societal relevance, which have achieved or are likely to achieve symbiosis with industry, commerce and government, and/or with local culture and society.
The operational procedure should be a two-step process.
First, each item should be evaluated and given a grade on a scale of A+/A/B/C/D. (Paragraphs 20 to 21)
Second, a 'score' should be assigned to the individual on a scale of 0 to 1, on the basis of the best three grades. (Paragraphs 22 to 24)
Although only the three best grades will be used, Panels are nevertheless asked to grade every item (including the exception item if one is submitted). This is to allow UGC to form a qualitative view on the statistical distribution of quality items in the different categories of scholarship. This additional statistical information will not affect the calculation of the index p.
For new researchers who first took up an academic appointment (anywhere) only within the last three years i.e. on or after 1 January 1996, their research outputs should be given special consideration by the panels.
Grading of items
Panels may wish to refer to the following scoring schema for the grading of individual items which was adopted in the RAE 1996 (except that Grade D has been added):
The quality of each item should be judged on its own merits and not solely in terms of its category (e.g., a journal paper is not necessarily of higher or lower merit than a book chapter, nor is a refereed article necessarily of higher or lower merit than an unrefereed one), venue or language of publication. Panels should recognize that there could be quality output items in venues that may not be prestigious. In these cases, and in any case when in doubt, the Panel (or designated member(s))should study the item in question and not judge it automatically according to the venue.
Panels should view the best 3 items submitted by each academic holistically, and should try, in the first instance, to see whether the individual can be said to have attained the agreed threshold. However, it is recognized that in some cases it may prove difficult to adopt a binary cut, in which case it will be up to individual Panels to consider whether a fractional score (i.e., lower than 1) should be assigned. The following conversion table is suggested, but Panels are urged not to operate mechanistically, but to use such tables only for guidance.
Three aspects of the scoring schema should be emphasized.
There is no need for 3 items to achieve a full score; for example, one 'A+' item could lead to a full score.
No more than the best three grades will be taken into account, so that quantity cannot make up for mediocre quality.
Once the individual items have been rated, the score should be blind to the category of scholarship, e.g., a combination of gradings of 'BBC'distributed in the categories of Dis/Dis/Dis should in general score no better and no worse than 'BBC' distributed in the categories of Dis/Tea/App.
With their subject expertise and knowledge of local circumstances, the RAE subject panels will be expected to translate these general definitions into more precise benchmarks appropriate to each discipline or group of disciplines. The Panels will also be expected to interpret 'international excellence' with due regard to the nature of those subjects that may, by their nature, necessarily have a strong local or regional focus. In the case of publications or other outputs of a local nature, the Panel will need to assess whether the item represents a contribution to the work of the international research community in terms of its intellectual content, as well as rigour of process and methodology.
The purpose of the RAE is to assess cost centres, not individuals. Therefore the principle of anonymity should be applied strictly throughout the process of evaluation. Records to be kept in respect of the grading of items and the scores of individuals should make no reference to the names of the academic staff concerned. Working papers kept by Panel Members should be destroyed after the completion of the exercise. The Secretariat will assist Panels in working out operational procedures that would comply with this principle.
In order to maintain the anonymity of individuals and also to keep a statistical record of the evaluation across the four categories of scholarship, a standard scoring form (Appendix E) will be used. As assessments are made, records should be kept by two designated members of the Panel on duplicate copies of this form and the relevant parts of the form which shows the individual names of the academic staff and their particulars should be destroyed immediately after the process.
Overseas members will also be asked to provide an impressionistic lateral comparison of the standard of the cost centres with those of other countries/regions. The views so obtained will not become part of the assessment and will not affect the scores of items, individuals or the research index of the cost centres in any way. To ensure that these impressions will not be so misused, the record should not relate to particular institutions. Further guidance will be provided to overseas members for this purpose.
-
A longitudinal comparison of the RAE 96 and RAE 99 results would also be desirable. Panel members who have served in the RAE 96 exercise are expected to give a general and qualitative comparison of the results having regard to the methodology and standard adopted in the two exercises. The comparison should be properly recorded and should form part of the Panel reports.
Panel Members may seek advice from other academics, but such referrals should be informal and kept to a minimum.
A workshop and two meetings will be held on 15 and 16 September 1999 for Panel Members to discuss and agree on the general principles and guidelines to be followed in the exercise:
on 15 September (afternoon), a workshop, with participation by the Carnegie Foundation (to be led by its President, Professor Lee Shulman), would be organised to brief Panel Members on the current thinking on the Boyer definitions of scholarship contained in the Special Report entitled "Scholarship Reconsidered" and to answer any questions on the subject;
on 16 September (morning), the Convenor of the UGC's QSC small group overseeing the RAE would outline the overall philosophy and framework of the RAE,including the points covered in this document;
on 16 September (afternoon), each Panel would hold a preliminary meeting to discuss and agree on further guidelines specific to the Panel and/or its sub-groups. The Panel Convenor would also assign cases for assessment to members at the meeting.
The Secretariat will distribute to Panel Members some samples of the summary sheets (Table 3) together with the supporting documents selected from the output items submitted for assessment for preliminary reading about a month before the meetings in mid September, so that Panel Members would have an opportunity to identify points/problems that may need to be addressed at the meetings. Thereafter, Panel Members are then expected to do the necessary detailed reading on the assigned cases and prepare for the full panel meetings which would be held for 1 to 3 days (depending on needs) during the period from 20 to 29 October 1999 (exact dates to be confirmed).
To increase the transparency of the RAE, these Guidelines, together with each individual Panel's guidelines on the Rating of Individual Items and Score Conversion Table (if different from those suggested above), will be published for information after the conclusion of the exercise.
-
Same submission submitted by a staff member for different cost centres under the same panel
The panel should decide how to make the assessment. The general advice is that the assessment may be based on the outcome of the cost centre with the larger share of the fte.
-
Different submissions submitted by a staff member for different cost centres under the same panel
The panel should assess the submissions separately.
-
Same submission submitted by a staff member for cost centres under different panels
The assessment should be conducted by the relevant panels. Difference in standards is defensible as the relevance of the subject matter may be different to different panels.
-
Different submissions submitted by a staff member for cost centres under different panels
Assessment should be made separately by the panels concerned.
-
No assessment should be made if the fte in a particular cost centre is less than 0.2 in all the above cases.
To allow the Panel to form a view as to the extent of the cost centre's involvement in the work, the item submitted for assessment should have listed all the authors in Table 3 and should have those belonging to the same cost centre underlined. The Panel should use this as a basis to assess (apart from the quality of the item in question) whether the co-authors involved have made a significant and substantial intellectual contribution to that item.
So long as the Panel is satisfied that the cost centre as a whole has made a significant and substantial intellectual contribution, there should be no discount or pro-rating. It is emphasized that the proportion of co-authors from within the cost centre would not be used as a multiplier in rating the item. Thus there should be no discouragement for genuine collaborations with each party contributing to the research output.
In the case of multiple submissions (i.e. a co-authored item submitted more than once) from the same cost centre of an institution, the staff members submitting that item for assessment should have provided additional information by filling in the declaration form (at Appendix E1 of the Guidance Notes). To avoid double-counting within a cost centre, the coordinator of submissions from each cost centre should have drawn multiple submission(s) from within the same cost centre to the Panel's attention by completing the form (at Appendix E2 of the Guidance Notes).
The possible scenarios and recommended mode of treatment are as follows:
if an item has n co-authors all from the same cost centre of the same institution, and it is submitted only once, then it should be treated as if it was a single-authored item by the submitting co-author, no matter how large n is;
if however such an item is submitted more than once from the same cost centre of the same institution, then the Panel should exercise its judgement to see whether each submitting co-author can be said to have made a significant intellectual contribution; and
if an item is submitted with some or many co-authors from other institutions,then the Panel should also exercise its judgement as to whether the submitting author has made a significant intellectual contribution.
| Grading | Rating of Individual Items |
|---|---|
| A+ |
- Masterpiece; cannot be ignored by anyone working in the field -Single item over 3-4 years is already good performance |
| A |
- Highly innovative and significant - Probably noticed by anyone working in the broad field |
| B |
- Innovative and significant; makes a valuable contribution to the field - Meets attainable standard of excellence common in the mainstream of the field |
| C | - A useful contribution; but possibly short of the attainable standard of excellence common in the mainstream of the field |
| D | - Standard below C |
Panels are requested in their preliminary meetings to recommend such adaptations or amplifications (with examples if possible) of this schema as they feel appropriate. An effort will be made to ensure broad comparability between similar disciplines.
Individual scores
Score conversion table (based on the best three items)
| Score | 1 | 2/3 | 1/3 |
|---|---|---|---|
| Grading | A+ | ||
| AB | A | ||
| BBB | BB | BC |
Using the above as a reference, individual Panels should work out their own scoring method. While some modifications may be required for individual Panels, it is not expected there would be great divergences in the scoring schema. Panels should record clearly the modifications and the reasons.
Anonymity
Scoring form
Calibration of Standards
External advice
Workshop and preliminary meetings
Consideration of RAE Submissions
Publication of Panel Guidelines
UGC Secretariat
September 1999
Appendix A
Scholarship as defined by the Carnegie Foundation*
The Carnegie Foundation considers that there is a more inclusive view of what it means to be a scholar -- a recognition that knowledge is acquired through research, synthesis, practice, and teaching. While scholarship means engaging in original research, scholarship now has a broader and capacious meaning. Beyond the age-old "teaching Vs research" debate, there are four separate, yet overlapping functions: they are the scholarship of discovery; the scholarship of integration; the scholarship of application; and the scholarship of teaching.
Scholarship of Discovery
The scholarship of discovery, at its best, contributes not only to the stock of human knowledge but also to the intellectual climate of an institution. It is a scholarly investigation, closest to what is meant when academics speak of "research", that confronts the unknown and creates new knowledge. Not just the outcomes, but the process, and especially the passion, give meaning to the effort.
Scholarship of Integration
It is a serious, disciplined work that seeks to interpret, draw together and bring new insight to bear on original research. The scholarship is closely related to discovery. Such work is increasingly important as traditional disciplinary categories prove confining, forcing new topologies of knowledge. This scholarship also means interpretation, fitting one's own research -- or the research of others -- into larger intellectual patterns. A variety of scholarly trends -- interdisciplinary, interpretive, integrative, are examples of scholarship of integration.
Scholarship of Application
It is a dynamic process of creating new intellectual understandings arising out of theory and practice. The term itself may be misleading if it suggests that knowledge is first "discovered" and then "applied". The process is in fact more dynamic; new intellectual understanding can arise out of vital interaction between theory and practice, and one renews the other.
Scholarship of Teaching
It is a process that transforms and extends knowledge while transmitting an intelligible account of knowledge to the learners.
Appendix B
RAE Panels in RAE 1999
| Cost Centre | Panel Title | Panel No. | |
|---|---|---|---|
| 6 | biological sciences | Biology | 1 |
| 9 | other biological sciences | ||
| 10 | agriculture | ||
| 22 | biotechnology | ||
| 1 | clinical medicine | Health Sciences | 2 |
| 2 | clinical dentistry | ||
| 3 | clinical vet studies | ||
| 4 | nursing | ||
| 5 | other para-medical | ||
| 7 | pre-clinical studies | ||
| 11 | physics & astronomy | Physical Sciences | 3 |
| 12 | chemistry | ||
| 13 | materials science | ||
| 14 | earth sciences (incl. oceanography, meteorology) |
||
| 15 | other physical sciences | ||
| 32 | mathematics & statistics | ||
| 17 | electrical engineering | Electrical & Electronic Engineering | 4 |
| 18 | electronic engineering | ||
| 33 | comp studies/science (incl. IT) | Computer Science /Information Technology | 5 |
| 16 | mechanical engineering | Engineering | 6 |
| 19 | chemical engineering | ||
| 20 | production engineering (incl. manufacturing & industrial engineering) |
||
| 21 | marine engineering | ||
| 23 | materials technology | ||
| 24 | textile technology | ||
| 26 | other technologies (incl. nautical studies) |
||
| 25 | civil engineering | Built Environment | 7 |
| 27 | architecture | ||
| 28 | building technology | ||
| 29 | planning | ||
| 30 | surveying, land | ||
| 31 | surveying, other | ||
| 34 | law | Law | 8 |
| 35 | accountancy | Business Studies & Economics | 9 |
| 37 | business studies (incl. management) | ||
| 38 | catering | ||
| 39 | hotel management | ||
| 40 | economics | ||
| 8 | psychology | Social Sciences & Education | 10 |
| 36 | public administration | ||
| 41 | geography | ||
| 42 | social work | ||
| 43 | other social sciences | ||
| 49 | communication & media studies | ||
| 57 | education | ||
| 58 | physical education | ||
| 44 | Chinese language & literature | Humanities | 11 |
| 45 | English language & literature | ||
| 46 | Japanese language & literature | ||
| 47 | other languages | ||
| 48 | translation | ||
| 50 | history | ||
| 51 | other arts/humanities | ||
| 52 | art | Creative Arts, Performing Arts & Design | 12 |
| 53 | performing arts | ||
| 54 | music | ||
| 55 | other creative arts | ||
| 56 | design | ||
Appendix C
Reference Guidelines on
Handling Submissions from Staff Members with Split Cost Centres
Appendix D
Guidelines on Co-authored Submissions