Home > UGC Activities > Research > Research Assessment Exercise > Documents on Previous RAEs > Meeting of Panel Convenors of Research Assessment Exercise 1999 held at 10:00 am on Friday, 17 September 1999 at the Conference Room of the UGC Secretariat

Meeting of Panel Convenors of Research Assessment Exercise 1999 held at 10:00 am on Friday, 17 September 1999 at the Conference Room of the UGC Secretariat

Present :
Prof Kenneth Young, Convenor of the UGC's QSC sub-group overseeing the RAE 1999
Prof William Massy, Member of the QSC sub-group
Prof Po S Chung, Convenor, Electrical and Electronic Engineering Panel
Prof Michael M T Loy, Convenor, Physical Sciences Panel
Prof Francis Y L Chin, Convenor, Computer/Information Technology Panel
Prof Joseph Hun-wei Lee, Convenor, Built Environment Panel
Prof Richard Yan-ki Ho, Convenor, Business Studies & Economics Panel
Prof Siu-lun Wong, Convenor, Social Sciences & Education Panel
Prof Ambrose Y C King, Convenor, Humanities Panel
Prof Bell Yung, Convenor, Creative Arts, Performing Arts & Design Panel
Prof S K Lam, Deputy Convenor, Health Sciences Panel
Prof Brian Duggan, Deputy Convenor, Engineering Panel

Absent with Apologies:
Dr David Korn, Convenor, Health Sciences Panel
Prof James Feinerman, Convenor, Law Panel
Prof Jeffrey Tze-Fei Wong, Convenor, Biology Panel
Prof Albert H Y Chen, Deputy Convenor, Law Panel

In Attendance:
Dr Lee Shulman, Carnegie Foundation
Dr Mary Huber, Carnegie Foundation
Dr Gene Rice, Carnegie Foundation

Secretariat:
Mr Nigel J French, Secretary-General, UGC
Mrs Brenda Fung, Deputy Secretary-General (2), UGC
Mr P H Li, Assistant Secretary-General (R), UGC
Ms Janice Yam, Office Executive (R), UGC

Major Points Covered

  • The Background Leading to the Need for a Broader Concept of Research

    In Hong Kong, the funding of an institution is the sum of funding for its two principal activities, namely teaching ('T') and research ('R'). The 'T' element provides support for teaching and, while the UGC's recurrent funding assessment methodology could apply factors for quality to this element, this has not so far been done. Instead, this aspect is being dealt with through the UGC's quality audit process called Teaching and Learning Quality Process Review (TLQPR). On the other hand, the RAE results form the basis for allocating a significant part of the 'R' portion. In the past two exercises, the submitted items fell principally within the category of traditional research, in the present terminology the scholarship of discovery (even though the Guidance Notes had made it clear that there was no such restriction). There has been concern that other types of research, e.g. applied research, research in pedagogy, etc., had not been sufficiently taken care of in this regard. At the same time, in keeping with the Government's increasing emphasis on innovation and technology, there is also a need to give proper and equal recognition to applied work in the context of research assessment.

    It is against this background that the new definition of research was broadened in RAE 1999 to cover other forms of scholarly/creative outputs which are also intended to be assessed in the exercise. The major purpose is to institute a shift of focus towards a broader sense of scholarship, but the shift is still an experimental and gradual process.

  • The Carnegie Definition of Scholarship

    The use of the four Boyer categories of scholarship is meant to stimulate discussion of what should constitute scholarship. In doing so, the definition has sought to give a broader concept of scholarship, encompassing the scholarship of discovery, the scholarship of integration, the scholarship of application and the scholarship of teaching. The traditional meaning of'research' seems to be confined to the scholarship of discovery and represents only a subset of the whole concept of scholarship. The Carnegie Foundation advocates the need to recognize and reward all the four categories of scholarship.

    It should be noted that the new definition of scholarship does not seek to open the floodgate by treating anything as scholarship. When evaluating the scholarship in a certain piece of work, one would have to make a judgement on a basis of a set of standards. Before forming a view, one might have to ask the following questions:

    1. Is the work driven by a problem?
    2. Is there a clear goal to be achieved?
    3. Is the work a process of inquiry?
    4. Does the work clearly connect the findings to the goal and has it added anything to the field of study?

    For example, there can be scholarship in a piece of teaching material if it is the outcome of a vigorous process of inquiry and is novel in the relevant field of study.

    A distinction should also be made between categorization and quality assessment: an item may fit the above criteria and be classified as research under a particular category of scholarship, but its quality (in terms of novelty, impact, intellectual rigour) may be low - in which case it should be given little or even no credit in the RAE.

    In particular, the meeting came to a clear understanding of the difference between (a) scholarly teaching, which is exemplary delivery of teaching and the preparatory process of reading, analyzing and synthesizing that goes into it, and (b) the scholarship of teaching, which is a process of inquiry concerning a problem in the teaching context. In the first instance, the beneficiaries are one's immediate students, in the latter the beneficiaries should be the entire scholarly community in that discipline, through output that is accessible by peers. In the UGC context, the former belongs to 'T' and should not be included in the RAE; the latter should be included in the exercise, but given credit if and only if its quality meets the standard set. It was also emphasized that this distinction may not have been clear in the original Carnegie formulation in its 1990 report, but that the Foundation's thinking has since sharpened, so that its current views are now not substantially different from the principles and philosophies to be adopted in the RAE.

  • Operationalization of the Concept in the RAE 1999

    There is a difference between research assessment and a scholarship assessment that encompasses all the activities of an academic. In RAE 1999, panels should continue to put the focus primarily on those aspects of scholarship that can properly be called research, broadly defined - namely that it is related to a process of inquiry that leads to novel and generalizable outputs accessible by peers, (as was the intention in the previous two exercises), rather than to expand the scope to the extent of including those activities and outputs that may be valuable but cannot be reasonably associated with the 'R' part of the UGC's funding. This focus is implicit both in the name of the exercise and in the underlying principles embodied in the Guidance Notes. It also is consistent with the Carnegie Foundation's current view of the matter.

    Panel members should note that the purpose of adopting a broader definition for research to include the other three forms of scholarship in RAE 1999 is to sensitize the local academic community (and panel members) to different forms of research, and in that context also to generate useful statistics on the forms of scholarship for future reference. It is hoped that, with the adoption of the new concept, institutions and their academic staff would also begin looking more critically at their academic work in the categories of scholarship other than traditional research narrowly defined. While it cannot be ruled out that future exercises may eventually take the form of scholarship assessment, this issue should not be a concern of panel members in the current exercise.

  • Standard of Assessment

    There is concern that the other three forms of scholarship (apart from the scholarship of discovery) may be judged with very different yardsticks. In this connection, the meeting confirmed that in evaluating a research output item, it is imperative to adopt a common set of standards across all categories of scholarly work. The most important consideration is whether the work is the product of a rigorous process of inquiry. If the process started with a clear goal to solve a problem, and the output is novel and generalizable, the submitted item should be regarded as a piece of scholarly work. Also, the output should normally have gone through the process of review by the proper community. In this regard, the standards of scholarly work as proposed by Dr Mary Huber during her presentation in the RAE workshop on 15 September 1999 can serve as a reference point (see Annex). Generally speaking, panel members in RAE 1999 should adopt a consistent standard of intellectual rigour to review and evaluate items irrespective of the category of scholarship. In reviewing the research output items, panel members should adopt and stick to the quality threshold standard as stipulated in the Guidance Notes for RAE 1999, which has been widely made known to the academic staff in institutions.

    Panel members should also note that, as promulgated in the Guidance Notes, the definition of research output items for the purpose of review in the RAE has in fact required that all items submitted must meet all the following criteria:

    1. the output contains an element of innovation;
    2. the output and the process involved contribute to scholarship;
    3. the output is publicly accessible; and
    4. the output is of interest to peers and is generalizable.

    This definition should apply equally to items in all the four categories of scholarship.

  • Need for Proper Documentation

    It is the first time that the four categories of scholarship have been adopted for classification purpose in the RAE. In order that the UGC can monitor and have a meaningful analysis of the experience, panels would need to document the whole process properly and reflect fully any problems or concerns they have experienced in their panel reports. Such information may give essential clues to how the next exercise should be carried out.

  • Statistics on Scholarship Classification

    While it might be the case that the statistics on the spread of research items and the gradings they receive amongst the four categories of scholarship are misleading in some instances, it should be stressed that they are primarily for internal reference and analytical use by the UGC. They are not meant for public use in any naive and simplistic way. For avoidance of doubt, the items should be logged in the column pertaining to the category of scholarship as declared, even though the panel may find that declaration to be inappropriate. If panels have any doubts about the classification, their views should be properly fed back to the UGC in their panel reports. Where appropriate, the UGC would seek to clarify any possible misunderstandings about the statistics that emerge.

  • Preliminary Panel Meetings

    As it is not possible to re-arrange very quickly the plenary meeting with all the panel members (which had to be called off due to the inclement weather), panel convenors should consider the best means to relay the general understanding reached at the meeting to their panel members so as to ensure consistency in the scoring work. Depending on need, individual panel convenors may need to consult their panel members and organize their own preliminary panel meetings prior to the actual scoring work to achieve this purpose.

  • Assignment of Items

    To allow time for the preliminary panel meetings, it was agreed that the deadline for submission of the assignment list (originally 18 September 1999) would be extended to 5.00pm on Friday, 24 September 1999.

UGC Secretariat
September 1999

Annex

Assessing Scholarly Work