"Am I too loud?"
Presentation by
Nigel J French Former Secretary-General UGC
at CUHK Research Grant Workshop 2000
10 August 2000
I had actually chosen the title for my presentation before I knew that Prof Chan Wing Wah, Chairman of CUHK's Department of Music, was to be another panellist. As I am sure he and probably others among you will recognize, it is taken from the title of the autobiography of Gerald Moore, possibly the greatest piano accompanist ever. He is particularly famous for accompanying Dietrich Fischer-Dieskau singing Schubert lieder.
The point of this piece of plagiarism is that I have all along regarded the role of the Research Grants Council Secretariat staff in RGC panel meetings as like that of an accompanist, supporting and enhancing the work of the star performers (the panel members and the panel chairs) without overshadowing them or influencing their artistic (that is to say, academic) judgement.
As you have heard, I was the first Secretary to the Research Grants Council when it was established in 1991. I was involved in drawing up the RGC's operational guidelines and practices, and have had the privilege and pleasure of working with all three RGC Chairmen and every panel chairman since then.
Particularly during the early years, when the volume of applications was not so great, I took part in all the panel meetings personally. Now that is no longer possible for the RGC Secretary because the sheer weight of numbers of applications has forced the panels to meet in parallel sessions. Brenda Fung, who took over the position of RGC Secretary from me in March 1997, now has to share the responsibility of "accompanying" panel meetings with other colleagues in the Secretariat. We nevertheless continue to provide administrative support for the workings of the panels.
From the very beginning, we supported the panel chairmen in stressing that the panels should focus on the quality of the proposals in scientific/scholarly terms first and foremost, before considering the budget or other administrative matters (eg a PI's failure to declare an association with a nominated external reviewer).
My view was and is that the Secretariat's role is to facilitate discussion, to help with maintaining comparability of standards and normalising the review process between panels as far as possible and appropriate, and to advise on administrative matters (esp. budget matters) in respect of individual applications - and, of course, to "keep score".
In the early days, I personally took a more active role in discussions mainly to help the panel chairmen to distil a decision from often very disparate views expressed by panel members, but very soon the panel chairmen became better at doing that on their own. I have, however, provided similar help for new panel chairmen in their first couple of meetings.
I am well aware, incidentally, that I overstepped these preset bounds on occasions, out of enthusiasm and interest (and sometimes sheer frustration!), but I was usually picked up for this by either the panel chairman or a member.
All along I, and my Secretariat colleagues, have tried to bear in mind Gerald Moore's question and attempted not be 'too loud' during panel meetings.
So what of the process itself? I am aware of and can, at least vicariously (I am not an academic and have never been one) appreciate the feelings of frustration and even anger that such a process can engender.
How is it possible that this group of so-called experts cannot appreciate the enormous value and academic merit of your latest research proposal?
Why does it take such an inordinately long time for your application to be processed?
How does the RGC expect you to undertake your vitally important research work with a budget one-quarter of the size you applied for and is evidently necessary?
I am sure we shall hear many more such questions later, and I will be happy to respond to them during the discussion period, in so far as I can. In any event I do not have the time to cover all possible aspects of the process now. I shall therefore leave a number of these issues open and confine my further remarks to some aspects of the process that are most often misunderstood.
These are :
- the selection of panel members;
- the selection of reviewers;
- the handling of grades; and
- budget justifications.
Selection of panel members
The selection of panel members is primarily the responsibility of the panel chair concerned in consultation with the RGC Chairman and colleagues in the relevant fields on the RGC or the panel already. Panel members are chosen for their subject expertise, but more so for their experience, impartiality and integrity. They are not, and should not be, representatives of their institutions, but are appointed in a personal capacity. Obviously, however, senior colleagues in the institutions are asked for advice from time to time about the standing of individuals being considered for appointment. The job of panel member is also, as any of those among you here who have served on a panel will attest, far from being a sinecure, so we need to find people who are prepared to put in the time.
From my own experience over the past ten years I can say that, while not all panel members have lived up to these high standards all the time - they are, after all, human like the rest of us! - they have all tried their darnedest to do so and have worked very hard. I have been very impressed by the dedication and commitment shown by all the panel members I have known.
Selection of reviewers
For each proposal, usually two panel members are selected by the panel chair as principal reader and co-reader, to be responsible for overseeing the review process. Since the principal reader is asked to choose external experts to whom the application is to be sent for review, he/she should ideally be knowledgable in the field concerned, but this is not always possible. Under such circumstances, it is both more difficult and more important to ensure that the right external reviewers are found.
The RGC maintains a database of experts in virtually the whole range of fields and disciplines covered by applications. This database has grown over the years, but now comprises more than 8,500 names, more than 85% of whom are outside Hong Kong in over 60 different countries. In addition, as you know, principal investigators are invited to nominate up to five external reviewers for their applications.
The guidance notes for panel members state: "At least 2 independent assessments should be obtained for each proposal, overseas or local. Where a project is of local relevance, it is desirable that one referee should have some local knowledge. External assessors should be chosen with care, to ensure that they provide truly independent assessment. They should be experts in the field. Care must be taken to avoid choosing local assessors who are themselves competitors for funds in a similar field."
As a matter of practice, the panels have also adopted as a guideline that as far as possible at least one of the external reviews should be obtained from an expert not on the PI's list.
In practice, the majority of proposals receive at least three external reviews.
Handling of grades
It will come as no surprise to you that the grades received from external reviewers are often not consistent with each other. As a first step, the principal reader must attempt to reconcile any discrepancies. For example, is reviewer #1 known to be a hard marker or a generous marker? Are reviewer #2's comments more or less favourable than his/her overall grading would imply? Does reviewer #3 point to a weakness in the proposal that other reviewers have apparently missed? And so on...
However, even with the input from the co-reader, some disparate gradings cannot be reconciled before the panel meeting. This, then, is one of the major roles of the panel meetings. The principal reader and co-reader present their findings and the panel tries to come to a view.
I stress that this process is wholly focussed on the quality and academic merit of the proposed research protocol. It does not at this stage take into account cost.
Once an overall grade is agreed by the panel, then and only then is the budget considered. For very highly graded projects, the RGC's policy is to support all justified expenditure in full - I stress "justified" and will revert to this later. At slightly lower grades, the level of funding support will be reduced somewhat, regardless of whether the expenditure is adequately justified. This may involve reducing the period of support from three to two years (or even less in some circumstances), or simply an arbitrary reduction in the level of support for equipment or consumables. However, in every case, the panels try to ensure that the level of funding finally agreed is sufficient (within the parameters of our dual funding system for research) for the project to proceed.
Budget justifications
It has been a constant source of amazement to me over the past ten years how many colleagues will obviously devote serious amounts of time to explaining and justifying the academic part of their grant applications, but then apparently not be prepared to spend even a relatively small amount of time and effort to explaining and justifying why they need large sums of public money to pursue their research.
I think the message on this score may be getting through at last, but to quote some examples from the budget justifications* section (paragraph 9 (a) of ERG 1) of this year's grant applications:
"Three years are planned for the project since there is a substantial amount of theory, implementation and experimentation work involved. Two Research Associates will participate in the proposed research. The first RA will be responsible for theoretical study while the second RA will be responsible for implementation." - cost $1,500K
"Two research assistants are requested to aid software development, knowledge acquisition and especially data collection." - cost $1,600K
"(The two items of equipment required for the project) are unavailable at the university, therefore we request the purchase of these items of equipment." - cost $1,400K
"An RA for 2 1/2 years is essential for assisting research in this project." - cost $400K
"The PI will supervise two research assistants on the progress of the research. Since this project involves theoretical and system implementation, we need at least two research assistants." - cost $800K
and, finally
"The post-doc will be responsible for the supervision, device design, fabrication and characterization study." - cost $760K - one wonders what the role of the PI might be?????
I stress that I did not have to look far to find these examples and they are are drawn from all subject areas. I could have drawn similar examples several times over from all of the last ten years* applications.
Even where the research proposal is otherwise of the highest standard, the panels have almost always agreed to reduce the funding provided for projects in cases like these. To put it bluntly - if you are not prepared to explain to the satistfaction of your peers why you need so many research support staff, so much equipment and so much other funding, then do not be surprised if you do not get it.
Finally, on a more positive note, I should like to reiterate what I said at the recent RGC dinner on 22 June, when I said my final farewells to Council and panel members in anticipation of my leaving the UGC Secretariat later this month.
I have greatly enjoyed working with colleagues in the academic community in Hong Kong and elsewhere, as well as with those in other organisations and institutions here and overseas, who are committed to the cause of advancing learning and improving the human condition through research. The academic networks have been an invaluable source of support and advice during my time with the UGC and the RGC, and I should like to thank you all most warmly for that. I hope that, wherever the next few years take me, I shall be able to maintain all of these friendships and collegial links, many of which stretch right round the world.