Home > UGC Activities > Research > Research Assessment Exercise > Documents on Previous RAEs > 1999 Research Assessment Exercise (RAE) Workshop (15.9.1999)

1999 Research Assessment Exercise (RAE) Workshop

Transcript of the recorded proceedings on 15 September 1999 of the RAE Workshop

Panel: Professor Kenneth Young, Convenor of the QSC sub-group overseeing the RAE Professor William Massy, Member of the UGC
Dr Lee S. Shulman, President, Carnegie Foundation
Dr Mary Taylor Huber, Senior Scholar, CarnegieFoundation
Dr Gene Rice, American Association of Higher Education
Mr Nigel French, Secretary-General, UGC Secretariat

Professor Young:I am pleased to welcome you all to this workshop on the Research Assessment Exercise. I am here on behalf of the UGC, as convenor of the subgroup responsible for the RAE. I should introduce another member of the subgroup, Prof Bill Massy. I am sure Bill is known to you all. He was the Chairman of the sub-group for the first RAE and has been closely involved with the process ever since. I guess you all know Nigel. I'll introduce the rest of the panel in just a minute. I wish to thank you all, not only for coming to these workshops, but more importantly for agreeing to contribute your time and your expertise for this exercise.

The shape of scholarship in the institutions of higher education in Hong Kong in the next few years will undoubtedly be influenced by how this RAE is run and how it is perceived. The purpose of these workshops is to ensure that we run this exercise in a consistent and fair way, setting high standards and pointing to the right directions.

These workshops will be spread over two days. Today's workshop will be mostly at the philosophical and conceptual level, providing the context and the theoretical underpinnings of some of the aspects of this exercise that we will embark upon, and tomorrow we shall deal with the more operational issues.

As you know, this is now the third RAE to be run by the UGC in Hong Kong. Going from the first to the second, the threshold standards were raised significantly. After the first two exercises there was a feeling that the RAE was steering academics largely into one kind of research, the kind of research that in our current terminology we would call the scholarship of discovery, and the kind of research that tends to be published in international referee journals.

Even though the guidance notes for the RAE have emphasised time and again to the contrary, the perception could not be entirely dispelled. That is why, in preparing for the current RAE, the UGC deliberately widened the definition of research in a very explicit way. In doing so, we made reference to the four categories of scholarship, first introduced in these precise terms by the Carnegie Foundation in its influential 1990 report, "Scholarship Reconsidered - Priorities of the Professoriate", and later elaborated upon in this 1997 report, "Scholarship Assessed - Evaluation of the Professoriate", assessment and evaluation being of course very close to what we are trying to do here. Copies of both publications have been made available to all RAE panel members. I don't know whether we will start with a pop-quiz as the evaluation of the professoriate here present.

It is a very great privilege and honour for us to have a team, a very senior team from the Carnegie Foundation, here with us today. The Carnegie Foundation for the Advancement of Teaching was established in 1906 by an Act of Congress. I was reminded that that was before any of our universities here in Hong Kong were established.

The Foundation is an independent policy and research centre whose vision, in the words of its charter, is:

    "To do and perform all things necessary to encourage, uphold and dignify the profession of teaching".

The Foundation's main activities of research and writing have resulted in published reports on every level of education. While the Foundation conducts most projects independently, its work occasionally includes partnerships with other institutions involved in educational research and reform, and we are very delighted that they have taken an interest in the higher education system here in Hong Kong. Before I hand the session over to the Carnegie Foundation representatives, let me give them a brief introduction.

Dr Lee Shulman is the President of the Foundation. His interest has focused on the improvement of teaching in K-12 and in university settings, on new approaches to the assessment of teaching and on the methods and quality of education research. Dr Shulman received his Ph.D. from the University of Chicago and he has taught at Michigan State University where he was Professor of Educational Psychology and Medical Education and Founding Co-Director of the Institute for Research on Teaching, and also at Stanford where he has been a Charles E. Ducomunn Professor of Education. Dr Shulman is a former president of the American Education Research Association (AERA) as well as Past President of the National Academy of Education. He has received AERA's career award for distinguished contributions to educational research and the American Psychological Association's E.L. Thorndyke Award for distinguished psychological contributions to education. Dr Shulman has also been a Guggenheim Fellow and a Fellow of the Centre of Advanced Study in the Behavioural Sciences.

Dr Mary Huber is a Senior Scholar at the Carnegie Foundation and has been there since 1985. She directs several projects on faculty scholarship including a study of cultures of teaching and learning in higher education and the study of scholarship of teaching in faculty careers. She also chairs the Advisory Committee for the Carnegie Classification of Institutions of Higher Education, directs the Professors of the Year Programme, and helps guide the Higher Education Programme of the Carnegie Academy for the Scholarship of Teaching and Learning. Dr Huber has written and edited several books on the academic profession and in particular is co-author of one of the two books I mentioned earlier, "Scholarship Assessed - Evaluation of the Professoriate".

Dr Gene Rice is from the American Association of Higher Education (AHHE) and was at Carnegie while the four scholarship model was developed. He has been a significant contributor to those ideas. In addition, in his role at AHHE, during recent years he has visited dozens of campuses that have been employing alternative models of scholarship in revising their systems of faculty review and education.

I now hand the session over to Dr Shulman.

Dr Shulman: Thank you very much, Professor Young. Thank you for adding the excitement of an impending typhoon to our otherwise pedestrian visit to Hong Kong. I hope it misses.

It's very exciting to come and join you in this effort, in large measure because many of the ideas that we're working with today are ideas that were developed initially at the conceptual and theoretical level and have been implemented at individual institutions around the world with a primary focus on providing a broader range of methods of assessing the quality of individual faculty work. But I think it's fair to say, as Professor Young has reminded us and I'm sure will again, that this is a pioneering effort to try to apply systematically these ideas to evaluating the quality of academic units and sub-units, and for us this becomes an important and very exciting test of these ideas.

Let me say that for 35 years until just two years ago, I sat exactly where most of you are sitting. I was a professor who spent my time doing teaching and research at universities. During one six-year period at Stanford, I chaired and served on a committee called the Advisory Board, which David Korn remembers quite well since we used to get into fights all the time, David, when you were Dean of the Medical School.

The Advisory Board's job was to review and either approve or disapprove every appointment and promotion in the university from the medical school to the philosophy department, from engineering to elementary education. I can tell you that among the frustrations of serving on that board was the sense that, when we reviewed faculty members for promotion for tenure, we were only looking at a particular slice of the lives of their minds and of their spirits, and only at a sub-set of the range of contributions they were making to Stanford as an institution, to their discipline, and to the community and the society. We simply had no way of acknowledging the broader range of contributions that people were making.

So I must say that two years ago, when given the opportunity to leave my teaching responsibilities and become President of the Carnegie Foundation, one of the attractions was the fact that this Foundation had been associated not only with the advancement of teaching in the 90 to 95 years of its existence, but more specifically under the leadership of my late predecessor, Ernest Boyer, with systematically attempting to broaden in a rigorous and intelligent way - not a frivolous way - the conception of what counts as scholarship in a college university setting.

During the more than 90 years of its existence - we're actually beginning to think already about our centennial five or six years from now - the foundation has concerned itself with issues of educational quality. For example, the first major project of the Foundation was something that those of you in medicine and the health sciences are probably familiar with, it was called the Flexner Report, a systematic study evaluating every single medical school in the United States and Canada against a set of standards. It was a devastating report because it named names and it was one of the major reasons - not the only one - why in the decade after the Flexner Report nearly half the medical schools in the United States closed.

I don't think that's going to happen here. Please don't see the Carnegie Foundation as the'grim reaper'. We are not playing that role. But it is a long term commitment to questions of quality. The Foundation created the first pension system for college and university faculty members for retirement so that they would not become indentured servants to the institutions that employed them. This has now evolved into the world's largest pension system, something called the Teachers' Insurance and Annuity Association (TIAA) which thankfully the Foundation gave away in 1918, though there are still something like nine widows whose pensions we continue to pay and may they live and be well.

The Foundation invented the Graduate Record Examination and helped create the Educational Testing Service. In many ways, its projects have focused on how do you assess quality, how do you maintain standards of quality, and when Dr Boyer, with the very strong collaboration I might add of Gene Rice, who will be speaking in a moment, proposed the four scholarships, this was quite consistent with the long term interests of the Foundation in combining high standards with a socially responsible conception of the responsibilities of institutions of higher education.

Let me say a word about the themes of today's afternoon workshop. First a word about scholarship: if there is a danger of having too narrow a view of scholarship, which Professor Young alluded to a moment ago, there's also a danger in having too broad a notion of scholarship such that you get to a point where anything anybody does gets called scholarship. We don't want to see that happen. We have a very simple conception of scholarship that we are using in our programmes at the Carnegie Foundation these days.

For us, any activity that calls itself scholarship must have at least three attributes. The first is that it must in some significant sense be public, it must be in the public domain, it must be what we've been calling'community property', that if what you are doing is absolutely wonderful and meets the highest standards but is somehow kept in the closet or kept hidden, it is not, by this definition, an active scholarship. An active scholarship is community property.

The second attribute of an active scholarship, in addition to being public, is that an active scholarship must be subject to peer review and critical evaluation by an appropriate intellectual or aesthetic or social community. It cannot simply be out there, it must be subject to peer review by the appropriate community of scholars.

The third attribute of scholarship is that it must be a kind of work that others can build upon.

Scholars are, by definition, I would argue, people who are attempting to do a task that no one individual can do alone. Scholars are members of intellectual communities where the members depend on one another to strengthen their work. Therefore, in that sense, scholarship must be generalisable in that others can build on one's work in order to advance their own, in order to advance the work of the field.

So if, for example, we take the contributions we make in public service or as great teachers, that work, as admirable and valuable as it may be in the doing itself, would not become scholarship until it became public, could be peer reviewed and others could build upon it in their own teaching, or their own service, or for their own theoretical purposes. Those are the definitions of scholarship that we have been working with at the Foundation and I think I find what we're discussing here in the next few days quite consistent with those notions of scholarship.

Well, what we're going to do today is explore historically, conceptually and with specific examples the varieties of scholarship that we are using in the reviews we're doing. Since most folks here are quite comfortable with what we've called the scholarship of discovery, the traditional conception of research, which is so highly valuable in higher education, we will be giving disproportionate attention to the forms of scholarship that are much less often discussed in these circles.

We're going to be doing this work and we're going to be alternating between the usual long periods of exposition and presentation which we have become used to, both as perpetrators and as victims of higher education, but what we will do several times during the afternoon is punctuate the lecturing by asking you to take a few moments to write down the ideas that you're wrestling with that are the topics of our discussion, and even to discuss them with your colleagues to your left or to your right.

We're doing this for a very simple reason. A lecture was once defined as a foolproof method of getting ideas from the notes of the lecturer to the notes of the audience without passing through the minds of either, and we would not like that to happen. We find that if you have a chance to jot down some ideas and discuss them with a colleague, it makes those ideas a little bit more problematic and a little bit more vivacious than if all you do is listen to them passively. So we will have alternating periods of lecture, and a little bit of reflection and writing and dialogue.

Let me begin then with the first presentation and it is a presentation by Gene Rice. As I said before, Dr Rice was at the Foundation actively collaborating with the team, working on the four scholarships during the period leading up to the publication of "Scholarship Reconsidered". Gene is, by his own training, a sociologist focusing for much of his work on the sociology of religion, which is why he has such great faith in higher education and its future. He has been an Academic Dean and is now serving as the Director of the program on faculty roles and rewards at the American Association for Higher Education. There's really no one better than Gene Rice to reconstruct for us the thinking that led up to the notion of the different kinds of scholarship and where those ideas have developed since then.

He will be followed by Dr Mary Huber. Dr Huber is a cultural anthropologist by training who has spent a number of years in Papua New Guinea studying aspects of culture in that society and has now shifted her attention from the exotic settings of Papua New Guinea to the even more exotic settings of universities and the native people who populate those universities called faculty and students.

So let me begin then by inviting to the microphone Dr Gene Rice to begin his review of the historical and conceptual basis for the scholarships. Gene?

Dr Rice: Thank you, Lee. I'm delighted to be here, although the weather is turning out to be much like it is at home. I don't know whether you've been watching the news but if you live along the East Coast we're expecting this major hurricane and I don't know what the difference is between a typhoon and a hurricane but I hope to learn.

I'm pleased to be here to talk with you about some of the same problems we're confronting in American universities. How do we get beyond that tired old teaching versus research debate? How do we put into place a multi-dimensional conception of excellence for faculty, while also enabling our universities to meet the changing educational needs of our students and provide the knowledge base required of a learning oriented society. As in the United States, this is what you are dealing with here in Hong Kong.

When I was at the Carnegie Foundation in the late 1980s working on "Scholarship Reconsidered", the central issue we were trying to address was the incongruity, the lack of a match, between faculty priorities, what faculty are recognised and rewarded for, and institutional mission. They'd come out of alignment and there was a call around the country for an effort to realign faculty priorities and institutional mission.

Like your institutions, American higher education has a wide range of sectors with faculty being called upon to perform different kinds of scholarly work and this diversity is seen as a major hallmark of both of our systems. We think of the diversity in higher education as a major strength, yet we have one, only one, rather narrow standard for measuring excellence.

What became clear in our deliberations was that faculty actually do what they're rewarded for. As faculty, we're smart people. Institutions were rewarding faculty work that could be documented and recognised as legitimate. The disciplinary research being rewarded was supported by a distinguished set of international associations, peer review journals, and a system for validating achievement. But there were many other scholarly tasks, especially teaching, that were not being recognised or sufficiently rewarded.

"Scholarship Reconsidered" was an effort to address this important professional and institutional problem. This whole effort to broaden our understanding of what counts as scholarly work was heuristic from the very beginning. It was intended to launch a discussion, a way of reframing the issue.

Ideas are still being generated. It's not something that was completed and then handed out as the best answer. That wasn't the intent at all. In fact, this year is the 10th anniversary of the publication of "Scholarship Reconsidered" and I direct the American Association for Higher Education's forum on faculty roles and rewards. The primary purpose there is to deal with the agendas generated by "Scholarship Reconsidered" and the changing role of faculty.

We actually are having an annual national conference that has grown every year in complexity and size. This February we're expecting about 1,200 people to gather in New Orleans. The theme of this year's conference is going to be "Scholarship Reconsidered" Reconsidered: Update and New Directions You're welcome to join us in New Orleans on February 3rd through 6th. In fact, we'll have people gathering there, teams sent from universities around the country, that are addressing the kinds of issues you're struggling with here. I know it's a long way. It's in the winter, however, and if you're running a conference in the United States to have one in New Orleans helps.

The places where this broader definition of scholarship has been the most useful are in the large land grant universities, places like Michigan State, Oregon State, Ohio State, where, in terms of mission, the application of knowledge has always been central. But when it comes to faculty scholarship, The application of knowledge is regarded as second rate or derivative.

Another group of institutions that are being particularly helped by this broader definition of scholarship are the comprehensive universities, both public and private; I think particularly of Indiana University, Purdue University in Indianapolis. It's especially in those large urban public universities where a multi-dimensional set of missions have been most apparent. Through the 90s it became increasingly evident that faculty work is going through a major transition and this is contributing to the re-examination of scholarship.

There are some major changes taking place. In the United States, one is a generational changing of the guard. Faculty my age who entered the profession in the 1960s are now reaching retirement age and are just beginning to retire in very large numbers We have this major bubble in higher education of faculty between 55 and 65. Now they're moving on to a different kind, a later life career - we don't talk about retirement - but they're moving on and new faculty are coming in. A very different kind of faculty is developing.

Technology is having an enormous impact. What we are learning about learning is critical. Changes in the function of knowledge in the workplace are making a difference. the knowledge boundaries are breaking down in the United States. The university doesn't have the corner on knowledge in the way in which it did in the past. Here are some of the transitions. [Overhead 1]

A basic change that's taking place is a movement from the focus on faculty, who we are and what we know, to a focus on learning. Think about the 1960s. In fact, there was a major book written by Jenks & Reisman in 1968 entitled "The Academic Revolution" and it documented the centrality of that focus on faculty. Now we're moving toward learning and we're being called upon to take seriously the learning needs of students and the demands of the larger community to commit ourselves to making that connection.

Another basic change is the movement from professional autonomy to institution building. I've enjoyed that professional autonomy most of my academic career but I see a change taking place. The way in which we work, we're moving from individualistic ways of working from a focus on "my work" to a call for greater collaboration and engagement, a focus on "our work".

There's a cultural shift there, a shift from a culture of unexamined assumptions to a new culture of evidence, and in the United States - and I think you're probably finding it here as well - this is having enormous influence. We're being held accountable and I think it's because the university has moved to central stage. We really are a knowledge-based society and so the university is becoming more central. Legislators and regents are looking over our shoulders. Accreditation bodies are holding us accountable. There's also a movement from - and we can talk about this later - career dependence to career resilience. We're interviewing junior faculty around the country, and they're saying what we need is the kind of career development that will give us resilient careers. They're not looking for the kind of careers that many of those in my age cohort have had. No Paragraph Then finally, there's the movement from universities as separate worlds to universities in which faculty engage in public life and take responsibility there.

I'm going to argue that beginning in the later 1950s and extending through the next decade and a half, a period often referred to as the golden age for higher education in the United States a powerful fiction, an image of what it meant to be an academic professional took hold. It was often reflected in institutional policy but most solidly ingrained in our own thinking of ourselves as professionals.

During those affluent days, that period of expansion, a consensus came together around what it meant to be an academic professional and here are the elements. They were imported from Germany, from Scotland from Oxford and Cambridge, but during that period of rapid expansion they came together and what you got was [Overhead 2] a focus on research, as the central professional endeavour. Quality was to be preserved through peer review and the maintenance of professional autonomy. Think of this. Now, this is in the 60s.

The pursuit of knowledge was best organised according to a discipline. It was organised for its own sake. Reputations were established in national and international forums, not locally. Professional rewards and mobility accrued to those who persistently accentuated their specialisations and then the distinctive task of the academic professional was to pursue - and this term was used carefully - cognitive truth. There was an epistemological assumption here. The image of the professional became normative for most of us. It dominated our thinking about academic work and specifically our thinking about what counted as scholarship. The problem was that most of us found ourselves in institutions that in fact had other priorities.

This professional vision, and the inter-related complex of assumptions on which it was based, contributed to major advances of knowledge. That was the period when research took off in a number of our fields. To this old set of assumptions, now, new demands are being made, new priorities have been added and emphasised. Now the old priorities are still there but, as higher education expands and changes, a new agenda has been put in place and in the interviews we are conducting with junior faculty we find that they see themselves as caught between these two worlds. [Overhead 3]

What you have on the left-hand of this overhead is that assumptive world of the academic professional. On the other side are priorities now that are getting very much emphasised, and I'm sure you're finding it in your environment just as we are in ours. We've got the research but also we've got to take teaching seriously. Professional service has to be honoured. Professional autonomy continues to be important but public accountability is being called for. We have peer review but also very sophisticated student evaluation and an ability to assess learning, a focus on the discipline but also the call that we cross knowledge domains. You can go on down that list:

  • Professional associations calling for us to attend to local needs.
  • Specialisation, but the call to integrate knowledge.
  • To build bridges between theory and practice.

So we've got faculty who are being called upon to meet all of these demands and in these interviews with junior faculty they're saying this is an academic career that's hard to sustain.

It is clear that what we need now is a more comprehensive vision of the scholarly work of faculty, one that will lead to a more complete and connected conception of what it means to be a scholar. This is what the Carnegie report, "Scholarship Reconsidered" was intended to be, a more inclusive way of thinking about faculty work, one that would be more appropriate for our institutions and for ourselves, more authentic and more adaptive for both the institution and the day-to-day working lives of faculty.

You're all familiar with the four forms of scholarly work [Overhead 4]. Let me just go through these briefly. I've arranged this in a Venn diagram because I do think it catches the inter-relatedness of these four forms of scholarly work with each containing a part of the other three. Also, each form of scholarly work can be assessed by similar criteria. Later, Dr. Huber is going to talk about this at some length in terms of what has been written in "Scholarship Assessed". But, as you will see,, we can use the same criteria across the four forms of scholarly work:

  • Is it fresh?
  • Is it innovative?
  • Can it be documented?
  • Can it be peer reviewed?
  • Will it have a significant impact?

The assumption is that the four forms of scholarly work are not only inter-related but are interdependent. When we look at the first form of scholarly work, the scholarship of discovery, this is the one with which most of us are familiar. I won't spend time talking about it, I've put this into a framework of learning - but it tends to fall into the more abstract, analytical, reflective quadrant, with physics and pure math, established patterns to be emulated when you think about this kind of scholarly work.

Communities of discourse have been established around specialised disciplines so you have the international associations, cosmopolitan conferences, the peer reviewed journals, and substantial, legitimate agreement about what counts as scholarly excellence.

What we are arguing is that every scholar needs to have an ongoing agenda of enquiry, an intellectual project grounded at least in one discipline. The scholarship of integration takes this beyond just the advancement of knowledge but calls on our developing the capacity to synthesise and integrate, to bring things together. This is probably the form of scholarly work that has been most neglected and needs to be encouraged. What is it that ties the pieces together, that gives coherence and meaning to the scholarly enterprise?

Faculty complain of disconnection in academic life. I think this is one of the reasons. Inter-disciplinary studies need to be attended to. We need to recognise and reward what is being done. Now to get involved in inter-disciplinary studies on lots of campuses is an enormous risk.

David Damrosch, Chair of the Comparative Literature Programme at Columbia University, has written a new book entitled "We Scholars" and in it he writes poignantly about the difficulty of supporting comparative literature programmes.

On a related topic, how can we encourage our best scholars to write textbooks. Is there a way to entice the leading scholars in our fields to write those basic texts?

One of the major complaints about academic scholarship is that our writing is not accessible, that it can't be understood beyond a narrow group of colleagues, that we've become, in fact, self-referential, that we don't or can't communicate with the larger public. Can we reward the scholars who manage to be public intellectuals in the best sense?

We're going to take the scholarship of teaching as an example of where you might go with this broader definition of scholarship. We've made the greatest progress in large part because of the leadership of Lee Shulman, and we're fortunate to have him with us here today, so we're going to spend the bulk of our time together looking at that.

Let me just say a word about the scholarship of application, what we're calling the scholarship of engagement.

One of the reasons I've begun to talk about the scholarship of engagement is that the term encompasses the application of knowledge as well as learning from practice, It's a two-way street. What you see here is a flowing between the discovering advancement in the application of knowledge and the learning from practice.

The work that is being done in this area is grounded in the fundamental challenge that people like Donald Schon have referred to as "institutional epistemology" and it has come to dominate scholarly work in American universities since the turn of the century. Its over-arching impact is that theory and research have precedence over application and that practice is regarded as secondary and derivative. Professional service and outreach can now be documented in ways that are persuasive.

I have with me here two books that have just come out, one by Ernest Lynton, "Making the Case for Professional Service" and one by Driscoll & Lynton "Making Outreach Visible, A Guide for Documenting Professional Service". Both books provide concrete examples of ways of documenting that kind of scholarship, the applied, the outreach, the engaged.

Again, it is the land grant universities and the large urban institutions that are moving forward with this kind of scholarship. At the University of California at San Diego, there's a major project on bridging the gap between academic knowledge and civic knowledge. Mary Walshok and Daniel Yankelovich are getting key research faculty to address serious regional problems, the border crossing issues between San Diego and Tijuana are ones that they're looking at now.

Let me end there. I think that takes you through the four forms of scholarly work and I hope that in our conversation we can come back to some of the misunderstandings that have been generated by "Scholarship Reconsidered" and maybe we can help you not make some of the mistakes that have been made elsewhere. Thank you. Mary?

Dr Huber: Hi, this is the first moment where we're going to take a break from our lecture mode and ask you to look, I think, in the papers you received when you registered. You'll find there a form that asks you two questions, and if you take three or four minutes to answer those two questions just with the thoughts you've had while listening to Dr Shulman and Dr Rice, that would be great. Then we'd ask you to turn to your neighbour, whoever you're sitting closest to, and talk to them about your answer. The two questions are:

    What was the most important thing you've learnt so far in this session?

and second and most important to us:

    What important questions remain unanswered?

We'll collect these sheets at the time of the break and we'll review the answers you've given and try to include them in the question and answer session later. So thanks. We'll take about 10 minutes for this whole exercise so if you'll write for three or four and then chat with your neighbour for about the same amount of time, that'll be fine.

[Pause in proceedings]

Dr Huber: In the years since its publication, "Scholarship Reconsidered" has joined and helped shape a lively debate about the work of professors in the United States, and to some extent abroad. As you've heard, the report appeared just as the balance between teaching and research in US colleges and universities became a subject of national debate.

By identifying the four scholarly functions: discovery, integration, application and teaching, "Scholarship Reconsidered" helped academics focus on how faculty roles and institutional missions could support each other rather than only on how they might conflict. This has led to a great deal of fruitful thinking about the nature of academic knowledge and the different ways in which it's produced. It has also challenged the academic community to acknowledge the full range of scholarly work.

What you're doing here in Hong Kong is a very important step in this direction. Broadening the kinds of research that can be submitted for the Research Assessment Exercise may help ensure that scholarly work in areas other than discovery can be appropriately recognised and supported.

Clearly, this task raises many important questions concerning the kinds of scholarship that the new definitions cover, the ways in which this work can be documented and the criteria that can be used to assess its quality. We're hoping that our experience in the United States will be of some help. Although we don't have a national research assessment exercise, we do have many occasions where faculty members' academic performance is evaluated and where colleagues have to ask and answer similar questions about what counts as scholarly work. In fact, you can find discussions of what counts as scholarly work in a wide range of academic settings in the US.

One of the most interesting is within the scholarly societies and professional associations to which most faculty members belong, the American Historical Association, for example, the American Chemical Society, or the Association of American Geographers. In the early 1990s representatives from these and several other societies met to consider what a larger idea of scholarly activity in their subject area might mean. Historians, mathematicians, geographers, sociologists, chemists, architects and professors in a variety of the visual and performing arts, business and journalism, proposed frameworks to encompass the full range of academic work that scholars in their field might do.

Reports from these groups show how conceptions of scholarship have widened beyond matters limited to the published results of basic disciplinary research. The mathematicians, for example, proposed to include research in core or applied areas that leads to new concepts and insights, discoveries and all of the rest. They included research that leads to the development of new mathematical techniques, research in teaching and learning that leads to new insights into how mathematical knowledge and skills are most effectively taught and learned at all levels.

They also included synthesis or integration of existing scholarship such as surveys, book reviews and lists of open problems; communicating mathematics to new audiences or to established audiences with improved clarity; the development of courses, curricula, or instructional materials for primary and secondary school as well as for the college level; and the development of software that provides new or improved tools for research in mathematics or its applications, for communicating mathematics, and for teaching and learning mathematics..

The historians likewise agreed that scholarship embraced more than traditional original research, documentary editions and translations. They also included integrated scholarship such as works of synthesis aimed at non-professionals. They counted as applied scholarship activities such as public history, including exhibits or tours in museums or other cultural institutions. They included professional service, editing journals and newsletters and community service that drew directly upon scholarship like public lectures.

Scholarly work in teaching, they said, might include research, writing and consulting in history education; the development of courses, curricula, visual materials, and teaching materials; and collaborating in content-based programs with primary and secondary schools.

Many professional fields like medicine and nursing have been intrigued by the broader notion of scholarship because it could place important activities like clinical work more firmly on the academic map. This includes a whole group of related fields in the arts. For example, the National Association of Schools of Art and Design list several activities under the title of creative work and research. They have creating art and design and studying art and design, but also advancing the pedagogy of art and design, and applying art and design and facilitating art and design activities. All of these they include as creative work and research.

Conversations about broadening the definition of scholarship have been particularly vigorous at higher education institutions where new issues relating to the changing priorities and rewards and responsibilities of the professors come up frequently.

To give you some idea of the activity that's been happening on campuses nation wide, I can tell you that when the Carnegie Foundation surveyed chief academic officers in 1994 - I call your attention only to the top bar there - they found that more than 80 per cent of the country's four-year colleges and universities had recently re-examined their systems of faculty roles and rewards or were planning to do so soon. Of these, again about 80 per cent reported that the definition of scholarship was being broadened to include the full range of activities in which their faculty were engaged. And again 80 per cent said that the definition of teaching was being broadened to include curriculum development, advising and instructional research. Applied scholarship was being clearly distinguished at that time from campus and community citizenship activity.

We like to keep in touch with what's happening on campuses and our conclusion is that in the past decade a few institutions have actually written "Scholarship Reconsidered's" four categories into their guidelines for promotion and tenure. But many others - many others - have modified existing categories to reflect a broader view. I want to give just two examples.

One is at Kent State University in Ohio. There the School of Nursing has been especially active and they negotiate a contract with each individual faculty member each year that states what per cent of their time should be spent in each of the five areas considered for tenure and promotion there:

  • teaching
  • discovery
  • integration
  • application, and
  • university citizenship.

This contract also determines how much weight will be put on each of the five areas during tenure and promotion deliberations.

At Oregon State University a different road was taken and they defined scholarship as any activity in discovery, integration, application, or teaching that involves new or creative work, that's communicated to others, and that's peer reviewed.

Their Vice President for Academic Affairs noted that most of the scholarship of teaching efforts that have been submitted under the new guidelines have involved curriculum development. As an example he described one web-based curriculum project that asked students and peers to evaluate the course through online surveys. "In this case," he said, "the peer responses had been submitted as evidence of peer review", but he also noted that this strategy was not completely acceptable to the particular committee involved.

Now, this last case illustrates that the move to broaden the kinds of work which are evaluated and rewarded in colleges and universities remains uncertain in its potential outcome. Successful cases for tenure and promotion have been made on the basis of a broader definition of scholarship. But they've not been easy to make because academics are often not confident in their ability to judge scholarly work that has not already gone through the traditional processes of peer review.

Likewise, one can note that even as scholarly associations propose more inclusive definitions of what might count as scholarly work, they rightly insist that assurances of quality must accompany changes. For example, the Joint Policy Board for Mathematics states: "The results of scholarly activities must be public and must be amenable to evaluation."

The Association of American Geographers emphasised that "teaching competence should be verified by vigorous peer review," and that geography programmes must "develop coherent systematic plans for evaluating and valuing outreach roles." The American Chemical Society concurred, adding that "the task force recognises that the mechanisms for gauging scholarship in areas outside of research are generally not firmly in place. We encourage the creative development of new approaches to measure scholarship in chemistry across a broad spectrum of activities."

Now, this concern about quality is evident in the efforts of many colleges and universities to develop new evaluation techniques. A 1991 task force on the reward system at the University of California, San Diego, for example, concluded that to do proper justice to teaching and service contributions within the reward system it is necessary to develop evaluation criteria and documentation methods to supplement those that are now in use.

The Carnegie Foundation's 1994 survey showed that over two-thirds of college and university academic officers reported that their institutions were developing new methods to evaluate teaching. About one-third of the country's colleges and universities also reported attempts to fashion new methods for evaluating applied scholarship.

There's a great deal of effort that continues to be directed towards finding better ways to make such work available for peer review. Integrated scholarship or synthesis is often directed toward a wider audience than discovery. Applied scholarship often emphasises interaction between scholars and people outside the academy, and the scholarship of teaching may be aimed primarily at students. When scholarly activity does not come out as a published article or a book directed at fellow specialists, what are one's colleagues to look at and what criteria will they use to judge the quality of what they see? Obviously, there's no single answer to this question because the products may vary so widely-- from a new course design to a text book, from a museum exhibit to an environmental impact statement.

For those of you who are interested in some of the best thinking in this area, excellent suggestions for documenting both applied scholarship and teaching can be found in a set of books published by the American Association for Higher Education. The two books on teaching that I'm going to illustrate (Making Teaching Community Property, by Pat Hutchings, and The Course Portfolio, edited by Pat Hutchings) look at a variety of ways in which to involve colleagues in peer review, focusing especially in the newest book on something called the'course portfolio' in which an instructor documents the unfolding of a course from the beginning to an end. This kind of portfolio actually has the potential of making classroom teaching available to peer review in a way that might even come under the guidelines of the research assessment exercise, if it were inquiry driven and had generalisable results.

The books on professional service by Ernest Lynton and by Amy Driscoll and Ernest Lynton are books that Gene has already mentioned in his presentation and they are wonderful books and will give many ideas for how to create portfolios on a service project or on applied scholarship, including sections which summarise the intellectual framework of a project, its purpose, its process, the presentation or delivery to the client, if it had a client, its outcomes, and reflections on its significance including unresolved issues for further work.

Well, exactly what one presents in a portfolio could depend on what criteria will be used to judge the quality of the work, and this indeed is a topic that is receiving renewed attention in the United States, spurred in part by the Carnegie Foundation's follow-up report to "Scholarship Reconsidered", entitled "Scholarship Assessed".

Your own guidelines state that work submitted for assessment in the Research Assessment Exercise, must contain an element of innovation, contribute to scholarship, be publicly accessible, of interest to peers and generalisable. Beyond this threshold, however, lie other criteria which can define more precisely whether work that meets those thresholds is of high quality throughout or not.

Traditionally, we've been told, architects judge design by its commodity, firmness and delight. When scholars judge works of scholarship the question is: what vocabulary of evaluation should they use? "Scholarship Assessed" says in effect that however variable their products the four kinds of scholarly activities must be held to the same standards of performance. In order to recognise discovery, integration, application and teaching as legitimate forms of scholarship, the academy must evaluate them by a set of standards that capture and acknowledge what they share as scholarly acts. Can we find some criteria of excellence that pertain, not just to research or discovery, but also to scholarship in teaching, integration and application?

This turns out to be a very hard question to answer, in part because there are so many candidate vocabularies around. If you look carefully at current evaluation practices, however, you can find the elements of a common frame already in use. We asked more than 600 universities to send us information about faculty evaluation in teaching, research, and service and we asked university presses, scholarly journals and granting agencies about the criteria that they ask reviewers to use in evaluating manuscripts and proposals for projects of all kinds.

As you can imagine these guidelines vary quite a bit among each other. Some are long, some are short, some are systematic and some are jumbled. Many include items tailored to highly specific needs. In the end, however, the most remarkable thing is not how much these sources differ from each other but how much they share.

Most of the vocabularies of evaluation we examined had in common a set of terms to capture the way in which scholarly professional work proceeds. Intellectual projects cannot be confined to any one product, moment or performance. Discovery is not just what one finds in the laboratory. Integration is not just insightful interpretation of others' findings. Application is not just clinical diagnosis of product design, and teaching, as Lee Shulman has argued, is not just the active interaction between the teacher and the student in the classroom.

In fact, we found that when people praise any work of scholarship they usually identify a whole staged sequence of activities, each of which embeds a very familiar virtue:

  • clear goals and adequate preparation
  • appropriate methods and significant results
  • effective preparation and reflective critique.
  • I think everyone will recognise them one by one but when they're taken together they provide a powerful conceptual framework to guide evaluation. Their very obviousness suggests their applicability to a broad range of intellectual projects while allowing the markers of what is clear, adequate, appropriate, significant, effective and reflective, to vary among different disciplinary communities and, to some extent, among projects of different types.

    Here I want to simply go over very quickly, because we're low on time, the kinds of questions one might ask to gauge the quality of a piece of work according to each of these criteria. For goals, the following questions apply: Does the scholar state the basic purposes of his or her work clearly? Does the scholar define objectives that are realistic and achievable? Does the scholar identify important questions in the field? All of these relate to the goals of projects of any kind.

    For methods one can ask if the scholar uses methods appropriate to the goals, if they apply methods effectively, and if they're ready to modify procedures in response to changing circumstances.

    In regard to significant results, you want to ask if the scholar actually achieved the goals he or she was aiming for; if the scholar's work added consequentially to the field; and if the scholar's work opened additional areas for further exploration.

    For effective presentation one would ask if the scholar uses a suitable style and effective organisation to present his or her work; if they use appropriate forums for communicating work to intended audiences; and if the scholar presents his or her message in all of these forms with clarity and integrity.

    Here I would like to just underline the fact that for the scholarship of teaching, the scholarship of application and the scholarship of integration, an important audience - a critical audience - if this is going to be evaluated as scholarship, is one's peers.

    For reflective critique you want to ask if the scholar has critically evaluated his or her own work; if they brought an appropriate breadth of evidence to their critique-for example, did they talk to other people, to their peers, to their students, to their clients; and did the scholar use evaluation to improve the quality of their future work.

    One could summarise the main argument of "Scholarship Assessed" with a simple grid which has the dimensions of quality down one side and the kinds of scholarship across the top. These dimensions probably allow enough flexibility to be applied judiciously to different types of projects from different disciplinary traditions while enabling one to keep in view the qualities that discovery, integration, application and teaching share as scholarly activities.

    Indeed, it's hard to imagine any scholarly work worthy of the name that didn't adhere to standards something like these. Clear goals and adequate preparation, appropriate methods and significant results, effective communication and reflective critique define important aspects of excellence for all scholarly work.

    I'm going to close with just a brief comment on the purpose of books like "Scholarship Reconsidered" and "Scholarship Assessed". They provide suggestions intended first and foremost to help move forward a discussion, and as I've tried to suggest, "Scholarship Reconsidered" has certainly had this effect.

    Several universities and groups of universities are writing reports right now on how they might best fill their service or outreach mission - Gene mentioned this in his comments - and they're taking into account in their conversation both the older and the newer Carnegie report. This doesn't mean that they're adopting all the ideas in them, but that they're finding them useful to think with and to think about.

    The discussions that you'll be having here about work that's been submitted to the research assessment exercise likewise have the potential to help strengthen a broader definition of scholarship in Hong Kong, and to give more precise dimensions to what you mean by quality when you're looking at the scholarly work.

    Many of our colleagues in the US will be very interested in your experience as will those in other countries with quality assessment processes that are similar to your own. It's certainly our hope that your experience will contribute to the current constructive debate about the role of the professoriate and from such discourse that common language will begin to emerge within the academy globally about the meaning of scholarship and how it can be authentically assessed. Thank you.

    We are a little late for our break but I think we'll take the break and if you would be so kind as to bring your answers to those questions along with you and we'll have someone collect them at the doors.

    Also, if you would keep in mind any questions that arose during my comments about "Scholarship Assessed" and about some of the recent developments in the US, keep those in mind because when we come back from the break we'll ask you to take just a couple of minutes to write those down.

    So thank you very much.


    Dr Huber: ...questions write out any -- again what was the most important thing you've learned in the last session and anything that's really important and remains unanswered. You don't have to limit it to my remarks but any new thoughts you've had about the whole first half. If you take just a couple of minutes to do that then hang on to them and we'll collect them at the end of the afternoon and we'll review them this evening and hopefully have a chance to respond to some of the most important things during the day tomorrow. Thank you.

    [Dr Huber's presentation may be viewed by clicking here.]

    [Pause in proceedings]

    Dr Shulman: Let me begin the next session then as you finish your writing. Just as a comment and a preface, the one thing we must always keep in mind is that whatever forms the scholarship takes, not only what kind of scholarship but what form it's presented in, I think Mary Huber's point here is again it isn't that everything is scholarship, it isn't that anything counts, but rather that rigorous standards of excellence must continue to be applied to the non-traditional forms of intellectual work just as to the traditional forms.

    Mary mentioned in passing that architects use three standards that they call: firmness, commodity and delight. I find those very, very useful.

    The firmness question is how well grounded is the argument, are the claims? If somebody challenges the argument or the claims or the description, how well is the evidence organised and presented to withstand criticisms so that if somebody says,'Why should I believe that?' there's something in the body of work that offers some response to that. That would rule out an awful lot of what loosely gets called scholarship right away.

    But we know that there is a lot of work in all our fields that meets the standard of firmness which is rigorous as a dead body a day after passing - stiff as a board - but it doesn't meet the standard of commodity. Commodity is what's its value? Who cares? What difference does it make? That's the question that we usually have in mind when we ask about research ,'Is there something really new here' or'Is this sort of pedestrian work?'

    Then there's that very difficult, but you know it in your field when you see it, standard of delight. Is there something surprising, is there something stimulating, is there something exciting about the work, over and above its rigour and its value; which we know often sets apart excellent work from great work.

    It doesn't matter whether you're talking about traditional forms of discovery oriented scholarship or accounts in society of some new ideas, or work that tries to examine teaching. Those kinds of standards have got to be applied and they've got to be applied relentlessly and without regret, if the standards of scholarship are to be maintained. So I don't think again, as we broaden the definitions of scholarship that we're calling for the relaxing of standards of excellence.

    At the same time, keep something else in mind though, and that is that with the newer forms of scholarship - newer in the sense that we have much less practice with them - they're probably going to be much more uneven for the first decade or so. There just aren't the same conventions.

    Just to give an example: when we review research we never really review research. We review an artificial representation of the research that we have developed some conventions for reporting and reviewing. So some one of you who is a lab researcher or bench researcher in the biological sciences might spend a year in the lab doing a set of experiments.

    Now, when I review your research that doesn't mean I go into the lab and have to spend a year watching you do your experiments. You can take that work and you can compress it and communicate it in a 10-page article. The reason you can is because over nearly a century we've developed codes, we've developed norms of communication in each of our disciplines so that the readers can fill in the blanks, can fill in what's not said, and in many ways can map back from the compressed version to what must have gone in the laboratory and critically review it and even build on it. But we also know that if we need more information we know where to find it. We can assume that it's back there with you.

    In a lot of these other fields we don't have those kinds of conventions yet, and it's going to take time to develop them, which is why perhaps in the next five years maybe most of the reports of the scholarships of integration - well, not integration but application and of teaching, will probably look a lot like conventional scholarship. They do right now from the artefacts that have been submitted for your panels. They tend to look like articles in research journals or monographs.

    But that could change. That could change. We could, just as the article, as we have become comfortable with it, is an invention that happens to fit nicely with a particular kind of intellectual work we have to be prepared for other inventions and some of them may take on a certain electronic reality. They may be represented in a variety of kinds of websites that are ways of reporting and representing data. Some of them may look more like exhibitions. I just don't know. But that's part of the challenge for the next generation of scholars to invent new kinds of representation.

    Well, what we promised to do in this last session is talk about the case of the scholarship of teaching as an example of one of the alternative forms of scholarship. I'd like to begin by telling you a few stories that I hope will begin to convey what the conditions for scholarship of teaching might look like in an academic unit and then what the actual representation of the work might look like.

    So first let me tell you about a meeting that I had in June in Shanghai. I was in Shanghai for part of a two-week visit to institutions in China with a group of specialists in mathematics and on the teaching and learning of mathematics. We spent one day in a high school, a secondary school in Shanghai, where every academic department also had its own organised teacher research group. I don't know if the secondary schools in Hong Kong also have the tradition of teacher research groups but it's a very well developed tradition in the People's Republic. We began by observing a one-hour class that was taught by a relatively new teacher in the department to a group of 11th grade high school mathematics students. It was an advanced algebra class.

    The first thing that was unusual to me as part of a foreign body viewing this group was that nearly every other member of the mathematics department was also observing this class. The scheduling was such that most of the other members of the math department were sitting in the back of the room with me and some of my colleagues in our group observing the mathematics class, and what we were given before the class was a design document that the teacher had written describing the rationale for the class, what he was planning to do, the kinds of examples he was going to use, and what he expected the students to understand at the end of the one-hour class.

    We all observed the class. We all took notes. One of the nice things about observing math classes is that if you know some math you don't have to know the Chinese. The blackboard tells the story of the mathematical argument. It's much more difficult for me to observe a class in the teaching of literature.

    That afternoon we all came together and the work of the teacher research group was the critical analysis of this hour of teaching. It was one of the loveliest hours of discussion I have ever been part of. First of all, it turned out that he had consulted with some of his colleagues about the design of his lesson ahead of time, and as the discussion proceeded there were some very deep questions raised. Why do you approach the teaching of this mathematical concept with these particular examples? Everyone knows, said one very senior teacher, that the first example you used is very confusing to students. Well, of course, if everyone had known that the teacher wouldn't have used it. But that was another matter.

    There were theoretical arguments about why certain kinds of examples were better than others and about why he used a visual example in one situation, where in fact a narrative example, a story, would have been more effective. There were questions about what did the students really understand, and since they had not yet assessed this understanding they were going to have to wait until the next week to have evidence from the students about what they understood or misunderstood about the topic.

    What was interesting was that it was taking teaching, which is usually in most of our lives, a private encounter between a teacher and a group of unsuspecting students, making it public, making it peer reviewed and his peers were reviewing both the design and the performance. But it was not yet in a form that could be exchanged with others outside that immediate high school, so that in a local sense you had scholarly work on teaching going on, which was strikingly effective, but I would argue that as wonderful as that was it would not be a form of scholarship until it had somehow been documented and could be exchanged in ways that people outside of that immediate local community could learn from, could critique and could build on themselves.

    Now, you may say I've got too narrow a conception of the scholarship of teaching. Perhaps. But I firmly believe that unless you get to that point of a kind of generalisability, beyond a very local context, that the claim of scholarship is a weak claim.

    On the other hand, the other aspects of the scholarship of teaching; its public nature and its critique, its peer review, were present in the daily life of that institution. If they wanted to move collectively or individually to more systematic writing and exchange more broadly, and many of the teachers in Chinese mathematics classroom already do some of that, they had all of the pre-requisites for doing it. That's one story, a story from Shanghai.

    My second story is very different. It's about my dentist back in California. Now, my teeth are pretty good but I go to the dentist regularly to keep them that way. My dentist is what I would call a scholar of dentistry. What do I mean by that? This is a dentist who fairly late in his career got very, very interested in experimenting with new laser techniques for the reconstruction of teeth.

    I have the great pleasure of being one of his research subjects and paying for it. I knew he was being very experimental. He told me about it. He asked my permission. But he was just doing this for himself. He was just very, very curious and he was early in doing very high grade photographs of before, during and after, in effect...

    Finally, he found the courage to describe this work to the monthly meeting of the local county dental society that meets at a local restaurant. It had never occurred to him before that dentistry was something you not only did but could present to a national meeting. But he did. And when he presented it the editor of a journal came up to him and said,'Would you be interested in writing this up for our journal?'

    Well, that was three years ago. Joh Karna, my dentist, now is an active scholar of dentistry systematically documenting, analysing, initially presenting orally, and eventually in writing, the results of his innovative work in dentistry, while still practising dentistry.

    I think the reason I like the story of my dentist is because his scholarship grows naturally out of the desire to do his clinical work very, very well. He has not stopped doing his clinical work but he has found a way to make the inventiveness of the work public, peer reviewed and work that other dentists can now build on. People now write him by Email from all over the country. He sends digital photographs of re-constructions that he does that aren't ready for publication yet. He's part of a community, a network, of other dentists who are exploring some of these problems.

    There is something about what Joh Karna does as a dentist that strikes me as analogous to some of the work that we can do in the scholarship of teaching, because not only is it consistent with the work he does as a dentist, it ends up enhancing the quality of his work and contributing more generally to the quality of work of others in his peer community. And he doesn't have to be a faculty member at a dental school to do it.

    I find this interesting because not all of us come from research universities. Many of us come from institutions that are what we call euphemistically in the United States'teaching intensive institutions' and we can treat the fact that we are in teaching intensive institutions as reasons why we don't have an active scholarly life or we can say we are enormously fortunate because we are in a laboratory that gives us almost daily opportunities to study what teaching and learning, what understanding and misunderstanding mean in our discipline, and it's all there for us to document, analyse, exchange and build upon.

    That's my second story. By the way, he does have serious problems with getting caps to stay on but he hasn't been studying that and so I go back every six to nine months and get him to glue it back on and I'm sure one day he'll get better at that too, but I'm prepared to make that concession.

    Third example, Tom Banchoff is Professor of Mathematics at Brown University. He recently spent a semester at Yale University as a distinguished visiting professor of pedagogy of mathematics because the Yale Mathematics Department has established a visiting chaired professorship for innovators in the teaching of mathematics. Banchoff is a very well known geometer. He has a life in the scholarship of discovery. He is a protoge and student of the great Chinese geometer and mathematician, Professor Cheung whom some of you in mathematics may know. He lives his life between Berkeley and Beijing, I think.

    Bantchoff, through his teaching of mathematics began to get progressively more experimental in using the Internet, the web, to support his math teaching. What he did was to develop a system where at the end of a lecture he would give out a set of very difficult problems - we all do that if we teach mathematics. The problem set at the end of the lecture is about as old as mathematics teaching itself - with the following difference: the students in the class have 24 hours to respond to those problems using electronic mail, using the Internet on his Website. And for the first 24 hours all the exchanges are between individual students and Banchoff or his teaching assistant. The individual feedback they get is often of the following kind:

    That's a very, very clever strategy you've used for this problem. When it becomes possible you should take a look at the way Ken Young is doing that problem. He takes a very different approach and between the two of you you're going to have an important insight.

    Something like that.

    At the end of 24 hours, everybody's work becomes available to everybody else. At that point the students are encouraged to start collaborating over the net as they use one another's proofs or analyses or representations, because as a geometer, very often what students come up with is very clever visual ways of representing a problem, and he gives special credit when students can build on each other's work. What he's done is to create in this course a mathematical community of scholars who learn from each other and build on each other's efforts.

    Well, he has been having astounding results with his course. Among the results is, first of all, he keeps a record of innovative proofs offered by students and he claims, though it's too early in his research to see if we can confirm the claim, that students in this class come up with unusual creative proofs by working together in this way at a much faster rate, much higher rate, than they do in conventional courses.

    He also finds that students will often, by working with each other, achieve understandings in the third week of class that they weren't supposed to achieve until the ninth week of class. That creates some interesting challenges for the teacher of mathematics when the students get ahead of the train. On the other hand, what greater pleasure could we have than having students begin to have insights that leapfrog across these divides.

    Now what Banchoff is trying to do - and Banchoff spent a couple of weeks with us this last summer as a Carnegie Fellow in a network of teacher scholars that we are trying to create an electronic connection among - is to systematically develop strategies to document, analyse, and communicate his experiments in the teaching and learning of mathematics.

    So here is someone who has spent all of his life on discovery research in geometry really, and now is increasingly moving from being a scholarly teacher of mathematics - I think he's always been a great teacher of mathematics but not engaged in the scholarship of teaching mathematics - to now beginning to contribute to the scholarship of teaching mathematics in ways that will move the field of mathematics teaching ahead.

    Interestingly, Banchoff is also now the President of the Mathematical Association of America and so has a position in the mathematics community that is going to permit him to move this kind of work in a significant way. Banchoff was part of the group that joined me in our visit to China in June.

    My last example - and this is the one that we're going to look at a little bit more closely - is the example of Professor Daniel Bernstein. Dan Bernstein is an experimental and social psychologist who has been at the University of Nebraska in the United States, a public land grant research university, for nearly 25 years. He is a superb scholar of psychology in the traditional sense of the scholarship of discovery. He has also done very good integrated scholarship. He's written some significant articles in which he reviews and pulls together different areas of psychology into new syntheses.

    But increasingly he has begun to ask a different kind of question and, as you will see on a videotape I'll show you, the question begins with something like this:'I've been teaching some courses for two decades and I have to confess that at the end of the course, when I look at my students' work, they really haven't been very good at learning what I teach them. Now, there is an easy resolution of this problem, one that many of us take, and that is,'Boy, the students just aren't as good as they used to be'. We see the analysis, the causes of this everywhere but in ourselves.

    But Dan began asking a different question. He began asking,'Is there something I'm doing as a teacher of psychology' - and he wins all kinds of awards, the students like his teaching, he gets high ratings - but he looks at their essays and says:

      'When I think of what it really means to understand what I'm teaching, most of them don't get it. What's going on? What's going on that I can make a difference in?'

    So Dan Bernstein began a series of studies which reflect the fact that he's a psychologist. If he were from a different discipline they probably would take on a different style, a different tone, so recognise this is a psychologist asking about his own teaching of psychology, but also then think about how this relates to the notion of a scholarship of teaching. And I should say again, Dan, like Banchoff is a Carnegie Scholar, he participates in our centre for advanced study for college and university teachers, and he is part of a growing network of scholars who are building on each others' work with regard to the study of teaching in a scholarly way.

    I'm going to show about 10 or 15 minutes of Dan Bernstein making a presentation to an inter-disciplinary group of psychologists, chemists, English professors, and business professors. And he's describing his research, where it's been and where he sees it going.

    I think it's a sort of a single living example of how a scholarship of teaching can be talked about and how it evolves. So if we can show that. The first person you hear talking and the only person you hear talking will be Dan Bernstein.

    [Videotape played]

    Dr Bernstein: I'm going to describe some work that evolved out of a teaching problem I had about three years ago. It'll also actually connect with my essay because what I'm going to go through very quickly is background as an example of what I call the formal scholarship of teaching. My actual project is much less formal, the part you're familiar with. So you'll see the range of things that I talked about in the essay but we'll do the formal stuff rather quickly simply as background. It's taken from work I've done with two graduate students who deserve an enormous amount of credit, Joyce Mackley and Jean Mason.

    The basic problem was, I teach a course called Conceptual Issues in Psychology. One of those issues has to do with measurement and reactivity of measurement; that is, when you measure something you change it. I wanted to relate that to the Heisenberg uncertainty principle. I also wanted to talk about quantum mechanics and the decisions that people make about in particular the famous Duroslid experiment and how that corresponds to decisions that psychologists make when they set up studies. Partially this is an inoculation against the people who mindlessly use that stuff to say'science is bunk' so psychology should return to some prior era.

    After eight or nine years of teaching this course, I was consistently discouraged with the results on the examinations. Students were simply not writing answers that were even approximating the kind of understanding I was hoping to promote. I had heard some presentations by a couple of people who were using computer based interactive technology and showing some good results, so I offered a challenge to a couple of students to try to take the material that was in that lecture and turn it into an interactive author-ware lesson and we would see whether it could do a better job than I was doing as a lecturer in getting students to that understanding.

    So there are readings for this part of the course. I have a well crafted and polished if, unfortunately, ineffective lecture. They spent an enormous amount of time producing what is fundamentally a home made interactive author-ware programme but it has some very nice features. It presents material visually, it presents material auditorially, it's got video clips, it has some mastery based questions in three separate units within the package. Students are allowed to jump out of answering questions to go back to where the material was in text or in video. So it was nicely planned if clearly home made.

    Then Joyce Mackley did a very nice Masters thesis doing three different studies comparing the effectiveness of that interactive programme with a videotape of my lecture with a live lecture and with different populations of students. So I'm going to quickly describe the data from her study as background which will help you understand why I decided to try the intervention I did with the web, that this was not just a crazy idea, there were some good reasons.

    Summary of the three studies: one of the interesting things was - and this is a product of good effective form of scholarship - reviewers of two studies said'you don't have enough control conditions' so we added as a control, reading the same material, just straight text. Here's 20 minutes, here's some pages of paper, read this and study it. We also had an irrelevant reading. Read something that has nothing to do with what you're going to be asked about. This is what you get when you enter into that community, additions of more conditions.

    I want to jump very quickly to a graph that summarises what we found. By the way, we analysed this in terms of Bloom's taxonomy, we had different formats of questions and different levels of understanding. But this is the key graph. This dimension here is basically achievement scores, 90s, 80s, 70, 60, 50s and below, and this is a distribution of students in each of the conditions who scored in that range. So the control condition read irrelevant material and the test apparently wasn't that hard because it looks like almost 50 per cent of them were in the 70 per cent range. But that's worth knowing. We want to know what our base line is. But a lot of people in the 60s and virtually nobody up here in the top level.

    My lecture - and this was a live lecture I gave in front of a borrowed introductory psychology class - and the teacher had told them that this stuff would be on a test though it was all done ethically with informed consent and optional conditions and everything you could possibly want. The lecture did slightly better at this end but it really showed the improvement in the 80 range and the 90 range, over irrelevant material.

    The reading group did a lot better in the 80 per cent range but was not as good as the lecture at the top end, but certainly diminished quite a bit, those people at the bottom, certainly from this group. Then the computer condition had the very fewest at the bottom end and really did quite well, got almost a third of the people up to the 90 per cent achievement range.

    So this was a really nice piece of work. It suggested that something about these questions that were used in the mastery part of this programme were in fact pretty simple multiple choice questions, though the assessments were essay. So we had some reason to think that even if you're just being given experience with pretty routine material in the computer part, it does generalise to some high order questions. If you look at the evaluative kind of questions they did quite well, and the computer did better than the lecture on those, interestingly enough.

    Jean Mason replicated this using commercially available products - statistics are fine but replication is the most important thing you can do - and we used conceptually the same idea but different materials and found the results generalised quite well with the possible exception, as you look at the total here, that readings did even a little better than computer. They were both certainly better than lecture. We've broken that down by different types of questions and by levels of understanding, application and evaluative.

    ...(indistinct) to read material scored significantly better on evaluative questions regardless of question type than students who listened to a lecture. If you ever have the luxury of knowing that students will spend half an hour reading and studying, it works quite well it turns out. Strangest thing.

    This gets us to where I come in. We're also very interested in efficiency. Donna, you were talking about the problem of time. One of the issues is if this were truly a more efficient way to teach it might solve some of those things. We can't change people's available time but if we can be more efficient and Joyce's dissertation actually is studying efficiency. She's very much interested in the ratio of time to learning.

    I was interested in - and here's back to the stuff you have read - a course I've been teaching in learning for a number of years. It has always used essay exams. NOTE AA I've never used a multiple choice test in 25 years of teaching. I was interested in offloading some of the time I spend going over readings in class. It's kind of tedious, it's the way I have developed to make sure students did the reading, but I wanted to have that happen somewhere else.

    So I got the idea that I could use a web-based system, which students are required to go to the web and answer questions about the readings before they come to class. The questions would look like the ones that Joyce and I had developed for that psychology and physics lecture. They would be multiple choice, computer gradeable with immediate feedback. They're not profound questions, but they provide the student with an occasion to do the reading, respond to the reading and find out if they're getting some part of what they were supposed to get out of the assignment. That's the part that came from my previous work.

    A second part got added to the project as a result of being here last summer. Talking with you all, especially interacting with Tom Hatch, I came up with a way of changing the assessment in these essay exams. I made them what I think of as harder because they became problem based exams instead of abstract exams. Don't give me a statement of a principle, show me you can use it in a particular case. Show me you have a generalised understanding that is not just what you read about but is something else.

    Then the third piece of this is I have always used repeatable testing in a variety of ways and with limited resources now and teaching - this is a senior level course with about 30 students - I can only give one alternate version of the exam. But I threw that in with the mix just to say,'Well, that's an old non-technological fix. What does that get us and are these other things getting us more or less?' sort of as a comparison.

    I should mention that I'm doing all this deeply embedded in conventional teaching. This is a course for largely full-time students. It's a course with readings, it is not a web-based course in any fundamental way. I do a little Email back and forth. So these are things -- how do I get students in a conventional text-based course to do better at that? Whether that would generalise to an entirely new vision of education would be a really great question and answer.

    The key to this is recognising something about the change in the questions. The form of question, for example, was to identify the reinforcing conditions in a study I did with some colleagues in a fast food restaurant to explain how we identified what those things were and why they ought to work. So that's telling me something in the reading.

    The new question looks like this: I give an entirely new context. This one happens to be a business context. It's different from the one they read about and I ask them key points:

    • Based on your understanding of the restaurant study, how would you improve the quality of employees' work by using access to activities as a motivator?

    • Describe the cost and benefits in the evaluation there.

    • Recommend whether or not you would implement it.

    That's the question that replaced,'Tell me what Bernstein did in the restaurant study.' I went through the entire course and revised all the questions in this way.

    Here's another example: the old question, behavioural context in which punishment is added must be considered. It's basically if you're punishing behaviour it's usually because it has some good consequences. It's being maintained by the world any way. If you're going to punish you've got to compete with the naturally good consequences, why people think about that. Drugs and sex are fun that's why kids do it. If you want to punish it you've got to find some heavily competing alternative.

    Instead of asking them to give me that abstract description I give them a specific case of binge drinking on campus and how would you build a programme to cut down on binge drinking, if you understand what we talked about when we talked about the context for punishment? The font gets smaller and smaller as the questions get longer and longer.

    Here's one: it's a great old study from the Soviet Union on classical conditioning and generalisation...

    [Playing of videotape stopped]

    Dr Shulman: I know you all want to see how it all comes out. If you do I'll give you the URL for the website at the Carnegie Foundation where Dan's full set of studies is in the process of being posted so that this community has a way of communicating with one another.

    But I simply want to call attention to some of the features of the scholarship of teaching as Dan Bernstein is pursuing it. The first feature is that it is what I would call instrumental and a lot of the scholarship of teaching is instrumental. It's asking the question:

    • Does something work?
    • Does something work better than something else?
    • Does the new thing I'm doing work better than the thing I've already been doing?

    But it's asking a question of comparative effectiveness. A lot of the scholarship of teaching will have this quality. But notice something: it begins with the presentation of doubt where we as teachers rarely present it, and that is:

      Do I really have any basis for claiming that my students truly understand what it is I teach so well?

    And so you have a whole set of instrumental kinds of studies; studies of the effectiveness of teaching.

    But notice that he's also got some process questions he's begun to ask, and these get tied up with:

      What are the students actually doing that either contributes to their learning or contributes to their failure to learn?

    And you saw there's a whole set of analyses he's doing in which he's essentially examining the hypothesis that maybe one of the simple reasons why students have difficulty is they simply don't seriously read the material before they come to class, ostensibly to discuss it.

    So he begins to do some small little interventions and it turns out that the most powerful one he did was this web-based intervention in a course that was otherwise totally conventional, where before they came to class the students had to answer on the web a set of fairly low level questions about the readings but in order to answer them you have to read the readings.

    He reports on part of the tape that we didn't see, that the quality of discussion in that class was so dramatically richer and more textured than in any previous class he had taught. It suddenly struck him the students before weren't more stupid than the students I had this year, most of them hadn't read the material when they came to class and they were faking it. And our students do sometimes fake it. Of course, we never know that if we don't give them a chance to participate, but that's another problem.

    But here again some of this may seem like fairly pedestrian work but as you watch this begin to accumulate into a programme of research on the teaching and understanding of psychology, it gets very exciting and I have to tell you the other faculty members sitting there were not only from psychology, but also from business, from Chemistry, and from English.. They're taking notes like mad because it's stimulating ideas about their own teaching that they already hadn't considered before.

    A lot of his work deals with assessment. NOTE AA see page 39 One of the things he's about to tell you - at the point where I interrupted the tape - is that when he shifted from using abstract questions to problem-based questions, the rate of student failure in the course increased dramatically, even though the teaching was actually getting better, which led him to conclude that he had been deluding himself using the abstract questions earlier about what the students really understood.

    Now he's exploring both the question of assessment and why it's important to begin giving students practice with the same kinds of assessment that he's going to use at the end of the course. Again, you can see the ways in which he's experimenting with the uses of technology in the teaching of the course in assessment and also in posting his results so that others can build on the results.

    Well, let me conclude this discussion of the scholarship of teaching and the cases I've given you by offering another kind of visual aid. I apologise that we have had to develop a physical artefact because I could not figure out how to do this on the web, or even with Powerpoint.

    Here what we're trying to illustrate is that, for example, if you're doing the scholarship of teaching, the form of scholarship you do, the purpose of your work can be the improvement of teaching itself, which is the sort of thing that the Chinese teachers were doing, essentially investigating their own teaching to improve their own teaching.

    But you could do the scholarship of teaching for the purpose of discovery, for the purpose of discovering new generalisable facts and principles about teaching and learning. I think Dan Bernstein is going to end up with a set of important discovery scholarship of teaching, new principles about what counts as good examples. What are the characteristics of good examples? What are the relationships between forms of assessment and forms of learning?

    The scholarship of teaching can clearly involve integration. I can see Dan Bernstein and his colleagues now looking across a whole body of work on generalisation from examples, and the difference between theoretical examples and applied examples, and doing some very important integrated essays and reviews that are also part of the scholarship of teaching.

    You can see with this wheel that this is ??a heuristic to show that these are not rigid categories but that these kinds of scholarship interact, which also means that let's say we take the discovery scholarship of teaching that Bernstein was doing. Well, how do you evaluate whether it's any good?

    Well, you can take the same kinds of criteria that Mary Huber was talking about an hour ago. You can ask: are the goals of his study clearly defined and is the design nicely done? Adequate preparation? Has he appropriately reviewed the literature? Has he considered a number of different alternatives? Are the methods clear? He talked about adding more control conditions because he wasn't accounting for enough alternative hypotheses. Are the results really significant, or are they yawn, who cares? Et cetera.

    So the purpose of these wheels, in effect, is to try to illustrate that these are not rigid categories. These are categories that flow into one another but also that standards of excellence can in some systematic way be applied to nearly any combination of types of scholarship of teaching.

    Well, let me then more or less sum up our contributions from this afternoon. What we've intended to do this afternoon is to offer, both in general and in particular, a set of discussions that are connected to what Dr Boyer called a more capacious conception of scholarship, a way of acknowledging the intellectual work of our departments and of our colleagues that will include a wider range of their efforts without loosening or compromising or corrupting notions of innovation, of novelty, of rigour, of value, and of delight, that are ultimately the standards for scholarship.

    And, with respect to the special case of the scholarship of teaching, I've tried to show how it is that a scholarship of teaching can be pursued in a rigorous way that grows out of the problems that we actually confront naturally when we start teaching reflectively and start teaching mindfully, and start asking some of the questions we don't traditionally ask about whether we really have evidence that our students understand, whether the innovations we've introduced really make a difference, and whether we've given enough thought to what really counts as understanding in our respective disciplines and professions.

    I don't know how well we have achieved those goals this afternoon. We'll perhaps get much more sense of that tomorrow when you move from these more theoretical conversations into actually operationalising the approaches in the review of research.

    But at this point I would like to hand things over to Professor Young. I'm just wondering now, we're running a little late, do we have time for some more questions and discussion, or do you want to wait until tomorrow. As Moses said to the Pharoah, when the exodus is well underway this is not a time for conversation.

    Professor Young: Thank you very much, Lee, and as Dr Shulman said we can have a more extended discussion and Q and A. We will try to squeeze some time in tomorrow, hopefully both in the plenary session and perhaps more extensively in your individual panel meetings.

    In closing, I want to thank Dr Shulman, Dr Huber, Dr Rice for leading this workshop. I believe we now have a better appreciation of the conceptual context upon which we shall embark upon this RAE and also a better understanding of how others have grappled with very much the same issues.

    But, before we all disperse and go home and mull over what we have learnt today in preparation for the operational workshop tomorrow, perhaps I should add a few remarks and I think this is important in view of some of the points that were brought to me at tea break concerning some differences between the Carnegie conceptualisation and the RAE.

    First, the Carnegie definition: at least one way of saying it - although I am aware that Dr Shulman has narrowed some of the definitions today - is that scholarship is about everything that professors do, what their priorities are, and how one assesses them.

    For the Hong Kong UGC a system has been devised such that the funding to each institution is at least notionally assessed on the basis of two elements: one portion T for teaching; one portion R for research; and the latter, although it has been given a wider definition, is the only element that we are here to assess through the RAE.

    Because of this division, the delivery of teaching is deliberately not part of this RAE, although it is part of the scholarship of teaching as defined by Carnegie, at least in one of its formulations.

    Therefore, in this RAE it is only those aspects of the scholarship of teaching, and indeed also of the other categories of scholarship, that are novel, generalisable and evidenced by outputs assessable by peers that will be recognised in this exercise; and of course, having met those criteria, also the further criteria of quality as are determined by each subject panel.

    Now, this does not mean in any way that other elements of exemplary teaching would not be recognised, but rather that it should be in the Hong Kong UGC scheme of things be recognised out of the T portion. How we can go about doing that better and more explicitly is another story for another occasion, but for the same Bill Massy.

    Another difference is that the Carnegie reports are more about how universities assess their individual professors, but in our RAE the individuals are only proxies for their cost centres, their departments, and we should not lose sight of the fact that we are really assessing cost centres of 10, 20, perhaps even 50 members of staff, and the level of precision and depth that we can aim for and that we can achieve would of course be quite different.

    With that I think we should have occasion tomorrow to go over some of these in greater detail. I hope you will bring the guidelines for panel members. We meet again at 9.30 tomorrow and will go until 3.30 with lunch provided. But it is important that, in going through the details, we do not lose sight of the philosophical underpinnings of how we conceptualise scholarship and the priorities of the professoriate. For this, let us thank the Carnegie team once more for their excellent contribution today.

    Thank you all. Any housekeeping announcements from the secretariat? Thank you then.

    UGC Secretariat
    October 1999

    [The programme originally scheduled for 16 September 1999 was cancelled due to inclement weather. Instead, a meeting of the panel convenors was convened on 17 September 1999 to wrap up the discussion on the subject. Readers of this document are advised to read also the notes of the meeting on 17 September 1999 for reference.]