The Management of University Research in the Intellectual Age by Kenneth E. Porter
Abstract The present time, a time of education, ideas and innovation, is referred to here, not as the Information Age, but as the Intellectual Age. Information is of little value for teaching or for invention until it has been transformed by ideas into understanding. The present university research culture is one of getting and spending. What is required is a radical change to an objective quality led process for spending research support, directed to those individuals who do the quality work, combined with the development of young researchers and incentives for universities to transfer new research thinking into entrepreneurial private companies.The paper describes a method of assessing researchers by an Attainment Correlation based on citations to the work of the professoriate. (The paper is part of a response to a request from the UK Government Department of Education to suggest ‘metrics’ methods of evaluating university research to replace peer review. It was submitted in October 2006.)
Contents
1.0 Introduction 2.0 The Intellectual Age 3.0 Back to basics 4.0 Identifying which people do high impact research 5.0 A metrics method for replacing the Research Councils 6.0 A metrics method for rating departments 7.0 Research rating of Universities 8.0 What is suggested is an improved version of traditional methods 9.0 Advantages of proposed methods 10.0 Concluding remarks Appendix 1 Some metrics for evaluating peer review. Appendix 2 A subjective peer review of peer review References
1.0 Introduction 1.1…In the U.K. the proposed 2008 Research Assessment Exercise [RAE], is in danger of collapsing beneath the weight of its bureaucracy. The Government now wishes to replace an elaborate system of subjective ‘peer review’, with a simpler system based on objective measurements of research activity, [D.E.S. (2006)]. It has suggested five alternative methods; all based on a university department’s research income, derived from data in financial records, It has invited proposals for other ‘metrics’ to replace or complement one or other of these alternatives. Many academics are arguing against such changes in funding practices which will lead to changes in ways of choosing which research to support and which to reject and inevitably change the behaviour of academics and the management of universities. The UK government spends a large amount of public money on academic research and it is disturbing that there has been no debate about what it is for and how it benefits the taxpayer. In the absence of market forces, any organisation inevitably becomes producer led, and both the existing funding allocation by peer review, and the proposed change to evaluation by income provide incentives to academics to spend more and more money to produce more and more papers irrespective of quality.
1.2…Some things which are obvious to many academics are being ignored both by those proposing a new metrics approach and by those resisting it.
For example, not all papers are equal. Ask a career researcher “How many of the papers published in your field do you read twice?” The answer is likely to be “Very few. I don’t have time to read them all, but some half dozen I read again and again.” This implies that a lot of academic research is unnecessary.
Not all researchers are equal. Say to a research team leader “Here is £10,000 to invite someone from overseas, an expert in your field, to come and meet your group. Who will you invite?” He/she will often give you an answer there and then, “Dr A” he might reply. “Is there nobody else?” you ask. “Well maybe Prof. B or even C, but A is the man for us.” He/she is unlikely to say “Anybody from the department of x at the University of California, Berkeley.” Individuals do high impact research. Every idea starts in one head. Some people are much better at it than others are. This implies that evaluating departments may not be the most efficient way of allocating support for research.
There is also a fundamental problem with the present system of using ‘peer review’ committees to share out research funds. This is that all the members of the committees have an interest in the result. Ask the research team leader if these committees are important and he will reply that they are of such critical importance that the survival of a department may well depend on their decisions. Ask if the decisions are always fair and he may well cautiously suggest that maybe the system is not robust enough to resist the pressures placed upon it. Ask what his university is doing about it and he will reply that they are doing everything they can to get their people on to the committees.
1.3…The best new system must surely be one which seeks to improve the quality of academic research and the way universities interact with activities in the wider community. Identifying such a system requires first defining what is meant by quality, why academic research should be supported and how research results are used by bodies outside the universities. These are the subjects of this paper.
The description below of universities in the modern world leads to the conclusion that it will be better to change from assessing the research of departments to assessing the research of individual researchers, so as to decide which of them are worthy of continuing support. Inevitably this means choosing which of them not to support. The research team leader mentioned above, in choosing to invite Dr A to visit his research team, bases his choice on the significance of the work Dr A has published in the past, work which has had time to influence other researchers in Dr A’s field. He makes his choice of guest in a similar way to the way he makes other choices, which writer’s books to read, which author’s plays to attend, which artist to choose to paint a portrait, which surgeon to choose for an operation. For all of these choices it is wise to select an individual on the basis of his/her previous record.
1.4…It is proposed that choosing which researchers are worthy of continuing support be made on the basis of citations to the researchers published work, a ‘metrics’ method to complement or replace metrics based on research income. A method of classifying researchers by comparison with the citations of the professoriate is described. The research rating of a department will depend both on how many [if any] frequently cited researchers work there, and on how many inexperienced academics work there, attempting to master the art of creating high impact research. This makes the assessment by metrics fall into line with how the real academic world works. Departments and universities gain respect in the wider world from the reputations of their members.
1.5…As part of the assessment, it is proposed to evaluate the dissemination of research results by counting the number of postdoctoral researchers who take jobs in entrepreneurial companies associated with the department.
2.0 The Intellectual Age 2.1…Nowadays, in the developed world, people work more productively; travel more and own more things. It is a time of change, driven by ideas. Ideas are the intellectual tools used to design new medicines, mobile phones and motor cars. According to science reviews in The Economist, we may soon be getting new medicines from DNA research, desktop super computers containing better computer chips developed for computer games machines, bigger aeroplanes built from new materials, and hand held x-rays (for on the spot diagnosis), a spin off from cold fusion research.…. Many of these inventions are to be derived from academic research which provided the ideas, the intellectual tools to be used in creating them. Governments now support academic research in the hope that it will lead to the creation of new business. All over the World, new Universities have been established and old universities have increased in size. During the last fifty years student numbers have increased some ten times and more and more people work with their brains rather than with their hands. Most academic study is now vocational in that students expect to recover the costs of their higher education from their increased earnings in graduate jobs.
2.2…The present time, a time of education, ideas and innovation, is referred to here, not as the Information Age, but as the Intellectual Age. We teach, not by rote but by concept. Information is of little value for teaching or for invention until it has been transformed by ideas into understanding. A new system for distributing support for university research would at its best, not only concentrate support on those who produce the influential ideas but also help drive the process of innovation. But just exactly what keeps the innovation process going?
2.3…A very recent paper by Bhide’ [2006] (Venturesome Consumption…) describes with many examples how discoveries in science are translated into new products for venturesome customers by entrepreneurs. (The customers must be venturesome to buy things which do not simply replace things they already had but which create opportunities which never before existed, Obvious examples are lap top computers, mobile phones, the internet and so on). It seems this process depends on at least four things: academic research, a free market, educated customers able to consume new products and a new type of entrepreneur, scientifically trained and thus able to see opportunities for translating the results of academic research into new opportunities. Bhide’ notes that nothing much happens if one or more is missing. Thus, for example, in the West, innovation was relatively slow in the Industrial Age because there was a free market but not enough research and education. It was slow in Soviet Russia which had a lot of research and education but no free market.
2.4…In discussing entrepreneurs, he refers to previous work on high tech firms, examining for instance how their internal R&D efforts help firms to use research produced in university labs. Bhide argues that too much importance is placed on supporting academic research by government intervention and more attention should be given to the downstream process. In his words, “the labor market may not have gotten it monumentally wrong and interventions that increase the supply of trained personnel for upstream innovators may impair productivity growth by reducing the availability of the personnel in downstream firms”. This paper also repeats Carter and Williams’s (1964) caution, of more than four decades ago, that “it is easy to impede growth by excessive research, by having too high a percentage of scientific manpower engaged in adding to the stock of knowledge and too small a percentage engaged in using it. This is the position in Britain today.”
2.5…An improved system of allocating Government support for University research should then not only find ways of concentrating support on the most effective researchers but also attempt to increase the number of scientifically up to date people working in entrepreneurial organizations outside the university.
3.0 Back to basics. 3.1…For more than thirty years research funding in the UK has been distributed by peer review committees. Peer Review of research is discussed objectively in Appendix 1. ‘Some metrics for evaluating peer review’ and subjectively in Appendix 2 ‘A peer review of peer review’. From both appendices it may be concluded that the assessment of departments and the allocation of research grants by peer review has gravely distorted the behaviour of academics. It has created in the universities an input-insider culture where older academics strive to get on the peer review committees and younger ones are rated by how much money they bring in. That is their future depends on how much money they spend. Not all publications are equal; a lot of money is wasted generating papers that nobody reads twice. Not all researchers are equal, by awarding grants on the basis of evaluating departments; money which would be best spent by an excellent researcher will be wasted on the research of others who are competent but uninspired. This compares badly with the situation in universities some forty or fifty years ago when there was no need for academics to compete for support and almost all research was done, not by expensive post doctoral fellows but by supervised PhD students working in departments with well founded laboratories. In general the supervisor had time to try and develop theory and would take pride in designing simple elegant experiments, which cost less than complicated ones.
3.2…It is then timely to address again the following questions: a. What is the relationship between teaching and research? Does academic research enhance teaching? Does teaching contribute to academic research? b. What is the difference between academic and practical research and how does academic research contribute to practical research? c. How much University research do we need? Should all academics be required to do it? If not, who should do it and how should we support them? Is there a more cost effective way of distributing financial support that will improve the quality of the output? d. Are there ways in which the Universities, still steeped in the academic traditions of a mediaeval agricultural society, may be made more responsive to the changing society of today? e. How may the objectives c and d above, to improve the effectiveness of distributing money for research and. the responsiveness of the universities to society, be accomplished by a simple metrics system? The system must identify those who will not receive financial support, [just as in selecting school leavers for places at a university, being selected is defined by not being rejected]. f. Are there ways, metrics ways, of evaluating the distribution of support by the peer review of the Research Councils and the Research Assessment committees?
3.3…Teaching and Research. It is difficult to use teaching to argue that all academics at all universities should spend time on research. Undergraduate students need to master the present state of the art before they are able to appreciate a new interpretation arising from recent research. Most undergraduate teaching, (probably more than 90%), is of well-established principles and procedures. Lectures on new interpretations are likely to take place only in final year, if at all. In any case, new interpretations are rare and most teachers do not do that sort of work. Preparing to teach a new interpretation made by someone else is part of the scholarship that supports all university teaching. External examiners, (appointed by a University to monitor the quality of its degree programs), are required to comment on examination papers and to interview students but not to report on whether the department research has contributed to the course. All this implies that academics can teach successfully by scholarship alone. That they teach provides no justification for supporting their research, nor, if they are reluctant to spend their time on research, should they be pressurised into it by the ‘getting and spending’ peer review committees. Nevertheless, in important ways teaching and academic research are complementary. The understanding created by the best academic research is used in teaching, and curiosity about things that are known but not understood may generate creative academic research, ‘If you can’t teach it, research it.’ This sort of academic research is an extension of scholarship. The search for understanding, which starts in the library and ends in the laboratory or as theoretical equations, derives from the curiosity of the teacher. It depends on him/her having the freedom to choose what he will study, what he will research and how he will teach it. Not all research at universities is the type of academic research described above. Most contributes information rather than explanation and some is done, not to increase understanding, but because it is believed by the researcher that the work may have a practical value.
3.4…The development of a subject over time. To distinguish between these different sorts of research, consider the development of a subject over time. For teaching purposes, information must be organised by ideas. Researchers testing ideas generate more information, and sometimes more ideas. New ideas that are widely accepted by other academics become part of the course. Some of them may become intellectual tools for creating new technology. This is the process shown in Fig.1. The rectangles are the ideas taught as part of a lecture course. The circles are the unexplained information that a lecturer may perhaps mention in passing. As time goes by, understanding increases until the conceptual base becomes too extensive for one undergraduate course and the subject fragments into specialities. In this model, academic research is of two forms, that which produces information and that which produces understanding. Papers reporting information are seldom read twice, while those presenting a new explanation may be read many times. This description explains how teaching may contribute to academic research, and how some academic research may eventually contribute to teaching.
The development of a subject over time.
3.5. Practical
Research and Academic Research
Practical research and academic research are
similar in that they use the same intellectual tools. They are different in
their purpose. The purpose of academic research is to increase understanding of
the subject. The researcher chooses his project because he is curious to find
out what is happening. The purpose of practical research is to make a profit
for the company supporting the work. The researcher chooses his project by what
he knows of the market. For practical research, competence is filing a patent
and excellence is making a fortune. For academic research, competence is
publication in an academically respectable journal and excellence is providing
a new explanation that is widely accepted by other teachers and researchers.
For both academic and practical research, the
choice of project often distinguishes between competence and excellence. It is
then difficult for a university teacher to choose an excellent practical
project because he usually lacks a sufficiently detailed knowledge of the
market and the costs of production. [Footnote]. However, he may do excellent,
curiosity driven academic research in a practical field, perhaps on a problem
suggested to him by practical researchers outside his university. His ideas may
then form new intellectual tools for those involved in market driven practical
research.
*Footnote.
A recent investigation of university units set up to commercialise research
found that many of them were not covering their costs.
3.6. Intellectual
Toolmakers A university
may strive to achieve a situation where all students receive some lectures from
an intellectual toolmaker, i.e. an academic who has produced ideas accepted as
a valuable contribution to his field of study. These teachers may tell the students
why they chose the topic, how they had the idea, what they did and why they did
it.
This is important so that students realise
that new ideas originate in human heads and are not found only in books or
carved on tablets of stone. Some of these teachers will be much older than
their students. The more ambitious student, with the arrogance of youth, may
believe that if that old fool of a teacher can produce such an important idea,
then so can I, and be prepared to spend a few years finding out if this is so.
Lectures from an established intellectual toolmaker may thus stimulate future
tool making.
However, these intellectual toolmakers are
relatively few, there are not enough to go round and many departments do not
have one. Most teaching is by academics who may be good scholars, but as
researchers, they are undistinguished
4.0. Identifying which people do high impact
research 4.1 Which brings us back to the questions of how
much academic research do we need, who should do it and how should we support
it? We need a way of identifying those few academics who are best at creating
understanding, those who write the papers that are read again and again, those
that receive unsolicited invitations to lecture overseas. The most direct way
of doing this is by citations to their published work.
A measure of
the impact made by a particular paper is the number of times other works refer
to it, that is, the number of times it is cited. Scholars soon realise that the
author of one interesting, (and well cited) paper is likely to have written
other significant works which may be relevant to their study.
4.2 A simple comparison of researchers by
citations to their publications may be criticised, because:
a] the rate of citation varies between
subjects and is different in different countries, [See the work of Gilli (2004)],
and
b] because it
provides insufficient guidance about how many citations represent recognition
by other workers for influential work.
The method
proposed here is based on a comparison of Professors as shown in Fig.2. This
provides the basis for a metrics method for choosing who should receive support
for their research work (to replace the peer review of proposals by Research
Council committees) and for evaluating university departments, (a metrics
method to replace the RAE). Fig.2 is an
Attainment Correlation, which shows how many citations represent a significant
impact. Fig 2 was created some twenty years ago as part of an evaluation of the
first UK
research assessment of chemical engineering departments. This is used in Appendix 1 below to
illustrate some of the methods proposed for evaluating peer review.
Today, Attainment Correlations may be constructed using the SCOPUS citation and
abstract data base, (SCOPUS).
Fig. 2. An Attainment Correlation. Professors are put in Rank Order of Citations in Reputable Journals to their published work.
4.3…Here the correlation of Fig 2 serves several purposes: First, it provides an anonymous standardisation of research achievement for a particular subject in a particular country at a particular time. That is, the research standing of academics in the same discipline that were not part of the professorial sample may be evaluated by where the number of citations to their work puts them on the curve of Fig 2. It provides a practical measure of what may be expected .
Second, it illustrates how research ability varies greatly, even between experienced researchers.
Third, it provides an opportunity to investigate the creative process and the differences between high impact researchers and no impact researchers. This may be important; not only for universities but also for the prosperity of the society that supports them.
Finally, it opens up new thinking about how to govern universities and about how to develop an objective cost effective, quality driven way of distributing financial support for research. It provides a basis for a metrics method answer to the questions of what research to support and who should do it?
4.4…The several advantages of Fig 2 are:
a. The research attainment curve may be generated from just two inputs, a list of researchers and a list of journals. The SCOPUS data base provides a way of doing this. At present it requires a tedious step by step procedure but it should be possible to replace this by a computer program which would present both the processed data and the correlation.
b. The advantages of basing the attainment curve on the professoriate in the subject under review are: i]. All Professors will have published many papers. It is rare to appoint anyone as Professor who does not have a long list of publications. The number of citations in fig. 2 is a measure of the quality of published works, not the quantity. ii]. All professors get research funding for their work. It is part of the job. In this example, the professors who received few or no citations had obtained about as much research money as those who received many citations, (See Fig.6 in Appendix 1 below). That means that the number of citations is not measuring spending on research.
c. The advantage of restricting plots of Fig 2 to one academic discipline, in this case Chemical Engineering, is that: i] It avoids any confusion resulting from the observation that the average rate of publication and of citation varies between disciplines. ii] Several management processes such as recruitment, research funding, promotion etc. are specific to a particular discipline.
d. A comparison of citations in particular reputable journals provides one way of defining a discipline.
4.5…Fig 2 illustrates that research ability varies greatly. In this sample, there are 40 professors from about 15 departments of chemical engineering. For ease of reference, they are split into five groups or Quintiles. Those in the most frequently cited group, Quintile 1, were cited, on average, about 50 times while those in the middle group, Quintile 3, were cited, on average, about 5 times. Those in Quintiles 4 and 5 are cited rarely if ever. The differences are such as to result in the straight line semi logarithmic correlation of Fig 2a. The Quintile 1 professors were those who produced papers read many times and who received unsolicited invitations to visit research groups overseas.[Note that it is mathematically impossible to include those professors who received no citations on this semi logarithmic graph.]
Fig 2a. The Attainment curve of fig 2 using semi logarithmic coordinates.
4.6…Self Evaluation by Academics
To
illustrate the value of having an anonymous standard of research
achievement in a particular subject, consider the case of the ambitious
young academic, wishing to find out if he/she has got what it takes to
be successful in academic research. After say, five years of choosing
his own research projects and publishing his results, the academic may
then evaluate himself/herself against the correlation of professorial
impact of fig 2. His chances of success (that is, of achieving Quintile
1 recognition) are small, probably no more than 1 in 20.
His
situation is similar to that of someone anxious to become a concert
pianist, who evaluates his chance of success by how many people turn up
to his concerts. He/she may then decide whether to go on doing what he
is doing or, if he is ambitious, seek some other occupation. Creative
ability in academic research is as rare as creative ability in other
fields, such as writing, painting, dancing or music. Many young
academics will soon realise that they lack the peculiar ability to
shine at academic research and will find something else to do,
sometimes outside the University. Why plod on producing papers that
nobody reads when there are so many other things to do in this exciting
world! They will wish to avoid the fate of Emeritus Professor
Serebryakov in ‘Uncle Vanya’, [Chekhov, (1897)], “Here he is in
retirement, and now one can see the sum total of his life: not a single
page of his labours will survive him, he’s completely unknown, he’s
nothing! A soap bubble!... “
Finding posts outside the
university should not be difficult even for Q5 people. Creativity is
important only in a relatively few situations. Many problems require no
more than professional competence. Nobody wants a creative accountant.
Consultants give advice based on best practice and their previous
experience, so do expert witnesses. Lawyers interpret a new situation
in terms of existing law. This also is the case for scientific and
technical innovation, Henry Ford is believed to have said “Invention is
1% innovation and 99% perspiration” and in support of this, a research
manager once commented, “one inventor goes a long way”. Even in Q1
research, after having identified a new question, much of the work of
finding the answer is applying existing ideas in a competent way and a
Q2+ researcher acting as an assistant to his Q1 colleague might help to
do this. Thus for the majority of University teachers rated Q2+, there
are many alternative occupations.
Indeed, one of the best ways
to integrate universities into modern society would be to encourage all
academics to have careers in both parts. Just as those outside the
university now expect to work in several different companies before
they retire, so should academics spend part of their career in jobs
within their discipline but outside the university. This would tend to
increase awareness by entrepreneurial companies, of recent academic
research.
4.7…Research into Creativity
One test
of the likely significance of a new idea is whether it stimulates new
thinking about things not directly connected to the original problem.
Perhaps the most exciting opportunity offered by Fig 2 is that of
exploring the nature of the creative process. The very great difference
in impact between the Quintile One [Q1] researchers and the others
suggests that they are doing something different. It would be
interesting to find out if there are any similarities between different
Q1 researchers in their research strategy, in particular:
How they choose research topics.
How they do the research.
Why they do what they do.
For
example, send each researcher a list of his/her three most frequently
cited papers, and ask how he chose the problem and how he decided on
the research programme. Ask him to comment also on his own three
favourite papers. [Note that asking a Q1 academic to describe how, in a
multi author paper, they did the work, would help in identifying which
one made the creative step]. Then look for similarities and differences
between researchers in their ways of working, comparing Q1 researchers
with other academics.
The correlation of Fig.2 applied to
other subjects, other countries etc. would provide data on large groups
of researchers and an opportunity to investigate differences between
populations of a statistically significant size. This research might be
more useful than an investigation of a few eminent individuals, e.g. by
reading the biographies of people such as Francis Crick or Albert
Einstein. It would provide information on two of the questions above,
how high impact researchers choose their projects and how they do their
research.
4.8…What sort of people do high impact research?
‘Why
they do what they do’ is a question of psychology, that is, of what
sort of people are likely to do Q1 research. This result may be
important in selecting people for universities and research schools. In
the absence of such investigations, it is only possible to speculate
and the speculation which follows is not part of the case for using the
Attainment Correlation of Fig 2 to develop metrics for research
evaluation. It is only for the purpose of stimulating research on the
questions raised by the shape of Fig 2, that is, the large difference
in impact between experienced researchers.
It may be that
the unique contribution of Q1 researchers is to create order in the
chaos of unstructured information by defining a new question which is
not only significant but which may be answered by existing techniques.
In the sciences, these would be experimental or mathematical
techniques. Defining such a question is an art that few have mastered
and usually more time will go into finding the question than in
answering it. Once a Q1 researcher has established and answered a new
question, other academics (Q2+) may answer it using very slightly
different techniques to get slightly different answers. All
professional academics are good at answering other people’s questions.
They are good at passing examinations, which is why they were
appointed. The easiest way of passing your teacher’s examination is to
do what teacher says, and look up to those set in authority over you
for your reward.
It is perhaps worth noting that those gifted in academic research may be different from the usual run of academics. They must have a high degree of self-confidence. They are in the business of changing the way people think, of introducing ideas that may be disruptive. This strength of character may well be associated with irreverence, with a lack of respect for their seniors and for the research of their colleagues. Thus the two outstanding academic researchers of the last century, Einstein and Crick, each had difficulties with their Professors, [Einstein unable to get an academic post and Crick in danger of losing his lowly one [Watson, J.D. (1968))], until each made an outstanding contribution which brought recognition, security and academic freedom. A gifted young academic may well suffer from the reaction of his colleagues and professor to his irreverent comments. This is the theme of the play and film Amadeus, a tale of how the virtuous, hardworking, devout and influential Salieri, envious of the gifted but immoral Amadeus Mozart, drove him to his death. It has been criticised as historically wrong. It may be bad history but still be good art. (Of interest here is to note that there are many conventionally good academics and others who, on seeing the film, sympathise more with Salieri than with Mozart.). This may all be no more than mischievous speculation. Nevertheless the questions raised are important enough to illustrate the significance of the research proposed and the value of an attainment correlation such as Fig. 2.
5.0 Choosing which Research to Support. A metrics method for replacing the peer review of the committees of the Research Councils
The most effective method of distributing public funds for academic research is:
5.1…Support Quintile 1 researchers to work on projects they choose, [who are we to tell these people what to do]. They should receive the support they need for their work, with perhaps a maximum of about three postdoctoral fellows and three doctoral students plus a fund for equipment and a contribution towards workshop, laboratory and administrative expenses.
5.2…Avoid wasting more money on the academic research of experienced people in Quintiles 3, 4, and 5. The savings would more than provide the costs of the Q1 researchers and that of new lecturers.
5.3…Support the research of recently appointed young academics while they attempt to master the art of producing high impact research. They might each receive say one PhD scholarship a year to offer to students who wish to work with them. After they have had between five and ten years of choosing research projects, evaluate their publications by citations, compare them with the fig. 2 attainment evaluation graph in their discipline and decide whether to continue the support. It is convenient to refer to these recently appointed academics as probationers.
Evaluating the research of Universities and their Departments. A metrics method for the Research Assessment Exercise.
6.0 Research Rating of Departments.
The evaluation should consider not only the quality of the research but the management of the people so as to achieve an appropriate turnover of probationers and effective cooperation with entrepreneurial companies in the private sector. For example only, use the following metrics to determine a research rating for each department
6.1…To measure research quality determine how many [if any] Quintile 1 researchers work in the department. That is, evaluate all the researchers, not just the professors, by comparing them with the Attainment Curve of Fig. 2. Let this be NQ.
6.2…To measure turnover count the number of probationers who have been there for less than five years. Let this be NP.
6.3….To measure cooperation with entrepreneurial companies, use a combination of metrics measurements of those practices which enable companies to become aware of recent research findings. For example: i] The number of probationers retained as consultants by companies,(nc). ii] The number of probationers who have left the university since the previous research valuation exercise and accepted posts in high tech. companies, (np). iii] The number of research students sponsored by companies, (ns). iv]The number of research students recruited by those companies who have contributed financially to the department research (ne). v] The total financial contribution to support research from the private sector, (F).
Then define a Company Cooperation number, (NC) where for example
NC= nc+np+ns+ne+F/100,000.
6.4 … The Department Research Rating, R, might then be calculated as R= NQ+NP/10+NC/50.
This calculation of R is intended to depend most strongly on the number of Q1 researchers in the department. That is, it will coincide with how the department is regarded by researchers in the international community.
7.0 Research rating of universities. This might be based on the proportion of its departments with a high department research rating and the management systems it uses to ensure that: 7.1…All of the money awarded to support the work of a Q1 researcher is spent as he/she wishes, that is, a management which avoids all forms of internal peer review. 7.2…Probationers and postgraduates are recruited by entrepreneurial private companies.
8.0… What is suggested is no more than an improved version of traditional ways.
8.1…A University department that employs a Q1 researcher gets international recognition for academic research and the reputation for research of a university depends on the number of Q1 academics working there. Serious researchers overseas neither know nor care who has gained a place on a committee to award grants or to evaluate research. They are interested in the published work.
8.2…The support for Q1 researchers to work on problems chosen by them is equivalent to providing them with endowments, capital sums or investments for them to use the income in whatever way they think fit. Endowments from alumni are an important feature of American Universities. In the U.K., endowments are only significant at the ancient universities of Oxford and Cambridge. A recent rating of University research by the Chinese, Shanghai Jiao Tong University, placed 17 American Universities in the top 20, which also included Cambridge and Oxford as the only British Universities. If there is indeed a dependence of high impact research on endowment, it seems likely that this is because endowment gives the endowed researcher the freedom to explore his curiosity-derived projects free from peer review intrusion. Any committee of senior academics appointing people to endowed posts should make such appointments dependent on a candidate’s previous record of accomplishment, established by citations to his published work.
8.3…Providing PhD scholarships for postgraduate students to work on projects suggested and supervised by young lecturers is not much different from the situation in British Universities in the 1950s/60s when the quality of the research was no worse than today and may well have been better. In those days, it was rare to employ post-doctoral fellows but there were many well-qualified Ph.D. candidates hoping to use their degree to get a work permit in the USA.
9.0 Advantages of the proposed methods.
9.1…The advantages of these proposed methods of management are that they are well focussed, efficient and cost effective. By making the peer review hierarchies redundant, they get rid of a vast amount of time consuming bureaucracy which is costly, not only in money but in lifetime. Distributing resources in response to this ‘bottom up’ free market in citations provides a way of escaping from the ‘top down’ system of peer review. It assumes that authors citing papers are analogous to customers purchasing goods or services and that distributing support in response to citation impact is similar, in principle, to investing in a company in response to its market share.
9.2…Implementation of these radical suggestions would cause a revolution in academic research and the management of universities. Academics would now become obsessed with the quality of their research and no longer waste time attending meetings of peer review committees or writing applications for support for their research. Young academics would take seriously advice from Q1 researchers. There would be demand for a Q1 Lectures Series, Q1 Master Classes and for the research into the nature of the creative process outlined above. 9.3…At present in the UK a lot of research is done by research fellows on temporary three year contracts. It is sometimes necessary for a research fellow to have to take three or more of these temporary appointments before he/she is able to get a permanent post as a university lecturer. During this time he will have had to work on research projects chosen by somebody else. It is these people who would be recruited as probationers under these proposals. They would lecture to students and choose research projects, both for themselves and for those PhD students who chose to work with them. 9.4… Perhaps the most effective way of ensuring a turnover of ambitious young researchers would be to insist that, after the probationary period, all academics, Q1 to Q5 find a post outside the university with no return for 5 years! This would also change for the better the interaction of a university with the society around it. Companies would get used to recruiting thirty or thirty five year old ex-academics, especially those who had been retained by the company as a consultant while they were at university. [This would not be so different from practice in those European countries where companies recruit thirty year old post graduates after five year degree programmes and six year doctorates.]
9.5…Non-STEM Subjects.
In presenting metrics methods based on research income, the Govt. Report [D.E.S. (2006)], noted differences between science, technology, engineering, mathematics and medical subjects (STEM subjects) and others such as sociology, management and humanities etc. which they have defined as non-STEM subjects. It has been claimed that citations to published papers may not be used to evaluate research in non-STEM subjects where work is published as books. If this is so then the Attainment Correlation of Fig 2 may not be used. However, the underlying principle that some researchers are a lot more effective than others may still apply, and departments may then be evaluated by the number of their high impact researchers and probationers. In consulting Non-STEM academics, ask the following questions: a) ‘If all expenses were paid, which person from another country you would invite to come and meet your research group?’ b) Followed by, ‘how did you select him/her, on what basis?’ and then, c) ‘How do we apply that test to UK researchers?’
10.0 Concluding remarks.
10.1…Those now responsible for distributing money for research and those responsible for evaluating the results try hard to be fair to everybody. Despite their good intentions, their own universities always benefit the most. Representatives from the older universities, (Oxford, Cambridge and Redbrick 1900 a.d. foundations) dominate the peer review committees. At the first research evaluation in 1986, 15% of the Universities provided 50% of the funding committees and received 50% of the grants. It seems likely that little has changed since then. It has been proposed by groups of the favoured universities that this result be made permanent by establishing elite research universities who will receive the lion’s share of research funding without the farce of peer review. One such group is the Russell Group of a dozen or so of the older universities; another is the Cambridge-London triangle. In all of these universities, many academics are Q2 plus, and all that is proposed is a continuation of what has gone on before.
10.2…What is required is a radical change to an objective quality led process for spending research support, directed to those individuals who do the quality work, combined with incentives for universities to transfer new research thinking into entrepreneurial private companies. In the real world, an elite university is one known internationally for the impact of its research, that is, by the number of Q1 researchers who choose to work there.
10.3…A reform of research funding, as described above, would permit any university to choose whether to become a research university or to concentrate on teaching. A research university would have to find ways of attracting Q1 researchers and of ensuring a turnover of young teachers striving to master the art of academic research. A teaching university would concentrate on recruiting teachers who had experience of practising their discipline in the world outside the university and gain its reputation from the quality of its graduates. Teaching universities are likely to attract ambitious students. Without the burden of supporting research, a teaching university might require lower fees from its students and concentrate on preparing them for well-paid jobs.
APPENDIX 1 Some metrics for evaluating
Peer Review
In 2008 the UK government will choose how to
spread 8 billion pounds across the country’s universities over a period of six
years. It is reasonable to ask if peer review is capable of spending this money
wisely. Those at present benefiting financially from the established funding
policy of peer review are likely to resist any change to a metrics based policy.
Their arguments in favour of peer review are another exercise in peer review,
amounting to no more than a statement that in their subjective judgement,
subjective judgement is superior to objective judgement.
The management of university research funding by peer
review has been used for at least twenty years in the U.K. and it is then surprising that
it has received little or no objective evaluation. It is not difficult to
devise simple ’metrics methods’ for evaluating peer review and those proposed
here have been used previously, one for evaluating research on animals and the
others for chemical engineering. These examples may thus be used to illustrate
methods which are suitable for other subjects.
Method 1. Evaluate publications. That is, obtain a list of all the publications arising from
the research proposals that were accepted by a peer review committee and see
how many times each paper is cited by later workers.
An example is the recent work of Andrew Knight and his
colleagues at Animal Consultants International [Knight A, Bailey J, and Balcombe J. (2006)],
who looked into the number of citations received by 749 scientific papers
published as a result of the peer reviewed allocation of support for invasive
experiments on captive chimpanzees. Half the papers had not received a single
citation up to ten years after their original publication! This despite the
fact that the peer review of this type of experiment is required to be
especially vigilant to avoid unnecessary suffering. [If this result is typical
of other subjects it confirms that much peer review supported research work is
insignificant.] A rank order correlation of papers in order of citation
frequency would test the significance of the comment quoted above, “I don’t
have time to read them all, but some I read again and again”.
Method 2. How dependent is a Research Evaluation on
‘Input’. Find out how much peer reviewed research grant money each
department received from Research Council committees in the period preceding a
Research Evaluation. Then put departments in rank order by research grant, and
plot a rank order graph on which each department position is shown by its
research rating. A separate graph is needed for each subject. Look for any
pattern in the results.
The example shown in Fig.3 is for departments of chemical
engineering at the first Research Evaluation Exercise in 1986.
This shows a strong correlation between research rating and Research
Council income. That is, with little disagreement, the research rating of a
department might have been decided only by income without further deliberation
by peer review.
[Thus began the cultural shift of emphasis from doing
quality research to writing grant applications.]
Fig. 3. Showing the dependence of a department’s 1986
Research Rating on its Research Council income.
Method 3. How important is Committee Membership for
getting grants.
For each of the
Research Councils’ peer review committees, (those which choose which
applications for funding to support and which to reject), calculate for each
year, the ratio of the average grant to a department represented on the
committee divided by the average grant to a department not represented on the
committee.
In the example
shown in Fig.4, this ratio is plotted against time for the period leading up to
the first RAE in 1986. In 1980, well before there was any suggestion of a research
rating, this ratio was 1.2 (a not unreasonable compensation for the time spent
on the committee). As the research evaluation approached, the ratio increased
to 3.2!
[Presumably this
is why there is so much competition to become a committee member. Figs. 3 and 4
taken together explain the present ‘Input-Insider’ culture of universities in
the U.K.]
Fig 4. The importance of committee
membership for obtaining research grants.
Method 4. Evaluate the research reputation of the grant
awarding committee members.
This may be discovered by using, for each subject committee,
the Attainment Evaluation correlation of Fig.2.
The example shown in Fig 5 is an evaluation of the committee members
awarding research grants to Chemical Engineering departments in the period
preceding the first RAE in 1986.
Most of the committee appear in Quintiles 3, 4 and 5, that
is, their published work had made little or no impact on the research
community. Only one or two committee members are high impact researchers in
Quintile 1. (It would seem that most Q1 researchers knew their time was too
valuable to spend on committee work.).
Method 5. Did the allocation of funds for research take
into account the reputation of the recipient?
Again, this may be determined by reference to Fig. 2, the
number of citations made to the recipient in the literature and the amount of research
grant he/she received. In the example shown in Fig. 6, from each researcher
located on the Research Attainment curve of Fig. 2, a vertical line is drawn
proportional to the amount of grant received. [In this example, it is shown
that no importance was attached to a recipient’s reputation for research.
Together, Figs 5 and 6 contradict any assumption that peer review is justified
because the clever people are awarding grants to other clever people. ]
Fig.
5.Using Citations to evaluate a committee responsible for distributing public
money for academic research.
Fig
6. Award of research grants compared to research reputation.
APPENDIX 2 A
subjective peer review of peer review.
A2.1 Fifty years ago, when there were relatively few university
students, relatively few academics were needed to teach them, and taking part
in research was a matter of choice. As one University teacher put it
“Research’? I’m not much interested. I publish something every two or three
years just to keep the Prof. happy, but I’m more into teaching”. Some idea of the
culture of academic research in that time may be formed from the absorbing
story of the identification by Francis Crick and James Watson of the double
helix as the structure of DNA, [Watson J.D., (1968)]. Neither Crick nor Watson
were expected to be doing this work, they did it from interest and ambition in
their own time, sometimes in the pub. Research was then largely self- motivated
and people had time to think for themselves. It was important for those seeking
recognition and promotion, (publish or perish!), but with a relatively small
number of people involved in research, there was enough money to go round.
A2.2 As
student numbers have increased, so has the number of academics who teach them.
A reasonable question is whether we need ten times the amount of academic
research just because we have ten times the number of students. Much academic
research seems to have no bearing on the way we live and indeed, in English, to
call something academic is to imply that it has no practical value. And yet it must
be accepted that in some way academic research confers great benefits on the
society which supports it. The difficulty is choosing which research proposals
to accept and which to refuse. All over the world, the solution has been to
leave this to the academics themselves, to let them evaluate each others
research proposals and select which of them to support. This then is the basis
for evaluation by peer review.
A.2.3 Peer review is a good way of evaluating scholarship. It is the
traditional method used by universities to make judgments about candidates
seeking entry into university or those wishing to graduate or to be awarded a
higher degree. Examination papers are subjected to the peer review of internal
and external examiners. These are all tests of the candidates’ mastery of
established knowledge.
In peer review of scholarship the
reviewer has no personal interest in the result.
A2.4 It is not at all obvious
that peer review is able to evaluate proposals for research work before the
result of the work is known. That is, that a method developed to evaluate
knowledge of the past is able to say anything sensible about the future. When
peer review committees were first given money to allocate to research proposals,
it seems this view was accepted, at least by two committees associated with
chemical engineering. In around 1967 I was awarded a significant grant for
research on the design of equipment for the condensation of mixed vapours in the presence of a non-condensable gas. This was
justified on the basis of some theoretical and experimental work I had already
published and which had been well received by experts in the field. Shortly
afterwards I joined a renowned authority in my field (commercial scale
separation processes) and we were awarded a very large grant to set up test
rigs. This was largely based on our many previous publications and that we were
based together in a university with a suitable laboratory. That is, both these
grants depended on the recognition given to past work. Mine had been done with
the assistance of research students, but without additional support
A2.5 In the U.K; resources are
allocated by two large bureaucracies, each supported by 50 to 100
sub-committees. The first, (The Research Councils), requires academics to send them
descriptions of what research they plan to do and then decides if they will be
given the money to do it. The other, (the Research Assessment Exercise, RAE,)
examines what they have done, decides how good it is and whether their
university department should be rewarded with additional resources. It is the
peer review of the RAE which is now to be replaced by objective measurements,
that is by metrics.
At this time of change it is not unreasonable to
examine also the competence of the peer review committees of the Research
Councils that decide which research to support. Indeed, both parts of the
process are linked; the RAE assesses research resulting from the research
proposals accepted by the Research Councils. Several objective, ‘metrics’,
methods for evaluating the success or failure of the peer review committees are
described above in Appendix 1.
A2.6 Research proposals are accepted [or rejected]
by the Research Councils’ peer review committees on the basis of what is
written in the proposal. In evaluating research proposals the committees are
able to evaluate the applicant’s professional competence, and knowledge of the
literature and the topicality of the paper or proposal. It is very difficult,
and in general not possible, to form a view from reading a proposal on whether
those seeking support are at all likely to have an original idea while spending
their research grant. Indeed, such is the accepted practices for getting grants
that it would have been quite impossible for Crick and Watson to have been
successful in getting support for their work.
A2.7 Departments are assessed by the RAE on the
basis of a judgement on the value of recent publications. In doing this, the
sub-committees are attempting to make a prediction about the future importance
of the work. The problem in judging papers is, not only insufficient time
available for the committee members to make a responsible evaluation [one of
the reasons for abandoning RAE 08] but that most papers have little real
significance. [For an objective evaluation of publications resulting from
research proposals supported after peer review, see recent work on animal
research projects, described in Appendix 1 above.]. The significance of a paper is only revealed
some three to five years after publication by when it will have either been
cited by other workers in the field or ignored. The RAE people are required to
make a there and then judgement.
A2.8 Another
problem is the degree of specialisation in modern research. No sub-committee
member can be an expert in all parts of his subject; indeed, he may well have
difficulty in understanding the relevance of papers from some of his own
department colleagues. What the RAE
committees can do is count the number of papers and the number of people able
to put forward four papers for evaluation.
Thus Universities must encourage
academics to obtain more research grants. Grants means more papers to impress
the RAE. Nowadays it seems that ‘Publish or perish’ has been replaced by ‘Getting
and Spending’. All departments and all university teachers are expected to get
in as much research grant money as possible. Careers depend on it.
Getting grants
is very time consuming. A mid career research team leader, expected to find
continuing support for three postdoctoral fellows must raise at least £100,000
a year. With a proposal hit rate of say one in three, this means, every year,
writing proposals to justify spending £300.000! Although more and more papers
are required, getting the grants may consume more time and energy than writing
the papers.
A2.9 After the first RAE in 1985, many people
complained about the ratings of their department. One comment was “that the
next time the game is played it would be better if everybody understood the
rules”. Since then, there have been several research ratings and several
changes in the rules. For example, ratings have been based on output i.e. the
number of publications, on papers in reputable journals, on funding from
government better than funding from elsewhere and now, no restriction on the
type of research to be evaluated. Each change in the rules gives an advantage
to those close to the committee members who may explain to them how the rules
will be interpreted, next time. What has not changed is the result. The older
universities dominate the committees; get the most money and the highest
research ratings. Like all other academic committees, peer review committees
work by consensus. Almost inevitably this will become ‘….I respect your
judgement about your people as you will respect my judgement about mine.’
A2.10 This explains why universities strive to get
their people on to the committees that decide between the departmental sheep
who get the money and the departmental goats who do not. Thus, for the proposed
next RAE in 2008, 1400 bodies nominated 5000 candidates of which 900 were
selected to serve on 67 committees.
A2.11 A serious criticism of the RAE is that it has
changed the research culture of the Universities to an ‘Input-insider’ culture,
where junior academics must dedicate themselves to getting grants and the more
senior, to getting on committees.
A2.12 But perhaps the most disturbing aspect of
the RAE by peer review is that by evaluating whole departments, it attaches
insufficient importance to the presence [or absence] in a department of gifted
individuals. Where there are gifted individuals, some of the support they
should be entitled to will inevitably be dissipated by energetic colleagues who
are no more than competent.
In the wider world, a department or university is recognised
for research by the reputations of the individual researchers who choose to
work there.
References.
Bhide’, A, (2006), ‘Venturesome Consumption, Innovation and Globalization’ Paper for a Joint Conference of CESifo and the Center on Capitalism and Society, “Perspectives on the Performance of the Continent’s Economies”Venice, 21 - 22 July 2006 www.bhide.net/publications.html
Carter, C.F. and B.R. Williams, (1964) “Government Scientific Policy and the Growth of the British Economy” The Manchester School, September 1964 pp. 197-214
Chekhov, A, (1897). ‘Uncle Vanya’. Act Two, (English translation by Peter Carson, Penguin Books, 2002, p163.) D.E.S. (2006),
‘Reform of higher education research assessment and funding’, Department for Education & Skills, Issued 13 June, 2006, rae.consultation@dfes.gsi.gov.uk
Gilli, G, (2004) ‘Bibliometric Tools for the Evaluation of Scientific Productivity’ Paper for a Conference, ‘Scientific excellence for economic growth’, on the occasion of the 600th Anniversary of the University of Torino, Torino, September 23-24th 2004, http://bs-d.unife.it/dip-chimica/varie.htm
Knight A, Bailey J, and Balcombe J. (2006),’Chimpanzee research: 1. questionable contributions to biomedical knowledge’. Altex: Alternatives to Animal Experimentation 2006; 2:107-8. [conference abstract]. http://www.animalconsultants.org/ See also review in Economist June 8, 2006.
A note by the author, Professor Emeritus Ken Porter.
The paper above, ‘The Management of University Research in the Intellectual Age’, refers, particularly in Appendix 1, to the results of an analysis of the first UK Research Assessment Exercise [RAE] which occurred in 1985. I have been asked why I took so long to write the paper, how I became interested in the subject, noting that I am an engineer rather than a sociologist, and indeed, was it ‘sour grapes’, written because the peer review committees were turning down my applications for financial support.
Before 1985 First some biography. I was born in 1930; my first degree is in Mechanical Engineering, an External Degree of London University, obtained by day release while working at Midland Tar Distillers, a coal tar processing company. A lot of the time I worked shifts supervising the operation of the plant. I then took a postgraduate diploma in Chemical Engineering at Birmingham University where I accepted a post as Lecturer.
My main field of research is the most frequently used commercial scale method of separating fluid mixtures, distillation. That is, what goes on inside most of those big towers seen in oil refineries and chemical works, how to design and improve the hardware and predict its performance. After twelve years research at Birmingham University and about three at UMIST I had by then achieved an international reputation for my academic research, [including the Moulton Medal “best paper” award of Institution of Chemical Engineers], but some of my research had become, in my view, a bit too abstract for practical engineering. I needed to get back to the real thing. I joined a metal bashing company, part of a big US group, to manage engineers designing, trouble shooting and inventing distillation column internals. After about ten years I returned to academic life as a Professor of Chemical Engineering at Aston University. About then I had written [with JD Jenkins] a paper “The interrelationship between industrial practice and academic research in distillation and absorption”. This was well received and formed the basis of a short course, “Developments in distillation” which practising engineers from the oil and chemical industry were happy to pay to attend. I was invited to present the course at the University of Salamanca in Spain, at KAIST in South Korea, at the Indian Institute of Petroleum, and, as mentioned below, at the University of Sydney, Australia. KAIST and Salamanca invited us to cooperate in research with exchange visits paid for by the British Council About this time I was awarded the Council Medal of the Institution of Chemical Engineers for my contributions to Symposia on Distillation.
Why I became interested in Research Evaluation I was drawn into the debate about academic research by chance. At the time of the first RAE in 1985 I was [as Chairman of the Midlands Branch] a member of the Council of the Institution of Chemical Engineers, [I.Chem.E.]. The new President of I.Chem.E., Rolf Prince, had been asked by several unhappy University Chemical. Engineering departments to debate the validity of the RAE results. He opened the debate by asking for a forthright Midlands view from Ken Porter. I didn’t have much of a view at that time but I remember saying something along the lines that the only people that believed in peer review were the peers doing the review. After the meeting, back at Aston University, I wondered if my comment been irresponsible and started to check it out. There was a lot of information available to the public and the result of my investigations was the series of graphs in Appendix 1. The situation in Chemical Engineering was far worse than I had imagined possible.
Professor Prince was our first Australian president, over in the UK on sabbatical leave from Sydney University, and that was his first Council meeting. It so happened that I had spent the previous summer in Australia at the invitation of a group of companies associated with Prince’s department of Chemical Engineering. I had lectured at the university and presented the short course on Developments in Distillation to Australian engineers. I suspect that one reason President Prince had turned first to me for a comment was that I may have been the first face he recognised. In effect, I was drawn into the work by chance.
My lack of interest in peer reviewed grants before the RAE In those years immediately before the first RAE, I had been far too busy to worry about the peer review committees. At the end of the ‘70s we had our first energy crisis and the chemical industry wanted to reduce the energy used by distillation columns by changing to a different type of column internal, packed columns. The problem was scale-up, scale-up failures of packed distillation columns were known to have occurred. The only outfit in the Western World that researched on full scale distillation columns was Fractionation Research Incorporated, (FRI), in California. Most companies in the distillation business paid a fee to belong to FRI, so FRI was asked to solve the problem. FRI then hired me as a consultant and I had a happy and fruitful time advising their steering committee and technical staff on the research programme. By 1983 the problem was solved and large diameter packed distillation columns became widespread. We probably saved more energy than all the research on the alternative separation methods put together. [The response of the peer review committees to the energy crisis had been to support research on separation methods other than distillation, mainly membrane separators and adsorption.]
I had attempted to insert some sense into this type of work. I was working with PhD students to explore ways of setting targets for the development of commercially viable, new separating materials based on the design and economic evaluation of the process of which the separating material would be a part. This was a new approach we invented. It is described in the paper Alternative Separation Processes, which incidentally was awarded the I.Chem.E. Hanson Medal for 1979. [ Also, later on, one of my doctoral students gained the first Richardson Award of the Society of the Chemical Industry (for researchers under 30) for his Ph.D work on designing a promising membrane for hydrogen separation in the acetic acid process]. We also did some contract research or testing of distillation devices in our test rigs for commercial companies. We used much of our earnings to build a larger test rig for research on distillation trays.
In brief, in the period before 1985 I had been far too busy to worry about the workings of peer review committees or to feel any resentment at their decisions; I don’t remember asking for their financial support, I didn’t need it.
After the first RAE After the first Research Assessment I noted that the rules had changed and I applied for and obtained research grants to support post doctoral fellows, we usually had three on different overlapping three year contracts. The ideas presented in the paper, The management of University research in the intellectual age, have developed over the last twenty years. I have presented some of them at various places, usually as part of an invited lecture. In particular the Attainment Curve and the plea that we get rid of the input-insider culture and concentrate our efforts on high quality, high impact research evaluated by citations. [See for example my paper presented in 1999 to the Annual Meeting of the American Institute of Chemical Engineers at a conference session they were kind enough to hold in my honour. Here I develop the idea of ideas as Intellectual Tools, based on examples from chemical engineering]. I soon realised that evaluation by citations was the last thing most academics wished to experience. I don’t waste my time pushing on doors that are firmly closed and I had plenty of other things to do. China and Russia were opening up and I received invitations to lecture and to cooperate in research [from The National Distillation Research Centre of China at Tianjin University]. I waited to publish [on the web] until this particular door was forced ajar by the desire of the UK government to look for a metrics method to replace peer review.
