REF: Can it measure me before I’m gone?

The UK is having a consultation on the REF, the “Research Excellence Framework”.   For those of you outside the UK, the REF is a system which decides how much government support goes to each university.  It distributes a substantial fraction of all the UK’s higher education money, an amount roughly comparable to the the total of all research grants.

It has good aims, of course.   It aims to “drive up quality” and various other things.  All well and good, but the “aims” are quite irrelevant: what you aim for and what you get can be very different things.  For instance, governments have aimed to eradicate poverty and drug use many times, but both persist.   Aims are, at best, cherished hopes.  At worst, they are a wordy cover for something else.

It is a long document; here, I’ll focus on the “impact” sections.

The flaw

The REF “Impact” process has a basic flaw: it cannot measure what it needs to measure.  The REF process is – in intent – a way to distribute money to departments that have the most productive people.  To do that, in any sensible manner, the entire cycle from research to REF evaluation to money needs to happen before the people leave.

But, on one hand, university faculty (and especially research staff) come and go in a few years, and it is the faculty and staff who determine the quality of the research.  On the other hand, research can be a long process.  A proposal needs to be written, the research needs to be done, and papers need to be published or patents filed.  How long does this take?

1. Proposals take weeks of full-time work.  Fitting this into the busy schedule of an average lecturer or professor can take months.  And, then, proposals have only a 1 in 4 chance of acceptance.  So, it can easily be a year or more from the initial ideas until four proposals are written so that one will be accepted.
2. Research may involve hiring someone.   That takes about four months.
3. The research itself will take a while, typically 1-3 years.  (And, since the proposals take so long to write, there is some incentive to plan longer research projects, if only to put off the pain of proposal writing a little bit longer.)
4. Only then, when the research is nearly done, does it make sense to spend a lot of effort thinking about impact.  After all, if it is a real research project, the outcome cannot be reliably predicted.   (If the outcome were known in advance, why would RCUK pay for it?)  Until the outcome is known, the impact cannot be even guessed at.  And, whatever you do to maximize the impact will probably depend on the outcome too.

Overall, it will likely be three years from a research idea to the point where the impact clock even begins to tick.  And then, impact does not normally happen instantly.  So, the REF needs to wait 3+N years (where N is however long it takes for impact to appear).  And then, it becomes a race between impact appearing and staff leaving.

The REF is caught: if it evaluates too early, it won’t know the real impact.  The evaluations will be based on a mixture of guesswork, bias, and intellectual fashion.  On the other hand, if it evaluates too late, it will be able to see the real impact, but the staff who caused the impact will have left.  Off to industry, retirement, or other universities.  And in the middle?  Well, if 3+N is bigger than the mean length of employment, the REF might be simultaneously too early to evaluate impact and too late because the staff have already left.

Notably, most research in the UK is not done by tenured staff: it’s done by research associates hired, typically for the duration of a single project.  (Much research may be run by tenured staff, and the tenured staff will have an influence on the research quality, but someone who spends perhaps 10% of their time involved in a research project cannot control it in detail.  The research associate will ultimately determine the success or failure of the project and also how trustworthy the results will be.)

So, the REF makes sense if and only if staff tenure at an institution is longer than the time it takes to execute a research project and for those results to have impact.

How long does real impact take?

A long time.

It is well understood that ideas do not immediately have impact.  For instance, patents are a limited monopoly to make something.  Patents are intended to encourage technological development, and they seem to work.  (E.g. most corporations will pay to patent inventions, and comparing nations, the rate of patent production is strongly correlated with GDP and economic growth.)  So, patents are a good model for encouraging R&D.

The logic underlying patents is that there is a benefit to society by encouraging invention and another benefit by encouraging the invention to be published widely.  In return for publishing the details of an invention, the inventor is granted 20 years to make a profit from his invention.  If that term were set too short, companies would prefer to keep inventions secret.  If too long, innovation and growth would be slowed.  The current 20 year term was set in the 1995 Uruguay round of the World Trade Organization negotiations; it was set recently and represents the best current consensus on how long it takes to profit from a patentable idea.

Patentable ideas are typically much closer to production and profitability than the research that RCUK funds, so that the 20 year term should be seen as a lower limit for the time to go from research result to profitability.

Objection: A patent term needs to be long enough for a company to actually make a profit.  That is a very strict standard for proving impact.  We can look at earlier stages of impact and get an answer faster.

Answer 1: Yes, but the time scale from research discovery to patent and then patent to profit can be much longer than 20 years.  One could point to the circa 70-year span from Einstein’s 1917 paper predicting stimulated emission to the point where the laser industry was making substantial profits.  One can safely say that before the invention of the Maser (1954, 37 years later) few people – even experts – would have claimed that Einstein’s idea had had much impact.  And, even then, the Maser had minor impact, except that it led to the laser.  So, even if impact is “obvious” half-way to profitability, that could still amount to 20 years, or nearly.

Answer 2: If we could actually predict the impact of a research idea at an early stage, then entrepeneurs would invest in high-impact ideas.  The fact that people generally do not get wealthy by buying and selling the results of research shows that its value cannot be predicted.  Even at a slightly later stage, when an idea becomes patentable, it is very hard to gauge its value.  If we could judge value then, people would only patent the valuable ideas, but in fact, most patents are never profitable.  Therefore people must be unable to accurately value ideas even at the patent stage.

In fact, even further down the line, start-up companies are notoriously hard to value.  Most fail.  Some of the failures are presumably due to a weakness in the ideas underlying the company.   So, even at that late stage in the intellectual property development process, judging the impact of an idea cannot be done reliably.  (If it could be done reliably, investors would only put money in those companies with high-impact ideas.)

Another legally recognized route from idea to impact is copyright.  This allows an author to profit for life+70 years (or life+120 years in the USA).  Even if this is supposed to be influenced by corporate lobbying, copyright has historically been long.  The Statue of Anne (1709) set it to 14 years, and USA copyright began at 26 years (in 1790) and has only lengthened.  So, ideas do have an impact for a long time, and sometimes it takes a long time for them to achieve their full impact.

For instance, the Communist Manifesto is arguably the most high-impact products of the British Library.  It was first published in 1848, but English translations before 1888 (Engels) are scarce, and apparently limited to small-circulation newspapers.  The impact of the Communist Manifesto continued to increase through the Russian Ocrober Revolution (1917), and the British Communist party was not even founded until 1920.   The work’s peak impact was reached perhaps in 1968, 120 years after publication.  Any plausible evaluation in the first few years would have been unlikely to estimate its eventual effect.

Objection: Yes, but most ideas show their value more rapidly than that.

Answer: Some do, but probably not most.  Consider most “new” things, like a MP3 player.  Typically, these are triggered by a new idea, but depend on a variety of older research outputs.  For instance, the novel ingredient that made MP3 players possible was the post-2000 availability of inexpensive, low-power processors.  However, an equally important ingredient was the MP3 compression algorithm itself, which was finalized in 1992, about a decade before its major impact arose.