21 December 2015
This response to the Green Paper focuses primarily on the proposals in ‘Part A: Teaching Excellence, Quality and Social Mobility’. Written by members of the Department of Educational Research, led by Paul Ashwin.

We welcome the commitment to supporting high quality teaching in universities, providing useful information for students on the quality of their programmes, and recognising socially-inclusive institutions as well as the diversity of institutions across the sector that is contained within the Green Paper.

There are a number of strengths about the approach taken in the Green Paper to supporting high quality teaching. In focusing on the quality of programmes the Teaching Excellence Framework (TEF) avoids the tendency to overly focus on individual teachers which so often undermines attempts to support the development of high quality teaching. It recognises that such judgements involve aggregating a number of factors that cannot be captured by metrics alone. The TEF also attempts to take disciplinary differences in teaching and learning seriously by seeking to develop subject based measures of teaching quality.

However, as a whole, the proposals in the Green Paper do not offer a workable methodology for supporting high quality teaching. This is for the following reasons:

1. The four levels of TEF will lead to much greater bureaucracy 

There are a number of problems with using the four levels of the TEF to support differential fees. First, if the levels are to act as an effective incentive then there will be large increases in tuition fees at the higher levels. This does not fit with the focus on value for money for students in the Green Paper.

Second, if fees are significantly different at the four levels of TEF, then this will create an unfair differential between those Level 3 institutions that just miss out on Level 4 and those institutions that just make it into Level 4. This does not cohere with current assessments of research quality which are much more discerning and differentiated. It will also place a great deal of pressure on the judgements made through the TEF, and these will have to stand up to a great deal of sustained interrogation. This is likely to make the TEF process much more bureaucratic than it is presented in the Green paper, as the threat of legal action by institutions force TEF panels to inspect the claims made by institutions to ever closer scrutiny.

Third, if there are not large differentials between the fees charged by Institutions at different levels of the TEF, then the systemic incentives to engage in the TEF will be weak for popular universities. It is difficult to understand why institutions who have healthy levels of student recruitment would engage in the TEF particularly as they will be expected to carry the cost of their involvement in the process. Whilst it has been argued that the benefits to reputation will be sufficient incentive for institutions to bear the costs of their involvement, this is based on a misunderstanding of the way in which reputation operates in higher education. Generally people have strong, largely evidence-free, preconceptions of which are the high quality universities and if a measure of quality does not meet these preconceptions then it is likely to be rejected as an effective measure. This means that if prestigious universities do not take the TEF seriously then it is unlikely to be seen as a rigorous measure of teaching quality.

2. TEF ratings will be based on past performance

TEF judgements will be based on evidence of institutions past performance. This means that if differential fees are charged, then students will be paying more based on the quality of a degree programme that previous students enjoyed. Thus it is perfectly possible that students will be asked to pay TEF4 level fees for what is later rated to be a TEF3, TEF2 or even TEF1 level of experience. Similarly students could be charged lower fees for a course that is later rated as TEF4. This is clearly inequitable and is likely to lead to student complaints.

3. The aggregation of quality to the institutional level

Whilst the Green Paper recognises that quality is a function of particular degree programmes rather than institutions, for reasons of bureaucratic efficiency, it proposes that these should be aggregated up to an institutional award. This will determine what level of tuition fees can be charged by an institution. This will mean that the fees charged to students will not reflect the quality of their particular course. So a student studying a much lower quality Economics degree in Institution X could end up paying much higher fees than students on a much higher quality Economics degree in Institution Y. This is both inequitable and is unlikely to support improvements in the quality of teaching. It is also likely to lead to complaints by students who are asked to pay more for lower quality programmes.

Whilst it is possible that institutions could vary the cost of their programmes according to their quality as measured through the TEF, there are three problems with this that make it very unlikely. First, as TEF level will set a maximum on fees charged, there will still be TEF4 courses that have to charge lower fees because of their institutional rating. Second, the issues of TEF being based on past performance become even more acute if institutions attempt to vary fees according to the ratings of individual courses. Third, it is not clear how joint degrees, or students studying single modules from a differently rated Department, could be handled within such an approach. Given these potential problems it would appear much safer for institutions to stick with fees that reflect their institutional TEF rating.

4. There is too little indication of how quality will be measured

There is very little detail of which measures will be used to inform the TEF. The Green paper indicates that a ‘technical consultation’ will take place in 2016, whilst the Office for National Statistics will review the robustness of potential data sources. This approach fails to take account of the fact that how we choose to measure quality will end up defining quality. This is because institutions will seek to improve their performance on these measures, which then become the embodiment of quality. Thus rather than being a technical issue the selection and use of measures will end up playing a key role in shaping teaching across the sector. They need to be the subject of rigorous debate and discussion of the kinds of quality and institutional gaming that they are likely to encourage.

Whilst the Green paper does provide some principles that will inform effective measures (they need to be valid, robust, comprehensive, credible, and current), there are key principles missing if the TEF is to lead to valid and equitable judgements. These are that any metrics:

  • are based on the established research evidence about what leads to high quality teaching and learning in higher education;
  • are measures of the quality of teaching offered by institutions rather than measures of institutional prestige, such as entry grades and graduate earnings;
  • require improvements in teaching practices in order to improve performance on the measure;
  • as a whole form a coherent set of metrics rather than a set of disparate measures.

5. It is not clear how the TEF will lead to enhancements in teaching quality

One of the primary aims of the TEF is to improve the quality of teaching in universities. In order for this to happen the system needs to be designed so that the assessment and measurement of quality lead directly to changes in teaching practices. Without this careful design the TEF is far more likely to lead to institutional gaming than changes in teaching practices. However, the Green Paper says nothing about how the system will be designed to link the measurement of quality to the enhancement of quality.

6. There is too much focus on outcomes at the end of, rather than during, the degree

Most of the potential TEF metrics discussed in the Green Paper focus on students at the end of their degree programmes. However, there is plenty of evidence that it is the first year experience that is crucial in student retention and success. This means that the TEF will not provide important information to students about how institutions design the first year experience in order to support students’ successful engagement with higher education.

7. It is not clear where postgraduate research students are located in the proposed new architecture

The Green Paper proposes the separation of HEFCE’s teaching and research elements between the Office for Students and a research funding body. What it is not explicitly addressed in any way is how postgraduate research students will be situated in this separation. The current proposals seem to imply that both the Office for Students and the new research funding body will have some responsibility for the quality of the postgraduate research student experience within this new architecture. This is likely to lead to increased bureaucracy in this area rather than a streamlined architecture.

8. It is not clear who will be responsible for universities as institutions

The splitting of HEFCE’s functions between a student and research body means that there will be no arms length body that has responsibilities for universities as institutions. This would be a major loss for the sector and is likely to make the handling of institutional crises much more complicated and bureaucratic in the future.