Viewpoint: QAA Reviews; Fact or Fabrication

Introduction

This article seeks to demonstrate, through relevant literature and worked experience, how QAA Higher Education Reviews for Alternative Providers (QAA HER(AP)) promote and nurture institutional and individual academics collusion in ‘fabrication of performance’ (Ball, 2001; Strathern, 2001). It can be seen as a necessary piece of action to survive the current quality surveillance climate and the ‘grisly reviews’ (Inglis, 2000). The theoretical framework is drawn from a critical policy analysis perspective. Suffice it to say, this is a personal viewpoint drawing on my thirty years’ experience in higher education of which the last twenty has been in senior academic management and being involved in accreditations and validations across the globe. In the last five years, I have been a QAA reviewer for REO and HER as well as qualified to hear institutional appeals against negative QAA judgments. In a legal situation, the criminal will lie on oath to avoid conviction. In the academy, Ball (2001) argues that academics rationally, lie and fabricate, when being pursued by internal or external quality auditors and are actively encouraged to do so by institutional management. It is hardly surprising that the institution behaves in this way. By fabricating academic standards and teaching and learning quality, funds and/or public kudos are potentially available, if the institution is able, through both internal self-evaluation and QAA inspection, to present and represent improved performance. For institutional management and academics alike, published evidence of poor performance, has to be avoided at all costs.

The Alternative Provider

For the Alternative Provider, generating less resources the need to evidence ‘improved performance’ is crucial to avoid the threat of a QAA lack of confidence, culminating in student recruitment being frozen by HEFCE or possible suspension of the College from any recruitment. Ball (2001) suggests, ‘the fabrications that organisations (and individuals) produce are selections among various possible representations or versions of the organisation or person. They are not made in a political vacuum or unconsciously but ‘are informed by the priorities, constraints and climate set by the policy environment’ (Ball, 2001: 216).

According to Walker (2001), there is a new cultural logic governing professionalism and its relationship with fabrication and academics have now found their performance and professionalism under scrutiny for quite a while. In these days of performativity, the reproduction of knowledge, applications and skills is not enough. Academics are expected to show entrepreneurial flair, to add value and to help the organisation both recruit and retain its ‘customers’. In order to achieve this multiplicity of aims, fabrication of quality outcomes is essential for Alternative Providers, as institutions have to look sound in the public domain albeit the public has little comprehension of the meaning of a QAA report.

Suffice it to say, the work academics engage in, has now been completely redefined as ‘marketable commodities’ (Morley, 2008) and the days of academic independence may be gone as management take over control of teaching, learning and assessment through the formation of a Teaching, Learning and Assessment strategy (Knight, 1998). This is juxtaposed with institutional strategy, business plans and mission statement. It is noteworthy that even today many senior managers are unaware of the power they hold through their right to produce a TLA strategy and they often make a poor fist of it. Suffice it to say, they are producing strategies that are gradually achieving management hegemony around teaching, learning and assessment, eroding the individual academic and his/her rights to determine how to do the job and measuring programme delivery and individual subject delivery. This managerialist approach and their right to manage teaching, learning and assessment is accepted and fully reinforced by the QAA HER(AP).

At the same time as Strathern (2000) suggests management are keeping central records of all modes of assessment and constantly emphasising ‘over teaching’ or ‘over assessment’, usually based on the assumption that students being asked to write an essay or do a presentation etc. more than once, is too much and contributes to high dropout rates. Alderman (2017) eludes to this point. Incidentally, reducing over teaching and over assessment has job threatening overtones. Morley (2008) states:

Quality assessment, accountability and the auditing of academic work have had a profound impact on reconstructing academic conditions of work and academic identities. The academic ‘habitus’ has been challenged. Academics have to be simultaneously self-managing and manageable workers ‘who are able to make themselves auditable within prescribed taxonomies of effectiveness’ (Morley 2003: 68).

In contrast, writers including Knight (1998), Jackson (2000), De Vita (2002) and Mutch (2002) argue otherwise presenting an alternative perspective. These writers have put forward arguments around approaches to assessment and effective assessment strategies that consider the multi-cultural student market, particularly appropriate today and this cannot be ignored.

However, for Thrupp (2003) they fall into the category of textual apologists. Knight (1998) and Mutch (2002) and have encouraged institutions to have learning and assessment strategies, enabling the re-direction of resources by mapping programme and unit learning outcomes (Jackson 2000), to assessment. Jackson (2000) put forward the idea of programme specifications, which are now common in every institution, akin to a brief motorcar brochure, available when one enters the showroom. The programme specification aims at selling the product to a student, re-defined as a customer and what is attractive, is it takes about five minutes to read. Thus, it’s a quick sell.

De Vita (2002) suggests that assessment should come through coursework rather than through unseen examinations as the examinations disadvantage multi-cultural students and reduce pass rates. This has brought about assignment writing factories that flourish today. Apologists constantly harp on about detecting plagiarism without acknowledging they are the cause of its rapid increase. Alderman (2017) suggests universities drop grades; all aimed at attracting more customers irrespective of falling standards. A concern for Alternative Providers as this effectively dips into their market. One can only see further fabrication at admissions to survive. Yet, most of these apologist writers argue that the concern must be for the student as a customer and they reflect on how we might deliver a more worthwhile student learning experience. In reality, it’s a concern for money and it is this sort of textual apology that has directed the approach to institutional review by the QAA. Let us explore how the approach to quality has changed over the last few years.

Originally, quality control was linked to the production values of manufacturing industry, employing the maxims ‘fitness for purpose’ and ‘pass or reject’ rates before the term ‘assurance’ became fashionable (Deming 1986). It checked ‘the quality’ of nuts or bolts, washing machines and cars. It has moved from the engineering shop floor to the academy in the space of about twenty years. It is highly adaptable to environments and Strathern (2001) argues that it has become a ‘freestanding system that can be injected into new organisational areas’. Whilst it may be useful and no doubt important to quantify standards and output of objects produced in industry, it is hard to see how this can readily be transferred to measuring the complex, qualitative output of an individual academic, let alone to create a valid and reliable measure of total institutional output.

Power (1997) states that it is difficult to see how quality assurance can be used as a tool in the production of ‘governable persons’ that might include academics. At the same time the QAA approach to review is largely desk-based and offers a brief snapshot of how an institution operates between two to five days [QAA HER(AP) Manual 2016], but never fully understands the workings of the Alternative Provider on a day-to-day basis. The model it operates is simplistic; has never been interrogated thoroughly and is cheap to operate.

Le Grand and Bartlett (1993) focussing on marketization, has been important work and Alternative Providers are constantly required to demonstrate value for money and run programmes with unrealistic resources per student head, being capped at £6000⁴ fees per student. This compares unfavourably with £9000⁵ per student in a university. When one takes the per student capita research funding on top, the realistic figure is closer to £20,000. Suffice it to say, terms from private enterprise such as ‘customer’, ‘marketplace’, ‘efficiency gain’ (Trowler 1998), are commonly used and competition between HE providers is encouraged, through the publication of tables, as well as with the selling of academic programmes of study.

Colleges are now less likely to share ‘good practice’ seeing their fellows provides as competitors. Smith and Ladlum (2001) state that staff in higher education are under pressure across the spectrum to do more work, but with fewer resources and this certainly applies to Alternative Providers. They have even less resources to draw on. In this way, it becomes virtually impossible for an Alternative Provider to compete on a level playing field when they are hugely disadvantaged once they enter the field of play. They cannot attract quality staff as they can neither afford the salary nor offer attractive pensions. Thus, they go for the cheaper option of hiring part-time staff or retired academics that have been discarded by the universities.

Pressure on the staff to perform within a framework of diminishing resources comes from within the institution and without. It would be fair to suggest that in days gone by they deserved their negative reputation as unscrupulous owners were keen to ‘take the money and run’. These days are long gone and there needs to be a greater trust of the Alternative Provider struggling in today’s HE climate.

Quangos such as the Quality Assurance Agency (QAA) and HEFCE control resource allocation and rankings. The QAA HER(AP) approach focuses on institutional self-regulation and the method changes every four or five years. A set of events at Warwick University in the Economics Department reported in the Guardian in January 2001 made interesting reading and pushed the QAA to embark on a more ‘softly-softly’ approach. Although the economics department at Warwick scored twenty-four out of twenty-four in every category of assessment, the department dismissed the results as ‘fictitious’ and stated that the method of self-evaluation has ‘nothing to do with quality’ (see Morley 2003). In fact, it is best remembered for its contribution to the felling of trees to generate more paper.

The modern method that focuses on institutions self-evaluation, is equally dubious as to validity, as panels are made up of different personnel and the QAA make no attempt to standardise outcomes. It is the luck of the draw whether a college gets positive or negative reviewers. The universities will now be able to avoid this scrutiny as fully self-evaluating bodies but Alternative Providers will continue to be subject to reviews that are dubious in terms of validity, reliability and generalisability.

Given the under-resourcing of Alternative Providers, it is little wonder that, as Ball (2001) states, through fabricating our quality we are perceived as efficient, able and as ‘part of the knowledge economy’. Bartlett (2003) suggests ‘we have an ideology of quality running through every academic institution’ and McWilliam and Hatcher (1999) argue that there is something quite ‘seductive’ about the terms used in quality assurance such as ‘peak performance’ and ‘excellence’. Today, we would include ‘enhancement’ ‘employability’ and ‘embeddedness’ [QAA HER(AP), 2016]. It is hard to imagine what the in-words will be in another five years if the QAA manages to survive but it is certain someone will come up with some nice Orwellian phrases.

Morley (2008) sees quality assurance “contaminating” departments and institutions with bureaucracy. As institutions create their own benchmarks on which to be measured by, deliver the programme, have QAA reviewers from other academic institutions monitoring output, [themselves’ having been regularly subjected to their own external reviews], it is difficult to see the full validity and reliability of the current quality assurance and enhancement systems. When reviewing academic teaching quality for example, some institutions, particularly within the new university sector, may be slower than others to realise that consumers value academic standards in terms of numbers of firsts and upper-seconds achieved, rather than the academic meaning and value behind the classifications. In that respect, finding methods to manipulate assessment outcomes, by adopting predictive assessment models, within assessment strategies (Mutch, 2002), operate as a clear institutional way of fabricating student performance, giving the appearance of raising standards. In order to do this, the institution simply analyses past results and then changes assessment weightings to increase pass levels. It is common practice when management realise the power they hold through control of the assessment strategy.

Furthermore, it is suggested another major issue centres on organisations becoming more transparent and representations of ‘transparency’ are becoming ‘highly sophisticated and complex, requiring specialist expertise’ (Ball, 2001; 1997). Strathern (2000) writing on what she suggests is the ‘tyranny’ of transparency suggests that ‘techniques for assessing, auditing and evaluating institutions are often defended on the grounds of transparency’ and suggests that it is all too easy to identify the ‘tyrannous’ side of what is presented as a benevolent and moral system. She suggests that in striving for, ever more sophisticated simulation and modelling techniques of measurement, the modellers are pushing the boundaries ‘beyond reality’. She asks whether we are talking about an instrument of knowledge or an instrument of control.

What is being tested seems to be the performance and productivity of academics, but ‘everyone’ knows that what is being tested is how amenable to auditing their activities are or how performance matches up to performance indicators. As the term accountability implies, people want to know how to trust one another, to make their trust visible, while (knowing that) ‘the very desire to do so points to the absence of trust’ (Strathern, 2000: 310).

In the pursuit of ‘transparency’, quality experts continuously reproduce a need for their skills with the development of ever more elaborate systems. The QAA’s move from REO to HER(AP) is a prime example. It is never ending. In effect, another ‘ivory tower’ is created to house quality assurance expertise and more complex methods of fabrication are necessary to ‘perform’ successfully. Models for measuring quality assurance change on a regular basis, whether it is OFSTED or the QAA and there is money to be made by QAE Officers operating more and more elaborate quality systems and writing about them.

In the current climate, many more tutors and course managers are using and interpreting simple statistical models of measurement, as well as employing quality officers who are adept at manipulating them or ‘spinning’ statistical argument, to convince external examiners of ‘improvements’ or accrediting agents for the purpose of achieving validation or passing formal monitoring events. As evidence, I have undertaken at least 30 validations or institutional accreditations in higher education some of which I have chaired and examined a range of substantial institutional documentation and entered in-depth dialogue with senior management and academic staff and I conclude that this can sometimes be the case.

Quantitative performance representation now in vogue with the collection of data under HESA is very attractive and a powerful tool for management as a simple way of convincing the public. Ball (2001) distinguishes between ‘trivial’ or what he defines as ‘representational’ fabrications and those that he describes as ‘constitutive’ or consciously organised. The presentation of statistics to show a ‘Performance’ of some sort would fall into the latter category.

Ball (2001) provides further useful examples of institutional fabrication, referring to a surgeon not performing a life-threatening operation, in order to maintain survival targets; the encouragement of underscoring in early school tests, to enable performance enhancement to occur, later in the year and busses missing bus stops to achieve punctuality targets. Rehearsals for QAA HER(AP) visitations are also common. In one of my recent institutions preparing for a QAA REO visitation, we spent a year ‘learning our lines’. Nothing has changed. In my present institution passing the QAA is all about learning the lines and practicing them until competent. It is an exercise for parrots.

As a quality manager myself, in the past I ensured the staff were ‘well trained’ before an important validation, ensuring that come the real event, any ‘rogue’ tutors were encouraged not to attend. The fear of going down in a QAA review can be seriously problematic and leading to possible institutional closure. Yet each QAA HER(AP) is different and lacks a level of predictive validity. Success depends on the quality of documents you present; how well one has been coached into answering the questions; the members of the panel you get; each member being different in attitudes, values and beliefs.

If an Alternative Provider has a panel with no members from Alternative Providers the college could be in trouble. There is absolutely no consistency of explanation for one panel passing a college and another failing and of major concern is the complete lack of standardisation. Imagine the ‘furore’ if students’ work was assessed in the same way. One finds HER(AP) outcomes where two colleges with the same type of provision are judged differently. I know one example where one review panel deemed UKRC B10 (collaborative arrangements) to be irrelevant to the outcomes and another was failed on B10, where both offered HND and university accredited programmes. At the same time, I have evidenced poor SEDs⁶ where the college has passed and competent SEDs where the college has failed. The review system needs a root and branch overhaul.

Why is QAA Review so inane? Effectively, prior to review, the QAA Review Coordinator comes to the college to tell it how to proceed. Following this, the QAA panel spends a few days reading all the Alternative Provider’s documents; meets together for a day to tidy up the paperwork and prepare questions; comes into the college for a few days to have an hour’s discussion with senior managers, academics, administrators and students, receiving what is no more than a snapshot of the college and spends the next day deciding on outcomes. As Inglis (2002) states, some of the reviewers are ‘drawn from colleges that are themselves failing’ and passing judgments on others.

When I worked as an external on the Open University Validation Committee between 2000 and 2008, its initial approach to the institutional accreditation and review and the validation of programmes of study with private providers, was similar. Open University Validation Services had drawn on the old Council for National Academic Awards (CNAA) model of approach to accredit polytechnics. The approach was aptly labelled as ‘big bang’ where a panel read the documents, visited for a few days and delivered a judgment, i.e. the panel comes in, passes or fails you, provides feedback of outcomes and then goes away again. It is a simple system to operate and cheap to run. It basically comes and does ‘quality to you’ (Ball, 2001) and the college learns ‘how to jump the hoops’ (Ball, 2001) and say its lines in the way the panel identifies with. It provides a review based around apologia for falling standards, as institutions move to greater and greater marketisation of the product. The reviews have reworked the role of the student from learner to critical observer and customer. Woe betide if a student criticises the college in front of a QAA panel.

An alternative method, that would be more valid and reliable, might be a process model of continuous development, where a QAA panel-member works for three or four days with the institution prior to preparation for final review. The successful outcome becomes guaranteed with both parties working together, working up the quality of the documentation and owning the process. In the current ‘big bang’ system little wonder that Alternative Providers are inclined to fabricate. Unfortunately, a serious model that works also costs money and the current QAA review method has to turn a blind eye to scarce resources that not only encourages but coerces academics to fabricate.

Supporting the concept of ‘fabricated representation’ (Ball, 1997, 2001; Strathern, 2000; Bartlett, 2003; Morley, 2003), Blackmore and Sachs (1999) suggest that for academic organisations the issue of ‘what is seen to be done, takes preference over what is actually done’ and tutors are learning to express themselves in relation to the performativity of the organisation or, as Ball (2001) suggests, they are becoming ‘adept in the art of convincing exaggeration’. He suggests they are ‘written into being’ with versions of the institution being constructed for different stakeholders and today this would include QAA, HEFCE, and accrediting universities. Ball (1997) argues that ‘documents produced in the technologies of surveillance become increasingly reified, self-referential and dislocated from the practices they are meant to stand for or account for’. In most cases to pass the QAA Review key documents have to be written by experts at writing QAA-speak, though not necessarily institutional academic staff. Writing convincing SEDs; validation documents and annual monitoring reports becomes an industry in its own right. We work ‘on ourselves and on each other through the micro-politics of representation / fabrication, judgement and comparison’ (Ball, 2001: 222)

A dominant discourse of ‘value for money and efficiency’ has emanated as a result and is central to government educational policy. Senior management in educational establishments are adopting New Public Management approaches to the way they run the academic institutions, operating from the perspective of performativity and using quality assurance systems to create what will be publicly perceived to be ‘objective’ and ‘transparent’ measures of the performance of both the institution and individual members of staff. Notwithstanding the fact, that it is difficult to objectify academic output; it is not a Vauxhall car or a Hoover washing machine, academics are fabricating outcomes, presenting a public image of institutional wellbeing to achieve funding, course accreditation and simply protect their own careers. These are necessary behaviours for Alternative Providers to survive. Ironically, as they embark on this misrepresentation, they effectively diminish their own status and professionalism, through the unethical practice of fabrication (Ball, 2001; Eraut, 1994; Hugman 1992), the acceptance of falling academic standards and the abandonment of implied trust and the psychological contract (Inglis, 2000).

In conclusion, the marketisation of higher education, the commodification of programmes and the customerisation of students have brought with it the unintended or unforeseen outcome of fabrication of quality and standards. This ranges from old universities dropping grades for under privileged students, who are then likely to drop out (Alderman, 2017) through to misrepresentation of both standards and quality within the Alternative Providers at QAA reviews. In the latter case, it is not flagrant misrepresentation but it is intentional and deliberate as the only defence to resource scarcity.

Alternative Providers are now being judged with the same benchmarks applied until recently, in the university sector. However, no consideration or mitigating circumstances are offered to account for a £3000 per student fee-income deficit compared to the universities.

In fact, at a meeting at which I was present with representatives of HEFCE and BIS in December 2016, the BIS representative stated that the Alternative Providers have to deliver the same quality as the universities before being judged competent to receive the fee increase to £9000. This is akin to stating the standard of living in a household with two children has to be the same on respective incomes of £20k and £30k before a government might consider awarding benefits to the family earning £20k per annum.