  |
EDITORIAL
Vol. 22 No. 1 March 2006
Recent changes within the New Zealand tertiary sector around the introduction of a performance-based system for funding research in degree-granting institutions have the potential to adversely affect both nursing scholarship and nursing education. The concept of a performance-based research fund (PBRF) to reward and encourage excellence in research and to ensure research continues to support degree and postgraduate teaching is a positive one. However measuring quality in a comparative way across a diverse range of disciplines requires strategies and measuring tools that are broad enough to recognise the unique differences and contribution of each discipline. I argue that in its current form the PBRF system is narrow and unbalanced in some aspects and that as a result nursing, along with other emerging disciplines, is being disadvantaged.
The background to PBRF began with changes to the Education Act in 1989 which established a new Tertiary Education Commission (TEC) along with the New Zealand Qualifications Authority (NZQA). One of the aims of the Act was a more seamless tertiary sector, and to achieve this Polytechnics were given the right to award degrees as approved by NZQA. This authority had previously only belonged to the country’s (then) seven universities. From 1992 nursing in New Zealand progressively adopted degree programmes to lead to registration. Entry to the profession is now only by degree.
The 1980s to 1990s saw increasing student numbers within the tertiary sector. This trend was accompanied by reduced funding, (a fall of 15 per cent per head over the decade), and a growing contribution required from the student (National Committee of Inquiry into Higher Education, 1997). Funding was based on a per student formula, using the term ‘equivalent full time student’ or EFTS funding, and depending on the category of the course a sum was paid to cover both teaching and research (and all other costs). How this money was apportioned was at the discretion of the institution, resulting in differing support for research and scholarship.
The Tertiary Education Advisory Commission (TEAC) in its November 2001 report, Shaping the Funding Framework, recommended the introduction of a performancebased research fund for tertiary education providers. In July 2002, a PBRF Working Group was established and their report, Investing in Excellence was delivered in late 2002, with Cabinet endorsing the recommendations in December 2002. These have formed the basis for the implementation of the PBRF. The aims of PBRF, such as to ensure that research continues to support degrees and to increase the quality of research, are based around a funding model that reduces the EFTS sum and establishes a research fund that is then distributed to “reward and encourage excellence’’ (TEC, 2003, p.13). There are three elements to the funding formula, 60% for quality evaluation, 25% for postgraduate research degree completions and 15% for external research income.
The basis for the quality evaluation is an evidence portfolio from each staff member which is rated in terms of their research outputs (ROs) and research contribution.The first round of quality evaluations occurred in 2003 and 22 of the eligible 45 tertiary organisations participated. These were the eight universities, two (of 21) polytechnics, four colleges of education, one wänanga and seven private training establishments. The evaluation process was a massive task. Over 8,000 staff members were assessed (TEC, 2004). Participating tertiary organisations undertook an initial assessment of the evidence portfolios (EPs) and those of a suitable standard (5,771) were submitted to TEC for assessment by one of 12 peer review panels representing 41 designated subject areas.
The EPs were given a rating of A, B, or C with an R status attached to those who did not achieve the standard. A high 39.9% were deemed as R, that is they have not yet achieved a “C” category. Nursing did particularly poorly with not one nursing academic achieving an “A” category (design and nursing were the only two subjects that did not receive an “A”) and nursing, with only three B rated people, was overall considered to have low levels of research quality or activity (TEC, 2004). As a degree based practice profession with increasing numbers of nursing professors and postgraduate students, this begs the question, is there an issue within nursing or is there a problem with the measurement processes used in PBRF?
The answer is something of both. Many would argue that as nursing is a relatively new profession, there is a strong need for more nursing research to underpin our knowledge base and that not enough nurses are writing for publication. It is particularly frustrating that many postgraduate students do exciting work that is never shared through publication beyond their thesis. We all know that there is an obligation to both participants and the profession to share new information in a range of forums, yet publication is still a scary option for many exhausted from the combined pressures of employment and study. As a profession we must continue to seek ways of supporting and encouraging both research and publication.
However it is essential that the assessment process is equitable across disciplines and subject areas as by 2007 all funding for research quality will have been transferred to PBRF, (an estimated $185 million) and then will be distributed by PBRF. Funding is distributed on an institution basis depending on the overall research quality of the individuals within that institution. Already there is anecdotal evidence of PBRF outcomes being used for and against individuals in employment opportunities so there is likely to be some flow-on effect to each discipline. It is also naïve to believe that disciplines that contribute to a greater share of PBRF money will not get greater support from their institution. Some changes have been made following an evaluation of the 2003 round (TEC, 2004), and it is positive to see
a category now included for emerging researchers. Yet with the second round of quality evaluations about to start there are still three important areas that in my opinion stand to disadvantage nursing as well as some other disciplines.
The first is the continued use of journal impact factors as a measuring tool. Impact factors are a flawed measure of rating a journal (rather than an individual article) through counting citations. The ISI ranking system is critiqued in the following reprinted article which explains its genesis from a tool for librarians to its use (and abuse) in today’s academic community. The major point for nursing is that currently ISI ranks only 39 of 547 CINAHL listed nursing journals and even these 39 have low rankings. Recently, after much lobbying, ISI has increased this by three journals and anticipates expanding their “coverage of nursing by about 20 journals in 2006” (Freda, 2006, p. 59). However the reality for nursing is that impact factors do not reflect the usefulness and impact of knowledge within the nursing community so their continued use for PBRF relegates nursing to a low score or rating. Additionally the pressure to publish in the few impact scored journals risks either pushing nursing knowledge into non nursing journals or else encouraging the demise of our broad
range of small speciality journals. Both would be detrimental to the profession. Little impact credibility is given to the large numbers (11 million worldwide) of practicing nurses who potentially read the wide and diverse range of nursing literature available to inform their practice.
Alongside the use of impact factors is the emphasis given to international publication. While this criteria is not stated explicitly in the ‘Guidelines 2006’ (TEC, 2005) this was identified in feedback sessions from the 2003 round. Ten principles guide PBRF (TEC, 2005, p. 13) and one of these, “comprehensiveness”, states “PBRF should appropriately measure the quality of the full range …..within the sector, regardless of its type, form or place of output”. Another principle is “cultural inclusiveness:….. and include the full diversity of New Zealand’s population” (p.13). Compliance with these principles mean that publishing within New Zealand should not only be rated and valued, but that often it is the best way to be culturally inclusive and appropriate. I was surprised to hear that the old adage of “its from overseas so it has to be better than what we have here” is still supported. Admittedly this appears to have greater weighting in some panels rather than others. That brings me to my third and final point.
Nursing must be submitted to the health panel and this panel appears to have the most draconian requirements of all the 12 panels. The health panel covers a broad range of subject areas including dentistry, nursing, pharmacy, sport and exercise science, veterinary studies and rehabilitation therapies such as physiotherapy. Despite the breadth and diversity of staff submitting to this panel the criteria for assessment appears to be the most specific. For example the expectation of the standard of evidence to be supplied states “research outputs will normally be peer-reviewed journal articles describing research studies. While other output types will be considered on their merits…a staff member should explain why these have been chosen as Nominated Research Outputs (NROs) instead of peer-reviewed journal articles” (TEC, 2005, p. 98). By contrast the education panel states “much of the work in education is designed to inform professional practice. Such work is entirely appropriate for consideration” (p. 86) while the social sciences panel states it “will include peer review for journals (including, where appropriate, on-line and e-journals), referee reports for books and conference papers and other equivalent” (p. 128). The health panel also states that it “expects that the majority of peer-reviewed journal papers will have multiple authors” (p. 99). Consequently nurses, and other subject areas submitting to this panel, are automatically disadvantaged. Not only does the health panel have restrictive criteria for outputs, the number of outputs required is also greater. To achieve a Tie-point of 6 (the highest) the health panel expects staff to provide evidence “of a major contribution
Nursing Praxis in New Zealand Vol. 22 No. 1 2006 Page 5 to four NROs and a minimum of 16 ROs published in major well-recognised journals” (p.100). By comparison the education and social science panels have no such quantity expectation and refer to the general tie-point guidelines on page 165 which focus more on the quality of the output rather than the quantity. In the 2003 round the health panel had a 67.6% R rating, meaning less than 33% gained any rating at all. In the 2003 evaluation (TEC, 2004) the health panel was identified as having the lowest ratings. This then does not fit with another of the ten guiding PBRF principles which is “consistency: …PBRF should be consistent across the different subject areas” (p. 13). Surely TEC has an obligation to explain how such different criteria can be reconciled when it has a clear principle of consistency.
Critique of PBRF both as an assessment of quality and as a funding model is increasing. There are also potential impacts on education. Small (2004) argues that PBRF risks creating a gulf between junior academics with high teaching loads and little time for research and senior researchers who can attract outside research grants and who do little or no teaching. Or it may lead to a common overseas model where students end up being taught by contract teachers and have no direct contact with academics.
While in principle I support quality assessment of research and scholarship, it has to be remembered that PBRF is primarily a funding method. The PBRF focus on comparative evaluation for funding allocation may not be the optimum research quality assessment process. The weakness is that the criteria on which assessment is based are not necessarily valid across disciplines. What we can gain from PBRF is a strong reminder on the importance of publication and that nursing, like all disciplines, needs to remain focused on quality outputs. What we also need to do is challenge the inequalities within the PBRF model, most particularly its reliance on the inappropriately used impact factors to rate uptake and impact of information, the unjustified elevation of international publication over New Zealand publication and the lack of consistency in requirements and expectations between subject panels.
Tina Smith, RN, BA, PGCert, MCNA(NZ)
Editor
References
Freda, M. (2006). Don Quixote, David and me. Nursing Outlook, 54(1), 58-59.
National Committee of Inquiry into Higher Education. (1997). Report of the National Committee of Inquiry
into Higher Education. Retrieved March 4, 2006, from http://www.leeds.ac.uk/educol/ncihe/
Small, D. (2004, April 3). Funding system threatens teaching and knowledge itself. The Otago Daily Times,
Retrieved March 4, 2006, from http://www.educ.canterbury.ac.nz/people/small_pub.shtml#funding
Tertiary Education Advisory Commission. (2001). Shaping the Funding Framework. Wellington: Author.
Tertiary Education Commission. (2002). Investing in Excellence. Wellington: Author.
Tertiary Education Commission. (2003). Performance-based research fund: A guide for 2003. Wellington:
Author.
Tertiary Education Commission. (2004). Performance-based research fund: Evaluating research excellence:
The 2003 assessment. Wellington: Author.
Tertiary Education Commission. (2005). Performance-based research fund: Guidelines 2006. Wellington:
Author.
|