Sarah Whalley and Candy Rowe from Newcastle University follow up their keynote at our Transforming Research Culture symposium 2024 by sharing their reflections on evaluation.
Discussing our values-led approach to research culture evaluation with peers from other institutions following our keynote at Advance HE’s Transforming Research Culture symposium, has enriched our thinking about evaluating research culture. Here, we share our post-symposium thoughts that will guide us as we reflect on and refine our measures.
Responsible evaluation
In 2021, we were tasked with ‘developing a basket of research culture measures’ to add to our suite of Research Strategy Key Performance Indicators (KPIs). At first, we hesitated. We recognised the challenges and risks associated with the metrification of research culture – we certainly didn’t want a box-ticking exercise. Research culture is also something that cannot be easily measured with numbers and figures. We have all heard the phrase: “not everything that counts can be counted”.
With the guidance of Lizzie Gadd and Tanja Strøm, we embraced the INORMS SCOPE process for evaluating research culture and the changes we wanted to make. This approach allowed us to begin our assessment by focusing on what truly matters. We engaged in conversations with our research community across disciplines, job families and career stages to define the features of a positive research culture. From this foundation, we collaboratively developed a values-led framework around four key attributes that informed both our research culture action planning and evaluation methods.
We’ve already discussed how we developed our Research Culture Index and the measures we finally included in our WonkHE blog. The SCOPE process provided the framework to develop measures that align with what we value, and probe them so they would be meaningful, feasible, and avoid any unintended consequences. Given the scale and complexity of our research culture programme of work, we recognised the need for multiple indicators to convey the breadth and nuance of our progress. We hoped that they would help to narrate a story of change over time. But can they?
Refining our measures
After three years of adopting our measures, we’re now entering the final stage of the SCOPE process. ‘E’ stands for Evaluation and involves evaluating your evaluation by assessing whether it has achieved its goals. As we prepared for our keynote, we started reflecting on how our measures are functioning within our institution, and how they will fit with future changes in our internal and external landscape. Here we share some reflections that will guide us in the coming year:
Realigning with our Research Strategy: do we need to revisit the ‘what matters’ for strategy as a whole?
During the symposium, a recurring theme was how we make research culture integral to higher education institutions through alignment with wider priorities. Following our keynote this was a particular feature of Julie Bayley and Jo Richardson’s interactive session on the role of impact leadership in sustaining a positive and supportive research culture. We all know that building a positive research culture supports wider strategic goals – from grant capture to creating an attractive environment for students and colleagues alike. But we agreed we can do more to weave the values of our research culture ambitions into the rich tapestry of institutional strategic planning.
In the Q&A session following our keynote, we had an incisive question about how we balance the values of our research community with the values of our institution, and how we ensure the values of the research community stay on the table. At Newcastle University, research culture is one of our top three priorities within our Research Strategy that is a core chapter of our wider Vision and Strategy.
This academic year, we’ll lightly refresh our Research Strategy, applying the SCOPE process to re-evaluate our KPIs and refine our research culture metrics. One thing we’ll be considering is how the values of the research community correspond to the values of the institution in the research strategy context – from financial sustainability through to research quality, we’ll be putting the ‘what matters’ back under the microscope. We’ll also revisit the scalability of our measures to ensure they translate well at different levels within the institution, and that they complement the indicators of our other cross-cutting strategies.
Reviewing selected metrics: do we need to change any?
In the context of our Research Strategy refresh, we will be asking: are our chosen measures effectively evaluating the change we want to see? As we prepared our keynote, we discussed this, and a few measures jumped out at us as ripe for review. One example was the proportion of grant holders who identify as being from minority ethnic backgrounds.
From some of our data analyses, in line with that from a wide range of funders, we now recognise that inequities lie in the success rates of applicants. This particular measure isn’t completely in our control, with biases in funder selection processes also potentially contributing. Although this is an issue, our Action Plan is addressing it locally, and externally, through collaboration with the Equity and Inclusivity in Research Funding Forum.
From a metrics point of view, the question for us this year is whether we need to re-focus our metrics on the success rates rather than funding rates, and whether a metric that lies only partly in our control is an appropriate KPI.
Expanding evaluation options: do we need to add any further indicators?
Have changes in the internal or external landscape opened up new opportunities for evaluation? Initially, we excluded some important potential measures – in some cases, our systems simply could not easily capture them, or they weren’t sufficiently tightly linked to the anticipated change. However, new projects, and dedicated colleagues working in them, allows us to reflect on these. For example, could our new Current Research Information System (CRIS) help us capture measures for collaboration? And does deeper thinking about tackling inappropriate behaviours in our priority ‘Behaviour Matters’ project lead to more informative indicators of behaviour change?
We’re currently looking to apply Theory of Change across our priority projects that could provide insights into better evaluation options we could use.
REF2029: How will this change the evaluation landscape?
Finally, like many in the sector we’re keeping a close eye on the outcomes of the REF2029 consultation on the new People, Culture, and Environment (PCE) indicators. We see the PCE as a welcome development that is boosting community engagement with our research culture initiatives. We’re eager to use this momentum to foster a shift in research culture. And our approach at Newcastle University is clear: we focus on building our research culture to enable our strategy and empower research communities, and trust that a strong REF result will naturally follow.
Having said that, we’re mindful that for this approach to work for us, our values-led approach and the approach developed through REF need to align. We’re optimistic about this, particularly given that the development of REF PCE indicators is also following the SCOPE process.
We’re excited to be joining the REF2029 PCE Indicators pilot this Autumn to see how we measure up. We’re hopeful that REF will allow universities to choose what works best for them and encourage progress in research culture that fits each university’s unique history and context. Measuring the journey travelled will enable us to continue to work on and progress projects we’ve already started, and each university to tell their own story.
Changing research culture is a team effort that spans the entire research ecosystem. It’s not about competing but about learning together and discussing successful approaches together in order to improve research culture for everyone. We’re keen to continue the conversation.
Candy Rowe is Professor in Animal Behaviour and Cognition at Newcastle University, and also Dean for Research Culture & Strategy there. She has responsibility for enhancing research culture across the institution, and the programme of work that she now leads on includes a priority project to release quality time for research. Over the last 10 years, she has held a series of leadership roles, with a particular focus on researcher career development and equality, diversity and inclusion.
Dr Sarah Whalley is Research Strategy Manager at Newcastle University where she works with senior leaders and colleagues to implement and develop the University’s Research Strategy. She drives forwards transformative change initiatives, including the Newcastle University Centres of Research Excellence, which aim to address global challenges through interdisciplinary collaborations. Her team are responsible for the delivery of various initiatives within the research strategy, including the Newcastle University Academic Track (NUAcT) scheme and the University’s Research Culture Action Plan.
