Researcher: Dr Hannah Griffin-James, Quantitative Researcher
The Accomplished Study Programme in Research Excellence (ASPIRE) is a reciprocal teaching programme, providing targeted packages of support and training for both students and academic staff based at Sheffield Hallam University and nearby higher education institutions (within 120-mile round trip of Sheffield) to address the Black PhD leaky pipeline.
The Black PhD leaky pipeline was first highlighted by Williams, Bath, Arday and Lewis in 2019. They reported that out of nearly 20,000 PhDs awarded by research funders, 245 were awarded to Black or Black mixed students, with 30 of those being from Black Caribbean backgrounds.
Sheffield Hallam University (SHU) and Manchester Metropolitan University (MMU) received a large package of support funded by the Office for Students (OfS) and Research England (RE) to design and deliver the programme, and commissioned Advance HE to independently evaluate it.
The aim of the independent evaluation was three-fold:
1) Process evaluation:
2) Impact evaluation:
3) Qualitative research:
This project is ongoing, and we will work with the ASPIRE team to conduct a longitudinal evaluation over four years.
In year one, a theory of change was co-constructed with the ASPIRE team, to identify specific and measurable outcomes and impacts. A mixed methods difference-in-difference design was adopted, including pre- and post-programme surveys, control groups, skill audits, diaries, and interviews. All evaluation materials were developed using the programme’s theory of change, tailoring the measures to capture the outcomes explicitly targeted by the ASPIRE programme.
Outcomes and impact
Through a series of innovative methods, Advance HE was able to evidence what worked well, and what needed to be adapted or changed (process evaluation) to further support the growing impact of the programme for Black students, staff and other students at SHU and MMU.
With regards to the impact evaluation, our evidence showed that the ASPIRE programme improved the experiences, engagement and career opportunities of Black students, finding students were confidently submitting doctoral applications, or had applied for a new role or successfully been promoted at work.
“They’ve upped my aspirations. They have made me realise there is nothing whatsoever I cannot achieve. My colour can never stop me. My gender can never stop me… sincerely, I just want to appreciate everybody that has brought this thing together. They’ve actually changed – and not just me. They’ve changed a whole lot of people. If they can change me, they have not changed me, they have changed my generation and I know my generation would be grateful to them.”
ASPIRE programme participant
Part of our work also involved providing individual anonymised examples to tell the story of ASPIRE to stakeholders. The evidence gathered has been compiled to share the journey of the students and staff in an understandable, useful way that has enabled evidence-informed decisions to be made by the programmes executive groups.
The evaluation is still underway and we are working closely with SHU and MMU to support the sustainable growth of the programme as well as evidence how students and staff experiences have been impacted by the programme.
McMaster Codiroli, N. (2021). Ethnicity awarding gaps in UK higher education in 2019/20. London: Advance HE.
Advance HE (2022). Equality in higher education: students statistical report 2022. London, UK.
About the Researcher
Dr Hannah Griffin-James, Quantitative Researcher
Hannah was part of the Advance HE Insights team from October 2021 to March 2023. She is an evaluation researcher with market research and higher education experience, and is a specialist in advanced quantitative analysis. She is an incisive critical thinker who is solution focused whilst remaining creative. She believes in demystifying research and presenting evidence to clients in a way that will enable maximum impact.
About our bespoke programme evaluations
Our bespoke programme evaluations support institutions to evidence the effectiveness and impact of targeted interventions. By building strong relationships, we support you to use evidence to shape and scale up initiatives.
We offer impact and process evaluation of both innovative or long-standing interventions. There are several types of evidence-based evaluations, using quantitative, qualitative or mixed design to evaluate and demonstrate impact.
- Difference-in-difference: a mixed methods design, to examine causal relationships between the programme and outcomes. Provides direction as to what works well, and what needs to be improved.
- Randomized control trial: a quantitative approach where the learners are randomly assigned to the teaching intervention or a control group (who also receive teaching). Great at identifying causality, helping staff know what is being achieved as a result of the intervention.
- Case study: a qualitative approach where the intervention is studied in depth with the educative activities detailed and analysed. A rich detailed account of the programme which can generate support programme evaluation.
Each method has its own strengths and limitations, which complement each other, when employed in tandem. In this example, we applied this approach to evaluate the impact of teaching programme. However, the difference-in-difference design could just as easily be used to provide an institution with practical insights into the impact of training, teaching and learning interventions as well as newly implemented policies.
Contact firstname.lastname@example.org to learn how our bespoke research consultancy can be applied to your context.