August 2022
In November 2021, an Advance HE integrative literature review was published (Austen et.al. 2021), which explored evidence of what is working to enhance student outcomes (access into higher education, retention and attainment whilst studying, and progression to employment or further study - ARAP). It found that impact reporting could be improved. Whilst some studies were evaluating impact of their interventions on short- or medium-term outcomes – such as learning gain or development of skills or behaviours – there was a lack of further evidence on longer term student outcomes.
Applying pedagogy to evaluation capacity building
The report specifically challenged sector stakeholders to fix the ‘leaky pipeline’ of impact evidence so that research and evaluation could enhance the outcomes evidence base. The review team recommended providing support for those designing interventions and evaluating impact, specifically to support users to complete a logical connection between evidence informed intervention design and student outcomes.
The authors of the literature review began to explore the design and creation of a game which could support better intervention design, evidence gathering, and impact on student outcomes. In addition to creating a resource for promotion of the Advance HE Student Success Framework, this project aimed to employ pedagogic principles to the development of evaluation literacy. How learning about evaluation is facilitated has been a notable omission in English HE sector noise which is increasing expectations for evaluation and impact evidence.
The game design uses a puzzled-based activity approach through the use of double-sided hexagonal playing cards that both informs and challenges game player understanding of evaluation. The hexagonal shape was adopted with a pedagogical purpose. As change is not always linear and can be complex, the hexagons can be placed in any order or pattern decided by the players. The hexagon shape encourages multi-faceted connections between the stages of a Theory of Change. There is no right answer; the conversations about placement and order are the most important to aid learning. Once the theory of change is agreed on by the group, players turn over the cards and explore evaluative questions about how outcomes/impact will be measured.
The ’ChangeBusters’ game
The innovative ‘ChangeBusters’ game, designed by the team from Sheffield Hallam University, constructs a ‘Theory of Change’ based on evidence from within the ARAP integrative literature review. This game aims to support users to understand the stages of a Theory of Change, build confidence, explore evaluative questions, and extend learning through application to their own context. The goal is better intervention design and evaluation, which in turn should lead to better student outcomes.
The intended outcomes for game players are as follows:
-
an understanding of the Advance HE ARAP literature review evidence.
-
support to understand the stages of a Theory of Change.
-
build confidence with theorising change by using pre-determined scenarios.
-
extend their learning through application to their own context.
At Sheffield Hallam we encourage the co-creation of a Theory of Change to enable discussion and critique by a group of people. The ChangeBusters game works best in groups of 4 or more but can be played by between 2 and 8 players. The game is aimed at academic and professional service staff in higher education institutions who are designing and implementing interventions in all areas of the student lifecycle. This might include a course or module leader designing an approach to teaching, an outreach practitioner designing pre-entry summer schools, or a careers advisor designing a mentoring programme. Game players could be existing teams who would like to develop their evaluative literacy or a range of players from different contexts who can bring diverse perspectives to the theorising of change. This game could also be of use for institutional leaders who have responsibility for designing and funding interventions and evaluators who are tasked with supporting practitioners to build an evaluative mindset and/or doing the evaluating.
There are three game scenarios to choose from which have been developed from the ARAP literature review: Financial Support Bursary; Peer Mentoring Scheme and Course-based Undergraduate Research Experience. During user testing it became clear that the scenario did not need to align with the game-players context for change to be explored.
One scenario pack of the ChangeBusters game contains:
-
one ‘Scenario’ card
-
one ‘Theory of Change’ menu
-
one set of 17 hexagonal cards with text on both sides
-
Side A: Intervention
-
Side B: Evaluation
-
five blank hexagonal cards
-
a set of instructions
-
team card
-
‘Principles’ card
-
whiteboard pen
-
packaging for storage and transporting
A Facilitator Pack is also available to download. A facilitator is recommended to support players with their discussions and decisions. The facilitator can also ensure that game playing aligns with the game principles:
In our intervention design and evaluation planning I / we will:
+ promote evaluation as a collective responsibility
+ acknowledge any differences in confidence and experience
+ appreciate different forms of knowledge and evidence
+ pay attention to the views of others, whilst thinking about the impact of what we say
+ welcome supportive critique and challenge
+ consider the needs of others in terms of access to materials and discussions
Game design
The design and delivery of this game emerged over several months. A design team from within Sheffield Hallam University was assembled to utilise the collective expertise of designers and evaluators.
Design sessions were facilitated by James Corazzo, Teaching and Learning Portfolio Lead – Art & Design and Principal Lecturer Graphic Design.
The content and scope of the game was led by members of the original ARAP review team: Liz Austen- Head of Evaluation and Research, Nathaniel Pickering, Caroline Heaton and Jill Dickinson – Senior Lecturers in Research, Evaluation and Student Engagement.
Jason Ruffell and Sam Smith, Learning Technology Advisors were also involved in the scoping of digital designs. Catie Varley and Sarah Graham, graduates from SHU Graphic Design, were employed as designers leading the development of the prototypes, with Catie delivering the final product design. Design sessions scoped potential users, product ideas, tensions and challenges and alignment to the brief via a problem statement. A Miro Board captured emerging designs and decisions.
Next steps
You can access the ChangeBusters game as a downloadable resource here
If you are interested in accessing the physical version of the game please contact consultancy@advance-he.ac.uk