Reflections on The Future of Evaluation in Health and Social Care Symposium

• Embrace complexity in evaluation. Portrayed by the black elephant standing in the pool of complexity (pg. 1 bottom left of the visual minutes), the image signifies the concept that both evaluation and policy-makers often seek to ignore or downplay complexity even when the complexities are anticipated, widely reported, and significant. Mike Kelly emphasised the importance of this to policy making it clear that we need to move away from seeking ‘easy answers’ and simple solutions to the inherently complex problems in health and social care evaluation and simple solutions to.  

• Recognise the limits of silos and hierarchies (disciplinary, evidence, and methods) and engage in collaborative, inter-disciplinary working to determine not only ‘if’ but also understand ‘how’ interventions or policies work (as represented by the ‘black box’ at the top of pg. 1 in the visual minutes) in order to develop ‘practical but contingent advice that is useful in the real world’. The visual minutes highlight moving away from a hierarchy of evidence, and instead towards a toolkit of evaluation approaches, each fit for purpose when used to answer the appropriate research question (pg. 1 top left of the visual minutes).  

• Don’t scrimp on the scope a term coined by our colleagues at BRACE (illustrated by the telescope pg. 2 of the visual minutes) to emphasise the importance of allocating sufficient time and resources to understanding and defining: what it is being evaluated, the range of perspectives and stakeholders involved, the evaluation questions that need to be addressed, and the evaluability of these questions based on the availability of historical and prospective evidence and data (among other things).  

• Learn from the past to guide the present and future avoid institutional amnesia (represented pg. 1 bottom left of the visual minutes), by documenting and synthesising previous evidence to learn from past mistakes, contextualise past decisions (recognising that these don’t occur in a political vacuum), and move toward a model of cumulative learning (illustrated as Ray Pawson waving farewell to the accreditation (of programmes) train, and thus moving away from pass or fail verdicts and towards cumulative learning, pg. 1 middle of the visual minutes).  

• Seek alternative explanations and be iterative in our designs to allow further exploration. Rachel Meacock used the example of community healthcare (pg. 2 top left of the visual minutes), to illustrate how different data and levels of data analysis can tell different stories, and illustrated the importance of theorising possible causal contextual and mechanistic explanations of how and why this might be to make sense of the results and inform future data collection; highlighting the need to theorise and to move between the empirical and the abstract to do so (pg. 1 bottom left of the visual minutes). 

• Take risks in commissioning and conducting evaluations including those borne by universities (employ researchers on secure contracts), funders and funding committees (fund larger interdisciplinary teams, longitudinal studies, and novel and live evaluations, employing stop/go criteria and staggered funding), and policy makers (accepting uncertainty in decision-making). represented by the checklist pg1 middle) (illustrated pg. 2 of the visual minutes). 

Each of these themes are reflected in, and will continue to inform, our approach to evaluation in IDEAS-NET. 

Thank you to everyone who attended or was involved in the delivery and organisation to make the event a success. We are overwhelmed by the positive feedback we have received. Watch this space…