Validated! The positive impact of using validated tools

Posted: 
14 October 2016
Portrait of blog author, Gina Floyd

Back to Expert series blog

In this Expert series piece, Gina Floyd, Evaluation and Impact Officer at RNIB, discusses the positive impact of using validated tools in service evaluations.

Whilst the majority (I should hope) of all social action projects are well intentioned, it is not always easy to prove their efficacy. And as charities, we need good evidence of impact to convince commissioners or funders to provide the resources we need to continue. Given this pressure, it would be easy to be swayed by the desire to use a measure that you know a particular funder favours. However, as evaluators and charities we must be confident that the measures reflect the intended outcomes of our own unique programmes and we are as realistic as possible.

From November 2014 and March 2016, the Cabinet Office and NESTA funded RNIB’s Time to Talk telephone peer-support service. In this time it supported over 1,000 people. The RNIB evaluation and impact team set out to evidence the impact of Time to Talk, working with The Social Innovation Partnership (TSIP) who supported the commissioning and oversight of the evaluation, and validated it against the Nesta Standards of Evidence. We started by carrying out a Theory of Change with the relevant project stakeholders. This helped us to map out the outcomes we expected and to choose validated measures; in this case the Short Warwick Edinburgh wellbeing 7 levels (SWEMWBS) and in house crafted measures to reflect the outcomes.

We used SWEMWBS to measure participants’ wellbeing before, after and 3 months after the Time to Talk sessions. We also measured the wellbeing of a comparison group who chose not to take part in the programme. We were able to demonstrate an increase in all of the wellbeing measures for Time to Talk participants and the impact of peer support upon confidence building, emotional well being and behavioural change. In contrast, the comparison group decreased in all of the areas of wellbeing.

The evaluation report has been used to provide strong evidence to clinicians of the social and emotional wellbeing benefits of peer support projects. The NHS Low Vision in Wales responded to new NICE guidelines around screening patients for depression by contacting Talk and Support and our Sight Loss Counselling service, to scope a depression referral route for Low Vision patients (estimated to be around 40% of patients they support). As a result, the services have been identified as an appropriate referral route and a pilot with clinicians is underway.

The evidence has also been shared with NHS commissioners in NI to support the roll-out of Time to Talk as part of the RNIB/Action Confidence building offering.

The validated measures and comparison group have been pivotal in this. If we had used another validated tool EQ5D that measures wellbeing, known to be favoured by NICE, we suspect we would not have been able to evidence the impact of the project we were evaluating. But we feel scoping for validated measures that match our outcomes was definitely worth the pay off in this instance. Using SWEMWBS to evidence outcomes has meant that NHS commissioners have paid attention to the proven benefits of peer support and RNIB are able to reach more blind and partially sighted people with support to increase their wellbeing, improve their confidence and connect with each other.

We will continue to make judgements of measures on a case by case basis so we can be confident the tools work to gather evidence of our intended outcomes – but if you find a match between validated measures and your programme outcomes it can ensure commissioning audiences stand up and take notice.

To read Time to Talk impact summary and full report visit www.rnib.org.uk/timetotalkevaluation 

To find out more about the Time to Talk service visit www.rnib.org.uk/time-to-talk

Join the network

Join the Knowledge Exchange Network for the Sight Loss Sector by emailing [email protected] with your name, job title and organisation and the subject line ‘Join network’. It is free to join, and as part of the network you’ll get regular research updates from us, be invited to join our LinkedIn group and attend professional development events.