Read on Twitter
We published our evaluation of Signs of Safety, looking at LAs funded through the DfE Innovation Programme - a joint piece of work with @abbotsky and colleagues @hscwru.
What did we find, and so what? https://twitter.com/whatworksCSC/status/1355131837669122051
What did we find, and so what? https://twitter.com/whatworksCSC/status/1355131837669122051
We looked at four outcomes - re-referral, escalation on re-referral, duration of assessment and use of Kinship care. We chose these based on discussions with local authorities and intervention developers, and the previous research by @abbotsky.
Our evaluation looked for evidence of *Impact* - answering "If I use Signs of Safety, what is likely to happen to X" - replace X with each of the four outcomes.
What did we find? We found a lack of impact on both re-referral outcomes, and on duration of assessments. Surprisingly, and worryingly, we found a 13% point *reduction* in kinship care.
Why is this worrying? For several reasons, including that there's pretty good evidence that kinship care leads to more stable placements. ( https://whatworks-csc.org.uk/evidence/evidence-store/intervention/kinship-care/) Others ( @CathyAshley @GPlusinfo) can give fuller answers!
What do we conclude?
There's not *A lot* of evidence of harm here, so we don't advise abandoning SofS if it's already in use.
We do think that anyone thinking about moving to SofS should pause for thought.
A thorough debate and more intensive look at Kinship care use is needed.
There's not *A lot* of evidence of harm here, so we don't advise abandoning SofS if it's already in use.
We do think that anyone thinking about moving to SofS should pause for thought.
A thorough debate and more intensive look at Kinship care use is needed.
You might still think SofS' values are the ones for you - and that's fine. But right now, there's not a lot to suggest that it will improve *outcomes*. More research will clearly follow, and your mileage may vary.
Before I get to the fully nerdy bit;
If you're interested, our summary report is here; https://whatworks-csc.org.uk/wp-content/uploads/WWCSC_Signs-of-Safety_summary-report_Jan-20212.pdf
The full report is here; https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/956625/Sofs_revised_evaluation_report_270121.pdf
If you're interested, our summary report is here; https://whatworks-csc.org.uk/wp-content/uploads/WWCSC_Signs-of-Safety_summary-report_Jan-20212.pdf
The full report is here; https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/956625/Sofs_revised_evaluation_report_270121.pdf
Nerdy bit;
To look for the impact of Signs of Safety, we used what's called a Matched Difference in Differences approach. First, we find local authorities that are statistically similar to the SofS ones, but which aren't using SofS. We look for LAs that are similar in trends.
To look for the impact of Signs of Safety, we used what's called a Matched Difference in Differences approach. First, we find local authorities that are statistically similar to the SofS ones, but which aren't using SofS. We look for LAs that are similar in trends.
This gives us similar, but not identical local authorities. There are differences. That's where the difference in differences come in. There are, as the name implies, two differences - one before delivery, and one after.
We look to see how differences between the SofS LAs and their matched comparators changes over time - the difference, in the differences. This eliminates differences caused by anything constant(ish) in the LA over the time of the evaluation - like demographics, workforce...
...which combined with the matching (and a lot of checks), lets us get at the impacts. This is one of a few methods that we use to try and do that, and I'm excited that the team was able to pull it off.
