Weekly Wonk: What if We’re Wrong About What Works?

Why the limits of evidence in child welfare matters for policy

From the Founder's Desk

Last week we explored where child welfare’s data gaps make it hard to see what’s happening in practice.

That surfaces a harder question: what does the evidence actually say works?

First-time contributor and child safety data expert Michael Cull takes that on in our Deep Dive.

He maps where the evidence is thin, why it got that way, and what it means for the leaders and policymakers building accountability structures on top of it.

His next piece will be available to our premium members, going into what leaders can actually do about it.

For our latest WonkCast I sat down with ACF Assistant Secretary Alex Adams to get his read on the big bet he’s making on what makes policy durable as he hits 6 months.

Let’s get into it.

Special thanks to Binti for their foundational sponsorship of WonkCast.


Weekly Wonk Deep Dive

Where We Go Off Course When We Assume Child Welfare Should Fly

Leading without certainty when our assumptions outpace the evidence.

By Michael Cull, PhD, MSN

Protecting children is not the responsibility of any single agency or profession.

It’s the product of decisions made across healthcare, housing, behavioral health, courts, education and social services, all operating under shared expectations of what systems can prevent and control.

Yet, when tragedies occur, accountability often collapses onto child welfare alone.

This reinforces a quiet assumption: that we largely know what works, and when things go wrong, it’s because we failed to apply that knowledge correctly.

This assumption is so central that it shapes policy, leadership accountability, and public critique. When tragedies occur, the search for error begins immediately.

But what if that assumption is wrong?

What if child welfare systems are being held accountable to a standard of certainty that the underlying evidence base cannot support?

This is the first in a two-part series.

Part I makes the diagnostic case — that child welfare’s accountability structures are built on a misreading of what the evidence can realistically support in complex, real-world conditions.

Part II examines what leading under those conditions actually requires of leaders.

What The Evidence Says

Start with substance use disorders (SUD). Roughly 19 million children in the U.S. live with at least one parent or caregiver with a SUD.

That statistic alone explains why treatment has become such a central feature of child welfare practice and policy.

When a parent is engaged in treatment, systems typically interpret that fact as proxy evidence that we’re addressing risk. The evidence does not support that.

Access to treatment matters, though in many communities it remains uneven or inadequate.

Even when treatment is available and engagement is high, the evidence on its effectiveness is limited.

Outcomes are modest and highly variable even under optimal conditions– stable housing, financial resources, social support, and consistent access to care.

A meta-analysis of 118 clinical trials found that the two best-supported drugs for alcohol use disorder require treating between 11 and 18 patients, respectively, to prevent one person from returning to any drinking at all.

That’s the ceiling under ideal conditions, without the other factors families face in child welfare.

Substance use disorders are widely understood to be chronic, recurring, and remitting. Relapse is not an exception but part of the clinical reality.

In practice, treatment offers population-level probabilities, not assurances, and those probabilities were generated under conditions that differ substantially from the lives of families involved with child welfare.

When One Gap Becomes Many

The evidence around the effectiveness of substance use disorder treatment is a window into a broader problem: the limits of the evidence underlying much of what child welfare systems rely on to ensure child safety.

The families child welfare serves aren’t navigating a single condition.

They’re navigating combinations — substance use, trauma, poverty, housing instability, domestic violence, mental health needs— that interact with each other and compound over time, often across generations.

Each added layer compounds uncertainty and further thins the evidence base guiding intervention (Mao, 2024).

It is neither ethical nor feasible to randomize families with complex needs into control groups that limit treatments.

Much of what child welfare is asked to do cannot be tested under the kinds of controlled conditions that produce certainty in other fields.

What we have instead are evaluations of discrete interventions in isolation.

Trauma Focused Cognitive Behavioral Therapy (TF-CBT), for example, is effective for trauma. SSRIs reduce anxiety and depression.

Parent-Child Interaction Therapy (PCIT) can improve parenting.

No study evaluates these treatments in combination, with families in the child welfare system, over the timeframes that actually matter for children’s lives.

Yet child welfare policy and practice are often built around the expectation that interventions can reliably resolve safety concerns in isolation, even when they struggle to produce durable outcomes even under the best circumstances.

This points to an uncomfortable truth: we do not have strong evidence for much of what child welfare systems are asked to do at scale, even though we act as if we do.

The Gaps in Our Flight Plan

The field’s response has been to standardize what it can, including through practice models and standardized assessment tools.

But the evidence there isn’t more reassuring.

Signs of Safety is perhaps the most widely implemented child welfare practice framework in the world.

One of the most rigorous program rating systems in the country, the California Evidence-Based Clearinghouse for Child Welfare, lists it as “Not Rated,” meaning it lacks the body of research required for even a preliminary evaluation.

The assessment tools structuring child welfare’s most consequential decisions were designed to support professional judgment, not replace it.

They were never designed to produce objective truth about risk (Lyons, 2022).

Like airlines and surgical teams, child welfare is being asked to predict low base-rate events that are as unacceptable as they are rare, like serious harm and fatalities.

We ask child welfare to do this across long timeframes, amid multiple chronic conditions.

That is an extraordinarily difficult prediction problem even with a strong evidence base. Child welfare doesn’t have one.

Mind The Accountability Gap

Child welfare leaders are asked to lead systems whose accountability structures are built around expectations that treatment science and social intervention cannot reliably support.

No system operating amid chronic conditions, layered adversity, and low-base-rate risk can guarantee that nothing ever goes wrong.

Even in highly controlled, well-resourced environments like hospitals, where risks like hospital-acquired infections are addressed over shorter timeframes, outcomes remain probabilistic rather than guaranteed.

Child welfare operates under far longer timelines, across multiple systems, and with far less control.

When the implicit expectation is perfection, the consequences are predictable.

Leaders manage to the expectation rather than the reality. Errors get concealed rather than examined.

Accountability becomes about finding who failed rather than understanding what conditions made failure more likely.

And institutional learning — the only mechanism that actually improves outcomes over time — becomes nearly impossible.

The accountability gap isn’t just unfair to leaders. It makes the system less safe.

None of this reduces the urgency of protecting children. It clarifies what it actually takes to protect them.

To Err is Human: A Workforce Already Past Its Limit

Even the modest gains evidence-based approaches can deliver depend on fidelity; the degree to which interventions are delivered as designed and tested.

The effect sizes reported in clinical trials are a ceiling of impact, generated under conditions of close supervision, adequate staffing, and practitioner competence.

When those conditions do not hold, the ceiling drops.

Over half of the U.S. population now lives in a mental health professional shortage area. 93 percent of behavioral health workers report having experienced burnout, with nearly half saying workforce shortages have prompted them to consider leaving the field entirely.

Annual departure rates in child welfare are estimated at 20 to 40 percent, with the average worker remaining in their position less than two years.

This is not a pipeline problem alone.

It is a structural condition of the systems being asked to deliver evidence-based practice.

The evidence base is already thin. The workforce needed to implement what evidence exists is in crisis.

The ceiling on what child welfare can reliably deliver is lower than the accountability structures governing it acknowledge — and it is getting lower.

The gap between what child welfare is expected to deliver and what the evidence can support isn’t a leadership problem.

It’s a structural one that we can actually do something about.

Until the field names it as such, the accountability structures governing child welfare will continue to make the system less capable of doing the very things they’re designed to ensure.

This is neither a dead-end nor destiny. There are shifts leaders can make that move the needle.

In Part II, we examine what leading under these conditions actually requires — and what it looks like when leaders build organizations capable of learning faster than they fail.

From the Wonk Briefing Room

Where the Weekly Wonk gives you a map of the terrain child and family policy faces, our premium resources aim at how to navigate it.

Last week, we held a member-only virtual roundtable, digging into the opportunities and challenges the Home for Every Child initiative pose for your work.

We also released a premium brief from Kurt Heisler, digging into what would create meaningful change in child welfare technology policy.

It’s an actionable guide to why prior federal efforts to modernize child welfare IT have stalled, and how to understand the new ACF Child Welfare Technology Incubator.

To read the full brief and access all our premium resources, join the Wonk Briefing Room. Individuals can sign up here, or get the team membership rate here

Wonkatizer

ACF Maps Prevention Push

What Happened

ACF has launched an interactive map and resource hub providing public access to Title IV-E Prevention Plans.

The tool shows whether each state’s plan is approved, under review, or not yet submitted, and which evidence-based services from the Title IV-E Prevention Services Clearinghouse it includes.

ACF is also signaling an expedited review process for states that adopt substantially similar language from an already-approved plan.Why it Matters

Why it Matters

Other sectors like Medicaid have long had public transparency for state plans and modifications to them.

This gives insight into patterns and trends, and makes it easier to adopt approved approaches. States can replicate them without starting from scratch.

The expedited review signal is consequential, and could accelerate uptake.

What to Watch

Whether the expedited review process produces measurably faster approvals in practice, and whether it accelerates adoption.

Of course, the real underlying rate limit is written into the law. ACF action can only go so far, so this also points to a longer-term deliberation as part of financing reform.

–

That’s it for this week. Stay sharp, Wonks.

~Z

Partners Making Your Weekly Wonk Possible