Children’s Bureau child welfare specialist Brian Deakins, left, was an organizer of the summit. Katherine Briar-Lawson, dean of the School of Social Welfare at the University at Albany-SUNY, right, spoke on the panel about research findings. “I would like a whole summit on intervention research,” she said.
What happens to research findings when they fail to reach expected outcomes?
A group of researchers debated that question during a panel discussion titled “The Rest of the Evidence: Where Have All the Null and Negative Findings Gone?” in late August at the second National Child Welfare Evaluation Summit in Washington.
The summit spotlighted promising innovations in child welfare research and evaluation across the country, but several experts discussed why some results fail to see the light of day.
The panel was moderated by Robin Perry, associate professor of social work at the Florida Agricultural and Mechanical University. Participants included Joan Levy Zlotnik, director of the NASW Foundation’s Social Work Policy Institute; Katherine Briar-Lawson, dean and professor at the School of Social Welfare at the University at Albany, the State University of New York; and Matthew Stagner, executive director of Chapin Hall Center for Children at the University of Chicago.
Stagner explained that research evolves, as does the policy that calls for research in the first place. “We’re quick to define a new direction,” he said.
He said results from studies that do not achieve anticipated outcomes often go unpublished, and other studies fail to surface for various reasons. “Peer reviews are not perfect but it is the best way,” he said.
Zlotnik said it often is misunderstood when findings don’t have positive outcomes.
“We may find that our innovation does no better compared to service as usual,”she said. “We need to better understand the control and treatment group when we have null findings. We need to gather both qualitative and quantitative data to examine the findings.”
Zlotnik, however, said she supports bringing negative and null findings to light: “If we only report the positive outcomes, we will repeat the same mistakes by not learning from what doesn’t work,” she said.
The call for research comes from different directions as well, Zlotnik noted. In her job, she said, it is important to monitor information supportive of social work effectiveness as well as outcomes that may not be positive. There may be misinterpretations that could be repeated during policy discussions on Capitol Hill, calling for further explanation and exploration.
Briar-Lawson said the research infrastructure needs to be re-examined. “I would like a whole summit on intervention research,” she said.
Perry said it’s important to encourage external funding sources to avoid perceived or actual biased results. Sometimes an agency supports a study but does not want to see the results published when they are not positive.
“As a researcher, my goal is to advance knowledge,” he said.
Zlotnik said she gained new insight from the meetings she attended. “Overall, I thought it was valuable to see how the field of child welfare has expanded with a diversity of methods to evaluate programs,” she said. “It’s not a one-size fits all and both process and outcome evaluations are essential.”