Program Evaluation Matters for ‘Bridge Programs’: Building evidence for Austin Angels’ programs
By Allie Long
In January of 2021, The Texas Institute for Child & Family Wellbeing was contracted to conduct a two-part program evaluation for the National Angels organization’s Austin Angels program. “This evaluation has really been a long time in the making,” shared Tym Belseth, Lead Research Coordinator for the Austin Angels Program Evaluation. “I first found out about Austin Angels several years ago, when I was at a community meeting presenting about the importance of normalcy in foster care, based on our work with the Texas Youth Permanency Study.”
It was there that Tym met Susan Ramirez, CEO of Austin Angels, who was also presenting on her organization’s work. “I thought they had a very unique approach and an innovative model to serving Texas families,” said Tym. “There is a growing number of ‘bridge programs’ that have flexibility in supporting families, but there is virtually no information about their effectiveness despite their potential to really improve the child welfare system.”
Austin Angels programming & what makes it unique
Austin Angels began in 2010 and now has two main programs: The Love Box and Dare To Dream. The Love Box program provides comprehensive support by matching a local foster family with a committed volunteer for at least one year who will spend quality time supporting the child in foster care and their biological and foster families. The Love Box Program creates personalized care packages based on practical needs, and offers relationship building and mentorship support to the entire family network. Dare To Dream is a program largely focused on providing direct mentorship to youth ages 15-22 through goal setting and preparation for independent living. Some examples include navigating higher education or getting a driver’s license. The purpose is to encourage positive outcomes at the transition to adulthood by offering consistent support. When volunteers are matched with children or youth in the foster care system, they will be partnered closely with that family for one year, even if the children are adopted, change placements or are reunited with their birth families during that time.
“When working with a program that is new to program evaluation, we want to meet programs where they are and help empower them to understand the process,” shared TXICFW Assistant Director of Research Swetha Nulu. “When I joined the Austin Angels project, I was intrigued by the way that this program really focuses on building the capacity of the volunteers. The focus on connection in Austin Angels programming is a core component. Volunteers for both The Love Box and Dare To Dream are connected to the birth families, siblings, and foster families— not just the foster child. Holistic support is the crux of this organization’s work.”
For years, Austin Angels has received positive feedback for their innovative programming. Up until recently though, feedback attesting to the effectiveness of their programs was largely anecdotal. It’s for this reason that a program evaluation is key to the future of program expansion.
Why building evidence matters
Program evaluations matter because they benefit both the organization itself and the field at large. “Building evidence for service effectiveness goes a long way in establishing credibility to what an organization does, outside of any narrative accounts previously gathered,” said Tym. While gathering anecdotal evidence is a necessary component to any program, it’s also important to provide concrete data on what’s working and what concrete outcomes will come from their services.
Program evaluations are also essential to social work practice because their scope allows researchers to tease out elements that have broader implications for practice. “It can be helpful for organizations doing so many different types of great work to have an unbiased third party assess where time is spent, what goals are effective for the program’s intentions, and how to best shift their activities based on desired outcomes,” said Swetha. “We do this work to help organizations learn what’s going well and what can be improved so that we can collectively fill service gaps being provided to families in the child welfare system.”
In the first five months, the TXICFW team has conducted interviews to better understand the two Austin Angels programs and developed a logic model to build a foundation for developing the strategic evaluation. “Instead of going into a program you don’t know and trying to evaluate it, we find that the process evaluation is a valuable tool for us to understand all the moving parts and how these programs work on the ground,” shared Tym. “As researchers, it’s important to truly understand the program to more empathetically gather and interpret the data in context.”
Once the process evaluation is completed, researchers will begin an outcome evaluation through surveys, interviews, and an analysis of underlying processes and impacts that these programs have on youth and families. The hope is to use these findings to expand services to foster and kinship families, build evidence of service effectiveness, and ensure that youth and families are supported in achieving their goals. “We look forward to data collection, starting in June, so stay tuned for more on this evaluation in a few months,” said Swetha.
Learn more about Austin Angels here.