Date August 25, 2002
To Kathi Marin-Bemm, Nancy Kaplan and Members of the Issaquah Network
Cc Trudy Arnold, Family Policy Council
From Kim Van Eyck
Memo
It was a pleasure to meet with all of you and to learn about the efforts of the Issaquah community to provide much needed programs for at-risk youth. I am very impressed with your enthusiasm and dedication to improving the lives of youth in your community. I hope that you will find the enclosed materials helpful as you set out to develop ways to evaluate the services you provide.
As I stated in the Network board meeting of August 21st, I believe that designing and implementing an evaluation plan is a great chance to demonstrate (with concrete evidence!) that programs such as Reach for the Sky July really make a difference in the lives of children. Finding ways to measure outcomes and to analyze the results facilitates program expansion, adaptation and improvement. Through increased clarification about the purpose of the program, for example, more focused and effective service delivery will result. Being able to demonstrate that programs are really making a difference can also help to gain additional funding, recruit new board members, to enlist and motivate volunteers.
With regards to the Reach for the Sky July program, I hope it will be clear from the attached materials that I believe that the program does significantly reduce risk and increase protective factors for Issaquah youth. I strongly recommend that additional resources be directed to expanding the program to two summer sessions and Friday sessions throughout the school year. I also urge the Family Policy Council and DSHS to consider assistance with grant-writing in order to fund this expansion. I have developed a draft of a project proposal including introduction, theory of change (or logic model), program goals, methodology/ program components and budget. I am also enclosing a template for a survey tool and for data collection. Increased funding should also allow for administrative/evaluation support for the program.
In my follow-up discussions with several people who participated in the evaluation of the other Network programs that took place last week, it was clear that you are all eager to make it easier to “see” the results of the programs you support and to be able to communicate them clearly to others. I encourage you to think about how to:
· Include the goals of the Issaquah network in writing at the top of the page on all program description and evaluation materials. This visual reminder may help to encourage providers (and board members) to continually reflect on how the work that they are doing fits into these areas of focus and may also help to keep people motivated when a meeting’s agenda includes a lot of administrative detail. (Perhaps the letterhead and printed meeting agenda could include the italicized portion at the top!)
Issaquah
has selected as its focus: child abuse and neglect, substance abuse and youth
violence. Their goals include:
developing community attachment, strengthening families, integrating schools
into strategies and involving youth. Its main strategies include parenting
training and skill building, mentoring for teen parents and academic supports
for at-risk children. (Greater Issaquah Public Health and Safety Network 10-year
Plan. Attachment O, 1996)
· Focus on progress: compare each program with its own past performance instead of making comparisons with other programs.
· Provide assistance to each program to be able to craft measurable outcomes and design effective evaluation tools. Perhaps a one-day training session could be held in which service providers learn how to write observable and measurable outcomes. (I am enclosing some supporting materials.)
· Provide assistance to each program to improve existing tools to capture outcomes. In the case of parenting classes, for example, measuring the learning of parents on pre and post-tests may be a more powerful indicator than measuring parents’ belief in what they have learned. If parenting skills are identified as the goal or desired outcome of the program, then it becomes easier to match the most appropriate tool—either a test, survey, observation or data collection. In several cases, it appears that pre and post testing or surveying would vastly improve the ability of the program to measure changes directly resulting from program participation. Perhaps this could be accomplished in a second day of training.
· Pick several key measurable outcomes to use to evaluate each program (instead of a much longer list). Include these in the Statement of Work. Although the annual Program Review Worksheet could include a few more considerations including cost effectiveness—I think the outcomes listed on the Statement of Work should serve as the primary focus of the evaluation. Perhaps the Review Worksheet could be streamlined to three or four additional considerations.
· Test all surveys to be sure that the wording is clear. Try administering the surveys to five individuals prior to giving them to the target group. On one of the surveys it was noted that the order of positive, neutral and negative choices should remain consistent throughout. (School psychologists have training in test and survey design and also have access to standardized tests that may serve as useful guides)
· Encourage service providers to collect data! Attendance data is particularly useful.
· Develop a standardized Services Report form that providers could use as a template. (see enclosed draft)
· Encourage providers to report aggregated data instead of data on each individual participant or survey and that data does not get separated from the original question. (Questions should be re-stated in written summary or report.)
Thank you again for allowing me the opportunity to learn about the Issaquah Network. Although I am leaving for Geneva in September, I hope there will be other (long-distance) possibilities to provide assistance to your programs in the future. I am most happy to respond to any questions or concerns you might have after reviewing the enclosed materials.