Process Evaluations for a Multisite Nutrition Education Program

Process evaluations are an often-overlooked component of evaluating health promotion interventions, but can be essential for interpreting program outcomes. The purpose of this study was to report the results of two types of process evaluations conducted for Food Fit, a nutrition education program implemented to 58 3rd through 5th grade children (67% Caucasian) in 5 YMCA after school programs. To evaluate program fidelity, a trained observer watched each lesson and recorded program adherence using a standardized checklist, outlining essential components of the intervention. Attendance was recorded by asking each child to complete a small task before and after each lesson. Results showed that program adherence was perfect in most cases and attendance rates varied, but were generally high. Attendance rates were not associated with improvement in nutrition behaviors. Implications and recommendations for future use of process evaluations are discussed. © 2012 Californian Journal of Health Promotion. All rights reserved.


Introduction
Obesity is a public health concern in today's society.This is especially true with regards to children, given that childhood obesity has tripled since the first National Health and Nutrition Examination Survey (NHANES I) taken in the early 1970's (Ogden, Flegal, Carroll, & Johnson, 2002;Ogden, Carroll, Kit, & Flegal, 2012).Obesity in early childhood is also a major risk factor for obesity in adulthood (Singh, Mulder, Twisk, van Mechelen, & Chinapaw, 2008).Health promoting interventions that can favorably impact behaviors associated with lowering the risk of obesity (such as a healthy diet) could help prevent unhealthy weight gain among youth, and spare them from the associated metabolic consequences, such as type 2-diabetes, high blood pressure, inflammation, as well as and psychological conditions such as depression, and low self-esteem (Daniels, Jacobson, McCrindle, Eckel , & Sanner, 2009).However due to methodological limitations, including inadequate use of theory when designing and evaluating health programs, mixed and modest outcomes have been regularly reported for many obesity prevention programs (Thomas, 2006).
Another commonly noted limitation for obesity prevention programs is inadequate reporting of process evaluations.Process evaluations assist researchers and practitioners in a number of ways.For researchers, conducting a proper process evaluation helps to determine if the program was sufficiently delivered (program fidelity) and whether program participants were adequately exposed to the intervention (program attendance).This in turn can help to strengthen study results, by assuring the program was delivered as designed.For practitioners, it helps formalize an intervention into a systematic series of tasks, which can help improve replication in the field.For example, a proper process evaluation for program fidelity will list critical program activities and how they should be implemented.By failing to monitor fidelity and attendance, researchers and practitioners run the risk of making what is known as a Type III error, where weak or null outcomes and results can be attributed to poorly executed or incorrectly implemented interventions (Windsor, Clark, Boyd, & Goodman, 2004).
Food Fit (FF) was a theory-based nutrition program, that has been described elsewhere (Branscum, 2008).Results from the pilot study were promising, as a number of psychosocial variables significantly increased, such as selfefficacy for choosing and consuming fruits and choosing lower calorie snack foods, and overall dietary behaviors increased, such as consuming raw vegetables and using the food label to choose healthier options.However in the report process evaluations were not addressed.The purpose of this study was to report the process evaluation of the implementation of Food Fit, to aid in the interpretation of the results (Branscum, & Kaye, 2009).

Study Design
The design of this study was a pre and post test treatment only design, with no control group.Approval from the Institutional Review Board was obtained from the sponsoring university.

Training
Fifty-eight undergraduate college students (researchers) enrolled in Nutrition Programs and Services in the Community (Human Nutrition 704) at The Ohio State University were trained to implement and evaluate Food Fit as a servicelearning component of their class.Students were randomly assigned to 1 of 10 groups.Each group of 5 to 6 students were then randomly assigned to a YMCA sponsored after school program.For the initial lesson, students were randomly assigned one of the following tasks: administer the program, administer pre and post testing, conduct process evaluation or administer the snack evaluation.For the following lessons, students systematically rotated through the assigned tasks.

Researchers attended weekly training sessions during class hours on Tuesdays (3 hours).
During the session, researchers were assigned one of the following tasks for that weeks' lesson: program facilitator who implemented the lesson, impact evaluator who implemented a pre and post test or snack assessment for the lesson, or process evaluator who evaluated the program facilitator on instructional fidelity.To improve program fidelity, the program facilitator participated in a training session using experiential learning and practiced performing the proper procedures for the assigned lesson.A detailed script, outlining the process for each lesson, was distributed weekly to each facilitator.The authors of this article then trained them using step-by-step instructions, and gave instructional feedback when necessary.At the end of each training session, program facilitators were also able to ask questions pertaining to the lesson.Lessons were implemented on the following Thursday of the same week.One-hundred percent of the researchers attended the weekly training sessions.

Impact and Process Evaluation Assessments
A number of evaluations were employed to assess the impact and process implementation of Food Fit.Fruit and vegetable consumption and the enactment of healthful eating behaviors were evaluated using a brief food behavior checklist (Branscum, Sharma, Kaye, & Succop, 2010).Fruit and vegetable consumption was evaluated using 7 items, 5 of which were scored as Yes (1) or No (0), and 2 were scored from 0 to 5, based on the response to the questions: How many servings of vegetables do you eat each day?; and How many servings of fruit do you eat each day?Scores for fruit and vegetable consumption ranged from 0 to 15, with scores closer to 0 indicating low consumption, and scores closer to 15 indicating high consumption.Enactment of healthful eating behaviors was evaluated using 2 items as Yes (1) or No (0).Items included Do you eat low-fat instead of high-fat food?and When choosing a food to eat, do you use the Nutrition Facts on the food label?Scores for this variable ranged from 0 to 2, with a score of 0 indicating children did not enact in either healthful eating behavior, a score of 1 indicating children enact one of the two behaviors and a score of 2 indicating children enact both behaviors.In addition to the dietary assessment, process evaluations measuring attendance and program fidelity were employed.Attendance was evaluated by having children complete a pre and posttest pertaining to the day's lesson.Completing both assessments signified that the child was present before the lesson began, and immediately after the lesson was completed, thus indicating they were present for the entirety of the lesson.This helped distinguish from children who arrived late, or left early.During each lesson the process evaluator evaluated the fidelity of program facilitator's instruction by using a standardized form, listing important subtasks (scored: Yes/No) needed for each lesson.Each lesson contained 50-70 sub-tasks.For example, each lesson started with the program facilitator giving a 'Personal Introduction' which was followed by them 'Stating the Purpose of the Lesson' and 'Defining Key Terms' such as Calories, and Food Label.At the completion of each lesson the number of implemented subtasks were divided by the number of possible subtasks to yield a percentage from 0% (implemented none of the lesson) to 100% (implemented the lesson in its entirety).Requests for the process evaluations can be made to the corresponding author of this article.

Results
Five YMCA after school programs were used for this study and all sites participated in the intervention for 6 weekly lessons.Fifty-eight children were enrolled in the study.There were more boys (n=33, 57%) than girls (n=25, 43%), and a majority was in the 3rd and 4th grade (n=48, 83%) and either 9 or 10 years old (n=44, 76%.Children in this study were mostly Caucasian (n=39, 67%), with some African American children (n=11, 19%) and the remaining self-identified as 'Other' (n=8, 14%).
Attendance rates were generally high, but varied from as low as 66% in Lesson 4 to as high as 79% in Lesson 6 (Table 1).Attrition rates per lesson also varied, indicating that for some lessons very few children left after the lesson began, and for some lessons many left before the lesson was complete.For example, for Lesson 2, 44 children started the lesson, with 42 completing it, indicating an attrition of only two children (attrition rate of 5%).For Lesson 3 however, 56 children started the lesson, with only 44 completing it, indicating an attrition of 12 children (attrition rate 21%).Among the forty-six children completing the dietary assessment, one attended 2 lessons (2%), eight attended 3 lessons (17%), nine attended 4 lessons (20%), fifteen attended 5 lessons (33%), and thirteen attended all 6 lessons (28%).
Results for program fidelity indicated that the intervention was generally implemented as planned.Sixty total process evaluations were collected from 5 YMCA programs (10 groups x 6 lessons).A majority reported perfect implementation (100% of tasks; 43 of 60), many reported very high implementation (at least 90% of tasks; 11 of 60) and one group reported implementation of 87% of tasks for one lesson.

Table 2 Changes in Dietary Scores for Children Enrolled in Food Fit
(Saunders, Evans, & Joshi, 2005)f limitations of this study that should be addressed.First,only two types of process evaluations were employed.After the completion of the study, it was apparent that an even more comprehensive process evaluation may have been warranted.For example, while program adherence measures whether or not elements of the program are implemented, it gives no information regarding the amount of time spent on each task, or each lesson.Future researchers should consider evaluating the amount of time facilitators spend implementing each lesson, since it is conceivable that this could impact results.Using comprehensive frameworks such as the Saunders model(Saunders, Evans, & Joshi, 2005), may be beneficial for future researchers, as it can shed light on why some interventions are successful and others are not.Another limitation was the small sample size.When stratified by attendance, only 1 child attended two lessons, 8 children attended three and 9 children attended four lessons.Having a small sample size, as well as limited variation in the outcome, lowers statistical power.Hence, it is difficult to find small effects that would generally require much larger samples.The final limitation of this study was that there was no comparable control group.Having such a group would have been useful in this study, as a comparison between children who had no exposure to the intervention could have been compared to children with little exposure and full exposure.