Evaluating healthy food access interventions
- Healthy food access initiatives can impact community health:
- Increasing fruit and vegetable consumption
- Improving nutrition knowledge and behavior
- Promoting healthy shopping and cooking
- Promoting healthy body weight
- Preventing or managing diabetes and other diet-related health conditions
- Healthy food access initiatives can also have valuable co-benefits:
- Supporting local food systems
- Promoting local economic development, vocational skills, and job creation in vulnerable communities.
- Evaluating community health interventions is vital in order to:
- Refine and improve program implementation
- Identify the most effective interventions for expansion or replication
- Provide evidence for funders to maintain financial support of programs
- Communicate the impact of community benefit initiatives to stakeholders and the wider community
- Complete community benefit reporting requirements
This brief contains recommendations to assist community benefit staff and community partners in conducting program evaluation for healthy food access programs. It contains guidance for evaluation planning and links out to validated, frequently used surveys and data collection tools to bolster support for programs nationally and develop a common set of indicators, measures, and methods.
Healthy food access initiatives can impact community health by increasing fruit and vegetable consumption and improving nutrition, promoting healthy shopping and cooking, promoting healthy body weight, and preventing or managing diabetes and other diet-sensitive health conditions. Healthy food access initiatives can also have valuable co-benefits of supporting local food systems and promoting local economic development, vocational skills, and job creation in vulnerable communities.
Evaluating community health interventions is vital in order to:
- refine and improve program implementation
- identify the most effective interventions for expansion or replication
- provide evidence for funders to maintain financial support of programs
- communicate the impact of community benefit initiatives to stakeholders and the wider community
- complete community benefit reporting requirements1
This guidance brief provides recommendations and resources for evaluating healthy food access interventions that hospitals support as part of their community benefit implementation strategy to address health needs in their communities. Evaluation is a vital component of community health improvement programs, requires a significant commitment of budget and program resources, and should be planned at the same time as programs are developed.
Engaging community partners and stakeholders
There are many community organizations and stakeholders that bring unique value, skills, and experience to the evaluation process. Community partnerships can be formed across all stages of evaluation including evaluation design, data collection, data analysis, report writing, and the dissemination of findings. Relevant stakeholders may be diverse and can include community-based organizations, local public health and other government agencies, community health and development coalitions, universities, farmers, local businesses, and external contractors.
- Public health departments are interested in initiatives that contribute to positive community health outcomes, and it is likely that health department staff have received some training in research methodology. Health department staff may be able to assist with evaluation design and data collection and analysis. Public health departments also have access to state and local data that can serve as the basis for selected evaluation measures.
- Public health, medical, and nursing school students and faculty may also be able to assist with evaluation design and data collection and analysis as an extension of their learning or part of their professional development.
- The U.S. Department of Agriculture (USDA) National Institute of Food and Agriculture (NIFA) administers the Food Insecurity Nutrition Incentive (FINI) grant program and develops the criteria awardees must adhere to when implementing their programs. The institute requires grant recipients to conduct a program self-assessment and also participate in the evaluation activities conducted by Westat, the external evaluator for the FINI grant program. Each year NIFA organizes and records a webinar outlining the evaluation expectations of FINI grant recipients.
- Fair Food Network and Wholesome Wave provide resources, tools, and offer technical assistance for evaluating fruit and vegetable prescription incentive programs.
It is vital to address challenges related to equity in program evaluation. Consider the following questions when developing your evaluation strategy:
- Who is at the table when the logic model is developed, and whose perspectives are missing?
- How is the program ensuring that it serves hard-to-reach populations?
- How is the program ensuring that its services are culturally appropriate?
- Who is left out of the evaluation methods you choose?
Find your baseline
Determine the baseline for program participants to better understand your target population and assess the impact of your intervention. Biophysical health indicators, such as blood glucose (hemoglobin A1c) levels, blood cholesterol levels, blood pressure, and body mass index (BMI), may be collected from participants before the program begins. Participants may be asked about their medical history to identify previous diagnoses of diet-related health conditions. (Note that evaluation and data collection plans must address privacy concerns and protected health information requirements, including obtaining consent from program participants.) Pre-intervention surveys may collect information regarding nutrition knowledge, food attitudes and beliefs, cooking and shopping skills, and food behaviors. Community health information from public health departments or local organizations can also help establish reference data.
Determine your evaluation objectives and questions
The evaluation questions that will be asked and the indicators that will be measured should be tailored to the specific initiative. In determining these questions and indicators, it is valuable to first clarify the program goals. It is critically important that the evaluation questions and indicators align with organizational strategies and program goals in order to highlight successes and identify areas for improvement. Having a few well agreed upon measures to evaluate may be better than a long list. It is also valuable to align your evaluation framework and indicators with those that are widely used to evaluate similar programs in order to be able to compare results.
Start by describing your initiative, including both basic program elements and activities and the goals and outcomes you expect to achieve.
Create a logic model to clarify and illustrate how your intervention works: the resources utilized (inputs), the activities you will undertake, and the expected short-, medium-, and longer-term outcomes (impacts) of your program activities.
The primary goals of your initiative should be linked to the health needs identified in the hospital's community health needs assessment. Increasing economic and physical access to healthy foods, improving dietary patterns, and increasing nutrition, shopping, and cooking knowledge all can address obesity, disease-related health conditions, and food insecurity. Additional goals may target community building co-benefits such as supporting local and sustainable food producers and workforce development in low-income communities.
The evaluation should assess how well or to what degree the program’s activities contribute to meeting program goals and producing desired outcomes. This can be measured by developing evaluation objectives and questions relating to both program implementation (process) and program impact (outcomes). Your logic model can capture the primary and additional goals targeted, relevant program activities, and the objectives that will be assessed in the evaluation process.
Implementation questions and objectives for healthy food access programs
Consider the ‘who,” “what,” “when,” and “how” of program activities and participation in order to improve program design and implementation.2
Example implementation questions and objectives
How many individuals completed the Cooking Matters curriculum?
60 individuals will complete our nutrition education program from June to August 2018.
How many patrons of the mobile market van were satisfied with the quality, variety, and price of the produce offered?
85 percent of mobile market users will feel satisfied or very satisfied with the quality, variety, and price of the produce offered.
How many individuals who screened positive for food insecurity were actively referred to community resources to address their food needs?
80 percent of those who screened positive for food insecurity will be actively referred to Hunger Free Colorado.
Which of eight potential sites for farmers markets are the most convenient locations for targeted community participants?
Surveys at community clinics will identify three optimal locations to host farmers markets to increase food access in underserved communities.
Fruit and vegetable prescription coupon utilization rate
What percentage of the coupons distributed were utilized at participating farmers markets and retail outlets?
80 percent of distributed coupons will be redeemed at participating farmers markets and retail outlets.
Pounds of produce donated
How many pounds of produce were donated to participating food banks and/or pantries?
In fiscal year 2017, 180,000 lbs. of produce will be donated to participating food banks and/or pantries.
Impact/outcome questions and objectives
Obtain information on the impact of the program on the targeted audiences. Questions should consider shorter- and longer-term outcomes of the initiative.
Examples of short- and medium-term outcomes
Diet and nutrition knowledge
Were participants knowledgeable about and able to use the USDA’s MyPlate Food Guide after completing the nutrition course?
75 percent of participants will be knowledgeable about and implementing the USDA’s MyPlate Food Guide in their homes after completing the nutrition course.
Did participants increase confidence in their ability to cook with fresh produce after attending the nutrition and cooking classes?
90 percent of participants will report being confident or somewhat confident about cooking with fresh produce after attending the nutrition and cooking classes.
Healthy shopping on a budget skills
Did participants increase their confidence about meal planning and shopping for healthy food that would adequately feed their families on a budget?
80 percent of participants will report increased confidence in their meal planning, shopping, and budgeting skills after the initiative.
Diet and nutrition attitudes and beliefs
How many participants are likely to try new fruits and vegetables after the initiative?
There will be a 50 percent increase in participants reporting they are likely or somewhat likely to try new fruits and vegetables after the initiative.
Did participants increase their consumption of fruits and vegetables?
70 percent of participants will report an increase in their daily servings of fruits and vegetables
Supplemental Nutrition Assistance Program and Women, Infants, and Children authorized food outlets
Did the number of SNAP and WIC authorized food outlets, including farmers markets, increase after the policy intervention?
Ten additional food outlets and farmers markets will become SNAP- and WIC-authorized after the policy intervention.
Farmers market and fruit and vegetable sales
Did participating farmers’ fruit and vegetable sales increase after the introduction of the fruit and vegetable incentive program?
- Farmers’ average fruit and vegetable sales will increase by 3 percent after the introduction of the fruit and vegetable incentive program.
- $10,000 will be spent on FVRx coupons at participating farmers markets in FY18.
Examples of longer-term outcomes
Healthy body weight
How many participants, diagnosed as overweight, reduced their weight?
60 percent of overweight participants will reduce their weight by 5 percent.
Prevent or treat diabetes
How many diabetic participants reduced their blood glucose (hemoglobin A1c) levels?
70 percent of diabetic participants will reduce their blood glucose (hemoglobin A1c) levels by .5 point or more.
Prevent or treat high blood pressure
Did at-risk participants lower their blood pressure post-intervention?
55 percent of at-risk participants will lower their blood pressure.
Prevent or treat high cholesterol
Did at-risk participants reduce their blood cholesterol levels?
60 percent of at-risk participants reduced their blood cholesterol levels.
Health care usage and cost
Did diabetic participants reduce health care usage after completing the program?
55 percent of diabetic participants reduced their hospital readmission rate following participation in the initiative.
Job creation in local food enterprises
How many new jobs were created as a result of the increased demand at farmers markets?
Fifteen new jobs will be created
Recommended indicators and data collection tools for healthy food access initiatives
Evaluation objectives and questions should inform which indicators and metrics will be collected in the evaluation process. Obtaining relevant data is critical for drawing valid conclusions and building continued support for the intervention. It is recommended that program staff collect biometric health data, knowledge and behavior change data, and, if relevant, data on policy, systems, and environmental change.
Knowledge and behavior metrics are valuable for demonstrating short- and medium-term impacts, and knowledge and behavior change outcomes can be clearly linked to program activities. Health outcomes often require longer than the intervention time period to achieve and are likely to be influenced by variables outside the scope of the intervention.
Achieving healthier eating behaviors
Because a growing base of evidence finds that increased consumption of fruits and vegetables contributes to healthier body weight and prevention or better management of diet-related health conditions, an initiative that achieves healthier eating behaviors can be considered a community health improvement success.
Recommended implementation indicators
- Number of individuals screened for food insecurity
- Percent of participants enrolled in program
- Number of participants who completed program
- Percent of participants satisfied or very satisfied with specific program elements
- Number of fruit and vegetable coupons distributed to food insecure individuals
- Percent of fruit and vegetable coupons redeemed
- Pounds of produce donated to food banks and pantries
Recommended impact/outcome indicators
- Percent of participants no longer food insecure
- Percent of participants who have increased diet and nutrition knowledge
- Percent of participants who are confident or very confident in new healthy cooking skills
- Percent of participants comfortable trying new fruits and vegetables
- Number of new Supplemental Nutrition Assistance Program (SNAP) and Women, Infants, and Children Food and Nutrition Services (WIC) authorized farmers markets
- Percent of increased fruit and vegetable consumption
- Income growth for participating local food producers or vendors
- Percent reduced BMI
- Percent reduced blood glucose (A1c) level
- Percent reduced high blood pressure
- Percent reduced high blood cholesterol
- Percent decreased emergency department use
- Percent decreased readmission rate
- Amount of money saved on health care costs
- Number of people who completed vocational skills training
- Number of new jobs created
Ideally, data should be collected pre-intervention, mid-intervention, and post-intervention.
- Pre-intervention information will allow program staff to collect baseline data against which progress can be measured.
- Collection of mid-stream data enables assessment of activities and individual aspects of the program that may be refined to maximize objectives.
- Post-intervention data, including 6+ months after program participation, demonstrates longer-term program impacts and sustained change.
While surveys may be tailored to the specific goals of the initiative, employing widely used and validated data collection instruments facilitates data comparison. See the following evaluation tools and instruments for examples.
Example evaluation tools
- Share Our Strength: Cooking Matters developed pre- and post- surveys for their food and nutrition education program to assess behavior and knowledge change for adults, parents, and families.
- CalFresh, the Supplemental Nutrition Assistance Program (SNAP) in California, developed “Plan, Shop, Save & Cook” and “Making Every Dollar Count” surveys for community members that qualify for federal assistance.
- The Food Attitudes and Behaviors (FAB) Survey was developed by the National Cancer Institute (NCI) to evaluate various factors that may influence fruit and vegetable consumption among adults.
- University of California Cooperative Extension developed a Food Behavior Checklist to assess behavior change for participants in food and nutrition education programs.
- The Wisconsin Farm to School Evaluation Knowledge, Attitudes, and Consumption Behavior Survey was adapted from the University of North Carolina-Chapel Hill Farm to School Evaluation, Wisconsin Fresh Fruit & Vegetable Program evaluation and the AmeriCorps Farm to School Survey. The survey is intended to be administered at the beginning and the end of the farm-to-school initiative.
- The Nutrition Assessment Shared Resource (NASR), created by Fred Hutchinson Cancer Research Center, developed the Snack and Beverage Questionnaire for distribution to children and teens ages 10-18.
- Share Our Strength: Cooking Matters developed pre- and post- surveys for their food and nutrition education program to assess behavior and knowledge change for children and teens.
Researchers at the University of Pennsylvania have developed a series of validated Nutrition Environment Measure Surveys and assessment tools.
Example evaluation tools (continued)
- The Hunger Vital Sign, a two-question screener developed by the Children’s HealthWatch, is a validated food insecurity screening tool that is widely used in clinical settings
- The U.S. Department of Agriculture’s Economic Research Service developed an 18-item food insecurity survey and also released an abbreviated 6-item food insecurity survey.
- The Center for Medicaid and Medicare Services developed and validated a 10-item social screening tool for clinical settings that incorporates the questions used in The Hunger Vital Sign tool.
Sample evaluation reports
- The Fair Food Network released a 5-year report on their Double Up Food Bucks program in 2017, which includes sections on program evaluation and impacts.
- Wholesome Wave published an annual report in 2014 and compiled a fact sheet of highlighted national programs from 2013-2017. Wholesome Wave offers evaluation tools and resources for members of their free National Nutrition Incentive Network.
- Kate Fitzgerald of Canepa Healthcare LLC, with funding from Wholesome Wave, developed a report of the 2015 Food Insecurity Nutrition Incentive (FINI) Grant Program.
It is important to communicate and share evaluation results with various stakeholders to demonstrate success and build support for program continuation and replication.
Stakeholders may include program partners, funders, hospital staff, public health agencies, nonprofit organizations working towards similar objectives, program participants, and the community at large.
Communicating results to hospital staff may help individuals feel more pride in their organization and can help improve employee satisfaction and retention.
It is also very useful to share lessons learned from unexpected or disappointing evaluation results. A key value of program evaluation is learning from mistakes or ineffective strategies in order to adapt and improve programs.
Ideally, data should be collected continuously in order to strengthen the support and evidence of effectiveness. This is particularly useful for current and future funders who look for consistent and sustained results when determining which projects they want to support. Evaluation reports may be posted to the public online, shared at community meetings and relevant conferences, and delivered to key stakeholders.
Example healthy food access program evaluation strategies
Oak Grove Community Fruit and Vegetable Prescription Program
- Program inputs/activities:
- Oak Grove Hospital contributes to funding the fruit and vegetable incentive (FVRx) vouchers and partners with several community organizations to implement the initiative.
- Hospital clinicians conduct food insecurity screenings at every patient visit in the emergency department and enroll patients experiencing food insecurity into the fruit and vegetable incentive program. Enrollment includes prescribing fruit and vegetable vouchers at a value of $7 per family member per week (up to $28 per week).
- Participants’ coupons are replenished biweekly at nutrition education/healthy eating classes. The classes begin with instruction from a dietetics (RD) student and continue with group cohort activities and discussions.
- Oak Grove Hospital partners with a community center to host the six-session nutrition education and healthy eating course. The community center is located in a low-income neighborhood where some program participants live and is also in close proximity to public transportation for those who do not live nearby.
- Ten local farmers markets and retail food outlets agree to accept both FVRx coupons administered by the program and federal Supplemental Nutrition Assistance Program (SNAP)/Women, Infants, and Children Food and Nutrition Services (WIC) benefits.
- Length of program:
- The program runs for three months spanning the growing season from June to August.
- Primary goals and anticipated outcomes:
- Reduce economic barriers to healthy foods
- Improve nutrition knowledge and dietary patterns of participants
- Reduce prevalence of obesity and diet-related health conditions
- Increase the number of community food outlets that accept SNAP/WIC
- Increase local food producers’ and vendors’ sales to strengthen the local economy
- Data Collection:
- Attendance is taken at each food and nutrition class to track active participation
- Program staff modified a widely-used survey that included questions on diet and nutrition knowledge, fruit and vegetable consumption, attitudes towards trying new fruits and vegetables, satisfaction with various aspects of the program design, and health care utilization related to diet-related health conditions. This survey was distributed at the beginning of the first healthy eating class, again at the fifth session, and once again at the last session.
- A clinician from Oak Grove Hospital was present at the first and last sessions to administer surveys and collect biophysical health indicators including BMI, blood cholesterol levels, and blood glucose (A1c) levels.
- Participating vendors, including farmers market vendors and grocery store managers, collected information on FVRx coupon redemption and increased sales.
Washington County Food Pantry Program
- Program inputs/activities: Washington Community Hospital collaborates with a group of local organizations in Washington County to help four local food pantries improve their offerings of fresh, healthy, and local food.
- Washington Community Hospital
- Plants extra garden beds in its large hospital garden to donate fresh produce to the food pantries.
- Commits staff time each month to participate in gleaning efforts at local farms.
- Provided a community benefit grant to purchase refrigerators to store fresh food.
- Partners with the local public health department to conduct program evaluation.
- Three local farms allow volunteers to collect surplus and unmarketable crops after the initial harvest.
- A local food rescue organization collects surplus/unsold foods from participating grocers and restaurants and delivers to local food pantries.
- Faculty and students at a culinary school offer nutrition education and healthy cooking classes at each food pantry location to complement food pantry activities on Sundays twice a month to increase knowledge and encourage sustained behavior change as new, healthy foods are introduced.
- Washington Community Hospital
- Length of program:
- The program runs year round. The food rescue organization has a larger role in the winter months when farmers are unable to grow crops outdoors.
- Primary goals and anticipated outcomes:
- Increase healthy food access for vulnerable populations by improving healthy food procurement in local food pantries
- Improve dietary patterns of food pantry clients
- Reduce food waste
- Data collection:
- The quantity of produce harvested, rescued, and donated is weighed and recorded.
- Each food pantry records the quantity and types of fresh produce clients receive.
- Washington Community Hospital partners with the local public health department to evaluate the initiative’s impacts.
- A hospital clinician attends each nutrition education and healthy cooking class and recruits food pantry clients to participate in an assessment study that will take place over the course of one year.
- Participants agree to attend three nutrition education and healthy cooking classes over the course of three months, in conjunction with utilizing the pantry’s food resources, and to share biophysical health indicators collected by the hospital clinician (BMI, A1c, blood pressure, and blood cholesterol), and complete a pre- and post-participation survey.
- Local public health evaluators modified a widely-used survey that included questions on diet and nutrition knowledge, fruit and vegetable consumption, attitudes towards trying new fruits and vegetables, utilization of food pantry resources, obstacles to healthy eating, and health care utilization related to diet-related health conditions
Identifying measurable goals and creating logic models
- Pell Institute developed an evaluation toolkit with sections that include step-by-step guidance for creating logic models and recommendations for how to best use logic models.
- Minnesota Department of Public Health offers guidance for developing SMART objectives and also provides links to additional resources, course, and online training.
- The Centers for Disease Control and Prevention (CDC) developed a resource for Developing Program Goals and Measurable Objectives.
- Health COMpass, a project of the Health Communication Capacity Collaborative, developed a how-to guide for developing logic models including templates, samples, and links to additional resources.
Additional resources for determining evaluation questions and objectives
- The Catholic Health Association (CHA) developed a resource for Evaluating Your Community Benefit Impact, which is free for members and available for purchase for nonmembers.
- Kaiser Permanente, the Center for Community Health and Evaluation (CCHE), and the Nutrition Policy Institute at the University of California developed a toolkit for utilizing the “dose” framework when planning and evaluating community health promotion strategies. Dose is a measure of both the reach and strength of an initiative on its target population.
- Cottage Health developed an evaluation toolkit, adapted from CCHE’s resource, which includes blank worksheets for each step and relevant case studies.
- The University of Kansas Center for Community Health and Development published an online Community Tool Box for professionals working to improve community health and initiate social change. The resource features multiple toolkits including one on “evaluating the initiative.”
- The RE-AIM framework includes 5 different dimensions (Reach, Effectiveness, Adoption, Implementation, & Maintenance) from which program questions and indicators can be developed. RE-AIM has a variety of digital resources including a dimensions items checklist and an example scoring of a farmers market program.
- The Supplemental Nutrition Assistance Program Education (SNAP-ED) Evaluation Framework and Interpretative Guide provides recommendations and general guidance for various indicators, outcome measures, and evaluation methodologies relating to nutrition, physical activity, and obesity.
- The Center for Training and Research Translation (Center TRT) developed an evaluation framework for nutrition and obesity initiatives and offers recommendations for developing evaluation questions, sample evaluation plans, logic models, and additional resources.
- The National Collaborative on Childhood Obesity Research (NCCOR) developed a Measures Registry User Guide: Food Environment to assist users in selecting and utilizing measurement tools specific to the food environment.
Additional resources for general and food-related evaluation design and guidance
- Project Star, the training and technical assistance provider for the Corporation for National and Community Service, offers guidance for program and evaluation design.
- The Economic Development Research Group and a member of the Economics Department at Northwestern University developed a report outlining best practices when assessing economic impacts of an initiative.
- The National Farm to School Network developed a cross-sectoral evaluation framework for farm-to-school programs including recommendations of common outcomes to be measured. Additionally, the Center for Food & Justice Urban & Environmental Policy Institute at Occidental college offers guidance on evaluating farm-to-school programs.
- The National Science Foundation offers recommendations for addressing cultural considerations when evaluating program activities and outcomes.