Inter-Rater Reliability of a Food Store Checklist To Assess Availability of Healthier Alternatives to the Energy-Dense Snacks and Beverages Commonly Consumed by Children

Abstract

Background:

Food stores have gained attention as potential intervention targets for improving children's eating habits. There is a need for valid and reliable instruments to evaluate changes in food store snack and beverage availability secondary to intervention. The aim of this study was to develop a valid, reliable, and resource-efficient instrument to evaluate the healthfulness of food store environments faced by children.

Methods:

The SNACZ food store checklist was developed to assess availability of healthier alternatives to the energy-dense snacks and beverages commonly consumed by children. After pretesting, two trained observers independently assessed the availability of 48 snack and beverage items in 50 food stores located near elementary and middle schools in Portland, Oregon, over a 2-week period in summer 2012. Inter-rater reliability was calculated using the kappa statistic.

Results:

Overall, the instrument had mostly high inter-rater reliability. Seventy-three percent of items assessed had almost perfect or substantial reliability. Two items had moderate reliability (0.41–0.60), and no items had a reliability score less than 0.41. Eleven items occurred too infrequently to generate a kappa score.

Conclusion:

The SNACZ food store checklist is a first-step toward developing a valid and reliable tool to evaluate the healthfulness of food store environments faced by children. The tool can be used to compare availability of healthier snack and beverage alternatives across communities and measure change secondary to intervention. As a wider variety of healthier snack and beverage alternatives become available in food stores, the checklist should be updated.

Introduction

Childhood obesity in the United States has increased dramatically in recent decades, making this one of our nation's most serious public health threats.¹ Children who are obese are at increased risk for a number of health conditions, including high blood pressure and type 2 diabetes, and are more likely to become obese adults.¹ Among the trends in eating patterns that have coincided with the rise in childhood obesity is the increased contribution of snacking (i.e., food eaten between regular meals) to daily energy intake.² Nationally representative survey data indicate that among 2- to 18 year olds, snacking accounts for approximately 27% of total daily energy intake.² Major sources of snacking kilocalories include desserts (e.g., cakes and cookies), high-fat, high-salt foods (e.g., chips), and sugar-sweetened beverages.²

Children's dietary behaviors are strongly influenced by their immediate food environments.^3,4 To date, most research in this area has focused on the school food environment as the primary venue for obesity prevention interventions.⁵ Yet, food stores clustered around schools may be important sites to promote health because of their potential effect on eating behaviors of children who walk to and from school.^6,7 A recent study of fourth to sixth graders in Philadelphia found that over half of children purchased snacks daily at convenience stores located near their schools.⁸ The most frequently purchased items were chips, candy, and sugar-sweetened beverages. Other studies have documented the lack of nutritious snacks in food stores located in close proximity to schools.^9–11 For example, one study found that among 452 items identified in 17 food stores located within four blocks of urban schools, 96.4% were highly processed foods, including salty snacks (e.g., chips) and refined sweets (e.g., cookies); none were fruits or vegetables.¹¹

Over the past decade, food stores have gained attention as potential intervention targets for improving children's eating habits.^12–15 In Philadelphia, for example, The Food Trust, a nonprofit organization, created Snackin’ Fresh–branded foods (e.g., fresh fruit salad) and beverages to encourage healthy snacking among children and adolescents.¹⁶ In Boston, a partnership between middle schools and corner stores aims to increase purchases of healthy beverages and decrease purchases of sugar-sweetened beverages.¹³ Though instruments with high reliability have been developed to assess food store availability of fruits and vegetables and other nutritious foods (e.g., 100% whole wheat bread and skinless boneless chicken breasts)^17–19 few include valid and reliable measures of healthy snacks and beverages in portions appropriate for children. Such measures are needed to systematically characterize the healthfulness of food store environments faced by children. Therefore, the aim of this study was to develop a valid, reliable, and resource-efficient instrument to assess the availability of healthier alternatives to the energy-dense snacks and beverages consumed by children in food stores located near elementary and middle schools in Portland, Oregon. This study was conducted in preparation for a larger, community-based participatory research project to compare availability of healthier snacks and beverages in food stores located near elementary and middle schools in urban and rural Oregon.

Methods

SNACZ Food Store Checklist

The SNACZ food store checklist was developed using a multi-step and iterative process. Snack and beverage items (e.g., cookie, cake, chips) consumed by children were identified by reviewing relevant literature and tools^{8,10,11,20,21} and surveying more than 750 children participating in an Oregon health fair about their snacking habits. The identified items were organized into three categories (i.e., beverages, snacks, and fruits and vegetables) and pretested in 10 food stores, located outside of the study area, in urban and rural Oregon. Energy-dense snack and beverage items with healthier alternatives (e.g., potato chips vs. baked potato chips) and ready-to-eat and single-portion fresh fruits and vegetables available in food stores were selected for inclusion in the checklist. Items (e.g., pepperoni stick) with no healthier alternatives available in food stores were not included in the checklist. “Healthier” was defined using Institute of Medicine (IOM)²² nutrient standards for competitive foods and beverages sold in schools for calories, total calories from fat and saturated fat, trans fat, total sugars, and sodium (Table 1). The checklist included 48 snacks and beverages. Twenty-four items were ready-to-eat and single-portion fresh fruits and vegetables (e.g., apple, basket of cherry tomatoes, and fresh fruit cup) and seven were beverages, including low-fat and nonfat milk, low-fat or nonfat flavored milk, soy milk, 100% fruit juice, and bottled water without flavoring, additives, carbonation, and/or caffeine.

Table 1.

Institute of Medicine²² Nutrient Standards Used to Identify Snacks and Beverages for Inclusion in the SNACZ Food Store Checklist

Snacks

— No more than 200 calories per portion as packaged

— No more than 35% of total calories from fat^a

— Less than 10% of total calories from saturated fat

— Zero trans fat (≤0.5 g per serving)

— No more than 35% of calories from total sugars, except for yogurt with no more than 30 g of total sugars, per 8-oz portion

— No more than 200 mg of sodium per portion as packaged

Beverages

— Water without flavoring, additives, or carbonation

— Low-fat and nonfat milk (8-oz portion); flavored milk with no more than 22 g of total sugars per 8-oz portion

— 100% fruit juice in a 4-oz portion as packaged

— Caffeine free

Fat content of nuts and seeds do not count against the total fat content of the product as long as other nutrient standards are met.²²

Observer Training and Certification

Two graduate students completed 5 hours of training (classroom and field work) in August 2012 in preparation to conduct observations using the SNACZ food store checklist. The classroom component included instructions on field procedures. The fieldwork involved using the checklist in two practice stores located outside of the study area. Several evidence-based strategies were used to promote high inter-rater reliability.^18,23 First, after each practice store, group debriefings were held to review results and allow observers to ask questions about the items. Second, inter-rater reliability scores (percent agreement) for each individual observer were compared to the “gold standard”—a coinvestigator who was also a registered dietitian—and computed immediately after each practice store and observers were offered oral feedback. Finally, at the end of their training, observers were certified to use the SNACZ food store checklist if they achieved the certification criterion of 90% agreement, as compared to the gold standard, on each section of the instrument in two test stores located outside of the study area. Both observers met this requirement after the first attempt.

Store Sample

A list of businesses in Portland classified by the North American Industry Classification System was obtained from the City of Portland Bureau of Technology Services.²⁴ Stores that fell into one of the following categories were extracted for further classification: general merchandise stores; convenience stores; food and beverage stores; fruit and vegetable markets; gasoline stations with and without convenience stores; specialty food stores; pharmacies and drug stores; and supermarkets and other grocery stores. Through a review of the list, stores that did not typically sell food for human consumption were identified and removed (e.g., discount variety store, pet store, and auto repair shop). Store proximity to an elementary and/or middle school was used as a selection criterion because children buy foods and beverages during, and on their way to and from, school.^6–8 Therefore, stores located within a 0.5-mile radial buffer from the geometric center of either a low- or high-income school (characterized as those with ≥ and <50% of students eligible to receive free or reduced-priced lunch)²⁵ were extracted for further examination (n=127); stores that fell into a buffer that included both low- and high-income schools were excluded. A ground-truthing procedure was used to confirm that the list of 127 stores sold snack foods, identify any stores that had permanently closed, and identify any additional stores in the study areas that sold snack foods; 42 stores were removed during this process and two stores were added. Letters describing the study and the voluntary nature of participation were sent to managers of stores remaining on the list. Sixteen store managers or employees declined to participate in the study. A list of 71 stores was ultimately identified for the study.

Overview of Data Collection

Data were collected over a 2-week period in August 2012. Upon arriving at a store, observers introduced themselves to the store manager/owner and obtained permission to conduct the observation. Once permission had been obtained, observers began completing the checklist. For each packaged snack and beverage item, availability of healthier alternatives was assessed by comparing nutrition facts labels with checklist criteria. For each item (e.g., granola bars), if at least one variety (e.g., Nature Valley brand Oats and Honey granola bar) met the criteria, the item was documented as available. For each fruit or vegetable, availability was assessed by documenting whether a single portion of the item was present in the store in a ready-to-eat form. For example, “apple” was documented as available if apples were sold as an individual item (i.e., loose) or sliced and in single-serving packages. The average length of time for an individual observer to conduct the observation using the SNACZ food store checklist was 16 minutes. To assess inter-rater reliability, two observers independently assessed a random subset of 50 stores at the same time. Oregon Health & Science University Institutional Review Board (Portland, OR) approved this study.

Statistical Analysis

The kappa statistic provides a chance-corrected measurement of agreement between two observers and ranges from +1 (perfect agreement) to 0 (no agreement above that expected by chance) to –1 (agreement less than that expected by chance).²⁶ The following guidelines, published by Landis and Koch and used in previous inter-rater reliability studies involving direct or systematic social observation,^18,23,26 were used to evaluate inter-rater reliability of the SNACZ food store checklist: 0.81–1.00, almost perfect; 0.61–0.80, substantial; 0.41–0.60, moderate; 0.21–0.40, fair; 0–0.20, slight; and <0, poor. All data analyses were completed using SAS statistical software (9.3; SAS Institute Inc., Cary, NC).

Results

Table 2 shows inter-rater reliability of checklist items. Overall, 73% of items assessed had almost perfect (0.81–1.00) or substantial reliability (0.61–0.80). Two items (i.e., nuts and seeds as well as grapes) had a kappa score between 0.41 and 0.60, indicating moderate reliability. No checklist items had a kappa score below 0.41. Eleven items occurred too infrequently to generate a kappa.

Table 2.

Inter-Rater Reliability of SNACZ Food Store Checklist Items

Category	Items	Kappa
Beverages	100% Fruit juice	^a
	Low-fat milk	0.85
	Nonfat milk	0.66
	Low-fat or nonfat flavored milk	1
	Soy milk	^a
	Bottled water without flavoring, additives, carbonation, or caffeine	1
Snacks	Chips	0.88
	Chex Mix	^a
	Pretzels	^a
	Cracker	1
	Rice cakes	^a
	Popcorn	^a
	Nuts and seeds	0.57
	Trail mix	^a
	Cookies	^a
	Graham/animal crackers	0.65
	Granola bars	0.80
	Bagels	^a
	Muffins	^a
	Popsicles or other frozen desserts	^a
	Yogurt	0.96
	Applesauce, unsweetened	0.66
	Other canned/bottled fruit in natural juice or water	0.75
	Dried fruit with no added sugar	0.78
Fresh fruits and vegetables	Apple	0.92
	Apricot	0.92
	Banana	0.76
	Blueberries	1
	Cherries	0.85
	Grapefruit	0.94
	Grapes	0.48
	Melon (e.g., cantaloupe, honeydew, watermelon)	0.75
	Nectarine	0.85
	Orange	0.96
	Peach	0.94
	Pear	1
	Pineapple	0.91
	Plum	1
	Strawberries	0.93
	Mixed fresh fruit	0.90
	Other ready-to-eat and single-portion fresh fruit	0.84
	Broccoli florets	1
	Carrots (baby)	0.73
	Cauliflower florets	1
	Celery sticks	0.65
	Cherry tomatoes	0.81
	Mixed fresh vegetables	0.84
	Other ready-to-eat and single portion fresh vegetables	0.73

Items that occurred too infrequently to generate a kappa.

Discussion

The SNACZ food store checklist was developed to evaluate the healthfulness of food store environments faced by children. Whereas reliable and valid measurement tools are available to assess the overall nutrition environment of food stores,^17–19 the SNACZ food store checklist is designed specifically to assess food store availability of healthier alternatives to the energy-dense snacks and beverages commonly consumed by children.

Overall, more than 70% of checklist items had inter-rater reliability scores above 0.60, indicating substantial to almost perfect reliability. The high inter-rater reliabilities obtained are comparable to those reported in similar studies.^17–19 Two items in the SNACZ food store checklist had moderate reliability: nuts and seeds as well as grapes. The wide range of nuts and seeds that did not meet the checklist criteria may have obscured the detection of those that did. It is possible that additional training may increase the inter-rater reliability of this item. Among fresh fruits and vegetables, grapes also had moderate reliability. This was likely a result of ambiguity over the definition of ready-to-eat and single-portion grapes. For fresh fruits and vegetables, ready-to-eat and single portion was defined as items that could be eaten as sold, such as an apple or fresh fruit cup, versus an item such as a watermelon that requires additional preparation. Except for when they are sold as part of a fresh fruit cup, grapes are typically sold in bunches. A more precise definition (e.g., 1 pint) for grapes may increase the reliability of this item. Overall, 11 items occurred too infrequently to generate a kappa score or meaningful score for percent agreement. For example, of 50 stores in this study, only one carried a muffin that met the nutrient standards. Although the 11 items occurred infrequently, including them in the checklist may be useful for tracking change in availability of healthier snack and beverage alternatives secondary to intervention. However, given their low frequency of availability, the inter-rater reliability of these items cannot be confirmed without more study.

To our knowledge, this is the first study to examine the reliability of an instrument designed specifically to assess food store availability of healthier alternatives to the energy-dense snacks and beverages commonly consumed by children. The high inter-rater reliability suggests that the observer training and data collection protocols were sufficient. In addition, efforts were made to improve the validity of the checklist by carefully choosing foods based on previous studies of snacking behavior among children and IOM nutrient standards for competitive foods sold in schools as well as by pretesting the checklist in food stores. Compared to other instruments^17,18 designed to assess food availability, the cost of personnel time to train observers and conduct assessments is lower because of the relative simplicity of the SNACZ food store checklist. The checklist therefore may be especially useful for limited resource organizations (e.g., public schools). Checklist data were analyzed by calculating a summary score for each of the three checklist categories (i.e., beverages, snacks, and fruits and vegetables) and a total score for the store. Available items are assigned 1 point; unavailable items are assigned zero points. Summary and total scores can be used to evaluate the healthfulness of food store environments faced by children and compare availability of items across stores and communities.

This study has several limitations. First, packaged snack items included in the SNACZ food store checklist met IOM²² nutrient standards for calories, fat, sugar, and sodium. However, the items did not always meet the standard that foods sold in schools also must contain at least one serving per portion of fruit, vegetables, or whole grains.²² The checklist, therefore, includes highly processed items (e.g., baked potato chips) that may not contain at least one serving per portion of fruit, vegetables, or whole grains. Though such items may be of questionable nutritional value, including them in the checklist may increase the potential usefulness of the SNACZ food store checklist as an instrument to measure relatively small, but meaningful, improvements in the healthfulness of the food store environment secondary to intervention (e.g., replacing fried potato chips with baked potato chips).

Second, the checklist captures variety across items, but may not capture variety within items. For example, granola bars that met the healthy snack criteria were available in many stores and were captured under “granola bars.” If a store sold multiple varieties of granola bars that met the checklist criteria, this variety was not captured in the checklist. Similarly, if a store sold only one type of granola bar that met the checklist criteria, this lack of variety also was not captured. Given limited resources, capturing variety within items was not possible. In reality, when healthier snacks and beverages were available, variety within items was very limited.

Third, this study is limited to reliability testing. Further research is needed to determine whether food store availability of healthier snacks and beverages predicts children's snacking behaviors and how this relationship is different in more versus less walkable neighborhoods, lower versus higher income neighborhoods, or rural versus urban communities. Given limits on how much data can be collected using an observational tool,²⁷ using the SNACZ food store checklist in conjunction with other research methods (e.g., interviews with store owners) is recommended when assessing change in food availability secondary to intervention.

Conclusion

Despite these limitations, the SNACZ food store checklist makes an important contribution to the measurement of food environments because it is designed specifically to evaluate the healthfulness of food store environments faced by children. The tool should be considered a first step toward a better understanding of the role of food stores in children's snacking behaviors. As a wider variety of healthier snack and beverage alternatives become available in food stores, the checklist should be updated.

Footnotes

Acknowledgments

The authors thank the Union County Fit Kids Coalition and Scott Ellis for their contributions to the work presented here and the journal editor and anonymous peer reviewers for their insights and suggestions. This study, and data collection in Portland, Oregon, was funded through grant support from the Oregon Health & Science University School of Nursing Betty Gray Rural Health Development Award.

Author Disclosure Statement

No competing financial interests exist.

References

Centers for Disease Control and Prevention. Childhood overweight and obesity. Available at www.cdc.gov/obesity/childhood/index.html Last accessed January 10, 2013.

Piernas

, Popkin

. Trends in snacking among US children. Health Aff, 2010; 29:398–404.

Dodson

, Hsiao

, Kast-Shors

, et al. Formative research for a healthy diet intervention among inner-city adolescents: The importance of family, school and neighborhood environment. Ecol Food Nutr, 2009; 48:39–58.

Kubik

, Lytle

, Hannan

, et al. The association of the school food environment with dietary behaviors of young adolescents. Am J Public Health, 2003; 93:1168–1173.

Gittelsohn

, Kumar

. Preventing childhood obesity and diabetes: Is it time to move out of the school?. Pediatr Diabetes, 2007; 8:55–69.

McDonald

. Active transportation to school: Trends among US schoolchildren, 1969–2001. Am J Prev Med, 2007; 32:509–516.

McDonald

. Critical factors for active transportation to school among low-income and minority youth. Am J Prev Med, 2008; 34:341–344.

Borradaile

, Sherman

, Vander Veur

, et al. Snacking in children: The role of urban corner stores. Pediatrics, 2009; 124:1293–1298.

Tester

, Yen

, Pallis

, et al. Healthy food availability and participation in WIC (Special Supplemental Nutrition Program for Women, Infants, and Children) in food stores around lower- and higher-income elementary schools. Public Health Nutr, 2011; 14:960–964.

10.

Gebauer

, Nelson Laska

. Convenience stores surrounding urban schools: An assessment of healthy food availability, advertising, and product placement. J Urban Health, 2011; 88:616–622.

11.

Lucan

, Karpyn

, Sherman

. Storing empty calories and chronic disease risk: Snack-food products, nutritive content, and manufacturers in Philadelphia corner stores. J Urban Health, 2010; 87:394–409.

12.

Holmes

, Estabrooks

, Davis

, et al. Effect of a grocery store intervention on sales of nutritious foods to youth and their families. J Acad Nutr Diet, 2012; 112:897–901.

13.

Hoffman

, Morris

, Cook

. The Boston Middle School Corner Store Initiative: Development, implementation, and initial evaluation of a program designed to improve adolescents' beverage-purchasing behaviors. Psychol Sch, 2009; 46:756–766.

14.

Gittelsohn

, Vijayadeva

, Davison

, et al. A food store intervention trial improves caregiver psychosocial factors and children's dietary intake in Hawaii. Obesity (Silver Spring), 2010; 18:S84–S90.

15.

Laska

, Borradaile

, Tester

, et al. Healthy food availability in small urban food stores: A comparison of four US cities. Public Health Nutr, 2010; 13:1031–1035.

16.

The Food Trust. Snackin’ Fresh. Available at http://thefoodtrust.org/snackinfresh/index.php Last accessed May 29, 2013.

17.

Glanz

, Sallis

, Saelens

, et al. Nutrition Environment Measures Survey in Stores (NEMS-S). Am J Prev Med, 2007; 32:282–289.

18.

Izumi

, Zenk

, Schulz

, et al. Inter-rater reliability of the food environment audit for diverse neighborhoods (FEAD-N). J Urban Health, 2012; 89:486–499.

19.

Gloria

, Steinhardt

. Texas nutrition environment assessment of retail food stores (TxNEA-S): Development and evaluation. Public Health Nutr, 2010; 13:1764–1772.

20.

Gittelsohn

, Suratka

, Song

, et al. Process evaluation of Baltimore Healthy Stores: A pilot health intervention program with supermarkets and corner stores in Baltimore city. Health Promot Pract, 2010; 11:723–732.

21.

Schwartz

, Novak

, Fiore

. The impact of removing snacks of low nutritional value from middle schools. Health Educ Behav, 2009; 36:999–1011.

22.

Institute of Medicine. Nutrition Standards for Foods in Schools: Leading the Way Toward Healthier Youth. The National Academies Press: Washington DC, 2007.

23.

Zenk

, Schulz

, Mentz

, et al. Inter-rater and test-retest reliability: Methods and results for the neighborhood observational checklist. Health Place, 2007; 13:452–465.

24.

Civic Apps for Greater Portland. Data sets: Business licenses. Available at www.civicapps.org/datasets Last accessed December 13, 2012.

25.

Oregon Department of Education. Students eligible for free or reduced lunch 2011–2012. Available at www.ode.state.or.us/data/reports/toc.aspx Last accessed December 13, 2012.

26.

Landis

, Koch

. The measurement of observer agreement for categorical data. Biometrics, 1977; 33:159–174.

27.

Zenk

, Schulz

, Izumi

, et al. Development, evolution, and implementation of a food audit. In: Israel

, Eng

, Schulz

, Parker

(eds), Methods for Conducting Community-Based Participatory Research for Health. 2nd ed. Jossey Bass: San Francisco, CA, 2013, pp. 277–303.