Validation of Good Agricultural Practices (GAP) on Minnesota Vegetable Farms

Abstract

The United States Food and Drug Administration and the Department of Agriculture jointly published the “Guide to Minimize Microbial Food Safety Hazards for Fresh Fruits and Vegetables,” which is used as a basis for Good Agricultural Practices (GAP) audits. To understand barriers to incorporation of GAP by Minnesota vegetable farmers, a mail survey completed in 2008 was validated with visits to a subset of the farms. This was done to determine the extent to which actual practices matched perceived practices. Two hundred forty-six producers completed the mail survey, and 27 participated in the on-farm survey. Over 75% of the on-farm survey respondents produced vegetables on 10 acres or less and had 10 or fewer employees. Of 14 questions, excellent agreement between on-farm interviews and mail survey responses was observed on two questions, four questions had poor or slight agreement, and eight questions had no agreement. Ninety-two percent of respondents by mail said “they took measures to keep animals and pests out of packing and storage buildings.” However, with the on-site visit only 45% met this requirement. Similarly, 81% of respondents by mail said “measures were taken to reduce the risk of wild and/or domestic animals entering into fruit and vegetable growing areas.” With direct observation, 70% of farms actually had taken measures to keep animals out of the growing areas. Additional, on-farm assessments were done regarding employee hygiene, training, presence of animals, water sources, and composting practices. This validation study demonstrated the challenge of creating nonleading and concise questions that are not open to broad interpretation from the respondents. If mail surveys are used to assess GAP, they should include open-ended questions and ranking systems to better assess farm practices. To provide the most accurate survey data for educational purposes or GAP audits, on-farm visits are recommended.

Introduction

It has been over 10 years since both the United States Food and Drug Administration (FDA) and Department of Agriculture (USDA) published good agricultural practice guidelines. In 1998, the FDA and the USDA jointly published guidelines for producing and handling vegetables to avoid contamination by microbial pathogens. These guidelines, titled “Guide to Minimize Microbial Food Safety Hazards for Fresh Fruits and Vegetables,” have been used as the basis for Good Agricultural Practices (GAP) audits, which are performed by state agencies and private companies. The importance of these guidelines is reflected by the number of farms producing vegetables and melons, of which 26% sold their products directly to consumers with sales of $335 million (Lucier et al., 2009). This trend continues with nearly a 100% increase in direct sales to consumers between 1997 and 2007. With the increased emphasis on “buy local” and farm-to-school programs, greater scrutiny of agricultural practices related to food safety at the local farm level is important.

To determine GAP training needs of Minnesota vegetable growers, a mail survey was conducted in 2008 to understand their use of GAP (Hultberg et al., 2012). This survey was based on the FDA GAP audit questions, with 14 questions used verbatim. Overall, the 246 respondents of this mail survey reported adherence to many of the GAP practices, but indicated deficiency in areas such as treating wash and processing water, taking measures to keep animals out of production fields, and cleaning and disinfecting harvesting tools and containers on a routine basis. Similar mail surveys assessing use of GAP recommendations have been done in the past in other states or regions of the United States (Rangarajan et al., 2002; Cohen et al., 2005; Jackson et al., 2007). Cohen et al. conducted a mail survey of 297 produce farmers in six New England states and found similar results of overall adherence but lagging in washing tools and containers as well as in water testing, record keeping, and use of bare hands. Rangarajan et al. conducted a mail survey of 213 produce farmers in New York and found results similar to those of Hultberg et al. (2012) and Cohen et al. (2005). Jackson et al. (2007) conducted a mail survey of produce farms in seven states across the country assessing both knowledge and implementation of GAP. The study showed that the majority of the 596 respondents were aware of GAP and that knowledge of GAP led to increased implementation of GAP. Alternatively, Ellis et al. (2005) conducted a study of nine produce farmers in Iowa using observation and in-person interviews, which had results similar to the previously described mail surveys.

The FDA GAP audit questions were intended for use by an in-person audit rather than a mail survey, and few studies have been done to validate this approach. To do so, we followed the mail survey with in-person visits to a subset of the responding farms to determine the extent to which actual practices matched perceived practices. It is well understood that the type of survey (mail versus in person versus telephone, for example) can greatly affect both the response rate and quality of data collected (Bowling, 2005; Dillman, 1978). Mail surveys are inexpensive with limited potential for interviewer bias, but often have low response rates with little control over who actually completes the survey. On the other hand, in-person interviews have high response rates but are time consuming, expensive, and subject to interviewer bias. Our goal was to compare responses from the mail and on-farm surveys to assess data accuracy. This has bearing on future educational efforts and farm audits for the growing market of small vegetable growers.

Materials and Methods

On-farm survey

An on-farm survey was designed to validate questions asked on mail survey (Hultberg et al., 2012) regarding food safety, hygiene, and manure/compost handling on farms as well as gain further knowledge regarding presence of farm animals, composting, fertilization, and irrigation practices (Tables 1 and 2). Fourteen questions that were identical or nearly identical to mail survey questions were used to validate mail survey responses. These 14 questions were a checklist of yes and no responses. In addition, there were specific questions regarding farm demographics (i.e., type of operation, products grown), presence/absence of animals, water source, use of compost, and composting practices if compost was used. Twenty-seven farms were selected from a list of 70 farmers who were willing to participate in an on-farm study. The farms were sorted by zip code and selected to provide broad geographical representation of producers throughout the state. Ten (37%) farms were located in the northern third of the state, 11 (41%) in the central third, and 6 (22%) in the southern third. Each farm was visited less than a year after completion of the mail survey by one of three surveyors. Surveyors had graduate training in food production and food safety and used a standardized assessment tool. Farm visits included a brief standardized survey, observation of farm practices, and a review of the physical characteristics of the farm.

Table 1.

Farm Size, Number of Employees, and Retail Outlet as Reported from Mail and On-Farm Survey Respondents

	Mail		Farm
	(n)	(%)	(n)	(%)
0–5 acres	138	56%	16	59%
6–10 acres	46	19%	6	22%
11–25 acres	29	12%	2	7%
26–100 acres	17	7%	3	11%
100+ acres	7	3%	0	0%
N/A	9	4%	0	0%
0–5 employees	167	68%	15	56%
6–10 employees	49	20%	8	30%
11–25 employees	13	5%	2	7%
26+ employees	11	4%	2	7%
N/A	6	2%	0	0%
Wholesale	13	5%	5	19%
Food service	1	0%	3	11%
Restaurants	14	6%	9	33%
Co-ops	1	0%	6	22%
Farmers markets	132	54%	15	56%
CSAs	11	4%	10	37%
U pick	60	24%	14	52%
N/A	22	9%	2	7%

N/A represents no answer for mail survey; not applicable for on-farm survey.

CSAs, Community Supported Agriculture.

Table 2.

Questionnaire Responses to On-Farm and Mail Surveys for the Farms That Completed Both (n=27) ^a

	On-farm survey (n=27)			Mail survey (n=27)
On-farm survey statements	% Yes	% No	N/A ^b	% Yes	% No	N/A ^b
Drinking water is available to all workers.^c	63	11	22	100	0	0
All toilets and restrooms have toilet paper, single-use towels, soap, and clean water for hand washing.	41	44	15	92	4	4
Visibly ill workers are kept from touching vegetables.^d	74	7	19	89	11	0
Clean first aid supplies are available for workers with cuts, abrasions, or other injuries.	93	0	7	100	0	0
All fruit and vegetable harvesting containers are kept visibly clean.^e	85	4	11	85	15	0
All hand-harvesting tools are kept as clean as possible.^f	81	0	19	55	41	4
A written standard operating procedure for what must be done when fruits and vegetables come into contact with chemicals, petroleum, pesticides, or other contaminants exist.^g	8	81	11	56	33	11
Measures are taken to keep animals and pests out of packing and storage buildings.	45	33	22	92	4	4
Packing containers are properly stored and sufficiently sealed to be protected from contamination by birds, rodents, or other pests.	48	26	26	74	22	4
Fruits and vegetables stored outside in totes, trucks, bins, etc. are covered and protected from contamination.	33	7	60	70	22	8
Fruits and vegetables are loaded and stored separately from other products that could spread human and plant diseases.	74	11	15	85	15	0
Manure is stored so that it is contained and cannot get on the fruits and vegetables.^h	27	22	61	81	4	15
Measures are taken reduce the risk of wild and/or domestic animals entering into fruit and vegetable growing areas.	70	30	0	81	15	4
Manure composted on the farm is properly treated or exposed to environmental conditions that lower the expected level of disease-causing bacteria.	37	0	63	78	11	11

Statements were identical on both surveys unless otherwise noted by superscript letter.

N/A for on-farm survey represents “not applicable”; N/A for mail survey represents “no answer.”

Mail survey stated “Clean and safe drinking water is available to all workers.”

Mail survey stated “Workers who have diarrhea or other symptoms of infectious illness are kept from touching vegetables.”

Mail survey stated “All vegetable harvesting containers are cleaned and/or sanitized before use and are kept as clean as possible.”

Mail survey stated “All hand harvesting tools (knives, pruners, etc.) are kept as clean as possible and are disinfected at least once every week.”

Mail survey stated “There is a standard operating procedure for what must be done when vegetables come into contact with chemicals, petroleum, pesticides, or other contaminants.”

Mail survey stated “Manure stored near crop growing areas is contained so that it cannot get on vegetables.”

Statistical analyses

The database for the on-farm survey responses was managed in Microsoft Access^® (Microsoft Corp., Redmond, WA) with summary analyses done using EpiInfo ver. 3.1 (Centers for Disease Control and Prevention, Atlanta, GA). Chi-square analyses were performed to assess the representativeness of the subset of farms visited compared to the farms that participated in the mail survey. Cohen's kappa analyses, a statistic measure of inter-rater agreement, were done to compare the agreement of responses from the mail survey to the on-farm survey for the 27 farms where both mail and on-farm responses were available. Degree of agreement was categorized as followed: ≤0=none, 0.01–0.2=poor, 0.21–0.4=slight, 0.41–0.6=fair, 0.61–0.8=good, 0.81–0.92=very good, 0.93–1=excellent (Dawson and Trapp, 1994).

Results

Farm descriptions

Questions on the mail survey that were validated by this study had been based on the FDA GAP audit and included nine sections covering Worker Health and Hygiene, Field Harvesting and Transportation, Packing and Storage, Animals and Manure, Irrigation, and Sewage Treatment (Hultberg et al., 2012). The locations of the 27 farms visited for this study were equally distributed among the northern (37%), central (41%), and southern (22%) regions of Minnesota. This differs from the 2007 Minnesota agriculture census, as most produce farms are located in the south (60%), 31% in the central region, and the remaining 9% in the north (National Agricultural Statistics Service, 2007).

The size of the farm, number of employees, and market outlets were similar between the two survey respondent groups (Table 1). Over 75% of the respondents from both surveys produced vegetables on 10 acres or less and had 10 or fewer employees. Over half of the respondents from both surveys used local, direct marketing (including Community Supported Agriculture, co-op, farmers market, and U-pick). Commercial marketing (wholesale, food service, and restaurants) was used by a third of on-farm respondents, but infrequently by mail survey respondents. There was no significant difference (p>0.05) in the growing acreage, number of employees, and market outlets between the survey respondents for these categories.

Seventy-seven percent (n=21) of the on-farm survey respondents farmed land that was either certified organic or in transition to become certified organic, whereas 27% (n=67) of the mail survey respondents considered their operation either certified or noncertified organic (p<0.001). Two thirds (n=18) of the on-farm survey respondents had animals on the property, most of which were poultry, followed by small ruminants, horses, cattle, and pigs. Presence of animals by type was not assessed on the mail survey.

Comparison of mail and on-farm survey responses

Fourteen questions were asked on both surveys. Percentages of positive, negative, and not applicable responses for the two surveys are shown in Table 2. The number of farms participating in the on-farm survey was 11% of the mail survey respondents.

There was variable agreement between mail and on-farm responses for the different questions. The results of the kappa tests, which compared mail and on-farm survey responses by each of the 27 farms visited in this study, are shown in Table 3. Only two questions were in excellent agreement for the two survey methods. Eight questions had no agreement, three questions had poor agreement, and one question had slight agreement (Table 3).

Table 3.

Question Response Agreement of Answers Given by the Same Farm on Both the Mail and On-Farm Surveys (n=27)

True or false questions	Kappa value	Description of agreement based on kappa value ^a
Clean and safe drinking water is available to all workers.	0	None
All toilets and restrooms have toilet paper, single-use towels, hand soap, or antibacterial soap, and clean water for handwashing.	0.09	Poor
Workers who are ill are kept from touching vegetables.	−0.1	None
Clean first-aid supplies are available for workers with cuts, abrasion, or other injuries.	Infinity	Excellent
All vegetable harvesting containers are kept as clean as possible.	−0.05	None
All hand-harvesting tools are kept as clean as possible.	0	None
There is a standard operating procedure for what to do when produce comes in contact with contaminants.	−0.04	None
Measures are taken to keep animals and pests out of packing and storage buildings.	0.15	Poor
Packing containers are properly stored and sufficiently sealed to be protected from contamination by birds, rodents, or other pests.	0.2	Slight
Vegetables stored outside in totes, trucks, bins, other containers, or in bulk on the ground are covered and protected from contamination.	−0.25	None
Vegetables are loaded and stored separately from other products that could spread human or plant diseases.	−0.15	None
Manure stored near crop-growing areas is contained so that it cannot get on the vegetables.	0	None
Measures are taken to reduce the risk of wild and/or domestic animals entering into vegetable growing areas.	0.16	Poor
Manure composted on the farm is properly treated or exposed to environmental conditions that lower the expected level of disease-causing bacteria.	Infinity	Excellent

Kappa value description ranges: ≤0=none, 0.01–0.2=poor, 0.21–0.4=slight, 0.41–0.6=fair, 0.61–0.8=good, 0.81–0.92=very good, 0.93–1=excellent (Dawson and Trapp, 1994).

Several other questions were asked on both surveys but not validated due to variation in wording between the two surveys. For example, only 7% of on-farm survey respondents had written worker training logs on proper cleanliness and hygiene practices available for review by the auditor, whereas 78% of mail survey respondents reported they provide training on proper cleanliness and hygiene practices to all workers (as the mail survey did not ask about documentation). Only 30% of on-farm survey respondents had hand-washing signs posted in restrooms and employee break rooms, compared to 73% of mail survey respondents, who reported that they required their employees to wash their hands before starting work (i.e., the mail survey did not inquire about posted signs). There were different options given on the on-farm and mail surveys regarding water use, with 19% of on-farm survey versus 24% of mail respondents reporting no irrigation for their crops, 59% and 57% using well water to irrigate, and the remaining 22% and 19% of respondents using another source such as pond, stream, or municipal water or did not answer the question.

A few questions designed to validate food safety, hygiene, and manure/compost handling on farms as well as gain further knowledge regarding presence of farm animals, composting, fertilization, and irrigation practices were added to the on-farm survey, which were not asked on the previous mail survey. The distance animals were kept from growing areas and water sources was assessed and found to be between 0 and 750 feet (0–228.6 m). Of the 13 farms that stored manure, the distance from where it was stored to growing areas was assessed and found to be between 50 and 1500 feet (15.2– 457.2 m). All farms used well water for drinking and hand washing except for one farm, which used municipal water. Similarly, 25 (93%) of the farms used well water for produce washing and processing, while 1 farm (4%) used municipal water and 1 farm (4%) used another source. Of the farms using well water, only 58% (n=15) had the water routinely tested.

Discussion

Standardized mail or in-person surveys looking at food or farm safety have been done on a small subset of farms (Rangarajan et al., 2002; Cohen et al., 2005; Ellis et al., 2005; Gomes da Cruz et al., 2006; Jackson et al., 2007). However, a weakness of mail surveys is that respondents may not understand the question or answer the survey with what is perceived to be the correct answer. To characterize the best methods to assess GAP, we validated a previous mail survey with in-person visits to a subset of the farms.

The representativeness of the on-farm survey respondents to the mail survey respondents was similar in regard to farm size, number of employees, and market outlets but differed significantly by percent of organic/noncertified organic versus conventional farms. The geographic distribution of the on-farm survey respondents was equally distributed in Minnesota. However, the overall distribution of produce farms in the state is skewed, with the majority in the south. Therefore, caution should be used when comparing farms by region or by organic versus conventional production methods.

Of the 27 farms assessed, there was excellent agreement on two questions regarding first-aid supply availability and properly composted manure. Most respondents likely had small first-aid kits in their houses, much like a family household normally would, and especially if they are working on small family farms. The other question asked “True or False: Manure composted on the farm is properly treated or exposed to environmental conditions that lower the expected level of disease-causing bacteria.” This question was a subjective assessment. Although most of these farms did not actively treat the compost piles, the piles were typically allowed to age for 6 months to 2 years to ensure sufficient composting.

There were three questions with poor agreement. The first of these questions asked “True or False: All toilets and restrooms have toilet paper, single use towels, hand soap, or antibacterial soap, and clean water for hand washing.” This question would have likely had excellent agreement had the term “single use” towels been omitted. Most of the farms were otherwise in agreement with the statement but used the house restroom for the employees, and they had cloth towels rather than single-use towels. In future surveys, the question would be more valid if each of the items is listed separately or if a “check all that apply” format is used. This likely represents the challenge of doing agricultural surveys where the respondents can be both small and large producers with varied crops and management practices. This has also been highlighted as a concern with the application of broad recommendations to the varied practices of growers and the dynamic nature of changing agricultural practices (Parker et al., 2012).

The second question with poor agreement was “True or False: Manure stored near crop growing areas is contained so that it cannot get on the vegetables.” This was a subjective assessment by the farmers, who apparently felt they were properly containing manure. With site visits, however, the manure was usually found to be uncovered and within about 200 feet (61 m) or less of the crops. This likely was a leading question in that it is unlikely that most farmers would intentionally store manure in a manner that could contaminate their product. After the fact, a better way of wording this question might have included options for what is considered “contained” or perhaps formatted as an open-ended question asking the respondent to describe how they contained their manure.

The third question with poor agreement was “True or False: Measures are taken to reduce the risk of wild and/or domestic animals entering into vegetable growing areas.” This question again was open to respondent interpretation. Several farms visited demonstrated that an answer of “true” has several interpretations. For example, one producer allowed his pet dog to run through the fields to keep other animals out. This would not be an acceptable GAP. Other farmers did not perceive dogs or chickens in or around the growing areas as a problem. The lack of fencing used to keep animals from entering growing areas may be explained by financial practicalities faced by small farm owners.

The remaining nine questions had no or only slight agreement and could be explained by the wording of the questions on the mail survey, which was taken nearly verbatim from the GAP audits. Wording such as “clean as possible” is leading and open to interpretation. Another question stated “True or False: Clean and safe drinking water is available to all workers.” While clean and safe drinking water was technically available to all workers on most farms, sometimes the water was distant and not readily available to workers.

Conclusions

This study demonstrates the value of on-site audits to assess GAP. The challenge of creating nonleading and concise questions that are not open to broad interpretation from the respondents is difficult with mail surveys. The initial mail survey was created using mostly closed-ended questions, primarily “yes/no” with a few “circle one/all that apply” or “fill in the blank” with numbers. The “yes/no” questions did not have an option for “not applicable,” which would have been useful, as it was learned during the validation visits that many of the farmers felt that many of the questions did not readily apply to them. This was especially true for the small operations with no workers or where family members were the only employees. It was readily apparent that many farmers do not consider family or close friends as “employees” and this was largely the reason for the disagreement between answers on the mail and on-farm surveys.

The mail survey was created to mirror the actual questions assessed on a GAP audit in order to determine level of compliance and barriers to compliance. However, the GAP audit questions were created with the intention of using them as in-person audits. Numerous questions on the mail survey were not specific and left room for broad interpretation. Possible solutions for continuing the use of a mail survey to assess GAP include the use of open-ended questions where the respondent could describe their situation, such as how they contain their manure. This would avoid any judgment of their approaches. However, this would likely result in longer questionnaires and challenges to data entry and analysis. Another option would be the use of ranking systems. For example, a respondent could score on a scale of 1–10 how well his/her manure pile is contained. This would require some description of the scale values but would likely have resulted in a higher level of agreement between the two types of surveys.

This validation study is limited by its small sample size and restriction to Minnesota growers. However, these 27 producers likely represent the diversity of small-acreage producers of vegetables in climates with shorter growing seasons. It is likely that across the country, there is a great diversity in the incorporation of GAP, especially by the local producer. On-site visits provide a valued means for producer education and production system assessment. However, as demonstrated by Tobin et al., merely providing technical knowledge and skills does not necessarily lead to change of on-farm practices (Tobin et al., 2013). We observed several GAP deficiencies on farms that were not recorded from the mail survey. This study demonstrates the continuing need to develop in-person tools that engage producers and encourage the incorporation of GAP on the diversity of vegetable production systems. The value of in-person, on-farm visits will improve survey accuracy and provide the best data to improve food quality and food safety.

Footnotes

Acknowledgments

This project was funded through the Healthy Foods, Healthy Lives Institute at the University of Minnesota. We thank the Minnesota producers who provided their time and insights.

Disclosure Statement

No competing financial interests exist.

References

Bowling

. Mode of questionnaire administration can have serious effects on data quality. J Public Health, 2005; 27:281–291.

Cohen

, Hollingsorth

, Olson

, Laus

, Coli

. Farm food safety practices: A survey of New England growers. Food Prot Trends, 2005; 25:363–370.

Dawson

, Trapp

. Basic and Clinical Biostatistics, 4th ed. New York: Lange Medical Books/McGraw-Hill, 1994.

Dillman

. Mail and Telephone Surveys: The Total Design Method. New York: John Wiley & Sons, 1978.

Ellis

, Strohbehn

, Henroid

. Assessing on-farm food handling practices of Iowa-grown produce and eggs in regard to food safety. Food Prot Trends, 2005; 25:758–761.

Gomes da Cruz

, Cenci

, Maia

MCA

. Good agricultural practices in a Brazilian produce plant. Food Control, 2006; 17:781–788.

Hultberg

, Schermann

, Tong

. Results from a mail survey to assess Minnesota vegetable growers' adherence to Good Agricultural Practices. Hort Technol, 2012; 22:83–88.

Jackson

, Archer

, Goodrich-Schneider

, Gravani

, Bihn

, Schneider

. Determining the effect of good agricultural practices awareness on implementation: A multi-state survey. Food Prot Trends, 2007; 27:684–693.

Lucier

, Dettmann

, Da Pra

. Direct-to-Consumer Sales of Vegetables Rising. Vegetables and Melons Outlook, VGS-333. Economic Research Service, 2009. Available at: http://www.ers.usda.gov/publications/vgs/#data, accessed September 14, 2010 .

10.

National Agricultural Statistics Service, United States Department of Agriculture 2007 Census of Agriculture, 2007. Available at: http://www.agcensus.usda.gov/Publications/2007/Full_Report, accessed September 14, 2010 .

11.

Parker

, Wilson

, Lejuene

, Rivers

, Doohan

. An expert guide to understanding grower decisions related to fresh fruit and vegetable prevention and control. Food Control, 2012; 26:107–118.

12.

Rangarajan

, Pritts

, Reiners

, Pedersen

. Focusing food safety training based on current grower practices and farm scale. Hort Technol, 2002; 12:126–131.

13.

Tobin

, Thomson

, LaBorde

, Radhakrishna

. Factors affecting growers' on-farm safety practices: Evaluation findings from Penn State Extension Programming. Food Control, 2013; 33:73–80.