Claim Assessment Profile: A Method for Capturing Healthcare Evidence in the Scientific Evaluation and Review of Claims in Health Care (SEaRCH)

Introduction

Many people make assumptions that the care they receive is grounded firmly in evidence. It was famously reported that only about 15% of medical practices are evidence-based. ¹ Since the 1980s, a push toward growing evidence-based medicine and practice has occurred. More recently, an analysis of common medical treatments determined that the effectiveness of 46% of medical treatments is unknown. ² Certainly, tried and true treatments with years of success and minimal adverse events may not warrant experimental testing (such as aspirin for headache), but many complementary and integrative health (CIH) practices and interventions or newly “discovered” or innovative technologies must be evidence-based.

However, a problem arises when results attained from the primary methods for studying causation (double-blind, randomized, controlled clinical trials [RCTs]) do not translate into the delivery setting or are impossible to conduct or resource. Further, evidence-based medicine that relies heavily on RCTs tends to exclude important qualitative and observational information about the use and benefits of some therapies, information that is often essential for patient-centered care. ³ The results of the Institutes of Medicine roundtable on evidence-based medicine found that beyond basic efficacy and safety, the use of gold standard trials and large outcomes studies to establish effectiveness and variation is often impractical. ⁴ It went on to suggest that “the speed and complexity with which new medical interventions and scientific knowledge are being developed often make RCTs difficult or even impossible to conduct.” ¹ Thus, new methods whereby data are collected from and applied in naturalistic healthcare contexts are needed. To fill these gaps in methodology, the Scientific Evaluation and Review of Claims in Health Care (SEaRCH) ⁵ is a systematic, stepwise, streamlined set of methods to evaluate CIH practice claims. It uses a sequenced, synergistic program of methods that includes field investigations for claims in practice; systematic reviews ⁶ for current evidence; and expert panels ⁷ for determining application, policy, or patient preference and the direction for further research. This sequence allows a progression through the incremental steps necessary for healthcare evaluation. A key component of SEaRCH is the Claim Assessment Profile (CAP), which studies treatments in their naturalistic setting and offers the first step toward evidence-based medicine for healthcare practices where only anecdotal evidence currently exists. The evolution of the CAP, its methods, and applications are described here.

Field Investigations for Healthcare Claims

Conducting research within a complex practice setting is challenging. In 1995, the National Institutes of Health's Office of Alternative Medicine created the Field Investigation and Practice Assessment (FIPA) program. Under FIPA, the Office of Alternative Medicine conducted dozens of site visits of CIH practices around the world. The major goals of this program were to (1) develop interest in research of complementary and alternative medicine practices that offered promising therapies for specific diseases, (2) assess the feasibility of conducting a practice outcomes evaluation, and (3) evaluate cases to see whether sufficient data existed to conduct retrospective and/or prospective outcomes studies. The Office of Alternative Medicine followed up on these site visits by contracting with the Centers for Disease Control and Prevention to conduct formal field studies of promising CIH clinical practices. The FIPA program was further developed in 2003 under the congressionally mandated Complementary and Alternative Medicine Research for Military Operations and Health Care Program and was renamed the Epidemiological Documentation Service. Run for several years by the Foundation for Alternative and Integrative Medicine (formerly named National Foundation of Alternative Medicine), the program was later transferred to Samueli Institute in 2008, where it was further developed, tested, and renamed SEaRCH. ⁵

Generally, field investigations are performed when the use of descriptive studies would be valuable for understanding practice use and generating hypotheses before application of other analytic study designs, or to determine when the existing practice data are sufficient to warrant further study. These initial field investigation methods have been formalized and enhanced to deliver the CAP within the SEaRCH program.

Methods/Design

The CAP provides an in depth, cross-sectional snapshot of the characteristics of a clinic, practice, program, or product to determine what the practice is, what it purports to do (the claim), and whether it would be feasible to collect and analyze data about the claim within the naturalistic setting of the practice. The CAP method requires expertise in program evaluation and qualitative research methods, with subject matter expertise in the claim being assessed. The CAP draws on several established methods, including rapid assessment processes developed over the last 20 years and program theory-driven evaluation science, which offer innovative and highly collaborative techniques for figuring out causal links. ^8,9 The CAP is conducted in the spirit of “appreciative inquiry,” in that it is a strengths-based collaboration with clients to discover “the best of what is.” ¹⁰

The CAP is not intended to be a comprehensive outcome evaluation; however, its methods are nonetheless systematic and rigorous. The approach is designed to (1) describe and clarify a claim, (2) gather information about the current practice, and (3) determine the practice's capacity to participate in further evaluation. This is done through a collaborative data collection approach with structured qualitative interviews, site visits, administrative/structural data, and other observational data gathered by the evaluation team. This method has been used previously to understand practice implementation and move claims along to the next level of study. By conducting field investigations via a CAP, clients are informed on how to articulate the structures and organization of their practice, how to frame research questions, and how to proceed with further research and evaluation. The outcome of a CAP is a descriptive report of the practice, its activities, its claims, and its readiness for future research.

Methodologic Summary of CAP

For implementation of a CAP, research evaluators meet with clients to discuss the scope of the claim and decide whether this method is appropriate for the client. An evaluator works with the client to design a clear research question and approach for the assessment, including the purpose, background, and specific aims. Then a team is assembled, including members from the practice and evaluators, in a collaborative effort, which is the keystone of rapid assessment process, program theory-driven evaluation science, and appreciative inquiry. ^9,11,12 A site visit is planned and a tailored questionnaire is created for the client.

During the site visit, interviews are conducted and observational and survey data are collected. An iterative data collection and analysis method is used, during which evaluation team members work in close collaboration with the staff at the site and use data triangulation to support and refute findings that emerge with other team members and the site staff to maximize their expertise. ⁸ The evaluation is team-based and appreciative in that it concentrates efforts in the study first and foremost on the positive elements of the program. ¹¹

Research questions, outcomes, and causal links are defined in collaboration with the client by using logic modeling based on program theory. ⁹ This approach uncovers the outcomes that are most impactful to participants and provides data from which to glean the most appropriate patient-centered research design.

A final report is produced. It describes the practice; what can be concluded about the research question formulated; what data exist that can support the claim; and what are the readiness, capacity, and willingness of the site staff for further evaluation.

There are criticisms that the findings can be less valid, reliable, and credible than those obtained through more formal methods. However, McNall and Foster-Fishman ¹³ countered these criticisms with an in-depth study of these rapid evaluation methods in which they highlighted the importance of increasing trustworthiness of findings through the use of a confirmatory and dependability audit trail process. That process was recommended by Guba and Lincoln (1989) as critical for ensuring that methods are logical and strategically employed and that findings accurately portray stakeholder perceptions. ¹⁴

Use of the CAP

The CAP is useful in a variety of settings and purposes (three examples are described below), but when does it make sense to use it? The rapid timeline of the method suggests the suitable context: when there is a vital need for information on which action will be taken soon. The findings are instrumental and not meant to provide hypothesis testing; rather, they are meant to develop explanatory models to inform hypothesis generation. The CAP provides information about evaluation readiness and capacity of the practice to conduct research. The data provide sufficient quality to assist clients in decision-making about next steps in the evaluation cycle. A required condition for a successful CAP is a client who is open to learning and ready to take action on the basis of findings. The CAP provides information to the client on the possibilities, readiness, and resources for further evaluation of their health claim. Further, the assessment results can engage expert review for such uses as setting research agendas, deciding on clinical appropriateness, determining patient relevance, or developing policies. It is useful for clinicians, practitioners, patients, administrators, researchers, or product developers wherever there is a health claim that is anecdotal or based on unknown evidence. A clinician or CIH practitioner may be seeing positive results from their treatments and are motivated toward evaluation as a way of understanding their own treatment effects, as well as for improving their practice and making it more widely used, accepted, and understood by the general public.

A case example of a CAP study is the assessment of a large integrative cancer clinic in Canada. ¹⁹ Samueli Institute was invited to work with Canada's first and foremost, MD-led, comprehensive cancer care center that uses an internationally recognized model of integrated care. Since 1997, it has provided care to more than 5500 British Columbians, with a primary focus on cancer. The center was interested in strengthening its evaluation efforts, and the Canadian government and a local nonprofit foundation were interested in knowing more about the clinic's practices and outcomes. Using the CAP method, evaluators helped to clarify, describe, and document the practices at the clinic in terms of care provided, delivery of services, and expected changes in specific outcomes for the practitioners and patients. Evaluation staff joined forces with center staff in logic modeling and causal path mapping exercises. Outcomes were defined and a path model was created, forming the basis for future rigorous scientific evaluation. Further, evaluators assessed the readiness and resources of the practice to engage in research through review of existing data collection capabilities. The CAP method clarified the claim and gathered information about the current practice in order to provide the public with an independent description of the practice as delivered, as well as determining the practice's readiness and ability to be involved in research. This work was instrumental in moving the center toward its next steps in outcomes evaluation, which was a prospective trial.

Another example is in the application of the CAP to define research questions and expected impacts for a resilience education training program within a behavioral health department. The program staff were creating and delivering contents to participants. but they did not have the research acumen needed to design an evaluation. Therefore, a CAP was conducted that described the program and provided a logic model that defined the structure, process, and outcomes of interest. The results from the CAP fed directly into a comprehensive mixed methods program evaluation.

A final example of the application of these techniques was conducted on a product that is a noninvasive, drug-free alternative therapy developed for individuals with fibromyalgia, delayed-onset muscle soreness, and phantom limb pain. The product consists of iron, nickel, and chromium fibers woven into a nylon fabric that can be custom made into socks, gloves, jackets, blankets, or limb covers. Various forms or sizes are applied to painful areas of the body. This product was being used in a solo physician practice with a chiropractic doctor, receptionist, and insurance/billing professional. The clinic had access to an ethics/independent review board committee for research but was not affiliated with a university-based collaborator. Samueli Institute implemented a CAP that defined the product, its application, and outcome claim in phantom limb pain. This led to a retrospective observational study followed by a randomized, placebo-controlled clinical trial.

Figure 1 offers a visual guide for the steps in the CAP.

OPEN IN VIEWER

FIG. 1.

Basic steps in the Claim Assessment Profile (CAP).

Data analysis

The types of data produced from the surveys and interviews implemented in the CAP dictate the data analysis approach. Descriptive data are analyzed by organizing the responses into topics to be reviewed and written up in a narrative report. This approach is based on Yin's work on case report research. ²⁰ The exploratory questions yield data that require an analytic coding technique to refine concepts for summarizing in a final report. This is based on the work of Miles and Huberman, whose analytical coding approach is illustrated in a graphical display. ²¹ If the client is oriented toward operationalizing outcome variables and the underlying causal links, then time is spent during the site visit for logic modeling in order to find what the staff and participants feel is most impactful. See Figure 2 for the model of the iterative and interactive data analysis approach.

OPEN IN VIEWER

FIG. 2.

Data analysis methods for iterative and interactive model.

Limitations

The limitations of this methods paper are that we were not able to provide specific examples of data from previous CAP reports because we do not have permission to publish data from client reports of programs that are beginning their evaluation processes.

Discussion

CIH practices vary considerably from site to site and practice to practice. The first step in evaluation of CIH is to accurately describe the practice. The CAP is a rapid assessment approach that describes the background, characteristics, and procedures of a clinic, practice, program, or product in order to define research questions and assess readiness and resources for engaging in research of the claim. The CAP is part of a suite of methods called SEaRCH, which was created to advance promising CIH practices into evaluation. This approach has been developed and tested primarily to aid in the rigorous and scientific evaluation of clinical practices that are already in use. However, the approach was designed with the flexibility for application beyond healthcare and wellness. There are numerous methods for evaluation of health practices, but many of the most rigorous scientific methods, such as RCTs or quasi-experimental studies, are expensive and difficult to conduct for clinical staff, who may not have the research background, capacity, experience, or resources to perform evaluation activities. The CAP process is the first step toward making research feasible for numerous health and wellness treatment modalities where clinical, anecdotal, or case-study evidence exists or the evidence is unknown.