An Initial Evaluation of a Concurrent Operant Analysis Framework to Identify Reinforcers for Work Completion

Abstract

Although functional analysis is a powerful tool for testing the function of challenging behavior, it is not always feasible or appropriate to include as a component of functional behavior assessment (FBA). Alternative experimental analysis methods are needed to inform individualized interventions in schools, particularly for students who engage in passive forms of problem behavior. We evaluated a concurrent operant analysis (COA) framework to identify reinforcers for appropriate replacement behaviors for four students referred for FBA and reported by teachers to engage in low levels of work completion. After completing two COAs per student (researcher-as-therapist and teacher-as-therapist), we used alternating treatments designs to compare the effects of an intervention matched with COA outcomes to intervention conditions that were not matched to COA outcomes on levels of work completion and task engagement. COA outcomes corresponded across therapists for three of four participants and intervention results validated COA outcomes for two of these participants. Although results of this initial investigation seem promising, more research on COA frameworks is needed to determine their utility to guide reinforcement-based interventions in schools.

Keywords

concurrent operants concurrent operant analysis choice assessment functional behavior assessment

The effectiveness of function-based interventions to reduce challenging behavior is well-documented (Goh & Bambara, 2012; Walker, Chung, & Bonnet, 2018), with evidence suggesting function-based interventions are more effective than those that are not matched to behavior function (e.g., Briere & Simonsen, 2011; Miller & Lee, 2013). Function-based interventions are designed from results of a functional behavior assessment (FBA). Of all the assessment strategies that can be included in an FBA, experimental analysis is the most rigorous, as it allows for hypotheses to be directly tested and confirmed (Hanley, 2012).

To date, the analysis method with the most empirical support for informing effective function-based interventions is the functional analysis (Beavers, Iwata, & Lerman, 2013; Gage, Lewis, & Stichter, 2012). In a functional analysis, levels of challenging behavior are compared between two or more conditions in which specific antecedents and/or consequences of these behaviors are programmed. Conditions that consistently produce higher levels of problem behavior are identified as evoking or maintaining these behaviors. Although research on practical variations of the functional analysis for school settings has increased markedly in recent years (Lloyd, Weaver, & Staubitz, 2016), functional analyses are rarely used in practice (e.g., O’Neill, Bundock, Kladis, & Hawken, 2015; Roscoe, Phillips, Kelly, Farber, & Dube, 2015).

Several barriers to completing functional analysis in schools have been identified. First, functional analysis requires evoking problem behavior to produce interpretable outcomes. This requirement can be seen as presenting unnecessary risk and disruption to instructional routines (Hanley, 2012; Roscoe et al., 2015). Second, relative to the population for whom functional analyses were originally designed (i.e., individuals with intellectual/developmental disabilities who engage in self-injury; Iwata, Dorsey, Slifer, Bauman, & Richman, 1982/1994), a wider range of students are being referred for FBAs in schools. In multitiered systems of support, students are referred for additional behavioral supports when they are not responsive to less intensive strategies (Crone, Hawken, & Horner, 2015). This means students with and without disabilities, and those who engage in a wide variety of behaviors that interfere with learning, may be referred for FBA (Erickson, Stage, & Nelson, 2006; Goh & Bambara, 2012; McKenna, Flower, Kim, Ciullo, & Haring, 2015). Applications of functional analysis to a heterogeneous student population—including a range of behavioral profiles—presents both practical and methodological challenges.

Particularly in academic contexts, a common behavioral concern is an absence of appropriate behavior during instruction (e.g., noncompliance, low levels of task engagement). Even though these behavioral deficits do not present an immediate safety risk, chronic patterns have been shown to negatively impact student achievement, teacher and peer relationships, and educational placements (Kalb & Loeber, 2003). These behavioral deficits also present methodological challenges to conducting a functional analysis. One challenge relates to defining and delivering consequences contingent on the absence, rather than the presence, of a target behavior. Another challenge involves designing an appropriate control condition when task demands must be present to preserve the opportunity for passive problem behavior to occur (Thompson & Iwata, 2005). Thus, although functional analysis is a powerful tool, there are conditions in which school teams may find the procedures unacceptable or unlikely to produce interpretable results based on a student’s behavioral profile. For these reasons, alternative experimental analysis methods are needed to inform individualized interventions in schools.

One alternative is a concurrent operant analysis (COA). A COA is an analysis in which two or more stimulus conditions are simultaneously available, with access to each condition contingent on some response (Fisher & Mazur, 1997). Rather than the occurrence of problem behavior, patterns of choice allocation are used to inform relative preference between two or more conditions. Results of preference assessments using concurrent operant arrangements have been shown to predict reinforcer value (Fisher et al., 1992; Piazza, Fisher, Hagopian, Bowman, & Toole, 1996), suggesting COAs can be used to identify reinforcers for appropriate behaviors.

COAs represent a promising alternative to the functional analysis for several reasons. First, assessment procedures do not involve evoking and reinforcing problem behavior, and interpretable outcomes do not hinge on whether consistent patterns of problem behavior are captured. Thus, practitioners may consider COA procedures more acceptable relative to those used for functional analysis (Lloyd, 2018). Second, the measurement of choice allocation (as opposed to problem behavior) represents a useful alternative for students who engage in behaviors that present measurement or procedural challenges to functional analysis. For example, Quigley et al. (2013) conducted a COA following an inconclusive functional analysis for a student with passive problem behavior. During the COA, the student consistently allocated his time to conditions that did not involve work, suggesting passive problem behavior was maintained by negative reinforcement.

Third, COAs may be advantageous when there is reason to suspect problem behavior is maintained by escape—a commonly identified function for noncompliance and other problem behaviors occurring in the context of academic instruction (McKerchar & Abby, 2012). Because response effort has been shown to impact compliance in young children (e.g., Wilder, Fischetti, Myers, Leon-Enriquez, & Majdalany, 2013), escape alone can be insufficient to reinforce compliance or work completion as a higher effort alternative to problem behavior (Neef, Shade, & Miller, 1994). In the study by Quigley et al. (2013), researchers designed an intervention in which the student chose between (a) working to earn access to adult attention and preferred items and (b) a break without access to these consequences. Even though escape from work was available as an option, the student shifted his responding toward the work alternative when work completion was reinforced with attention and preferred activities. In another example, Gardner, Wacker, and Boelter (2009) used COAs to demonstrate that two typically developing children with escape-maintained problem behavior chose to complete academic work when it was paired with high-quality attention, even when escape from demands was available for an alternative response. Thus, COAs may be an efficient strategy for identifying potential reinforcers for replacement behaviors that are more effortful than problem behavior. A final related advantage is that COAs have potential to inform intervention packages that do not require escape extinction (i.e., a procedure in which interventionists withhold escape when escape-maintained problem behavior occurs). This procedure is difficult for teachers to implement consistently, can require intrusive forms of prompting, and has been shown to produce unpleasant side effects (e.g., Ward, Parker, & Perdikaris, 2017).

In contrast to the expansive literature on functional analysis, the literature on COAs includes a small collection of studies using varied procedures, conditions, and sequences of comparisons to address specific questions about study participants’ preferences (e.g., Berg et al., 2007; Harding, Wacker, Berg, Lee, & Dolezal, 2009; Quigley et al., 2013). To date, no published studies have proposed or evaluated a standard COA framework to identify potential reinforcers for appropriate behavior. In an unpublished dissertation study, Casey (2001) developed a COA framework based on conditions included in a study by Harding et al. (1999). The framework included a series of pairings of up to eight conditions using concurrent operant arrangements to assess the relative value of attention, tangible, and escape as reinforcers for appropriate (i.e., choice-making) behavior. Casey completed COAs and brief functional analyses for 23 children with high-incidence disabilities who were referred to an outpatient clinic for noncompliance and other problem behaviors. Results of brief functional analyses were inconclusive for 10 participants, whereas results of COAs were conclusive for all participants. Moreover, for the 13 children for whom functional analyses produced conclusive outcomes, results matched or partially matched those of the COA for nine children. This COA framework has been used successfully by state-funded technical-assistance teams who provide behavioral consultation and related professional development services in public school settings (e.g., Bassingthwaite & Casey, 2016; Mews, Bassingthwaite, & Ausenhus, 2018; Simcoe, Staubitz, Gregory, & Juarez, 2018). However, no studies to date have attempted to replicate this COA framework and evaluate its utility to inform individualized interventions in school settings.

The purpose of the current study was to evaluate the utility of a COA framework (Casey, 2001) to identify reinforcers for work completion for students referred for FBA and reported to engage in low levels of work completion. The following research questions guided the study:

Research Question 1: Does the COA framework lead to an interpretable outcome (i.e., the identification of a potential reinforcer for work completion)?

Research Question 2: With minimal training, can teachers implement COA conditions with fidelity, and do student outcomes differ depending on whether research staff or teaching staff serve as therapists?

Research Question 3: Does an intervention based on COA outcomes produce higher levels of (a) work completion and (b) task engagement relative to intervention conditions that are not based on COA outcomes?

Method

Participants

After obtaining study approval from Vanderbilt University’s Institutional Review Board, we recruited student participants who (a) were in grades K–5, (b) received special education services or were identified by school personnel as at risk for a disability (e.g., special education referral), (c) were referred for individualized assessment and intervention (i.e., FBA and behavior support plan), and (d) were nominated by their classroom teacher as a student who engaged in low levels of compliance and/or work completion during one or more instructional routines. Inclusion criteria were evaluated based on interviews with teachers, record reviews, and an initial classroom observation. We obtained informed consent from each student’s parent and teacher(s) and verbal assent from each participating student before collecting any study data.

Four students and four teachers participated in this study. All four students scored in the Abnormal range on the Strengths and Difficulties Questionnaire (SDQ; Goodman, 2001), which was completed by their teachers. Demetrius was a 9-year-old Black boy in fourth grade who was diagnosed with attention deficit hyperactivity disorder (ADHD) and had average intellectual functioning. He attended a public special day school for students with emotional/behavioral disorders. Demetrius’s classroom teacher was a 25-year-old White woman who was in her first year of teaching; she had a master’s degree in special education and was a Board Certified Behavior Analyst. Donny was a 9-year-old Black boy in fourth grade who attended a public elementary school. He had been referred by his school’s student support team for special education evaluation, but was not receiving services at the time of participation. His teacher was a 27-year-old White woman with a bachelor’s degree, elementary teaching certification, and 5 years of classroom teaching experience. Charles and Braden were two 6-year-old boys who received special education services under developmental delay at a public elementary school. Charles was Black and Braden was Black/American Indian. Their kindergarten classroom teacher was a 26-year-old White woman with 3.5 years teaching experience and certification in both K–6 general education and K–12 special education. Their special education teacher was a 53-year-old White woman with 15 years of experience and certification in special education.

Settings and Materials

Demetrius’s COA sessions were conducted in a small conference room at his school. For Donny, Charles, and Braden, researcher-as-therapist COAs were conducted in empty auxiliary rooms in the school building and teacher-as-therapist COAs were conducted in their usual classrooms while other students were in art or physical education classes. Intervention sessions were conducted in the usual classroom during scheduled 1:1 instruction (Demetrius), at a desk in the hallway outside the usual classroom (Donny), and in a teacher planning room (Charles and Braden).

We arranged each COA at a table with a line of masking tape dividing the space in half. We also placed lines of tape on the floor to clearly demarcate two contiguous choice areas. Materials associated with each choice condition (i.e., preferred items, neutral/low-preferred items, task materials; see Table 1) were placed on corresponding sides of the table. Materials used during intervention sessions included: 8.5 × 11 in. (21.59 × 27.94 cm) laminated token boards, three sets of 1 × 1 in. (2.54 × 2.54 cm) laminated tokens, and academic worksheets.

Table 1.

COA Procedures by Condition and Participant.

	Participant
Condition	Demetrius	Donny	Charles	Braden
A: Demand without attention	Student sits alone and completes work independently
Materials	Math worksheet	Math worksheet	Phonics worksheet	Phonics worksheet
B: Free play with attention and preferred items	Student plays with therapist and preferred items (no prompts)
Materials	iPad and LEGO™ toys	iPad and LEGO™ toys	Play-doh, LEGO™ toys, bubbles, puzzle, ball, toy car	Laptop, play-doh, LEGO™ toys, bubbles, puzzle, ball, toy car
C: Directed play with preferred items	Therapist delivers prompts on how to play with preferred items (i.e., “play my way”)
Materials	iPad and LEGO™ toys	iPad and LEGO™ toys	Play-doh, LEGO™ toys, bubbles, puzzle, ball, toy car	Laptop, play-doh, LEGO™ toys, bubbles, puzzle, ball, toy car
D: Free play with preferred items and without attention	Student plays alone with preferred items
Materials	iPad and LEGO™ toys	iPad and LEGO™ toys	Play-doh, LEGO™ toys, bubbles, puzzle, ball, toy car	Laptop, play-doh, LEGO™ toys, bubbles, puzzle, ball, toy car
E: Demand with attention	Therapist delivers prompts and assistance to complete academic task
Materials	Math worksheet	Math worksheet	Phonics worksheet	Phonics worksheet
F: Alone	Student sits alone in area without any items or activities
Materials	NA	NA	NA	NA
G: Free play with attention and low-preferred items	Student plays with therapist and neutral or low-preferred items (no prompts)
Materials	Pot holder, coaster, coffee filter
H: Free play with low-preferred items and without attention	Student plays alone with neutral or low-preferred items
Materials	Pot holder, coaster, coffee filter

Note. COA = concurrent operant analysis.

We worked with each student’s teacher to select academic tasks to include in the COA and intervention comparisons. We asked teachers to identify tasks that were within each student’s skill repertoire but for which low levels of work completion were commonly observed. For Demetrius and Donny, we selected brief (140–180 words) fiction passages with a set of 10 comprehension questions. Passages were on a third-grade reading level and accessed from ReadWorks®. To control for task difficulty across conditions, we randomly assigned passages to intervention conditions and used the same set of 10 comprehension questions (e.g., Who is the main character?) across sessions. For Charles and Braden, we selected worksheets on matching pictures to letter sounds. To control for task difficulty across conditions, we randomly selected 10 letters for each worksheet and randomly assigned worksheets to conditions. During intervention sessions, we used a Time Timer™ to track work time and reward time.

Measurement System and Response Definitions

During each COA, we measured student choice allocation. We used Lily data collector (Tapp, 2010) on Dell™ electronic tablets to measure duration of choice allocation via timed-event sampling. There were three mutually exclusive duration codes: Choice Area 1, Choice Area 2, and No Choice. Allocation to each choice area was defined as having both feet in one of the two choice areas. No Choice was coded when one or both feet were outside a choice area or on the dividing tape. Changes in choice allocation were coded based on a 3-s offset (e.g., if a student left Choice Area 1, Choice Area 2 was coded only after the student remained in Choice Area 2 for at least 3 s). The dependent variable was the percentage of session time spent in each choice area, calculated as the number of seconds allocated to each area divided by the total session duration (s) and multiplied by 100.

During intervention sessions, we measured work completion via permanent product and task engagement using timed-event sampling. We measured work completion as the total number of questions answered or worksheet items completed in each 5-min session. We consulted with each student’s teacher to determine requirements for completed items. For Demetrius and Donny, answers to comprehension questions had to (a) begin with a capital letter, (b) end with punctuation, and (c) be related to passage content (i.e., contain at least one context-specific word related to the passage). For Charles and Braden, a letter had to be matched correctly to the corresponding picture (e.g., a line drawn from the letter “B” to a picture of a bumblebee) to be counted as a complete response. We defined task engagement as being oriented toward the academic materials and demonstrating one or more of the following behaviors: (a) writing (or erasing) with pencil on the worksheet in a manner consistent with the academic activity, (b) reading aloud, (c) scanning the page and/or turning the page over, or (d) asking a question pertaining to the worksheet (adapted from Kamps, Wills, & Wehby, 2012). We coded engagement duration using a 5-s offset rule. We calculated the percentage of the work period the student was engaged as the total number of seconds engaged divided by the duration of the work period(s) and multiplied by 100.

Experimental Design

For each COA, we used a simultaneous treatments experimental design (Barlow & Hayes, 1979) and a predetermined sequence of comparisons (Casey, 2001; see Figure 1) to evaluate choice allocation among concurrently available conditions. We changed conditions when the student allocated 70% or more of the 5-min session (i.e., 3.5 min) to one choice area; we repeated the same condition if less than 70% of session time was allocated to one choice area.

Figure 1.

Decision framework for concurrent operant analysis.

For each intervention comparison, we used an alternating treatments design (Barlow & Hayes, 1979) to compare effects of three potential intervention conditions with differing reinforcement contingencies on work completion and task engagement. With few exceptions, we completed one set of block randomized sessions (i.e., one session per condition, in a randomly determined sequence) per day. We used visual analysis to evaluate response differentiation among the rapidly alternating conditions. We identified one condition as more effective than another if the level of work completion or task engagement was consistently higher across three or more series of conditions within the same phase (Wolery, Gast, & Ledford, 2018). We hypothesized students would complete more work and be more engaged in conditions corresponding with their COA outcomes relative to the other conditions.

Study Procedures

Indirect assessments

We used a combination of open- and closed-ended interview and questionnaire materials (i.e., modified Functional Assessment Interview [FAI; O’Neill et al., 1997], SDQ [Goodman, 2001], researcher-developed COA materials interview) to collect information from teachers on student problem behavior and potential contributing factors, preferred activities, and representative instructional activities. Teacher responses informed the stimuli (e.g., tangible items, forms of attention, instructional tasks) included in each student’s COA and intervention comparison (see Table 1).

Concurrent Operant Analysis

We completed two COAs for each participant 1 to 2 weeks apart. Each COA was completed within a 1-hr visit or across two 30-min visits on different days. A research team member served as therapist during the first COA; the student’s teacher served as therapist for the second COA. For Charles and Braden, the general education teacher conducted the teacher-as-therapist COAs with one exception: the final session of Braden’s COA was completed by the special education teacher based on availability. Aside from therapist role, all procedures were identical across both COAs. Prior to the teacher-as-therapist COA, we held a brief meeting with the teacher to review COA procedures and role-played the different forms of interaction the teacher might be asked to provide depending on student choices.

We replicated the branching framework used by Casey (2001) to select, compare, and sequence all COA conditions. The COA framework included a total of eight possible choice conditions (see Table 1), which were paired against one another in a hierarchical sequence to assess the relative value of attention, tangibles, and escape as potential reinforcers (see Figure 1; Casey, 2001). Each participant was exposed to at least five choice conditions. During each COA session, we presented two concurrently available activities on opposite sides of a table. Before each session, a graduate research assistant (i.e., the facilitator) explained each available choice. When attention or prompts were programmed in a condition, a therapist (either a second graduate research assistant or the student’s teacher) was seated on the side of the table associated with that condition. The facilitator checked for understanding and asked the student if he had any questions before beginning the session. If a student asked a question, the facilitator answered him and demonstrated what would happen in each choice area via role play. The facilitator also told the student he could switch between the two activities at any time. The facilitator then prompted the student to stand at a location equidistant to each area that was marked with an X. The facilitator instructed the student to “Make a choice” and the session began. At least once during the 5-min session, the facilitator reminded the student he was free to change sides at any time.

Intervention comparison

Prior to conducting COAs, we designed three intervention conditions to assess the validity of COA outcomes. Each intervention condition was based on one of three outcomes (i.e., tangible, attention, escape). Our goal was to create intervention conditions that only differed with respect to the type of reward earned for task completion. We programmed these reward contingencies in a token economy based on its potential to be incorporated in classroom instruction. Research assistants implemented intervention sessions. Across conditions, sessions consisted of a 5-min work period during which up to 10 tokens were delivered contingent on work completion. Following the work period, students exchanged the tokens they earned for access to a potential reinforcer. Each token was worth 30 s of access.

Before beginning intervention, the interventionist introduced each student to the token board and each set of tokens and described what they could be exchanged for. She paired tokens with each reward by prompting the student to exchange each type of token and delivering 30-s access to the corresponding reward. The interventionist began each session with a contingency review to specify (a) what the student would be working for, (b) the work requirement to earn one token, and (c) when the student could exchange his tokens. She told the student he could do as much or as little work as he wanted and asked if he had any questions. After reviewing the rules, the interventionist set a visual timer for 5 min and the session began. For Demetrius and Donny, each completed comprehension question resulted in a token delivery. For Charles and Braden, one token was delivered for every three worksheet items completed. If a question or item was partially completed, the interventionist provided a verbal reminder of the completion requirement. However, if the student chose not to complete work, she did not deliver any further prompts. She responded to student questions related to the assigned task, but did not respond to other bids for attention. When the 5-min work period ended, the interventionist counted the tokens aloud and indicated how much reward time was earned. She then set the visual timer to match the total reward time earned and provided access to the reward. Following sessions in which the student did not complete any work, the interventionist indicated that the student did not earn any tokens this time, but could choose to work for tokens in the next session.

Tangible condition

In the tangible condition, the student worked to earn tangible tokens, which had a yellow border and an image of each student’s preferred items. Worksheets were printed on yellow paper to facilitate discrimination among conditions. When tangible tokens were exchanged, the interventionist provided access to the same highly preferred tangibles evaluated in each participant’s COA, but did not otherwise interact with the student.

Attention condition

In the attention condition, the student worked to earn attention tokens, which had a green border and a speech bubble icon. Worksheets were printed on green paper. When attention tokens were exchanged, the interventionist provided continuous attention without delivering prompts; preferred items were withheld. The type of social attention varied across students based on preference, which was informed by both teacher and student report.

Escape condition

In the escape condition, the student worked to earn escape tokens, which were white with a black border and no icon. Worksheets were printed on white paper. When escape tokens were exchanged, the interventionist provided a break without attention or preferred items. During these breaks, she oriented away from the student. Although we conceptualized the escape condition to represent an intervention matched to an escape outcome, we also designed it to inform whether the provision of tokens alone increased work completion. This is because participants could freely access escape during all sessions. If a participant chose to complete work to earn tokens across conditions, including escape tokens that could be exchanged for breaks that were already freely available, this pattern might suggest the delivery of tokens alone (regardless of what they were exchanged for) reinforced work completion.

Attention + tangible condition

We added a condition for Donny based on results of the COA and information from the first series of intervention sessions. In this condition, Donny worked to earn attention + tangible tokens, which had a blue border and images of a speech bubble and iPad. Worksheets were printed on blue paper. When attention + tangible tokens were exchanged, the interventionist provided access to both preferred items and social attention (i.e., access to the iPad while the therapist provided comments on the games he played).

Choice probe

For three of four participants, we completed choice probes intermittently or at the end of the intervention comparison. During a choice probe, the interventionist presented all three token types and asked the participant which kind of token he wanted to work for. When the participant made a choice by either naming or touching the token type, we conducted that condition as described in the previous sections. We attempted a choice probe for Charles, but were not able to complete it due to the occurrence of problem behavior when the choice was presented; due to time constraints, we were unable to attempt additional choice probes.

Interobserver Agreement

For each participant, we collected interobservor agreement (IOA) data on a minimum of 33% and 30% of COA and intervention sessions, respectively. For COA sessions, we calculated point-by-point agreement on choice allocation by dividing the number of seconds in which both observers agreed on choice allocation by the total number of seconds in the session (300). Mean agreement was 98.2% (range = 92.8%–100%) across participants and conditions. For intervention sessions, we calculated point-by-point agreement on work completion and task engagement. To estimate agreement on work completion, two coders independently scored each item on the worksheet as complete or incomplete. Agreement on complete responses was calculated as the number of items both observers scored as complete divided by the total number of items. Mean agreement on work completion was 100% for Demetrius and Donny, 84.3% (range = 75.0%–100%) for Charles, and 96.8% (range = 90.0%–100%) for Braden. For Charles and Braden, disagreements occurred due to difficulty determining whether lines correctly connected corresponding letters and pictures when multiple lines were drawn on top of each other. To calculate agreement on task engagement, we divided the number of seconds in which both observers agreed on whether the student was or was not engaged by the total number of seconds in the work period (300). Mean agreement on task engagement was 97.9% (range = 94.6%–100%) for Demetrius, 97.7% (range = 88.5%–100%) for Donny, 89.7% (range = 77.0%–98.2%) for Charles, and 94.5% (range = 83.3%–100%) for Braden. When disagreements occurred, we relied on the primary observer’s codes for data entry; however, observers discussed and resolved discrepancies between sessions to prevent similar disagreements in subsequent sessions.

Procedural Fidelity

We used paper-and-pencil data sheets to collect procedural fidelity data during COAs. We collected data on the occurrence of the following set-up variables for each session: Correct materials were in each choice area, facilitator explained rules for each choice area, and student began each session at the designated choice location. During each COA session, we collected 30-s partial interval data on student behaviors (i.e., choice allocation, item interaction), therapist behaviors (i.e., attention delivery, directed play prompts, academic prompts), and facilitator behaviors (i.e., rule reminders, redirections). Because some of these behaviors were performed on an as-needed basis, we coded the presence or absence of each behavior during the COA session. Then, following the session, we scored each interval as correct or as containing at least one procedural error based on the student, therapist, and facilitator behaviors coded as present. We summarized fidelity data as a percentage of correct implementation (i.e., number of intervals scored as correct divided by total number of intervals [10], multiplied by 100). We collected procedural fidelity data during all COA sessions. Across participants, mean procedural fidelity exceeded 99.0% for both researcher-as-therapist sessions (range = 88.9%–100%) and teacher-as-therapist sessions (range = 94.4%–100%). We collected IOA data on procedural fidelity for 47.8% of researcher-as-therapist sessions and 62.5% of teacher-as-therapist sessions. Mean IOA for procedural fidelity data was 99.7% (range = 96.4%–100%) and 99.1% (range = 96.4%–100%) for researcher- and teacher-as-therapist sessions, respectively.

We used tablets to collect timed-event data on procedural fidelity during intervention sessions. At the beginning of each session, we coded the occurrence of the following interventionist set-up behaviors: (a) explained rules, (b) presented correct materials, and (c) set visual timer for 5 min. During each session, we coded the start and end points of each work and reward period, occurrences of interventionist attention, and the presence of correct materials. We used these data to score whether the provision of attention and materials was correct for both the work period and reward period. Following each session, we used permanent product data (i.e., student worksheet and token board) to evaluate whether the correct number of tokens was delivered for work completion, and whether the total reward period duration was correct based on the total number of tokens earned. For each session, we calculated a percentage of total intervention components implemented correctly. We collected procedural fidelity data during a minimum of 67% of intervention sessions per condition and participant. Across participants, mean fidelity exceeded 97.0% for each intervention condition (session range = 86.7%–100%). We collected IOA on intervention procedural fidelity data for at least 50% of sessions per condition. Mean IOA on intervention procedural fidelity met or exceeded 93.0% for attention, escape, and tangible conditions (session range = 66.7%–100%). For Donny’s attention + tangible condition, we collected IOA on fidelity in only one session (11.0%), and agreement was 66.7% (one observer neglected to record occurrence of interventionist set-up behaviors).

Results

Demetrius’s COA results are presented in Figure 2. In both COAs, Demetrius consistently chose to allocate all time toward one alternative; no sessions were repeated in either assessment. Regardless of whether a research staff member or his teacher served as therapist, Demetrius consistently chose to spend time in assessment areas in which his preferred tangibles (i.e., iPad, LEGO™ toys) were available—even when he was directed on how to play with them. Based on these choice patterns and the framework shown in Figure 1, we identified access to tangibles as the most likely reinforcer for work completion.

Figure 2.

Results of Demetrius’s concurrent operant analyses with researcher-as-therapist (top) and teacher-as-therapist (bottom).

Results of Demetrius’s intervention comparison are shown in Figure 3. The total number of worksheet items completed is graphed in the top panel, and the percentage of session time with task engagement is graphed in the bottom panel. During the first three series of conditions, we noted Demetrius was engaged in work for 100% of the 5-min work period in three sessions, yet completed no more than three comprehension questions due to his handwriting pace. Because we were reinforcing completed items, rather than time engaged, we were concerned Demetrius would not contact the relevant contingencies at therapeutic levels when writing complete sentences was required to earn a token. After discussing this with his teacher, we adjusted the response requirement such that single words or phrases could count as complete answers. Following this change, we observed consistent differentiation in level between the tangible condition and each of the other conditions for both work completion and task engagement. That is, in four out of six series, Demetrius was engaged in work for 100% of the time and completed between five and 10 items in the tangible condition, but was engaged at zero or near-zero levels and completed no items in the escape and attention conditions. In the final three choice probe sessions (marked with asterisks), Demetrius consistently chose the tangible condition and completed all 10 comprehension questions. Based on the differentiation in level for time engaged and work completed, and the consistent selection of tangible during choice probes, we interpreted results of Demetrius’s intervention comparison to support his COA outcome.

Figure 3.

Effects of intervention conditions on work completion (top) and task engagement (bottom) for Demetrius.

Donny’s COA results are presented in Figure 4. One condition had to be repeated during each COA. In the teacher-as-therapist COA, we chose to repeat a condition even though the 70% response allocation was met due to a procedural fidelity error made in that session (Session 5). Across both assessments, Donny consistently allocated his time to conditions in which both attention and access to tangibles were available. When he had to choose between attention and tangible, he allocated time to both choice areas, but ultimately chose attention with low-preferred items over no attention with high-preferred items regardless of who served as therapist. According to the framework shown in Figure 1, we interpreted Donny’s choice patterns in both COAs to indicate attention as the most likely reinforcer for work completion.

Figure 4.

Results of Donny’s concurrent operant analyses with researcher-as-therapist (top) and teacher-as-therapist (bottom).

Results of Donny’s intervention comparison are shown in Figure 5. Across the first five series of conditions, we observed an increasing trend in item completion and high levels of task engagement across conditions. During intermittent choice probes, Donny chose to earn attention tokens (Session 10) and tangible tokens (Session 14). Because Donny demonstrated a preference for both tangible and attention rewards in the choice probes (as he had during the COA), we hypothesized that tokens that could be exchanged for time with both adult attention and preferred items might be the most effective reinforcer. Thus, we developed a synthesized attention + tangible condition. To minimize the potential for multi-treatment interference, we alternated the attention + tangible condition with the original escape condition only (see second phase in Figure 5). When we compared these two conditions, the increasing trend in item completion continued and the percentage of time engaged remained high in both conditions. In a final series of choice probes, Donny chose the synthesized condition in four of six opportunities and completed all work during those sessions. Because we did not identify consistent differentiation in level for the number of items completed or the percentage of time engaged, results of Donny’s intervention comparison provided no further evidence supporting his COA outcome.

Figure 5.

Effects of intervention conditions on work completion (top) and task engagement (bottom) for Donny.

Results of Charles’s COAs are presented in Figure 6. In both COAs, Charles consistently chose to allocate all time toward one alternative; no sessions were repeated in either assessment. Although Charles’s choice patterns did differ between researcher-as-therapist and teacher-as-therapist COAs, he ultimately chose to spend time in assessment areas in which access to his preferred tangibles (e.g., Play-doh, LEGO™ toys, bubbles) were available. Based on these choice patterns and the framework shown in Figure 1, we identified access to tangibles as the most likely reinforcer for work completion.

Figure 6.

Results of Charles’s concurrent operant analyses with researcher-as-therapist (top) and teacher-as-therapist (bottom).

Results of Charles’s intervention comparison are presented in Figure 7. Across all series, Charles completed more work and spent more time engaged in the tangible condition relative to the attention and escape conditions. His performance varied, however, across sessions in the tangible condition. Charles completed between one and 24 items and was engaged between 11% and 100% of work periods during tangible sessions. Unfortunately, Charles’s intervention comparison was cut short due to the end of the school year. We were neither able to evaluate whether additional exposure would have led to more stable responding in the tangible condition nor to present choice probes to evaluate Charles’s preferred intervention. Because we did identify consistent differentiation in level that favored the tangible condition for both items completed and time engaged, we interpreted results of Charles’s intervention comparison to support his COA outcome.

Figure 7.

Effects of intervention conditions on work completion (top) and task engagement (bottom) for Charles.

Braden’s COA results are displayed in Figure 8. Braden’s patterns of choice allocation were relatively consistent; only one condition in the researcher-as-therapist COA was repeated. Results were similar across COAs for the first four choice comparisons. He consistently chose to allocate all time in assessment areas in which his preferred tangibles were available. However, in the final comparison, Braden chose directed play with preferred tangibles over free play with low-preferred tangibles when the researcher served as therapist, yet made the opposite choice when the teacher served as therapist. Thus, based on the framework shown in Figure 1, the researcher-as-therapist COA suggested tangible as the most likely reinforcer for appropriate behavior, whereas the teacher-as-therapist COA suggested escape.

Figure 8.

Results of Braden’s concurrent operant analyses with researcher-as-therapist (top) and teacher-as-therapist (bottom).

Results of Braden’s intervention comparison are presented in Figure 9. With the exception of the first escape session, Braden completed zero items, and was engaged at zero or near-zero levels, in the escape condition. During two of three attention sessions and three of four tangible sessions, Braden completed between 12 and 19 items and was engaged for 83% or more of the work period. We interpreted these data as demonstrating emerging differentiation by level between both the attention and tangible conditions and the escape condition. However, no consistent differentiation was observed between the tangible and attention conditions, and performance was variable. During choice probes, Braden chose tangible (Session 12) and escape (Session 13), yet only completed work to earn tokens in the tangible condition. As with Charles, the end of the school year prevented us from conducting more intervention sessions to determine whether increased exposure to contingencies would have produced more stable levels of responding within condition. We interpreted Braden’s intervention comparison to provide preliminary support for the researcher-as-therapist COA only.

Figure 9.

Effects of intervention conditions on work completion (top) and task engagement (bottom) for Braden.

Assessment and intervention comparison outcomes are summarized by participant in Table 2. Outcomes of the researcher-as-therapist and teacher-as-therapist COAs corresponded for three of four participants. Of these three participants, higher levels of work completion and task engagement were observed in intervention conditions matched to the COA outcome than conditions not matched to the COA outcome for two participants (Demetrius, Charles); results of the intervention comparison were undifferentiated for the third participant (Donny). For Braden, COA outcomes did not correspond between therapists; results of the intervention comparison partially supported the researcher-as-therapist COA outcome.

Table 2.

Summary of Assessment and Intervention Comparison Outcomes by Participant.

Participant	COA-researcher	COA-teacher	Intervention condition(s) with highest work completion
Demetrius	Tangible	Tangible	Tangible
Donny	Attention	Attention	Undifferentiated
Charles	Tangible	Tangible	Tangible
Braden	Tangible	Escape	Tangible and attention

Note. COA = concurrent operant analysis.

Discussion

The goal of this study was to evaluate the utility of a COA framework (Casey, 2001) to identify reinforcers for work completion. Although COAs have been used to inform behavioral interventions in previous research (e.g., Piazza et al., 1997; Quigley et al., 2013), procedures have varied markedly across studies. The current study differs from previous studies in its evaluation of a proposed framework for conducting COAs to identify potential reinforcers for appropriate behavior. Rather than comparing outcomes of this COA model to outcomes of functional analyses (Casey, 2001), the current study represents an initial attempt to validate outcomes of the COA framework by examining effects of matched interventions. We also repeated each COA with a teacher serving as therapist to evaluate the reliability of outcomes across implementers and the feasibility of involving teachers in this process.

All applications of the COA framework produced interpretable outcomes with respect to identifying a potential reinforcer for appropriate behavior. That is, all participants complied with facilitator prompts to make choices and engaged in patterns of behavior that suggested an understanding of contingencies for each choice (e.g., actively contacted or participated in activities that were available on the side they chose, sampled both choices before allocating a majority of session time to one). With minimal preparation, teachers were also able to implement conditions involving adult interaction with fidelity.

Results of the researcher-as-therapist and teacher-as-therapist COAs corresponded for three of four participants; for Braden, the one participant with different outcomes by implementer, choice allocation differed in the final session of the COA. The similarity in choice patterns across implementers was somewhat surprising given the different learning histories associated with each type of therapist. Interestingly, Braden’s COA outcomes differed by therapist such that he chose directed play with preferred items when the researcher served as therapist, yet chose the alternative (i.e., free play with low-preferred items) when the teacher served as therapist. This may suggest a higher tolerance for demands presented by a novel therapist relative to his usual teacher.

Results of the intervention comparison validated COA outcomes for two of the three participants whose COA outcomes corresponded across therapists (i.e., Demetrius and Charles). For Donny, however, levels of work completion were variable and showed increasing trends across conditions, with levels of task engagement remaining high across conditions. This pattern may indicate that for Donny, the delivery of tokens alone reinforced work completion, regardless of the activities for which each token could be exchanged. For Braden, researcher-as-therapist and teacher-as-therapist COAs pointed to different potential reinforcers. Results of the brief intervention comparison did favor the researcher-as-therapist COA, yet this may have been due to the fact that a research staff member implemented the intervention conditions.

The only previous attempt to validate this COA framework was done by comparing COA outcomes to results of brief functional analyses (Casey, 2001). Results of COAs matched or partially matched those of functional analyses for nine of the 13 participants for whom interpretable functional analysis outcomes were identified. In the current study, COA outcomes were validated by intervention outcomes for two of four participants, with partial validation for a third participant. Even for participants who showed response differentiation, however, levels of work completion and task engagement were variable in the more effective intervention condition. We designed the intervention comparison to isolate one component of intervention that was directly informed by the COA (i.e., reward delivered for tokens earned). Additional individualized intervention components (e.g., prompting, feedback, functional communication) may have produced more consistent levels of work completion. Taken together, our results indicate a need to further evaluate the conditions in which this COA framework can inform effective individualized interventions.

A final contribution worth noting is that the COA-informed intervention conditions did not require escape extinction as a treatment component. The interventionist told the student he could complete as much work as he wanted and did not deliver any prompts to complete work during sessions. Thus, despite the availability of escape, all participants chose to complete work in one or more intervention condition. This outcome points to the potential utility of COAs in classrooms where teachers are unable or unwilling to use escape extinction.

Limitations

Results of this study should be considered in light of the following limitations. First, we did not collect initial baseline data on work completion or task engagement prior to the intervention comparison. Thus, we can only make comparisons among the three token-based interventions and cannot draw conclusions regarding whether any of these conditions improved performance relative to business-as-usual. Second, with the exception of Demetrius, research staff implemented the intervention conditions, which may have biased outcomes toward the researcher-as-therapist COA for one participant for whom COA outcomes differed across therapist (i.e., Braden). Third, we primarily relied on teacher report to select task levels and types for the intervention comparison; we did not directly assess student performance on these tasks prior to beginning intervention sessions. However, we observed high levels of task completion in at least a subset of sessions across participants, suggesting these tasks were indeed in students’ repertoires. Fourth, although we took several steps to minimize differences in task difficulty across intervention conditions (see “Settings and Materials”), the instructional stimuli varied across sessions and conditions and may have contributed to variability in responding. Finally, data from Charles’s and Braden’s intervention comparisons were abbreviated due to teacher scheduling preferences and the end of the school year. Continued exposure to intervention conditions might have produced more interpretable and replicable outcomes.

Future Research

With respect to applying this COA framework in academic settings, researchers might adapt one or more conditions to better represent aspects of students’ usual classroom contexts in future studies. For example, the directed play condition might be modified to a directed work condition in which a teacher provides 1:1 prompting and support as the student completes a task. Or, for students who value peer attention over adult attention, including a peer in one or more choice conditions may be needed, especially if it is feasible to incorporate peer attention in the behavior support plan. Procedural modifications to increase experimental control for the COA framework may also prove useful in future studies focusing on validating outcomes. For example, two or more consecutive sessions with a demonstrated preference might be required before moving on to the next comparison. Or, implementers might systematically counterbalance which condition is assigned to which choice area to rule out idiosyncratic response biases. Depending on outcomes of future validation studies, research on procedural modifications designed to maximize the feasibility of COAs for school personnel (e.g., simplified data collection procedures, shortened session duration) might also be warranted.

With respect to intervention, additional research is needed to evaluate the effectiveness and maintenance of COA-informed interventions. In the current study, we rapidly alternated brief, single-component intervention conditions to efficiently assess the validity of COA outcomes. Other single case demonstration designs (e.g., multiple baseline designs across participants and/or settings; changing criterion designs) may be used to evaluate both efficacy and maintenance of one comprehensive, classroom-based intervention relative to business as usual. If these interventions are effective, evaluations of social validity—with respect to both the intervention and the assessment process that informed it—will be needed to inform acceptability and feasibility of implementation.

Conclusion

Although further research on applications of COAs in school settings is clearly needed, the current study provides preliminary support for a COA framework (Casey, 2001) to inform individualized interventions to reinforce appropriate behavior. Given the barriers of incorporating functional analysis in school-based FBAs, alternative experimental analysis methods are needed—particularly for students whose primary behavior-support needs involve behavioral deficits. In these cases, the COA may represent a promising alternative for analyzing choice allocation to identify conditions most likely to support appropriate student behavior.

Footnotes

Authors’ Note

This research was part of the second and fifth authors’ theses submitted in partial completion of the MEd degree at Vanderbilt University.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grant R324B160010 to Vanderbilt University and the Office of Special Education Programs, U.S. Department of Education, through Grant H325K140308 to Vanderbilt University. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education.

ORCID iD

Blair P. Lloyd

References

Barlow

D. H.

Hayes

S. C.

(1979). Alternating treatments design: One strategy for comparing the effects of two treatments in a single subject. Journal of Applied Behavior Analysis, 12, 199–210. doi:10.1901/jaba.1979.12-199

Bassingthwaite

Casey

(2016, April). Training school districts to conduct functional behavior assessments. Invited paper presented at the 6th Annual Convention of the Association for Professional Behavior Analysts, Washington, DC.

Beavers

G. A.

Iwata

B. A.

Lerman

D. C.

(2013). Thirty years of research on the functional analysis of problem behavior. Journal of Applied Behavior Analysis, 46, 1–21. doi:10.1002/jaba.30

Berg

W. K.

Wacker

D. P.

Cigrand

Merkle

Wade

Henry

Wang

(2007). Comparing functional analysis and paired-choice assessment results in classroom settings. Journal of Applied Behavior Analysis, 40, 545–552. doi:10.1901/jaba.2007.40-545

Briere

D. E.

Simonsen

(2011). Self-monitoring interventions for at-risk middle school students: The importance of considering function. Behavioral Disorders, 36, 129–140.

Casey

S. D.

(2001). Comparing functional outcomes between brief functional analyses and concurrent operants assessments (Doctoral dissertation). Available from ProQuest Dissertations and Theses Database. (UMI No. 2002-95020-222)

Crone

D. A.

Hawken

L. S.

Horner

R. H.

(2015). Building positive behavior support systems in schools: Functional behavioral assessment (2nd ed.). New York, NY: Guilford Press.

Erickson

M. J.

Stage

S. A.

Nelson

J. R.

(2006). Naturalistic study of the behavior of students with EBD referred for functional behavioral assessment. Journal of Emotional and Behavioral Disorders, 14, 31–40. doi:10.1177/10634266060140010301

Fisher

W. W.

Mazur

J. E.

(1997). Basic and applied research on choice responding. Journal of Applied Behavior Analysis, 30, 387–410. doi:10.1901/jaba.1997.30-387

10.

Fisher

W. W.

Piazza

C. C.

Bowman

L. G.

Hagopian

L. P.

Owens

J. C.

Slevin

(1992). A comparison of two approaches for identifying reinforcers for persons with severe and profound disabilities. Journal of Applied Behavior Analysis, 25, 491–498. doi:10.1901/jaba.1992.25-49

11.

Gage

N. A.

Lewis

T. J.

Stichter

J. P.

(2012). Functional behavioral assessment-based interventions for students with or at risk for emotional and/or behavioral disorders in school: A hierarchical linear modeling meta-analysis. Behavioral Disorders, 37, 55–77. doi:10.1177/019874291203700201

12.

Gardner

A. W.

Wacker

D. P.

Boelter

E. W.

(2009). An evaluation of the interaction between quality of attention and negative reinforcement with children who display escape-maintained problem behavior. Journal of Applied Behavior Analysis, 42, 343–348. doi:10.1901/jaba.2009.42-343

13.

Goh

A. E.

Bambara

L. M.

(2012). Individualized positive behavior support in school settings: A meta-analysis. Remedial and Special Education, 33, 271–286. doi:10.1177/0741932510383990

14.

Goodman

(2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 1337–1345. doi:10.1097/00004583-200111000-00015

15.

Hanley

G. P.

(2012). Functional assessment of problem behavior: Dispelling myths, overcoming implementation obstacles, and developing new lore. Behavior Analysis in Practice, 5, 54–72. doi:10.1007/BF03391818

16.

Harding

J. W.

Wacker

D. P.

Berg

W. K.

Cooper

L. J.

Asmus

Mlela

Muller

(1999). An analysis of choice making in the assessment of young children with severe behavior problems. Journal of Applied Behavior Analysis, 32, 63–82. doi:10.1901/jaba.1999.32-63

17.

Harding

J. W.

Wacker

D. P.

Berg

W. K.

Lee

J. F.

Dolezal

(2009). Conducting functional communication training in home settings: A case study and recommendations for practitioners. Behavior Analysis in Practice, 2, 21–33. doi:10.1007/BF03391734

18.

Iwata

B. A.

Dorsey

M. F.

Slifer

K. J.

Bauman

K. E.

Richman

G. S.

(1994). Toward a functional analysis of self-injury. Journal of Applied Behavior Analysis, 27, 197–209. doi:10.1901/jaba.1994.27-197 (Reprinted from Analysis and Intervention in Developmental Disabilities, 2, 3–20, 1982)

19.

Kalb

L. M.

Loeber

(2003). Child disobedience and noncompliance: A review. Pediatrics, 111, 641–652. doi:10.1542/peds.111.3.641

20.

Kamps

Wills

H. P.

Wehby

(2012). CW-FIT game: Group on-task and off-task definitions. Unpublished Technical Manual, The University of Kansas, Lawrence.

21.

Lloyd

B. P

. (2018, March). Practitioner perspectives on hypothesis testing strategies in the context of functional behavior assessment. In Lloyd

B. P.

(Chair), New directions and advances in research on functional behavior assessment and intervention. Symposium conducted at the 15th International Conference on Positive Behavior Support, San Diego, CA.

22.

Lloyd

B. P.

Weaver

E. S.

Staubitz

J. L.

(2016). A review of functional analysis methods conducted in public school classroom settings. Journal of Behavioral Education, 25, 324–356. doi:10.1007/s10864-015-9243-y

23.

McKenna

J. W.

Flower

Kim

M. K.

Ciullo

Haring

(2015). A systematic review of function-based interventions for students with learning disabilities. Learning Disabilities Research & Practice, 30, 15–28. doi:10.1111/ldrp.12049

24.

McKerchar

P. M.

Abby

(2012). Systematic evaluation of variables that contribute to noncompliance: A replication and extension. Journal of Applied Behavior Analysis, 45, 607–611. doi:10.1901/jaba.2012.45-607

25.

Mews

J. B.

Bassingthwaite

B. J.

Ausenhus

J. A.

(2018, May). Evaluating functional analysis and choice analysis outcomes of students through a challenging behavior training project. Poster presented at the 44th Annual Convention for the Association for Behavior Analysis International, San Diego, CA.

26.

Miller

F. G.

Lee

D. L.

(2013). Do functional behavioral assessments improve intervention effectiveness for students diagnosed with ADHD? A single-subject meta-analysis. Journal of Behavioral Education, 22, 253–282. doi:10.1007/s10864-013-9174-4

27.

Neef

N. A.

Shade

Miller

M. S.

(1994). Assessing influential dimensions of reinforcers on choice in students with serious emotional disturbance. Journal of Applied Behavior Analysis, 27, 575–583. doi:10.1901/jaba.1994.27-575

28.

O’Neill

R. E.

Bundock

Kladis

Hawken

L. S.

(2015). Acceptability of functional behavioral assessment procedures to special educators and school psychologists. Behavioral Disorders, 41, 51–66.

29.

O’Neill

R. E.

Horner

R. H.

Albin

R. W.

Storey

Sprague

J. R.

Newton

J. S.

(1997). Functional assessment and program development for problem behavior: A practical assessment guide. Pacific Grove, CA: Brooks/Cole.

30.

Piazza

C. C.

Fisher

W. W.

Hagopian

L. P.

Bowman

L. G.

Toole

(1996). Using a choice assessment to predict reinforcer effectiveness. Journal of Applied Behavior Analysis, 29, 1–9. doi:10.1901/jaba.1996.29-1

31.

Piazza

C. C.

Hanley

G. P.

Bowman

L. G.

Ruyter

J. M.

Lindauer

S. E.

Saiontz

D. M.

(1997). Functional analysis and treatment of elopement. Journal of Applied Behavior Analysis, 30, 653–672. doi:10.1901/jaba.1997.30-653

32.

Quigley

S. P.

Peterson

S. M.

Frieder

J. E.

Brower-Breitwieser

Mullins

Ivers

Acker

(2013). Example of a function-based choice assessment for a student with “passive” problem behavior. Journal of Evidence-Based Practices for Schools, 14, 81–102.

33.

Roscoe

E. M.

Phillips

K. M.

Kelly

M. A.

Farber

Dube

W. V.

(2015). A statewide survey assessing practitioners’ use and perceived utility of functional assessment. Journal of Applied Behavior Analysis, 48, 830–844. doi:10.1002/jaba.259

34.

Simcoe

Staubitz

Gregory

Juarez

A. P

. (2018, May). Oh the places you’ll go: Concurrent operant assessment in school-based consults. In Lloyd

B. P.

(Chair), Applications of concurrent operant assessments in public school settings. Symposium conducted at the 44th Annual Convention for the Association of Behavior Analysis International, San Diego, CA.

35.

Tapp

(2010). Lily data collector. Nashville, TN: Vanderbilt Kennedy Center.

36.

Thompson

R. H.

Iwata

B. A.

(2005). A review of reinforcement control procedures. Journal of Applied Behavior Analysis, 38, 257–278. doi:10.1901/jaba.2005.176-03

37.

Walker

V. L.

Chung

Bonnet

L. K.

(2018). Function-based intervention in inclusive school settings: A meta-analysis. Journal of Positive Behavior Interventions, 20, 203–216. doi:10.1177/1098300717718350

38.

Ward

Parker

Perdikaris

(2017). Task as reinforcer: A reactive alternative to traditional forms of escape extinction. Behavior Analysis in Practice, 10, 22–34. doi:10.1007/s40617-016-0139-7

39.

Wilder

D. A.

Fischetti

A. T.

Myers

Leon-Enriquez

Majdalany

(2013). The effect of response effort on compliance in young children. Behavioral Interventions, 28, 241–250. doi:10.1002/bin.1362

40.

Wolery

Gast

D. L.

Ledford

J. R.

(2018). Comparative designs. In Ledford

J. R.

Gast

D. L.

(Eds.), Single case research methodology: Applications in special education and behavioral sciences (pp. 283–328). New York, NY: Routledge.