Abstract
The vast majority of assessment instruments developed to assess children facing special healthcare challenges were constructed to assess children within a limited age range or children who face specific conditions or impairments. In contrast, the interRAI Pediatric Home Care (PEDS HC) Assessment Form was specifically designed to assess the long-term community-based service and support needs of children and youth aged from four to 20 years who face a wide range of chronic physical or behavioral health challenges. Initial research indicates that PEDS HC items exhibit good predictive validity–-explaining significant proportions of the variance in parents’ perceptions of needs, case managers’ service authorizations, and Medicaid program expenditures for long-term community-based services and supports. In addition, PEDS HC items have been used to construct scales that summarize the strengths and needs of children facing special healthcare challenges. Versions of the PEDS HC are now being used in Medicaid programs in three states in the United States.
The Health of Children
Historically, much of the international focus on child health and welfare has been on child mortality and infectious disease. 1 More recently, this focus has expanded to children with chronic health challenges and disabilities. Unfortunately, comparable international data and information on children with disabilities around the world are largely unavailable. At this point, one must be satisfied with bits and pieces of survey and programmatic data and hope to build a more solid and much wider informational foundation to guide future international efforts to improve the health and opportunities of children with disabilities. 2
Most higher income countries, however, have for years invested in relatively elaborate data systems for gathering and retaining information on children's health. 2 For example, in the USA, the National Survey of Children's Health (NSCH) has, for over a decade, provided information on the health of American children. The most recent fielding of that survey in 2011–2012 gathered data from the parents of almost 96,000 children in order to develop national and state-level estimates of measures related to children's health. 3
As one would expect, much of the emphasis on children's health in the NCSH revolves around the prevalence of, and the services provided to, children with special healthcare needs or chronic health challenges. In the latest NSCH, parents of 23.6% of children reported that their child suffered from one or more chronic diseases. Parents of almost 12% of the children reported that their child's condition(s) had a moderate or severe effect on the child. 3 Using 2013 population estimates, these percentages translate into 17 million children with one or more chronic conditions and over eight and half million for whom such conditions constitute a moderate or severe chronic health challenge. 4
Like all care for chronic conditions, the care for children or youth with chronic illnesses is quite expensive. As early as 2000, researchers noted that over half a million children in the USA received home care services that resulted in expenditures of over $5 billion dollars. Over three-quarters of these expenditures were covered by the Medicaid program, a national– state collaboration that provides health services for citizens with lower incomes. 5 In 2011, heathcare expenditures for those under 18 years of age totaled over $117 billion; almost 30% of those expenditures were made by state Medicaid programs. 6
As noted above, children and youth in the USA who are members of lower income families often receive their medical services through state Medicaid programs. Often these services focus on the timely administration of vaccinations and assuring the availability and periodic use of well-child services. However, those lower income children with the greatest health challenges often receive Medicaid services through a special Medicaid program, the Early Prevention, Screening, Diagnosis, and Treatment (EPSDT) Program, which provides all necessary medical services for qualifying children until they are 21 years of age. 7
Average annual per child expenditures for children and youth receiving the usual array of Medicaid services are dwarfed by per capita Medicaid expenditures for recipients in EPSDT programs. In Texas, for example, the average Medicaid expenditure for a child in state fiscal year 2009 was $1,834. For children receiving personal care plus any other services through the EPSDT program, the average Medicaid expenditure was over 18 times higher ($33,628). 8 The data displayed in Table 1 clearly indicate why such an expenditure differential occurs; the prevalence of serious medical, psychological, and developmental challenges among the EPSDT population is much higher than the prevalence among the general population of children. 9
Comparing the prevalence of common conditions in a national sample of children and a sample of children in the Texas Medicaid EPSDT population.
Assessing Children Facing Special Healthcare Conditions
As vulnerable and costly as this population may be, comprehensive assessments of their health challenges and service needs have not been regularized to the degree that one finds in programs providing long-term services and supports for adults and older persons. 10 For example, all residents in nursing homes participating in Medicare and Medicaid must be assessed with the Minimum Data Set (MDS) for Nursing Home Resident Assessment and Care Screening at admission and at regular intervals. 11 Home care agencies participating in the Medicare program must also assess clients using a standardized assessment and outcome measurement tool. 12
For children facing chronic health challenges, the closest one that comes to a standardized assessment is the survey-based five-item CSHCN Screener. The instrument is part of the instrumentation of the National Survey of CSHCN, and since 2001, it has been used to collect data on over a million and half children in the USA. As one might suspect with a screener, the instrument is best suited to identifying a group of children or youth who may need further assessment to determine their service needs. 10
The passage of the Children's Health Insurance Program Reauthorization Act in 2009 called for measuring and improving health services’ quality and the outcomes of services for children and youth. In pursuit of that endeavor, federal agencies have invested considerable research funds in developing a core set of potential quality measures broadly aimed at a variety of aspects of pediatric care, both for well-child services and services for some children facing special healthcare challenges. 13
This paper presents an approach to the assessment of children and youth facing special healthcare challenges from a somewhat different, more focused, perspective. It presents information concerning neither a screener nor a set of suggested service outcomes or care processes. Instead, it provides information on an assessment tool designed for use in assessing children and youth facing special healthcare challenges to determine their need for long-term community-based living services and supports. Later, we describe the development of the instrument and the results of its use in initial research efforts.
Developing the interRAI Pediatric Home Care (PEDS HC) Assessment
Unlike many assessment tools, the interRAI Pediatric Home Care (PEDS HC) Assessment had its genesis in the world of public policy, not academia or clinical medicine. In 2005, the Texas Health and Human Services Commission (HHSC) settled a lawsuit concerning the state's administration of its EPSDT program. 14 As a result of this settlement, the HHSC entered into an agreement with researchers at Texas A&M University (TAMU) to develop an assessment tool to determine the personal care needs of children in the Texas EPSDT program.
Prior to the development of this assessment, the state agency had assessed the care needs of children in the EPSDT program using an assessment instrument developed and used for determining the home care needs of older persons. All instrument development and other project activities undertaken by TAMU staff during this effort were reviewed and approved by the TAMU committee charged with protecting the rights and safety of research participants.
The TAMU research team began its activities by reviewing the variety of instruments used by other states in their EPSDT programs. These instruments varied dramatically in both their level of detail and their comprehensiveness. Following this review, the research team focused on the development of a new instrument for use in the Texas EPSDT home care program. In this effort, senior members of the TAMU team drew on their experience as part of interRAI, an international consortium of researchers committed to the development of a set of “seamless” assessment tools for use across a variety of health services settings and with a variety of vulnerable populations. 15
The interRAI instruments are meant to be seamless in the sense that a standardized, common core of assessment items constitutes the common foundation for these different tools. This core set of items is then supplemented and, if necessary, revised to fit the needs of clinicians and service providers in a specific service setting or to reflect the characteristics of persons facing different types of health challenges. 15
InterRAI members include 96 researchers and clinicians from 34 countries. Members of interRAI have collaborated in the development and testing of over a dozen assessment tools for use by clinicians in residential long-term care, home care, acute care, palliative care, residential mental health services, and community-based mental health services.15,16 TAMU team members participated in the development of interRAI assessment tools designed to assess the care needs of nursing home residents, 17 adults seeking or receiving home care, 18 and persons seeking or in need of mental health services. 19
Within the United States, interRAI's Home Care instrument, the interRAI HC, is now used in over 15 state Medicaid home care program for adults, including some Medicaid managed care organizations in Texas. It has also been officially mandated for use by social services in the Provincial Government of Ontario, Canada, and in New Zealand. 15
Because of the potential to develop a “picture” of home care users at an early age that could easily be integrated into the assessment and clinical processes for those who also might later need services as adults, the HHSC accepted the TAMU team's proposal to use the interRAI HC as the core of the EPSDT personal care assessment and revise its elements so that they were more appropriate for use in children receiving community-based services and supports in the EPSDT program.
After reliability testing and instrument revision in 2007– 2008, two interRAI HC-based assessment instruments were implemented in the Texas EPSDT program. 20 The Personal Care Assessment Form 4-20 (PCAF-4-20) and the Personal Care Assessment Form 0-3 (PCAF-0-3) were implemented in Texas in 2008. The PCAF-4-20 was used to assess the service needs of children older than three years of age. The PCAF-0-3 was used to assess the service needs of children less than four years of age. Texas currently continues to use both PCAF instruments in its EPSDT program.
Based on the PCAF effort, the interRAI Pediatric Home Care Assessment Form (PEDS HC) was then developed from the PCAF-4-20. The focus was restricted to assessing children over three years of age because available evidence indicated that they constituted over 90% of children receiving community-based services and supports through the Texas Medicaid program. 9 One aspect of this development process involved expanding the PCAF, which had been used to determine personal care needs, to fully capture a wider range of potential health challenges.
The second aspect of the process was to assure consistency of the PCAF items with similar items in other instruments in the interRAI suite of instruments that were developed to assess the strengths and service needs of a variety of populations in a variety of settings–-adults in need of home care, individuals facing intellectual or developmental disabilities, individuals with mental health problems, or children or youth with mental health problems. 15 When this process was complete, the interRAI Pediatric Home Care Assessment Form (PEDS HC) was formally added to the interRAI suite, and the instrument and manual were published for interRAI in 2014. 21
The end result of this process is an in-person assessment tool that can be used by a social worker or a healthcare professional to assess a child/youth's needs for long-term community-based services and supports. Such assessments in the USA appropriately require that the assessment be completed during a visit to the child/youth's primary residence. Those whose presence is required at the assessment are the child/youth and their primary caregiver (ie, the responsible adult). Other families, caregivers, or advocates may be included in the assessment conversation at the request of the client or caregiver. The PEDS HC manual explains how the assessment is to be conducted and provides detailed information on each PEDS HC item. 21
Reliability Testing
Information on the reliability and validity of the PEDS HC items is, at this point, less complete than one would hope. In part, this situation derives from the genesis of the instrument. The HHSC was largely concerned with the interrater reliability of PCAF items, most of which were later included in the PEDS HC. As a result of this concern, a reliability trial was conducted that involved dual PCAF assessments of 236 children in 2007–2008 to determine their needs for EPSDT personal care services (PCS) in their home. PCS are those services provided by Medicaid programs to compensate for impairments in function that affect the client's performance of activities of daily living (ADLs) or instrumental activities of daily living (IADLs). 20
The usual efforts one sees to establish concurrent validity by having study participants assessed using multiple instruments were not feasible in this project environment. The HHSC had no interest in the concurrent validity of the instruments; the agency's validity concerns were limited to face or content validity established through expert review. In addition, the agency staff felt that the time burden associated with the use of additional assessment tools on already taxed families caring for a child with serious health challenges was too great, as was the potential burden of such activities on the data collection resources the HHSC had allocated to the PCAF project.
Reliability concerns revolved around interrater reliability. This commitment to assuring interrater reliability was not driven by concerns about the science supporting the instruments but by a policy issue–-reports and complaints that children with similar health challenges were receiving very different levels of assistance.
Even the testing of interrater reliability using state program staff presented a number of operational difficulties. The state staff assessors were case managers in the Department of State Health Services, who were almost exclusively social workers, and the second assessors were contract social workers previously or currently employed as contract case managers by the agency. State case manager's assessments were to begin immediately after training, but the assessment data indicate that the median number days between training and a state case manager's first assessment was 38 days, more than a month. The dual assessments were to be completed in the same week, but the data indicate that the median number of days between assessments was 12 days, or almost two weeks. Such are the realities of research in the context of an operating social program, rather than in a laboratory or a research environment where the research team has control over all field staff.
Reliability testing of individual items involved the use of kappas, weighted kappas, or interclass correlations, depending on the type of indicator being considered. If the level of inter-rater agreement for categorical items was 90% or greater, no reliability statistics were calculated. As a general rule, items with reliability coefficients less than 0.30 were dropped. Items with kappa values ranging from 0.30 to 0.39 were either dropped, or, if the items were thought to cover crucial information, the items were revised, based on follow-up interviews with assessors. Items with at least moderate reliability scores (kappa > 0.39) were retained, usually without revision. 20 Unfortunately, project time and financial constraints would not allow for a second reliability trial to test revised items.
Predictive Validity–-Needs, Services, and Expenditures
One might at this point, given the available data, wonder how one can evaluate the usefulness of the PEDS HC. We believe that this evaluation can be based on the review of, and reflection on, the results of the initial research published using data collected from PEDS HC items collected using the PCAF.
It is important to remember that we test the reliability and validity of an instrument to be certain that it can do what it was designed to do. Thus, here we present research results that emphasize two of the main tasks for which the instrument was developed. We review research that focused on two issues. First, how well do PEDS HC items predict home care needs and the costs of home care? Second, what useful scales that summarize dimensions of a child/youth's status or challenges can be constructed using PEDS HC data?
Brown and Bourke-Taylor reviewed 37 recent articles in the American Journal of Occupational Therapy, which focused on instruments assessing children and youth. They noted that the most common validity concerns for these assessment tools focused on descriptive and discriminant validity. These articles rarely addressed predictive validity–-the ability of the information to predict some future event or status. 22
Soo and her colleagues’ work on the Paediatric Care and Needs Scale (PCANS) provides an excellent example of the standard validity assessment one finds in the literature on assessment tools for children and youth. 23 The validation sample involved participants with a limited age range (15–18 years). The focus was on participants with a specific condition or impairment, in this case acquired brain injury. The PCANS construct, convergent, divergent, and discriminant validity was investigated by comparing participants’ scores and rating on the PCANS with their scores and ratings on three other commonly used scales. Predictive validity was never mentioned.
In the policy and health service world, however, the predictive validity of assessment tools is a crucial issue. Needs assessments should lead to predictable service allocation decisions. A wide range of factors may affect an agency assessor's or case manager's final decisions concerning service authorization. However, one expects the assessment information available to the decision maker to play a major role in determining which clients receive how much service. An instrument's ability to capture variation in such decisions is its predictive validity.
Three articles, each using somewhat different databases from the Texas EPSDT project, developed structural equation models (SEM) predicting some element of home care services (Table 2). One used the reliability test data that included 236 clients. The model developed using these data included two behavioral scales, an ADL scale, information on the severity of a child/youth's intellectual or developmental disability (IDD), the child/youth's health conditions, continence, and diagnoses. The research team developed a SEM with the number of personal care hours requested by the primary caregiver serving as the dependent variable. The model explained 30% of the variance in caregivers’ requests for PCS and 29% of the variance in PCS hours authorized for a client by the state case manager. 24
PEDS HC predictive validity.
Another analysis used data collected in 2008–2009 on 2,759 children with special healthcare needs seeking or receiving PCS services. In this instance, the SEM predicted 28% of the variance in PCS hours authorized by state case managers. 25 Using only those clients in this sample who had reported intellectual or developmental challenges, a similar SEM explained 26% of the variance in hours of care authorized by case managers. 26 In a more recent effort, Adepoju and her colleagues developed an SEM that explained 36% of variance in total annual Medicaid home healthcare expenditures for those children and youth who were part of the 2008–2009 data collection. 27
In addition, one of the assessment items included in the study's two large-scale data collections (n = 5,016) was whether the primary caregiver or client believed the personal care hours authorized were in line with their own view of what was needed. Most made no request for a specific amount of personal care (63.7%); others said that the authorization equaled or exceeded their request (32.4%); less than 4% (3.9%) indicated that the authorization was for fewer hours than requested. 9
The reported explained variances displayed in Table 2 constitute measures of the predictive validity of the PEDS HC items. Explained variances in this research ranged between 26% (r = 0.51) of the variance in personal care hours requested to 36% (r = 0.60) of the variance in total home care costs. These levels of explanatory power may seem modest. However, these levels of predictive validity are very much in line with other research modeling variance in home care services for other populations.
Previous research predicting home care expenditures for older persons have yielded very similar level of explanatory power. This earlier research used variants of the interRAI HC and the Resource Utilization Groups for Home Care (RUG-III/HC). Using data collected using an early version of the interRAI HC with home care recipients in Michigan, researchers explained 38.1% of the variance in the total of formal and informal home care costs. 28 More recent work using home care data from Ontario used a RUG-III/HC-based case-mix model that explained 20.5% of the variance in formal home care costs. 29
Summary Scales Developed with PEDS HC Items
Scale development using PEDS HC items has focused on four dimensions of health status: IADLs, ADLs, problem behaviors, and cognitive function. In each of these domains, data from PEDS items have been used to develop scales with strong claims to reliability.
The PEDS HC Cognitive Sum scale was developed using five items–-long-term memory, short-term memory, procedural memory, decision-making, and interpretive communication. A simple sum of these items resulted in a scale that exhibited excellent internal consistency (alpha = 0.82) in our sample of over 2,000 children in the EPSDT program. 30
Two ADL scales have been developed and evaluated. The first scale, ADL Sum, simply summed a client's score on all those ADLs affected by the child/youth's conditions. The second scale, ADL Hands-On, focused on the number of ADLs in which the child/youth received hands-on assistance as a result of their health challenges. Research has shown that both exhibit excellent internal consistency (average alpha greater than 0.90) across gender, age, and type of client (medical problems only, psychosocial or development problems only, or both types of problems). The scales also exhibited excellent construct validity and predictive validity. Of special importance was the discriminant ability of these scales; neither was correlated (Pearson's r ranged from -0.012 to -0.025) with the clients’ age. 30
The scales measuring function were especially constructed to reflect impairments only in those areas of function affected by a child/youth's conditions. For each of the IADL and ADL items, respondents were asked whether the child/youth's performance in the activity was affected by their conditions. Only those activities or tasks that were affected by a condition were used to construct the two functional scales.
PEDS HC scale characteristics.
This strategy was utilized because the EPSDT program only provides PCS to ameliorate the effects of special healthcare needs on daily living. So, service planning in this program can only include assistance with activities affected by those special needs. The logic behind this policy is that public services should not be used to supplant the assistance to children that is part of everyday parenting. These scarce public resources should be used to assist families laboring under an unusual burden.
This strategy was also used in measuring IADL performance. The IADL scale (IADL Sum) summed a client's scores on seven items, including such activities as meal preparation, medication administration, house cleaning, and travel outside the home. The scale displayed excellent internal consistency with an alpha of 0.88 in the development sample and an alpha of 0.87 in the validation sample. 30
One might think that IADL assistance is of little importance in a pediatric assessment instrument. One expects parents to provide all those services to children or youth, no matter their health status. Tasks such as meal preparation, housecleaning, and laundry are normal parental fare that support all those living in the household.
However, some older clients may be emancipated minors or young adults living on their own or with a partner or caregiver. Even among the youngest clients where all IADL tasks are usually performed by the family, formal IADL assistance may be needed. A child/youth's special healthcare challenges may affect the time required to do these tasks, the number of times these tasks must be performed, or the number of adults required to do the task safely. These factors must be considered when building an official service plan that contains all necessary services related to the child/youth's health challenges.
The Texas data also allowed the research team to construct two measures of a child's behavioral problems. The first scale (Externalizing Behavior Scale) reflected the number of 18 potential behaviors exhibited by the child/youth, which were characterized as “externalizing.” These behaviors included behaviors such as persistent expressions of anger, compulsive behaviors, and any abusive behavior; this externalizing behavior scale demonstrated excellent internal consistency (alpha = 0.87). Another scale, Internalizing Behavior, summed the number of 11 potential behaviors exhibited by the child/youth, which were characterized as “internalizing.” This scale also displayed excellent internal consistency (alpha = 0.83) and included such items as sleep pattern problems, withdrawal, and diminished concentration. 24
Using six PEDS HC items, researchers also developed an additive scale (Caregivers Barriers) summarizing the level of the primary caregiver's limitations in their ability to provide care. Such items are very important as health service providers attempt to develop service plans that allow for some tasks to be performed by informal caregivers. The items in this scale included barriers to care resulting from work and school attendance, as well as physical and mental health conditions. This scale demonstrated good, though not excellent, internal consistency (alpha = 0.68).
The Finalized PEDS HC
To date, the interRAI PEDS HC has been implemented to assess the home care needs of children in Medicaid home care programs in the state of New York and the state of Maryland, both of which also use interRAI's home care assessment for adults, the interRAI HC. Additional state Medicaid programs in which the interRAI HC is used, or is under consideration, are now reviewing the PEDS HC. Table 4 displays the information on the specific sections included in the PEDS HC, displays the number of items in each PEDS HC section, and indicates the source of the items.
PEDS HC sections, item examples, and item sources.
PCAF, Personal Care Assessment Form (Texas); 9 HC, interRAI HC; ID, interRAI ID; MH, interRAI MH; ChYMH, interRAI ChYMH; LTC, interRAI LTC.
As indicated, the PEDS HC drew, in part, on more comprehensive assessment tools such as the interRAI HC, the interRAI Mental Health Assessment (interRAI MH), the interRAI Child and Youth Mental Health Assessment (interRAI ChYMH), the interRAI Long-Term Care Assessment (interRAI LTC), and the interRAI Intellectual Disability Assessment (interRAI ID) focused on a broader range of issues required for providers of more varied services. 15
Thus, many items from these other interRAI instruments were added to the PEDS HC, both to create a more comprehensive assessment tool and to make the PEDS HC consistent with other interRAI assessment tools. Many of these items and sections have proven themselves reliable and valid, but not specifically for children and families facing special healthcare challenges.
16
(Copies of the PEDS HC are available, when requested, from interRAI at
Conclusions
A wide variety of assessment instruments for children and youth have been developed and tested. However, the bulk of these tools address the needs of children within a limited age range or children with a specific condition or impairment. Validity concerns in these development efforts have revolved almost exclusively around construct and discriminant validity. 22
In contrast, The PEDS HC was developed with fundamentally different concerns. Its development was driven by the needs of health service providers attempting to understand the home care needs of children and youth. The age range required for use in an EPSDT program demanded that the instrument can be useful in assessing the needs of children and youth with ages ranging from four to 20 years. The instrument had to be used to assess the service needs of children and youth with very different health challenges and very different levels of service needs. The instrument items were expected to be, and proved to be, important drivers of parental perceptions of needs, case managers’ service authorizations, and program expenditures for services.
The PEDS HC, though its core elements had been used to drive personal care service provision for years in the Texas EPSDT program, was only added to the interRAI suite of assessment tools in 2014. It is already in use in the Medicaid programs in two other states in the USA and under consideration for adoption in Medicaid programs in other states, especially those states using or adopting the interRAI HC for adults receiving home care services.
Creating a clinical and administrative database that allows one to use common items and measures to evaluate client outcomes or program impact should be an appealing concept. 31 As with the interRAI HC and other interRAI instruments, the use of the PEDS HC in these settings will lead to a wider range of investigations concerning the psychometric properties of the instrument and its sections and, over time, revisions to the tool to enhance its usefulness, reliability, and validity.
At this time, the interRAI PEDS HC occupies a unique position among pediatric assessment instruments. It is one of the few purpose-built and tested instruments designed to assess the long-term community services and supports needed by children facing special healthcare challenges. In addition, it rests on a foundation of a core of common items shared with assessment tools that have been used in nursing homes, assisted living, home health care, and acute care. As such, it offers an assessment that should fulfill the needs of public programs providing health services to children and youth in the community, while fitting within an entire suite of instruments that cover a much broader spectrum of populations and settings.
One of the future steps in the evolution of the PEDS HC may be the “modularization” of the tool. This strategy would involve choosing a core of items from the PEDS HC for administration to all clients. Depending on results of that core assessment, an assessor might then complete one or more additional, more focused item modules. These modules or supplements might focus on a more detailed inquiry into a child/youth's nursing needs, psychosocial challenges, or special issues arising in providing services to children and youth with intellectual or developmental challenges. These supplements would use both items currently used in the full PEDS HC and additional items reflecting the specific focus of the supplement. Other steps might include the development of tools that use the PEDS HC data for the development of service planning to meet conditions or impairments and a case-mix classification system for use in resource allocation or payment.
The information reported here does have limitations. The PEDS HC is not yet in widespread use, and its rollout to different environments may demand revision and reevaluation. Additionally, more work is needed on the psychometric properties of many of the items in the PEDS HC and scales developed using those items.
Finally, the instrument has not been tested outside the USA and has not been translated into languages other than English. Such efforts always pose potential problems. However, the foundational instrument for the PEDS HC is the interRAI HC, which has thus far been successfully translated into Dutch, German, French, Spanish, Estonian, and Italian. These efforts may bode well for the eventual translation and testing of the PEDS HC in international settings.
Author Contributions
CP and CH are responsible for all the aspects of the article. Both the authors reviewed and approved the final manuscript.
Footnotes
Acknowledgments
The authors would like to acknowledge the assistance of Mary James, University of Michigan, in the preparation of this article. Acknowledgements of TAMU staff and faculty for work on the original PCAF instrument development are extensive and appear in the instrument user's manual.