Establishing Reliable Cognitive Change in Children With Epilepsy

Abstract

The goal of this study was to investigate reliable cognitive change in epilepsy by developing computational procedures to determine reliable change index scores (RCIs) for the Dutch Wechsler Intelligence Scales for Children. First, RCIs were calculated based on stability coefficients from a reference sample. Then, these RCIs were applied to a sample of 73 children with refractory epilepsy who were tested twice with the WISC-RN^NL/WISC-III^NL after a mean interval of 2.3 years. Results indicated that children with refractory epilepsy are at risk for cognitive decline over time: 26.0% of the children showed reliable losses on Verbal IQ and 16.4% on the full scale IQ (expected rate = 5%). Declines on performance IQ were within expected limits.

Keywords

Wechsler Intelligence Scale for Children epilepsy in children reliable cognitive change cognitive decline longitudinal study

It remains uncertain whether cognitive declines occur over time during the course of epilepsy (Devinsky & Tarulli, 2002). Two updates on longitudinal studies of children with epilepsy indicate that evidence of cognitive decline in epilepsy continues to be sparse and inconclusive (Dodrill, 2004; Seidenberg, Pulsipher, & Hermann, 2007). Numerous studies, in fact, report nonsignificant changes over time in groups of children with epilepsy (Aldenkamp, Alpherts, De Bruine-Seeder, & Dekker, 1990; Bjornaes, Stabell, Henriksen, & Loyning, 2001; Bourgeois, Prensky, Palkes, Talent, & Busch, 1983; Jones, Siddarth, Gurbani, Shields, & Caplan, 2010; Oostrom, van Teeseling, Smeets-Schouten, Peters, & Jennekens-Schinkel, 2005; Rodin, Schmaltz, & Twitty, 1986). These findings of no significant decline, though encouraging, are generally associated with nonreferred samples of children with relatively uncomplicated epilepsies and high rates of seizure remission (Jones et al., 2010; Oostrom et al., 2005). However, severity of epilepsy syndrome—particularly epileptic encephalopathies and epilepsies with a known (symptomatic) or suspected (cryptogenic) cause—has been associated with (a) low IQ (Bulteau et al., 2000; Nolan et al., 2003) and (b) increasing deviation from the developmental curve (Berg et al., 2004). Furthermore, within an epilepsy syndrome, variability among individuals has been reported to be high (Reijs et al., 2006).

To gain insight into the heterogeneity within a clinical sample, and to increase the clinical meaningfulness of research data beyond the study of group means, studies should also provide information on individuals (Chelune, 2002; Martin et al., 2002). Studies of children with epilepsy that do report rates of individuals with cognitive change are rare, both in samples without (Rodin et al., 1986) or with epilepsy surgery (Westerveld et al., 2000). These occasional studies of cognitive change in children have rarely used the psychometric properties of the test to determine reliable cognitive change.

In adult literature, approaches to quantify reliable cognitive change have used normal or clinically stable samples as reference groups. From these references samples, reliable change has been computed applying regression-based procedures (Cysique et al., 2011; Martin et al., 2002) or, alternatively, the reliable change index (RCI; Chelune, Naugle, Lüders, Sedlak, & Awad, 1993; Woods et al., 2006). These RCI formulas tend to perform as well as more complex regression formulas (Heaton et al., 2001) and to be more “user friendly” (Chelune, 2002, p. 424). The RCI is a cut-off score and ±RCI yields a confidence interval (typically 90%). Descriptions of procedures to establish RCI appear in the literature (Chelune et al., 1993; Maassen, Bossema, & Brand, 2009). When retesting is conducted, “change scores” that fall within the confidence interval are interpreted as “common” or nonmeaningful differences; on the contrary, change scores outside the interval are deemed uncommon or “reliable cognitive changes.” RCI formulas are based on the standard deviations of the mean scores at first or “baseline” testing (Chelune et al., 1993) or at both baseline and retesting (Chelune, 2002; Maassen et al., 2009). The formulas are also based on the stability coefficients. Where practice effects are known to occur, the RCI cut-off score is adjusted accordingly (Chelune et al., 1993; Woods et al., 2006).

Woods et al. (2006) underscored the importance of examining the validity of this RCI approach in clinical samples expected to present cognitive change, whether improvement or decline. Although epilepsy is considered an ongoing neurological condition that can potentially lead to cognitive change, these kinds of validation studies barely exist in the epilepsy research on children. Thus, reliable cognitive change in children with epilepsy was the focus of the present study.

Statistical Properties, Test Familiarity and Test Version

Changes in IQs at retesting vary depending on familiarity with the test, length of the interval between measurements (Kaufman, 1994), IQ at baseline (Schittekatte, 2005), and test version used (Canivez & Watkins, 1998). A period of 9 to 12 months is often considered sufficient to counter practice effects in referred children (Canivez & Watkins, 1998); however, some evidence exists that in children with epilepsy, the impact of practice effects may level off within 6 months (Neyens, Aldenkamp, & Meinardi, 1999). When the same test is administered at both measurements, long-term studies on referred children without epilepsy—retested after a 2.5- to 3-year interval—generally show stable scores, with differences that are either not significant or not clinically meaningful (Canivez & Watkins, 1998; Pesch & Ponsioen, 2004). These test–retest studies are based on data from the WISC, the WISC-R, the Dutch WISC-R (to be called WISC-R^NL), or the WISC-III. Stability coefficients over time have been found to be high for the three IQ scales that comprise the various versions of the WISC, verbal scale (V-IQ), performance scale (P-IQ) and full scale (FS-IQ; Canivez & Watkins, 1998; Schittekatte, 2005). For the index scores, Canivez and Watkins (1998) also report high stability coefficients; for the Dutch Wechsler tests, however, coefficients were based on small numbers of children (Schittekatte, 2005) and are of limited utility.

Purpose

To establish the rate of children with epilepsy who show reliable cognitive change, the present study focused on the three IQ scales of the Dutch Wechsler Intelligence Test for Children in a sample of referred children with refractory epilepsy who were tested twice. The following research questions were addressed:

Given the presence of two sets Wechsler IQs obtained at Time 1 (T1) and Time 2 (T2), what magnitude of T1–T2 change is sufficient to denote a significant change, that is, a “reliable cognitive change”?

How often do these reliable changes occur in children with epilepsy, and is this percentage larger than expected?

The principal hypothesis is that referred children with epilepsy show decline over time in cognitive function (Seidenberg et al., 2007). It is hypothesized that the study of individuals will uncover elevated rates of children who demonstrate reliable cognitive decline during the course of epilepsy relative to a clinical control reference sample without epilepsy.

The study included two phases: (1) testing the utility of reliable change formulas and establishing RCIs (90% confidence intervals) based on Dutch Wechsler test–retest data from a reference sample; and (2) applying the RCIs to establish the proportion of children with epilepsy who show reliable cognitive changes at retesting. Phase 1 is described in the Method section with phase 2 presented in Results.

Method

Participants

The sample comprised 73 Dutch children who met the following criteria: (a) they had a diagnosis of epilepsy, (b) they were tested (T1) with either the WISC-R^NL (van Haasen et al., 1986) or WISC-III^NL (Wechsler, 2005), (c) they were retested (T2) after an interval of 12 months or more on the same test version that was used at T1, and (d) no epilepsy surgery had occurred in between measurements. The children presented at a Dutch tertiary center for epilepsy or at a special school providing services for children with epilepsy. Reasons for testing were concerns expressed by parents, schools, or neurologists about the child’s cognitive development. Wechsler testing generally occurred as part of a comprehensive neuropsychological evaluation and took place at the start of the evaluation or after a break. Commonly, results of the assessment were used in applications for special financial and educational services for the child. Thus, this study dealt with a selected sample of referred children who were tested with the same WISC on both occasions, either the WISC-R^NL or the WISC-III^NL. No exclusionary criteria were applied in terms of epilepsy type or IQ at T1.

The sample was selected from 420 children with epilepsy who had completed a WISC. Of these, 290 children had been tested once. From the 130 children who had been tested twice, 57 were not eligible because: (a) retesting had taken place within 12 months (n = 23), (b) epilepsy surgery was conducted (n = 2), or (c) different versions of the Dutch WISC were administered at T1 and T2 (n = 32). The selected sample (n = 73) and the sample of children who were not eligible for this study (n = 347) were overall similar. ANOVAs or χ²-tests using an α level of 0.01 to control for familywise error rates, showed that the samples did not differ in sex, handedness, V-IQ, P-IQ, or FS-IQ, age at onset of epilepsy, duration up to first testing, number of antiepileptic drugs (AED) used, seizure type, severity of epilepsy syndrome, or rates of children with documented MRI abnormalities. The children of the sample of interest, however, were tested for the first time at a younger age (selected sample: mean = 9.1, SD = 2.2; not selected sample: mean = 10.2, SD = 2.8). The difference in age was significant, F(1, 419) = 9.84, p = .002, r = 0.15.

Test Versions

Comparison of the 41 children tested twice with the WISC-R^NL with the 32 children tested twice with the WISC-III^NL revealed overall equality. ANOVA and χ² (with α set at 0.01) revealed no significant differences between the subsamples on any IQ scale at T1 or T2; on T1–T2 change in IQ; on any demographic variable, namely sex, handedness, age at T1 or age at T2; or on any epilepsy variable (age at onset, duration of epilepsy up to T1 or T2, T1–T2 time interval, number of AEDs used, seizure type, epilepsy syndrome severity, seizure status at T2, MRI status).

Epilepsy Variables

The characteristics of the sample are presented in Tables 1 and 2. Data collected on epilepsy relate to age at onset of epilepsy; side of seizure onset (left hemisphere onset, right hemisphere onset, or bilateral seizures); type of seizures (focal vs. generalized); presence of documented brain lesion on neuroimaging (MRI+); and epilepsy syndrome severity, classified according to the 10-point Syndrome Severity Scale for Children with Epilepsy (Dunn, Buelow, Austin, Shinnar, & Perkins, 2004). Seizure status was divided in active or inactive epilepsy; inactive epilepsy was defined as seizure freedom at T2 of 12 months or longer.

Table 1.

Epilepsy Variables at Time 1 (T1) and Time 2 (T2).

	Mean	SD	Range
Age at onset of epilepsy (years)	5.4	3.0	0.1-13.2
Age at T1 (years)	9.1	2.2	6.0-15.9
Age at T2 (years)	11.4	2.3	7.8-16.9
Duration epilepsy to T1 (years)	3.7	3.0	0.2-13.4
Duration epilepsy to T2 (years)	6.0	3.1	1.6-15.8
Interval T1−T2 (years)	2.3	1.2	1.0-7.5
Antiepileptic drugs used at T2 (n = 65)	1.3	0.8	0-4
Antiepileptic drugs tried at T2 (n = 65)	2.3	1.3	0-5
Epilepsy syndrome severity, min = 1, max = 10 (n = 70)	6.0	1.5	2-8

Table 2.

Demographic and Epilepsy Variables.

	N	%
Sex
Boys	38	52.1
Girls	35	47.9
Handedness (n = 63)
Right-handed	54	84.1
Left-handed	9	15.9
Seizure type and side
Focal	51	69.9
Left hemisphere	24	32.9
Right hemisphere	12	16.4
Bilateral or multifocal	15	20.5
Generalized	6	8.2
Uncertain	11	15.1
Unknown	5	6.8
MRI+ (positive findings)	21	28.8
Seizure status
Active epilepsy	41	56.2
Inactive epilepsy	16	21.9
Uncertain	7	9.6
Unknown	9	12.3

Clinical Control Reference Sample

The use of a referral or clinical control sample rather than a sample of normal controls in clinical studies has been recommended (Cysique et al., 2011; Woods et al., 2006). Following this line, to calculate RCI values for the Dutch Wechsler tests, the referral sample from the test manual (Wechsler, 2005), based on the study of Schittekatte (2005), was used as a clinical control reference sample.

Schittekatte’s study appears as a valuable reference for the present purpose. It is based on a large sample of 353 children tested with the Dutch Wechsler tests with a mean interval between testings of 3 years and a mean FS-IQ in the low average range (82) at T1. The sample consisted of children referred for cognitive and learning problems. One major advantage of the clinical control reference sample is that it does not primarily relate to children with ongoing neurological conditions possibly associated with cognitive loss. The previous study differs from the present study in two main aspects—in Schittekatte’s study (a) a change in test version had taken place from WISC-R^NL at T1 to WISC-III^NL at T2, and (b) the T1–T2 time interval was longer (mean difference of 7.9 months) and had a wider range (range 0.2 to 8.7 years). Both differences, changes in test version and longer time intervals, lead to lower stability scores (Sattler, 2001). Therefore, the reference sample is likely to yield conservative RCI estimates and, therefore, less likely to lead to type I errors.

Thus, a neurologically “uncomplicated” clinical control reference sample provides the standard for evaluation of the neurologically compromised sample. If elevated rates of decline are found, as hypothesized, they are more likely to be related to the ongoing epileptic condition itself and less to other factors such as familiarity with the test or nonspecific effects of remediation programs.

Coefficients of Stability: Comparison Between the Reference Sample and the Sample With Epilepsy

For the reference sample, Schittekatte (2005) reports coefficients of T1–T2 stability of 0.87, 0.81, and 0.88 respectively for the verbal, performance and full scales (see Table 3). For the sample with epilepsy, these values were 0.75, 0.76, and 0.77. The coefficients for the sample with epilepsy were significantly lower than the values found by Schittekatte for the verbal and full scales, but not the performance scale (using Fisher’s z for independent samples and α set at p < .01 to control for multiple comparisons).

Table 3.

Reliable Change Indexes (RCI) Based on Coefficients of Stability R_T1T2.

	American WISC-III
			Expected according to		Dutch WISCs
	Observed Canivez		Chelune	Maassen	Based on Schittekatte
Wechsler scale	R _T1T2	RCI	RCI	RCI	R _T1T2	RCI
V-IQ	0.87	13	13	13	0.87	14
P-IQ	0.87	14-15	14	15	0.81	18
FS-IQ	0.91	11	11	11	0.88	14

Establishing and Testing RCI Formulas

Similar to Schittekatte (2005), Canivez and Watkins (1998) collected a large sample of referred children tested twice with the WISC-III. The sample included 667 American children tested for special education eligibility. The authors calculated coefficients of stability for the V-IQ, P-IQ, and FS-IQ. Notably, the authors also constructed an empirically derived base-rate table that presented frequencies of changes in IQ (Canivez & Watkins, 1998, p. 289). These data provided an excellent starting point to test RCI formulas.

Chelune et al. (1993, pp. 45-46) published an RCIs formula based on the coefficients of stability and the standard deviation from the first test administration. Maassen et al. (2009, p. 340, formula 2) expanded the formula to include the standard deviation of both the first and second testings. To compare the formulas from the two studies, they were applied to the coefficients of stability from Canivez and Watkins (1998, p. 287). As can be seen in Table 3, both formulas provided accurate estimates: they yielded RCI values within 1 point of those found empirically by the authors (Canivez & Watkins, 1998, p. 289). In the present study, the formula from Maassen et al. was applied as follows:

S E_{diff} = \sqrt{(S D_{T 1}^{2} + S D_{T 2}^{2}) (1 - R_{T 1 T 2})}

where $S D_{T 1}^{2}$ and $S D_{T 2}^{2}$ are the squared standard deviations at T1 and T2, respectively; and R_T1T2 is the stability coefficient between measurements at T1 and T2. The value for the 90% CI: ±(SE_diff × 1.64).

Cut-Off Values for the RCIs

After applying the formula to Schittekatte’s coefficients of stability, the established RCI cut-off values were 14 for V-IQ, 18 for P-IQ, and 14 for FS-IQ (rounding off to the nearest integer). The last two columns of Table 3 present the stability coefficients and the RCIs used to establish reliable change in the present study. Chelune et al. (1993) suggested that the differences between T1 and T2 be adjusted with the T1–T2 changes found in the reference sample to control for practice effects. Given the intertest interval of at least 12 months for the present sample, no practice effects were expected.

Analyses

Temporal stability (mean change) was calculated with paired samples t tests for the IQ scales. Two-tailed tests were applied, α was set to 0.01 to correct for multiple comparisons, and r was used as a measure of effect size; r ≥ 0.5 was interpreted as a large effect size (Field, 2005). The percentage of children with epilepsy outside the 90% reliable-change interval was established. For each scale, a single χ²-statistic was used to test whether the observed rates for no reliable change (±RCI), gains (≥+RCI), and losses (≤–RCI) differed from: the expected rates of 90% (normal gain or loss), 5% (reliable gain), and 5% (reliable loss). Again, α was set at 0.01.

Results

Changes Over Time on Wechsler Scales

As presented in Table 4, mean decline between T1 and T2 on the verbal scale was 7.2 IQ points; a paired sample t-test indicated this was significant (p < .001, effect size r = 0.5). The performance scale showed a nonsignificant decline of 0.6 IQ points. The full scale showed a significant decline of 4.4 IQ points (p = .001, r = 0.4). Changes ranged from −52 (loss of 52 IQ points) to +21 (gain of 21 points) on the verbal scale, −38 to +29 on the performance scale and −48 to +17 on the full scale.

Table 4.

Wechsler IQ Scores at T1 and T2 and Change Over Time (ΔT2−T1) With Paired Samples t-Test and Percentages of Children Showing Reliable Gains, No Reliable Change and Reliable Losses.

	IQ at T1		IQ at T2		ΔT1−T2					Reliable change
	Mean	SD	Mean	SD	Mean	SD	t	p	ES (r)	Gain (5%)	n.s. (90%)	Loss (5%)	χ²	p
V-IQ	90.89	15.5	83.68	16.3	7.21	11.3	5.435	<.001	0.5	1.4	72.6	26.0	68.933	<.001
P-IQ	87.45	16.3	86.86	17.8	0.59	11.9	0.423	n.s.	—	5.5	89.0	5.5	0.075	n.s.
FS-IQ	88.10	15.9	83.73	17.0	4.37	11.1	3.358	.001	0.4	2.7	80.8	16.4	20.531	<.001

Reliable Cognitive Change in Epilepsy

Table 4 presents the rates of children showing reliable gains and losses, as well as the results of the χ² analyses for the three IQ scales. The table shows that the percentage of children who displayed a reliable gain on any IQ scale is close to the expected 5%, while the rate of children with significant loss is elevated in relation to the expected 5% on the verbal and full scales. On the verbal scale, 53 children (72.6%) presented with T1−T2 differences within the reliable change interval; one child (1.4%) showed a reliable gain; and a substantial 19 children (26.0%) showed a reliable loss. The proportions differed significantly from the expected values (χ² = 68.93, p < .001). The values found for the performance scale—gain in four children (5.5%) and loss in four children (5.5%) – were not different from those expected (χ² = 0.07, p > .01). On the full scale, two children (2.7%) showed a gain and 12 (16.4%) a loss; values were significant (χ² = 20.53, p < .001).

Discussion

The aim of this study was to establish reliable cognitive change in children with epilepsy tested twice with the WISC-R^NL/WISC-III^NL, applying a 90% RCI. The present study differed from earlier research on cognitive change in child epilepsy (Aldenkamp et al., 1990; Westerveld et al., 2000) as it considered the psychometric properties of IQ changes on the Wechsler Scales to predetermine RCIs. It was shown that observed changes could be predicted with great accuracy, providing support for the usefulness of these formulas to determine cognitive change and confirming that empirical data on change can be estimated from psychometric data (Chelune et al., 1993; Maassen et al., 2009).

Lower coefficients of stability found in the sample with epilepsy compared to the reference sample suggested more variability in IQ scores in children with epilepsy from T1 to T2. Indeed, compared to the expected values of 5%, elevated rates of reliable loss at T2 were seen in epilepsy on the verbal scale (26.0%) and on the full scale (16.4%). Rates for gains did not exceed expected values on any IQ scale. The present data support the hypothesis that the seizure condition is associated with an elevated risk for cognitive decline (Dodrill, 2004; Seidenberg et al., 2007). Notably, this decline is only seen on the verbal and full scales and not on the performance scale.

The differential changes for the verbal and performance scales may be partially related to the higher initial V-IQ than P-IQ. It is unlikely, however, that they were related to regression to the mean phenomenon (Heaton et al., 2001), which more often affects scores in the extremes, because the present sample had low average IQs at first testing. A VIQ > PIQ difference has been reported regardless of seizure onset side (van Iterson & Augustijn, 2006). A somewhat more favorable course of the performance scale than the verbal scale at retesting has also been described in samples without epilepsy (Kaufman, 1994; Schittekatte, 2005), as well as in children who have undergone epilepsy surgery (Westerveld et al., 2000), and is possibly due to the decreased “novelty” of the performance tasks at T2 (Canivez & Watkins, 1998).

Epilepsy Variables

Seidenberg et al. (2007) state that there is a dearth of studies on cognitive decline, particularly studies that include epilepsy variables. The breakdown of the present sample according to epilepsy variables yielded small subsample sizes, and did not permit conducting meaningful analyses. Earlier longitudinal studies on children with epilepsy suggested that patterns of change were independent of seizure laterality (Westerveld et al., 2000), type of epilepsy, and antiepileptic drugs (Oostrom et al., 2005). Mixed results have been reported on cognitive development over time in relation to persistence of seizures (Bjornaes et al., 2001; Jones et al., 2010; Oostrom et al., 2005).

Limitations of the Study

The sample in the present study was available from tertiary epilepsy settings, which, by nature, deal with more difficult to treat epilepsies and therefore possibly with children with worse prognosis. The lower age at T1 of the children included in the sample, compared to those not selected, also points to the inclusion of “worse” epilepsies (Bulteau at al., 2000) and may limit the generalizability of the results.

Future Directions

Future studies with larger samples may allow insight into the possible impact of epilepsy variables on cognitive decline. Alternatively, more specific epilepsy subsamples (e.g., specific epilepsy syndromes) could be studied with similar methods. Also, given that epilepsy surfaces at various ages and may last for a prolonged period of time, it is important to obtain data on reliable cognitive when different intelligence tests are used at T1 and T2 (e.g., a childhood version of Wechsler’s scales followed by an adult version).

Clinical Implications

The results of the present study contribute to the literature on the cognitive course of epilepsy in children and should be of value for clinicians and researchers. The procedure described for the Dutch Wechsler tests can readily be applied to other languages and cultures, provided that coefficients of stability and standard deviations are available from a reference sample. Clinicians may want to apply the reliable change cut-off values when retesting a child. For the American WISC-III, and for the Dutch Wechsler tests, the data presented in Table 3 would yield appropriate estimates. After a change in test version (WISC-R^NL to WISC-III^NL), adjustment with the differences reported by Schittekatte (2005) is pertinent; applying the more stringent criteria of 19 (V-IQ), 18 (P-IQ), and 17 (FS-IQ) points for reliable loss is recommended to account for the predictable Flynn effect. The present study used a statistically sound methodology to help addressing the question, “Is this child presenting a reliable cognitive change at retesting?” The applicability of the procedure goes beyond children with epilepsy to other neurodevelopmental disabilities potentially associated with cognitive decline.

Footnotes

Acknowledgements

The authors thank Alan S. Kaufman for his valuable contribution to this study.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Aldenkamp

A. P.

Alpherts

W. C. J.

De Bruine-Seeder

Dekker

M. J. A.

(1990). Test-retest variability in children with epilepsy—A comparison of WISC-R profiles. Epilepsy Research, 7, 165-172.

Berg

A. T.

Smith

S. N.

Frobish

Beckerman

Levy

S. R.

Testa

F. M.

Shinnar

(2004). Longitudinal assessment of adaptive behavior in infants and young children with newly diagnosed epilepsy: Influences of etiology, syndrome and seizure control. Pediatrics, 114, 645-650.

Bjornaes

Stabell

Henriksen

Loyning

(2001). The effects of refractory epilepsy on intellectual functioning in children and adults. A longitudinal study. Seizure, 10, 250-259.

Bourgeois

B. F.

Prensky

A. L.

Palkes

M. A.

Talent

B. K.

Busch

S. G.

(1983). Intelligence in epilepsy: A prospective study in children. Annals of Neurology, 14, 438-444.

Bulteau

Jambaqué

Viguier

Kieffer

Dellatolas

Dulac

(2000). Epileptic syndromes, cognitive assessment and school placement: A study of 251 children. Developmental Medicine and Child Neurology, 42, 319-327.

Canivez

G. L.

Watkins

M. W.

(1998). Long-term stability of the Wechsler Intelligence Scale for Children—Third edition. Psychological Assessment, 10, 285-291.

Chelune

G. J.

(2002). Making neuropsychological outcome research consumer friendly: A commentary on Keith et al. (2002). Neuropsychology, 16, 422-425.

Chelune

G. J.

Naugle

R. I.

Lüders

Sedlak

Awad

I. A.

(1993). Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 7, 41-52.

Cysique

L. A.

Franklin

Abramson

Ellis

R. J.

Letendre

Collier

Heaton

R. K.

(2011). Normative data and validation of a regression based summary score for assessing meaningful neuropsychological change. Journal of Clinical and Experimental Neuropsychology, 33, 505-522.

10.

Devinsky

Tarulli

(2002). Progressive cognitive and behavioral changes in epilepsy. In Devinsky

Westbrook

L. E.

(Eds.), Epilepsy and developmental disabilities (pp. 133-149). Woburn, MA: Butterworth-Heinemann.

11.

Dodrill

C. B.

(2004). Neuropsychological effects of seizures. Epilepsy and Behavior, 5, S21-S24.

12.

Dunn

D. W.

Buelow

J. M.

Austin

J. K.

Shinnar

Perkins

S. M.

(2004). Development of syndrome severity scores for pediatric epilepsy. Epilepsia, 45, 661-666.

13.

Field

(2005). Discovering statistics using SPSS (2nd ed.). London, UK: Sage.

14.

Heaton

R. K.

Temkin

Dikmen

Avitable

Taylor

M. J.

Marcotte

T. D.

Grant

(2001). Detecting change: A comparison of three neuropsychological methods, using normal and clinical samples. Archives of clinical neuropsychology, 16, 75-91.

15.

Jones

J. E.

Siddarth

Gurbani

Shields

W. D.

Caplan

(2010). Cognition, academic achievement, language, and psychopathology in pediatric chronic epilepsy: Short-term outcomes. Epilepsy and Behavior, 18, 211-217.

16.

Kaufman

A. S.

(1994). Intelligent testing with the WISC-III. New York, NY: Wiley.

17.

Maassen

G. H.

Bossema

Brand

(2009). Reliable change and practice effects: Outcomes of various indices compared. Journal of Clinical and Experimental Neuropsychology, 31, 339-352.

18.

Martin

Sawrie

Gilliam

Mackey

Faught

Knowlton

Kuznieky

(2002). Determining reliable cognitive change after epilepsy surgery: Development of reliable change indices and standardized regression-based change norms for WNS-III and WAIS-III. Epilepsia, 43, 1551-1558.

19.

Neyens

L. G. J.

Aldenkamp

A. P.

Meinardi

H. M.

(1999). Prospective follow-up of intellectual development in children with recent onset epilepsy. Epilepsy Research, 34, 85-90.

20.

Nolan

M. A.

Redoblado

M. A.

Lah

Sabaz

Lawson

J. A.

Cunningham

A. M.

Bye

A.M.

(2003). Intelligence in childhood epilepsy syndromes. Epilepsy Research, 53, 139-150.

21.

Oostrom

K. J.

van Teeseling

Smeets-Schouten

Peters

A. C.

Jennekens-Schinkel

(2005). Three to four years after diagnosis: Cognition and behaviour in children with ‘epilepsy only’. A prospective, controlled study. Brain, 128(Pt 7), 1546-1555.

22.

Pesch

Ponsioen

(2004). Flinterdunne en flagrante Flynn-effecten bij licht verstandelijk gehandicapte kinderen (Tiny and flagrant Flynn effects in mildly mentally retarded children). De Psycholoog, 29, 64-68.

23.

Reijs

R. P.

van Mil

S. G. M.

van Hall

M. H. J. A.

Arends

J. B. A. M.

Weber

J. W.

Renier

W. O.

Aldenkamp

A. P.

(2006). Cryptogenic localization-related epilepsy with childhood onset: The problem of definition and prognosis. Epilepsy and Behavior, 8, 693-702.

24.

Rodin

E. A.

Schmaltz

Twitty

(1986). Intellectual functions of patients with childhood-onset epilepsy. Developmental Medicine & Child Neurology, 28, 25-33.

25.

Sattler

J. M.

(2001). Assessment of Children. Cognitive Applications (4th ed.). San Diego, CA: Author.

26.

Schittekatte

(2005). Overzicht van onderzoek met de WISC-III (Overview of research on the WISC-III). Tijdschrift Klinische Psychologie, 35, 214-230.

27.

Seidenberg

Pulsipher

D. T.

Hermann

(2007). Cognitive progression in epilepsy. Neuropsychology Review, 17, 445-454.

28.

van Haasen

de Bruyn

Pijl

Poortinga

Y. H.

lutje Spelberg

H. C.

Vandersteene

Stinissen

. (1986). WISC-R. Wechsler Intelligence Scale for Children Revised. (Dutch version). Lisse, The Netherlands: Swets Test Services.

29.

van Iterson

Augustijn

P. B

. (2006). Unilateral hemisphere involvement associated predominantly to VIQ > PIQ discrepancies in children with (refractory) epilepsy. Epilepsia, 47, 158-159.

30.

Wechsler

(2005). WISC-III^NL. Wechsler Intelligence Scale for Children. Derde editie NL. Handleiding en Verantwoording. London: The Psychological Corporation/Nederlands Dienstencentrum NIP.

31.

Westerveld

Sass

K. J.

Chelune

G. J.

Hermann

B. P.

Barr

W. B.

Loring

D. W.

Spencer

D. D.

(2000). Temporal lobectomy in children: cognitive outcome. Journal of Neurosurgery, 92, 24-30.

32.

Woods

S. P.

Childers

Ellis

R. J.

Guaman

Grant

Heaton

R. K.

(2006). A battery approach for measuring neuropsychological change. Archives of Clinical Neuropsychology, 21, 83-89.