Developing Situated Measures of Science Instruction Through an Innovative Electronic Portfolio App for Mobile Devices: Reliability,Validity,and Feasibility

Abstract

Adoption of new instructional standards in science demands high-quality information about classroom practice. Teacher portfolios can be used to assess instructional practice and support teacher self-reflection anchored in authentic evidence from classrooms. This study investigated a new type of electronic portfolio tool that allows efficient capture of classroom artifacts in multimedia formats using mobile devices. We assess the psychometric properties of measures of quality instruction in middle school science classrooms derived from the contents of portfolios collected using this novel tool—with instruction operationalized through dimensions aligned to the Next Generation Science Standards. Results reflect low rater error and adequate reliability for several dimensions, a dominant underlying factor, and significant relations to some relevant concurrent indicators. Although no relation was found to student standardized test scores or course grades, portfolio ratings did relate to student self-efficacy perceptions and enjoyment of science. We examine factors influencing measurement error, and consider the broader implications of the results for assessing the validity of portfolio score interpretations, and the feasibility and potential value of this type of tool for summative and formative uses, in the context of large-scale instructional improvement efforts.

Keywords

generalizability theory measures of instruction performance assessment portfolio assessment validity

A wealth of research shows that instructional practices in the classroom are a better predictor of student achievement than teacher experience or degrees, and that improvements to teacher practice can have a considerable impact on students compared with other interventions (Kane & Staiger, 2012; Rockoff, 2004; Seidel & Shavelson, 2007; for recent challenges to this perspective, see, for example, Podolsky et al., 2019). A growing focus in studying the mechanisms relating teaching and learning within classrooms has renewed interest in the development of tools and instruments to systematically study instruction. From a measurement perspective, the construct instruction presents unique challenges as it requires examining how different types of knowledge and skill interact in context with various student, classroom, and social factors to influence the types, variety, and quality of the educational experiences offered to students in classrooms (Gitomer, 2009; Grossman & McDonald, 2008; Zeichner, 2012). A range of measurement tools have been used to derive indicators of instruction, including among the most notable teacher and student surveys, observation protocols, and teaching portfolios (Correnti & Martinez, 2012; Raudenbush & Sadoff, 2008). Building on our prior work, we developed a new type of electronic portfolio tool which can support efficient collection of multimedia evidence of classroom instruction through mobile devices. We report the results of a pilot validation study that examined first, the psychometric properties (i.e., reliability and validity) of indicators of quality instruction in middle school science classrooms derived from evidence in these portfolios, and second, the broader feasibility and potential of the portfolio tool to support summative and formative assessment of instruction.

Measuring Instruction using Teacher Portfolios

The properties and relative strengths of different methods and instruments for measuring instruction have been a persistent presence in the literature dating back more than half a century (Medley & Mitzel, 1963). The perceived gold standard is expert ratings based on standardized classroom observation rubrics (Cohen & Goldhaber, 2016), but used at scale, these present difficult choices between volume and quality of evidence, timeliness, depth, and scope of feedback (Knapp, 1997). Teacher and student surveys have gained popularity as a cost-effective alternative for collecting data about a broad range of aspects of instruction at scale from key actors (Desimone & LeFloch, 2004; Schweig, 2014). But survey self-reports are also subject to a variety of sources of measurement error, bias, and strong social-desirability effects, and are not well suited for capturing richer, qualitative aspects of instruction (Mayer, 1999). Various other methods attempt to balance depth and breadth of coverage, reliability, validity, bias, and cost-efficiency. For example, teacher logs (short surveys administered repeatedly) can improve reliability and address social desirability to better capture instruction over time (Rowan & Correnti, 2009). Instructional vignettes elicit situated descriptions and explanations of practice in realistic classroom scenarios (Stecher et al., 2006). Finally, tests of teacher pedagogical content knowledge (PCK) can be used to assess constructs closely related to instructional practices (Kennedy, 1999; see also Kersting, 2014, for more recent efforts to measure applied PCK using video prompts.)

Teacher portfolios have long been used for teacher induction and certification, and there is growing interest in using them to monitor and improve instruction at scale. Indeed, the two best known teaching portfolios in the United States (the National Board, and edTPA) are used to assess thousands of beginning and experienced k-12 teachers each year (see, for example, National Board for Professional Teaching Standards [NBPTS], 2019; Stanford Center for Assessment, Learning, and Equity [SCALE], 2018). As measurement tool, portfolios can provide a rich evidentiary basis to monitor instruction in context, offering richer information than surveys, at lower cost and with better coverage than observations (Authors, 2012; Ball & Forzani, 2009). Evidence from our own prior work and that of others suggests that portfolio instruments can support judgments about complex features of mathematics and science instruction with reliability comparable to that of classroom observations, and significant relation with classroom observation scores and student achievement (Authors, 2012; Borko et al., 2006; Matsumura et al., 2008).

Traditional teacher portfolios are not without significant drawbacks, however, ranging from the burden of collecting and annotating classroom artifacts, to logistical challenges for large-scale collection, handling, and review of portfolios, and psychometric challenges in deriving reliable and valid portfolio scores. A final and critical substantive limitation of paper-based portfolios has been the inability to reflect evidence of key aspects of live instruction, classroom interactions, and discourse. Modern computer technology is helping address some of these challenges; in particular, electronic portfolios are being increasingly used to compile multimedia evidence of instruction that can include not only documents but also images, audio, and video of live instruction (Strudler & Wetzel, 2011). In practice, the broad term e-portfolio comprises a range of tools and instruments with varying structures, underlying assumptions, and intended uses. Importantly, however, e-portfolios have been traditionally conceived as performance assessments: involving instructional projects that teachers develop, prepare, implement, and analyze over a span of months, anchored on a small number of critical pieces of evidence (e.g., a unit plan, test, and video of a lesson). This type of portfolio assessment requires extensive engagement from teachers over long periods, and has thus been limited to one-off use for certification or distinction (e.g., the Educative Teacher Performance Assessment, or NBPTS portfolios).

Continuing prior work with paper/pencil and electronic portfolios, for this study, we developed a new type of e-portfolio tool enabling middle school science teachers to efficiently collect rich multimedia evidence of instruction in their classrooms (i.e., documents, images, and video) using a tablet computer or other mobile devices. This tool is based on an alternative model that conceptualizes portfolios not as (optimal) performance assessments, but as detailed documentary records of authentic (typical) instruction (Shulman, 1998). Instead of completing extended essays explaining and reflecting materials and video from a specially prepared lesson, this model of portfolio asks teachers to collect a variety of classroom artifacts frequently and in site, trying to reflect instruction as experienced by students during an instructional unit or series of lessons. Implemented through modern mobile technology, the tool addresses important shortcomings of paper and pencil and other traditional electronic portfolios. First, teachers can capture rich evidence of live instruction and classroom dialogue typically out of reach in traditional paper portfolios. Second, compiling evidence in the portfolio is substantially more efficient, exploiting the capabilities of modern mobile technology to enable teachers to contextualize classroom artifacts using on screen survey prompts (or voice-to-text transcription for short-answer open-ended prompts). Third, portfolio collection and the measures of instruction derived have better coverage, may be more useful and informative (and thus support more valid inferences), as they relate to bodies of evidence covering an entire unit of instruction, or other realistic instructional period of interest. Finally, the process of systematically documenting and reflecting on day-to-day classroom practice over a period of time could hold particular value for supporting professional learning, compared with traditional portfolio performance assessments (Kloser, 2021).

We also developed a set of scoring rubrics to systematically assess the evidence contained in these e-portfolios against a model of instruction aligned to the Next Generation Science Standards (NGSS; National Research Council, 2012). The NGSS aim to address the excessive focus on contents, facts, and procedural manipulation in science education (Krajcik et al., 2014), offering exposure to realistic scientific experiences (Scruggs et al., 2013), and social discourse and literacy practices key to scientific knowledge (Latour, 1986; Lemke, 1990). Importantly, assessing the degree to which different features of this rich model of science instruction are present in a given classroom ideally requires considering evidence from different sources. Figure 1 maps evidence from multimedia artifacts captured in this novel portfolio tool to key features of classroom instruction and assessment in the NGSS.

Figure 1.

NGSS Instruction and Types of Evidence Captured by the Portfolio App.

We conducted a pilot validation study to examine the reliability and validity of scores derived from the evidence contained in portfolios collected by a sample of middle school science teachers. As with other measures that rely on human judgments of complex behavior, portfolio-based measures of instruction can be affected by significant measurement error from inconsistencies across judges (i.e., portfolio raters)—or in the contents of the portfolios themselves. Specifically, we investigated the following research questions:

Research Question 1 (RQ1): What is the reliability of ratings of science instruction assigned to portfolios by trained raters? What are the key sources of measurement error affecting the scores?

Research Question 2 (RQ2): What is the factor structure of portfolio ratings? Is there evidence of general and specific instructional factors underlying the scores?

Research Question 3 (RQ3): Do portfolio ratings relate to other measures of instruction as expected? Do they relate to valued student academic and socioemotional outcomes?

Research Question 4 (RQ4): What demands are placed on teachers collecting portfolios, and what benefits do they perceive from this process?

In the closing section, we consider the psychometric findings in the context of a broader validation argument (Kane, 2006) that considers the accuracy of judgments about the quality of instruction, their relation to concurrent indicators, and predictive power on student outcomes, along with the potential formative value of the tool for supporting professional learning, and it feasibility for supporting efficient use on a large-scale by individuals and groups.

Method

Participants and Procedures

Sample

In total, 31 eighth grade science teachers in a large urban school district in southwestern United States participated in the study. Half of the placement schools served predominantly Latino students (85%–99%), and the remaining schools served mixed student populations. About 70% of participating teachers were females, half held bachelor’s degree, and half a master’s degree or more. On average, teachers had 11 years teaching experience and 9 years teaching science. Most teachers taught on a regular schedule (only six taught block schedules). In addition to participant teachers who collected portfolios, we recruited five experienced middle school science teachers who were trained and served as portfolio raters.

Portfolio Software

We developed a new type of e-portfolio tool based on a general purpose open-source software platform for distributed data collection in mobile devices (Tangmunarunkit et al., 2015). The e-QIS portfolio app (Quality Instruction in Science) enables teachers to efficiently capture multimedia artifacts as evidence of planning, instructional delivery, and assessment in their classroom using an Android or iOS phone or tablet (University of California, Los Angeles, 2018). Teachers use the app installed in their phone or tablet device to upload artifacts reflecting important aspects of instruction they want to reflect in their portfolio. These can include a variety of documents, and files, images, and short video clips stored in their device or online (e.g., google drive, box), each contextualized upon capture using a combination of multiple choice, and open-ended questions completed via the screen, keyboard, or voice recognition/transcription. Because the app is often used in classrooms during instruction, teachers are able to temporarily store artifacts in a parking lot, to complete the annotation prompts and upload at a later time. A companion web-based portal allows storing, organizing, and editing the contents of electronic portfolios—other authorized users (e.g., portfolio raters, teacher educators) are also able to review and input comments and feedback for teachers. Figures 1 to 3 in the supplemental appendix present screen captures of the portfolio app dashboard, sample prompts used to annotate artifacts for upload, and the web portal—interested readers can refer to the online documentation or the researchers for further detail and to download the app (see University of California, Los Angeles, 2018).

Portfolio Collection

Portfolios were structured around a 10-day sequence of lessons (a typical science unit) as the conceptual unit of collection and analysis of evidence of instruction. Teachers used the app to document two instructional units from the eighth grade California Science Standards, each comprising approximately 10 days of instruction (or equivalent in block schedules) in physics or chemistry in fall and spring.¹ Portfolio collection guidelines required that teachers collect (a) initial reflections and artifacts to offer context about their class, their students, and the science unit covered in the portfolio, (b) at least one instructional artifact and one assessment artifact for 10 consecutive days of instruction, or 6 days for block schedules, and (c) concluding reflections and artifacts at the end of the unit. We defined instructional artifact as any document, image, or video clip with relevant evidence of instructional planning, content, or delivery, whereas assessment artifacts were samples of student work (also in any format) used to gauge student learning or progress.

Portfolio Scoring

Training for raters spanned 2 days; on the first day, they became familiar with the portfolio app and web portal, the dimensions of science instruction, and the evidence in the portfolios in relation to each dimension. Raters also discussed and scored a first training portfolio in pairs, and reviewed the evidence for their ratings against master ratings with the research team. On the second day of training, raters independently scored two teacher’s portfolios, and discussed their ratings for all dimensions in relation to the evidence in the portfolio and master ratings provided by researchers. In our study, each rater scored a total of 24 portfolios (on average 12 teacher portfolios from physics units and 12 from chemistry units), and each teacher received ratings from two or three raters, organized in an incomplete crossed (or matrix) design depicted in Table 1. Raters were able to review and score the contents of the portfolios remotely from their computers, using the web portfolio portal. Portfolios took on average 90 to 100 minutes to score, with raters spending an average total of 36 hours scoring, spread over 3 to 4 weeks.

Table 1.

Scoring Design. Raters Randomly Assigned to Physics and Chemistry Portfolios.

Teacher id	Rater id
Teacher id	1	2	3	4	5
1	x	x
2	x	x
:	:	:
10	x	x
11	x	x
12			x	x	x
13			x	x	x
:			:	:	:
24			x	x	x
25			x	x	x
Total no. of ratings	12	12	13	13	13

Measures

Portfolio Scores

Alongside the software tools described earlier, we developed a detailed conceptual model comprising four domains and 11 dimensions of instruction aligned to the NGSS framework, with input from a national advisory group of educators and science education experts. For each dimension, we also developed scoring rubrics mapping portfolio contents to instructional practices of different levels of quality or rigor on a 5-point scale (1 = Not Realized to 5 = Fully Realized). Figure 4 in the supplemental appendix presents the complete set dimensions as defined in the scoring rubrics.² We also developed detailed training materials and examples relating the varying images of instruction depicted in the rubrics to different types of multimedia evidence contained in a portfolio—for example, assessing the extent to which students develop and investigate their own scientific questions might bring together evidence from a lesson plan, worksheets, pictures of physical models constructed by students, and a short video of a group discussion tied to a lab or investigation.

A total of N = 31 teacher portfolios were scored by multiple trained raters on all 11 dimensions and a summary rating reflecting overall quality of instruction aligned to the NGSS model. A subset of 25 teacher portfolios additionally received consensus master scores from the research team for comparison to trained rater scores. Raters also kept detailed record of the evidence they used to assign scores—for example, a rater may find evidence relevant to assess Dimension 4 (Engaging Students in Productive Talk) from artifacts collected on Days 1 to 4, 6, 9, and 10. These records were then systematically quantified and coded to reflect the volume and types of evidence raters reviewed in assigning scores to each dimension.

Teacher Self-Efficacy

Each participant teacher completed a survey inquiring about a range of teacher background characteristics, education, experience, and attitudes, as well as the types of instructional models and practices they use in their science classrooms. The survey also measured teacher perceptions of self-efficacy on the same 11 dimensions of science instruction in the model above. The indicators derived from this survey were used to assess the concurrent relationship of portfolio scores to constructs of pedagogical practice, and self-concept, which an extensive literature shows are related to quality of instruction (for a review, see Zee & Koomen, 2016).

Assessing Teacher Learning About Science Teaching (ATLAST) Scores

Participant teachers completed two assessments of PCK in physics and chemistry before completing portfolios in each subject. The ATLAST test was developed to capture domains of teacher knowledge theoretically closely related to instructional practice (Smith, 2010). We derived number-correct scores in each test as concurrent criterion indicators hypothesized to relate to portfolio ratings of quality instruction.

Student Outcomes

To more thoroughly assess the claim that indicators of instruction derived from portfolios can predict valued student outcomes, we collected three different types of variables as criterion measures. First, the district provided scores on the eighth grade state science test, and seventh grade mathematics scores the previous year as covariates; whereas common in the literature, standardized test scores are typically not aligned to curriculum or very sensitive to instruction (Ing, 2018). As a complementary criterion measure, we also collected course grades assigned by teachers at the end of the school year, which are often suspect on reliability grounds, but can be predictive of broader achievement, especially for some groups of students (Pollio & Hochbein, 2015). Finally, students in participating classrooms completed a survey inquiring about their instructional experiences in the class, as well as their attitudes and perceptions of self-efficacy in science learning, and enjoyment of science which have been widely shown to be important mediating outcomes for science achievement (see, for example, Schunk, 1989). In total, we collected survey data from 768 students in 30 classrooms.

Debriefing Interviews

We conducted structured debriefing interviews with participant teachers after completing each portfolio to gather information about the process of artifact collection, relative ease or burden, clarity of the annotation prompts, missing information to be collected, among others, along with user feedback to help enhance the software tools. In addition, the interviews asked teachers to reflect on the potential value of the portfolio collection process for informing their professional self-reflection and improvement. Finally, we also conducted exit interviews with participant raters to collect evidence about the process of portfolio review and scoring, the clarity and consistency of the scoring rubrics, and the cognitive load of specific types of artifacts in relation to these rubrics.

Analytic Approach

We used a set of complementary psychometric and statistical analysis techniques to assess the measurement properties of indicators of science instruction derived from the electronic portfolio tool. In a validity argument framework, these are claims that portfolios can be scored to make reliable distinctions among teachers, measure common dimensions of instruction, and correlate or predict other concurrent indicators and outcomes as expected.

We used a series of generalizability models to understand the sources of measurement error influencing portfolio scores, and the reliability of aggregate scores assigned by raters. We estimated inter-rater agreement indices and error variance components in portfolio scores using MINQUE estimation in SPSS Version 24 (Shavelson & Webb, 1991). These G-studies follow a two-facet design with teachers as object of measurement and raters and topics as the facets of measurement (sources of error). The design (denoted p x r x t) isolates true score (teacher) variance, error variance across raters and topics ( $σ_{P}^{2}$ , $σ_{R}^{2}$ , $σ_{T}^{2}$ ); interactions ( $σ_{PR}^{2}$ , $σ_{RT}^{2}$ , $σ_{PT}^{2}$ ); and residual measurement error ( $σ_{PRT, e}^{2}$ ). Complementary decision (D) studies estimated reliability under varying scoring scenarios that alternatively treated topics as random or fixed facets. In turn, Exploratory Factor Analysis (EFA) framework analyzed patterns of interrelation among portfolio ratings and identified common underlying traits or dimensions of instructional quality. Models were estimated in the Mplus software Version 7.4 (Muthén & Muthén, 2012) with promax rotation and maximum likelihood estimation, assessing fit via chi-square and root mean square error of approximation (RMSEA) indices.

Correlational analyses investigated concurrent and predictive validity of portfolio ratings. Measures included master consensus scores assigned to portfolios by the research team, teacher scores in the ATLAST test of PCK, and indicators of instruction and self-efficacy derived from teacher surveys. Multilevel models investigated how portfolio ratings related to three different student outcomes: state science test scores, course grades, and survey indicators of self-efficacy and interest in science. Models were estimated in the HLM software Version 7.0 (Raudenbush & Bryk, 2002) to take into account the clustered structure of the data. Finally, we undertook qualitative analyses of portfolio contents and interviews with participant teachers and raters to investigate the feasibility and perceived value of the portfolio tool.

Results

Table 2 presents descriptive analysis for portfolio ratings of the 11 dimensions of science instruction—the 12th dimension is an overall holistic assessment of the degree to which the evidence in the portfolio reflects instruction consistent with our NGSS-aligned model. Ratings were highest for Dimension 4 (accuracy), and lowest for Dimension 10 (feedback); these two dimensions have very skewed distributions and low variation, suggesting strong floor and ceiling effects. Overall, ratings were slightly higher (3.1) for physics than for chemistry (2.79); both ratings were slightly higher than simple average scores across the 11 dimensions (2.95 and 2.61, respectively). Reviewing scores by subtopic shows variation associated to the match between science content and certain dimensions—for example, portfolios covering topics related to atoms and periodic tables received lowest scores in Dimensions 6 and 7, likely reflecting the difficulty of implementing instruction that emphasizes student-driven experimentation and data analysis in those topics. The arithmetic average of the 11 dimensions is slightly lower than the overall quality rating (D12), suggesting that raters did not equally weigh evidence from all dimensions for assigning a holistic rating—this gap widens if Dimensions 4 and 10 are eliminated.

Table 2.

Descriptive Statistics of Portfolio Ratings by Topic (Across Teachers and Raters).

Dimensions	Physics unit (N = 63)				Chemistry unit (N = 57)
Dimensions	M	SD	Min	Max	M	SD	Min	Max
1. Eliciting Student Initial Thinking	2.76	1.06	1	5	2.00	1.10	1	5
2. Supporting Needs of Diverse Learners	2.70	1.07	1	5	2.61	1.22	1	5
3. Engaging Students in Productive Talk	2.44	1.13	1	5	2.09	0.87	1	5
4. Accurate Content Knowledge	4.95	0.21	4	5	4.86	0.44	3	5
5. Multiple Ways of Knowing	3.17	0.93	2	5	2.74	0.92	1	5
6. Investigating Scientific Questions	2.73	0.83	1	5	2.14	1.03	1	5
7. Supporting Data Analysis/Interpretation	2.73	1.02	1	5	2.14	1.11	1	4
8. Construction/Use of Scientific Products	2.29	0.91	1	4	2.16	0.92	1	5
9. Constructing Diverse Assessments	3.33	0.98	1	5	3.23	1.00	1	5
10. Providing Feedback	1.81	1.01	1	5	1.54	0.57	1	3
11. Using Information to Improve Teaching	3.54	1.04	1	5	3.19	1.16	1	5
12. Overall Quality of Science Instruction	3.14	0.78	2	5	2.79	0.92	1	4
Average of Dimensions 1 to 11	2.95	0.82			2.61	0.91

Reliability/Generalizability

Table 3 presents agreement indices (exact and within-1) for each of the 11 dimensions in the model of instruction. Exact agreement ranges from a low of 28% (D11: Using Information) to 78.4% (D4: Accuracy). Agreement indices within 1 point in this 5-point scale are naturally much higher—in the 70% to 90% range. The table also presents the results of generalizability studies of portfolio scores of each dimension. Because of differences across topics noted above, these models treat topics as a fixed rather than random facet. The table shows variance components for teachers, raters, and error—averaging together results from the physics and chemistry portfolios. Across dimensions, systematic differences in rater stringency ( $σ_{R}^{2}$ ) accounted for comparatively small percentages of variation, ranging from 1% to 12% of variation across dimensions—residual error accounts for the largest proportion of error variance. The final two columns in the table show reliability coefficients for scenarios with two and three raters. With two raters, reliability ranged from .20 (D4: Accuracy) to .84 (D6: Investigation). With three raters, seven of 11 dimensions had reliability above .65, five higher than .75. The Overall Quality rating had reliability of .69 and .77 with two and three raters, respectively. Among the dimensions with low reliability, Accuracy (D4) and Feedback (D10) had problems of misalignment between the rubrics and portfolio contents. The rubric for accuracy framed high scores in terms of absence of evidence of inaccuracy, which guaranteed very high scores for most teachers—not necessarily because there were no instances of inaccuracy in those classrooms, but because none were documented in the portfolio. Conversely, teachers collected little evidence of written or oral Feedback given to students in their portfolios, which resulted in very low scores on Dimension 10. These do not necessarily indicate that teachers in our sample offered extremely poor feedback to their students, but instead likely reflect a combination of lack of fidelity to the portfolio collection protocol, along with real logistical difficulty of collecting evidence to reflect verbal feedback offered to students.

Table 3.

Generalizability Study Results by Dimension (Topic as a Fixed Facet).

Dimension	Agreement %		Variance components (Estimate and %)						Reliability
Dimension	Exact	w1	$σ_{T}^{2}$	%	$σ_{R}^{2}$	%	$σ_{TR, e}^{2}$	%	ρ2	ρ3
1. Initial Thinking	33.3	79.2	0.24	30.6	0.03	3.70	0.51	65.6	.48	.58
2. Diverse Learners	47.0	91.7	0.32	45.6	0.07	10.5	0.31	43.7	.68	.76
3. Productive Talk	37.3	87.5	0.46	54.4	0.01	1.65	0.37	43.8	.71	.79
4. Accuracy	78.4	95.8	0.01	11.2	0.00	1.60	0.05	87.2	.20	.28
5. Ways of Knowing	33.8	81.3	0.20	33.6	0.04	6.11	0.36	60.2	.53	.63
6. Scientific Investigations	30.0	85.4	0.45	65.9	0.06	9.03	0.17	25.0	.84	.89
7. Data Analysis	34.0	72.9	0.57	64.6	0.05	5.76	0.26	29.6	.81	.87
8. Scientific Products	33.8	81.3	0.18	38.9	0.02	4.15	0.27	56.9	.58	.67
9. Assessment	33.2	87.5	0.37	48.2	0.08	10.0	0.32	41.7	.70	.78
10. Feedback	56.7	93.8	0.13	29.9	0.06	12.7	0.25	57.3	.51	.61
11. Using Information	28.0	91.7	0.47	60.9	0.02	2.06	0.29	37.0	.77	.83
12. Overall	27.9	79.2	0.24	44.7	0.08	15.2	0.22	39.9	.69	.77

Dimensionality

We found robust correlations in the .3 to .8 range between ratings of the 11 dimensions of instruction. However, an EFA of all portfolio ratings showed that a unidimensional solution was relatively weak fit (RMSEA = 0.21, χ² = 103.5, p < .000).³ Fit improved after removing accuracy and feedback, the two dimensions with lowest variance and reliability in Table 3 (RMSEA = 0.12, χ² = 74.9, p < .05). Average item-factor correlations observed in this solution were .57, with 42% of the total variance accounted by the first factor. Two- and three-factor solutions, while a marginally better fit to the data, show a considerable amount of cross loadings, and they are difficult to reconcile with expected patterns of relationship among domains of instruction.

Concurrent and Predictive Relation With Other Measures

Table 4 presents correlations for scores assigned to a subset of portfolios (N = 25) by trained raters and consensus master scores assigned by the research team. The average scores are similar for most dimensions, and the correlations generally show substantial convergence among rater and master scores with the exception of accuracy and feedback—the two dimensions with limited variation discussed previously. The table finally shows a moderate to high correlation between the portfolio ratings and the number of pieces of evidence recorded by the raters across the 10 days of collection, suggesting that the volume of evidence collected was a significant driver of rater scores.

Table 4.

Descriptive Statistics and Correlations for Raters and Master Scores.

Dimensions	Trained rater scores		Master scores		Correlation
(N = 25)	M	SD	M	SD	Rater and master scores	Mean rating and no. of pieces of evidence
1. Initial Thinking	2.05	1.04	2.53	1.07	.543	.759
2. Diverse Learners	2.57	1.20	2.65	1.10	.789	.679
3. Productive Talk	2.11	0.83	2.53	0.91	.549	.582
4. Accuracy	4.89	0.36	3.51	0.95	−.101	−.55
5. Ways of Knowing	2.79	0.86	2.62	0.77	.591	.312
6. Scientific Investigations	2.16	1.02	2.03	1.12	.784	.626
7. Data Analysis	2.17	1.06	2.71	1.08	.876	.634
8. Scientific Products	2.19	0.88	2.51	0.95	.557	.325
9. Assessment	3.24	0.95	3.51	0.80	.502	.424
10. Feedback	1.54	0.59	1.78	0.84	.201	.674
11. Using Information	3.33	1.19	3.16	1.14	.815	.307

Table 5 shows limited convergence of portfolio ratings with teacher scores in the ATLAST test of PCK; only Dimensions 1 (Eliciting initial student thinking) and 11 (Using information to improve teaching) showed significant correlations with ATLAST scores in physics, and no significant correlations were observed with chemistry portfolio ratings. Portfolio ratings for Initial Thinking (D1), Productive Talk (D3), Scientific Investigations (D6), and Overall (D12) were significantly associated to teacher reports of self-efficacy for teaching students with different levels of achievement, learning disabilities, and limited English proficiency. A number of additional correlations in the same direction were not significant, which may reflect in part low reliability of some portfolio ratings, and limited statistical power of the final sample of portfolios included in the analyses (n = 27–30). Interestingly, negative correlations were observed for portfolio ratings of Support of Diverse Leaners (Dimension 2) and all indicators of teacher self-efficacy. Although not statistically significant, the consistent direction of these coefficients across the board could suggest that teachers working the hardest to serve their diverse students also perceive the highest barriers to effective teaching and learning.

Table 5.

Correlation Between Portfolio Ratings, ATLAST Scores, and Teacher Self-Efficacy Reports.

	ATLAST scores (0–30 scale)		Teacher self-efficacy^a (0–4 scale)
Dimension	Physics^b	Chemistry^c	Varied achievement	Learn disab.	Phys. disab.	ELL	GATE	NGSS
1. Initial Thinking	.451*	−.076	.430*	.385*	.06	.391*	.02	−.05
2. Diverse Learners	.29	−.101	−.09	−.18	−.19	−.29	−.26	−.31
3. Productive Talk	.097	−.136	.413*	.29	.00	.381*	.24	.01
4. Accuracy	.356	−.013	−.02	−.17	.06	.14	.25	.07
5. Ways of Knowing	.241	−.394	.28	.23	−.06	.28	.14	−.05
6. Scientific Investigation	.037	−.126	.430*	.423*	.01	.32	.10	.03
7. Data Analysis	−.109	−.151	.26	.34	−.19	.22	.10	−.10
8. Scientific Products	.097	−.245	.25	−.09	−.06	−.08	.420*	.07
9. Assessment	−.196	−.329	.21	.13	.02	.27	.18	−.03
10. Feedback	−.061	−.047	.08	−.20	−.11	.05	.17	−.08
11. Using Information	.43*	−.002	.27	.11	.06	.19	.20	.08
12. Overall	.212	−.192	.362*	.18	−.01	.15	.24	−.01
N	28	27	30	30	29	30	30	30
Mean (SD)	15.7 (3.9)	13.9 (1.7)	2.1 (0.6)	1.8 (0.7)	1.4 (0.8)	2.0 (0.6)	1.97 (0.8)	1.8 (0.8)

Note. ATLAST = Assessing Teacher Learning About Science Teaching; NGSS = Next Generation Science Standards.

Average scores across both subject portfolios and raters. ^bAverage scores for physics portfolio across raters. ^cAverage scores for chemistry portfolio across raters. *p < 0.05.

Table 6 shows a number of significant correlations between portfolio ratings of two dimensions (Ways of Knowing and Scientific Investigations) as well Overall portfolio ratings, and aggregate student survey reports of instructional practices. Specifically, higher portfolio ratings consistently relate to students reporting more frequent engagement with a range of NGSS-aligned practices, including discussing scientific ideas in groups, planning experiments, real world modeling/design, explaining scientific claims, among others. This pattern of convergence is observed across 12 NGSS-aligned instructional practice items, and is reinforced by nonsignificant correlations with items that capture divergent instructional practices (e.g., memorizing vocabulary, teacher lecturing, carrying out experiments planned by the teacher).

Table 6.

Correlation Between Portfolio Ratings and Student Survey Reports of Instruction.

	Mean (SD) (0–4 scale)	Correlation to portfolio ratings
Dimensions	Mean (SD) (0–4 scale)	D5: Ways of Knowing	D6: Scientific Investigation	D12: Overall Quality
a. Discussed scientific ideas with class or small group	1.93 (0.81)	.478*	.507**	.434*
b. Wrote reflections about experiments or investigations	1.75 (0.93)	.536**	.475*	.327
c. Used/created models to explain what happens in nature	1.53 (0.88)	.393*	.333	.312
d. Discussed or critiqued other students’ scientific claims	1.39 (0.91)	.545**	.491**	.354
e. Memorized science vocabulary and/or facts	1.82 (0.84)	.149	.124	.154
f. Completed assignment/homework (short-answer, essay)	2.13 (0.8)	.100	.131	.025
g. Completed assignment/homework (mc, t/f, fill blank)	2.02 (0.93)	−.032	.016	−.111
h. Listened to a lecture or presentation by the teacher	2.34 (0.79)	.109	.134	.155
i. Planned experiments to answer my/our own questions	1.27 (0.96)	.548**	.466*	.455*
j. Completed investigations planned by the teacher	2.17 (0.75)	.296	.296	.244
k. Designed an invention to solve real world problem	0.88 (0.96)	.644**	.592**	.456*
l. Prepared written reports for labs or experiments	1.52 (0.86)	.489**	.475*	.504**
m. Spent time thinking about what I learned in class	1.68 (0.92)	.455*	.460*	.269
n. Completed pretests so teacher knows what I know	1.31 (0.94)	.315	.281	−.059
o. Collected or analyzed data	1.99 (0.83)	.424*	.396*	.272
p. Explained scientific claims using evidence	1.89 (0.83)	.600**	.571**	.494**
q. Received written comments on tests or assignment	1.59 (0.99)	.625**	.590**	.428*
r. Received feedback from classmates to help me learn	1.4 (0.96)	.449*	.430*	.398*

Note. Item wording streamlined for space. See the supplemental appendix for full text of student survey items. *p < 0.05.

Table 7 presents the results of multilevel linear models investigating the relationship between portfolio ratings of science instruction and three types of student outcomes. The results in the table showed no relationship between portfolio ratings of any of the 11 dimensions of instruction (or Overall Quality ratings), and either student scores in the state’s eighth grade standardized science test, or course grades assigned by the teacher at the end of the year—to adjust for preexisting differences in student achievement across classrooms, the models include student scores in the seventh grade state mathematics test as covariates (there is no state science test in seventh grade). Interestingly, portfolio ratings of two closely related dimensions (Scientific Investigations and Data Analysis) were positively associated to a composite indicator of student self-efficacy and enjoyment of science. This is consistent with growing evidence of positive effects of student-driven scientific investigation on student engagement in science (see, for example, Hushman & Marley, 2015).

Table 7.

Hierarchical Linear Models Relating Portfolio Ratings to Three Student Outcomes^a.

Variable	Course grades		Test scores		Self-efficacy
Variable	Estimate (γ₀₁)	p value	Estimate (γ₀₁)	p value	Estimate (γ₀₁)	p value
SBAC_seventh grade (covariate)	−0.006	.000	0.488	.000
1. Initial Thinking	0.081	.342	5.99	.482	−0.002	.972
2. Diverse Learners	−0.043	.626	−5.64	.500	0.075	.087
3. Productive Talk	0.057	.532	−4.31	.620	0.045	.449
4. Accuracy	0.388	.538	−78.37	.180	−0.11	.644
5. Ways of Knowing	0.112	.357	4.06	.729	0.009	.891
6. Scientific Investigations	0.176	.107	−6.71	.526	0.14	.009
7. Data Analysis	0.103	.263	−4.05	.641	0.12	.016
8. Scientific Products	0.121	.362	−6.18	.621	0.002	.975
9. Assessment	0.110	.259	14.34	.117	0.04	.431
10. Feedback	0.167	.203	−9.72	.474	−0.07	.341
11. Using Information	0.031	.802	−.286	.981	0.05	.305
12. Overall	0.101	.445	−.837	.947	0.10	.141

Separate models estimated for each dimension, models summarized in one table for purposes of space.

A final set of analyses investigated the quality of evidence collected by teachers in portfolios as the basis for portfolio scoring. We asked portfolio raters to tally the number of pieces of evidence in a portfolio relevant for scoring each dimension of instruction. Based on these tallies, a typical portfolio had 12 pieces of evidence related to Multiple Ways of Knowing (D5), and seven pieces for Providing Feedback (D11), but less than two pieces of evidence relevant for Eliciting Student Initial Thinking (D1), and no evidence at all for Accurate Content Knowledge (D4). However, the volume of evidence collected was only weakly related to the reliability of the scores—a correlation of .29 and .19, respectively, for the physics and chemistry portfolios. Moreover, rater notes and debriefing interviews indicate that for some dimensions (e.g., Multiple Ways of Knowing, Data Analysis) raters found artifacts with relevant evidence on most or all days covered in the portfolio, whereas for other dimensions (e.g., Initial Thinking, Accurate Content Knowledge) evidence was much more sparse. Rating reliability was moderately related to the percentage of days with evidence (.33–.45 across dimensions) suggesting that this, more than the total number of artifacts, was important for informing reliable ratings.

Discussion

For this study, we developed an electronic portfolio app for mobile devices that enables teachers to compile, review, and store high quality, context-rich evidence of instruction in science classrooms. This overcomes key limitations of traditional portfolios and contributes to research in the measurement of instructional practices in several important ways. First, multimedia capabilities allow the portfolios to compile evidence of critical live and interactive aspects of instruction like eliciting and building on student thinking (e.g., Windschitl et al., 2012) and facilitating argumentative discourse (e.g., Kuhn, 2010). Second, interactively collecting and annotating evidence in site, in a familiar handheld device using screen survey prompts and voice-to-text reduces the burden of portfolio collection for teachers, and can improve the quality and granularity of evidence collected. Third, the app and accompanying web portal can substantially reduce the logistical challenges of large-scale portfolio collection, management, and storage. Finally, on the substantive side, this project contributes a formalized tool and set of rubrics for measuring instruction aligned to the three-dimensional framework set forth by the NGSS.

As a measurement instrument, the portfolio app is intended to support reliable and valid inferences about the quality of instruction in middle school science classrooms, and this study aimed to provide initial evidence to support a validity argument for the measures derived. We hypothesized that trained raters using detailed portfolio scoring rubrics would be able to produce scores that reliably reflect important aspects of quality instruction in science aligned to the NGSS, and relate to expert ratings of the same portfolios, to other measures of instructional practice, and to variation in student outcomes. The study produced mixed but promising results. For a number of dimensions (and Overall Quality judgments), generalizability coefficients were adequate or strong, indicating that portfolio ratings may be able to reliably differentiate instruction of varying levels of quality. However, for several other dimensions, reliability was low and decidedly outside range for high-stakes (and even low-stakes) uses. For two dimensions (accuracy and feedback), low levels of reliability did not arise from rater error, but instead low score variance reflecting uniformly high or low scores. Upon closer inspection, this revealed misalignment between the rubrics and the contents of the portfolio—the rubrics aimed too low for one dimension, and little to no evidence was collected in portfolio for the other. These results highlight challenges faced by teachers in documenting certain kinds of practices (e.g., most portfolios contained little to no evidence of verbal feedback) and by extension the importance of offering detailed guidelines and expectations for fidelity of collection to teachers for compiling evidence into their portfolios. The results overall also suggest that effectively documenting some key aspects of instruction require consistent and detailed collection of classroom artifacts, and certain types of artifacts in particular. For example, teachers may be instructed to collect pictures of assignments or assessments to capture written feedback, and short video artifacts that specifically reflect verbal feedback.

Examining the daily evidence raters considered in assigning scores to portfolios offers an interesting look into the process of evaluating portfolio evidence in relation to scoring rubrics. Rater logs and exit interviews suggest trained raters did not systematically tally, but instead mentally compiled or aggregated the entirety and variety of multimedia evidence captured in a portfolio over 2 weeks, which was then weighed to assign high inference scores to portfolios. This approach to scoring represents a substantial cognitive challenge, and is susceptible to inconsistency and thus measurement error, as some pieces of evidence will inevitably be weighted differently by different raters, in relation to some dimensions and scoring rubrics. On the contrary, a different approach using lower-inference daily scores would present their own challenges and sources of inconsistency, as it would force raters to consider artifacts in isolation from the broader context reflected in the portfolio. The advantages and disadvantages of these approaches to handle multiday evidence from artifacts require much additional research. In a project currently under way, we are investigating the reliability of ratings of portfolios configured to cover units of different lengths (for another example, see Meyer et al., 2011).

Our findings point to a unidimensional structure underlying portfolio ratings (excluding the two dimensions with extremely very low variance). This is of course not surprising considering the deep conceptual connections between the dimensions of instruction in our NGSS model, and their close interdependency when enacted in practice in classrooms. In terms of convergence with other measures, we found strong correlations between portfolio scores assigned by raters and expert master scores, but limited convergence with other measures of instruction collected from teachers, including ATLAST PCK scores, and teacher self-efficacy, and reports of instruction. However, we found substantial convergence between student reports of NGSS-aligned instruction and portfolio ratings of two key dimensions in our model (Ways of Knowing and Scientific Investigations) as well as the Overall Quality portfolio rating. Divergent correlations are not significant for a number of non-NGSS practices, providing further evidence that this is not a case of a single strong correlation and a halo effect. Finally, we also found significant relationship between these portfolio ratings, and higher student enjoyment of and better attitudes about science and science learning.

There was no relation between portfolio ratings, student test scores, and course grades. This could reflect a small sample size, but also likely restriction of range in instructional practices (teachers were a select group of volunteers from one district), and of the criterion measures themselves—lack of alignment and sensitivity for test scores, reliability and endogeneity for course grades. Additional work is needed to adjudicate these shortcomings to issues of portfolio design, structure, and guidelines for consistent evidence collection, rubric design, and rater training. Subsequent study using refined instruments and a larger sample of teachers more experienced with both portfolios and NGSS instruction would help clarify these results.

This study contributes to our understanding of key issues in the burgeoning subfield of measurement that focuses on the development of reliable and valid indicators of instruction for purposes of instructional improvement and accountability in science and other areas in k-12 and higher education (see, for example, Ing, 2017; Polikoff et al., 2020; William T. Grant Foundation et al., 2015). Our findings add to a growing body of evidence that support a validity argument for portfolio use that rests on evidence that these instruments can offer a robust evidentiary basis to support reliable judgments about the quality of instructional practices in STEM subjects, and that they relate to other measures of instruction, and some valued student outcomes. Much more research is needed to investigate specific validity arguments for these scores, for specific uses and contexts. Of particular interest are differences in the requirements for high-stakes summative inferences and uses (e.g., certification or evaluation), compared with lower-stakes formative uses (see, for example, Ing et al., 2020).

A final set of considerations relates to the feasibility and practical value of using this type of tool in practice. On one hand, thoroughly compiled and annotated evidence of instruction is needed for portfolios to be scored reliably. On the other hand, the burden that portfolio collection imposes on teachers can be substantial—debriefing interviews indicate that participant teachers invested 10 to 20 hours collecting a complete portfolio. Importantly, though still significant, this is a qualitatively different type of time commitment relative to well-known portfolio performance assessments used for teacher certification. By comparison, the NBPTS portfolio can require hundreds of hours and extend over months (and sometimes years), and the edTPA portfolio still requires dozens of hours and potentially months to complete.

Portfolio assessments such as those are simply too onerous for frequent real-world usage, and thus commonly limited to major barrier reviews like teacher certification (edTPA) or professional distinction (NBPTS). Importantly, however, portfolios can be alternatively conceived as documentary records reflecting authentic classroom practice over a period of time (Shulman, 1998), and used formatively outside of high-stakes professional evaluation structures. In this model, portfolios are crucially not passive data collection instruments, or exhaustive teaching performance assessments, but powerful platforms for sustained teacher intellectual engagement and fruitful professional reflection around instruction (Stefani et al., 2007).

Indeed, qualitative analyses of participant teacher and rater interviews suggest that portfolio collection (and portfolio scoring too) are cognitively rich and demanding tasks that required both groups to reflect on their own instructional practices, and ultimately led to shifts in perceptions of self-efficacy. Interestingly, in subsequent work with teachers using the portfolio platform, participants also highlighted the value of joint discussion around artifacts and portfolios in the context of professional learning communities (Kloser). Thus, a more nimble and efficient type of portfolio tool could enable realistic implementation of systems of professional reflection and development organized around classroom artifacts. In this context, the time spent compiling and annotating evidence in a portfolio is not merely a data collection burden, but instead the main avenue for engagement in professional reflection, discussion, and learning (Darling-Hammond et al., 2017).

Thus, while the psychometric properties of portfolio scores reported here would not support their use for high-stakes summative assessment of instructional practice, the preliminary evidence of reliability and validity presented in this study suggests that the tool could hold promise for formative assessment of classroom instruction and teacher practice. From a measurement perspective, the development of more efficient portfolio tools can address a critical need in education systems, enabling efforts to systematically monitor and improve instructional practice, and use consistent evidence to inform the professional development of teachers. One way to assess the cost-benefit of this type of tool is to consider the amount of time that would be required for an outside observer to collect a volume of evidence comparable with that of a portfolio—and to note that the body of evidence collected would likely be less nuanced and revealing than that compiled by the teacher in this portfolio model. More broadly, efficient portfolios that yield granular, richly contextualized indicators of instruction could also help educators build the knowledge base correlating student learning outcomes and instructional practices and strategies teachers implement in the classroom (Correnti & Martinez, 2012). Beyond measurement considerations, an extensive body of research suggests that teachers need time and support to align their instruction to the NGSS, and that this process requires extensive analysis and reflection based on multiple sources of information and evidence of instruction (Allen & Penuel, 2015). Evidence from this and other studies suggests that electronic portfolios can support reflection around classroom artifacts for individual teachers and professional learning communities (Kloser, 2021). Much research is still needed to investigate the potential uses and value of this type of modern electronic portfolio tool in this context.

Supplemental Material

sj-pdf-1-epm-10.1177_00131644211064923 – Supplemental material for Developing Situated Measures of Science Instruction Through an Innovative Electronic Portfolio App for Mobile Devices: Reliability, Validity, and Feasibility

Supplemental material, sj-pdf-1-epm-10.1177_00131644211064923 for Developing Situated Measures of Science Instruction Through an Innovative Electronic Portfolio App for Mobile Devices: Reliability, Validity, and Feasibility by José Felipe Martínez, Matt Kloser, Jayashri Srinivasan, Brian Stecher and Amanda Edelman in Educational and Psychological Measurement

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Support for this project was provided by the National Science Foundation (DRL-REAL 1420252).

ORCID iD

José Felipe Martínez

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Allen

C. D.

Penuel

W. R.

(2015). Studying teachers’ sensemaking to investigate teachers’ responses to professional development focused on new standards. Journal of Teacher Education, 66(2), 136–149.

Ball

D. L.

Forzani

F. M.

(2009). The work of teaching and the challenge for teacher education. Journal of Teacher Education, 60(5), 497–511.

Borko

Stecher

B. M.

Martinez

J. F.

Kuffner

K. L.

Barnes

Arnold

S. C.

(2006). Using classroom artifacts to measure instructional practices in middle school science: A two-state field test (No. 690). Center for Research on Evaluation, Standards and Student Testing, University of California.

Cohen

Goldhaber

(2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45(6), 378–387.

Correnti

Martinez

J. F.

(2012). Conceptual and methodological issues in the study of teaching: Mission statement for improving instruction at scale. Educational Assessment, 17, 2–3.

Darling-Hammond

Hyler

M. E.

Gardner

(2017). Effective teacher professional development. Learning Policy Institute.

Desimone

LeFloch

(2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26(1), 1–22.

Gitomer

D. H.

(2009). Measurement issues and assessment for teaching quality. SAGE.

Grossman

McDonald

(2008). Back to the future: Directions for research in teaching and teacher education. American Educational Research Journal, 45(1), 184–205.

10.

Hushman

Marley

(2015). Guided instruction improves elementary student learning and self-efficacy in science. The Journal of Educational Research, 108(5), 371–381.

11.

Ing

(2018). What about the “instruction” in instructional sensitivity? Validity issues in research on instructional sensitivity. Educational and Psychological Measurement, 78(4), 635–652.

12.

Ing

Chinen

Jackson

Smith

(2020). When should i use a measure to support instructional improvement at scale? The importance of considering both intended and actual use in validity arguments. Educational Measurement: Issues and Practice, 40(1), 92–100. https://doi.org/10.1111/emip.12393

13.

Kane

(2006). Validation. In Brennan

R. L.

(Ed.), Educational measurement (4th ed., pp. 17–64). American Council on Education and Praeger.

14.

Kane

T. J.

Staiger

D. O.

(2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Bill & Melinda Gates Foundation. http://www.metproject.org/reports.php

15.

Kennedy

M. M.

(1999). Approximations to indicators of student outcomes. Educational Evaluation and Policy Analysis, 21, 345–363.

16.

Kersting

N. B.

Sherin

Stigler

J. W.

(2014). Automated Scoring of Teachers’ OpenEnded Responses to Video Prompts: Bringing the Classroom Video Analysis (CVA) Assessment to Scale. Educational & Psychological Measurement, 74(6), 950–974.

17.

Kloser

Edelman

Floyd

Martinez

J.F.

Stecher

Srinivasan

(2021) Interrogating practice or show and tell?: Using a digital portfolio to anchor a professional learning community of science teachers. Journal of Science Teacher Education, 31(2) 210–241.

18.

Knapp

M. S.

(1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning. Review of Educational Research, 67(2), 227–266.

19.

Krajcik

Codere

Dahsah

Bayer

Mun

(2014). Planning instruction to meet the intent of the next generation science standards. Journal of Science Teacher Education, 25(2), 157–175.

20.

Kuhn

(2010). Teaching and learning science as argument. Science Education, 94(5), 810–824.

21.

Latour

(1986). Laboratory life: The construction of scientific facts. Princeton University Press.

22.

Lemke

J. L.

(1990). Talking science: Language, learning, and values (Vol. 1). Ablex Publishing Corporation.

23.

Martinez

J. F

Borko

Stecher

Luskin

Kloser

(2012). Measuring classroom assessment practice using instructional artifacts: A validation study of the QAS notebook. Educational Assessment, 17(2-3), 107–131.

24.

Matsumura

L. C.

Garnier

H. E.

Slater

S. C.

Boston

M. D.

(2008). Toward measuring instructional interactions “at-scale.” Educational Assessment, 13(4), 267–300.

25.

Mayer

(1999). Measuring instructional practice: Can policy makers trust survey data? Educational Evaluation and Policy Analysis, 21(1), 29–45.

26.

Medley

D. M.

Mitzel

H. E.

(1963). Measuring classroom behavior by systematic observation. In Gage

N. L.

(Ed.), Handbook of research on teaching (pp. 247–328). Rand McNally.

27.

Meyer

J. P.

Cash

A. H.

Mashburn

(2011). Occasions and the reliability of classroom observations: Alternative conceptualizations and methods of analysis. Educational Assessment, 16(4), 227–243. https://doi.org/10.1080/10627197.2011.638884

28.

Muthén

B. O.

Muthén

L. K.

(2012). Mplus (Version 7.0) [Computer software].

29.

National Board for Professional Teaching Standards. (2019). Scoring guide: Understanding your scores. https://www.nbpts.org/wp-content/uploads/2021/05/NBPTS_Scoring_Guide.pdf

30.

National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academies Press.

31.

Podolsky

Kini

Darling-Hammond

(2019). Does teaching experience increase teacher effectiveness? A review of US research. Journal of Professional Capital and Community, 4(4), 286–308.

32.

Polikoff

M. S.

Gasparian

Korn

Gamboa

Porter

A. C.

Smith

Garet

M. S.

(2020). Flexibly using the surveys of enacted curriculum to study alignment. Educational Measurement: Issues and Practice, 39, 38–47. https://doi.org/10.1111/emip.12292

33.

Pollio

Hochbein

C. D.

(2015). The association between standards-based grading and standardized test scores as an element of a high school reform model. Teachers College Record: The Voice of Scholarship in Education, 117, 1–28.

34.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE.

35.

Raudenbush

S. W.

Sadoff

(2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154.

36.

Rockoff

J. E.

(2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review, 94(2), 247–252.

37.

Rowan

Correnti

(2009). Studying reading instruction with teacher logs: Lessons from a study of instructional improvement. Educational Researcher, 38, 120–131.

38.

Stanford Center for Assessment, Learning, and Equity. (2018). Educative assessment & meaningful support: 2017 edTPA administrative report. http://edtpa.aacte.org/news-area/2017-edtpa-annual-administrative-report.html

39.

Schunk

D. H.

(1989). Self-efficacy and achievement behaviors. Educational Psychology Review, 1, 173–208.

40.

Schweig

J. D.

(2014). Quantifying error in survey measures of school and classroom environments. Applied Measurement in Education, 27(2), 133–157.

41.

Scruggs

T. E.

Brigham

F. J.

Mastropieri

M. A.

(2013). Common core science standards: Implications for students with learning disabilities. Learning Disabilities Research & Practice, 28(1), 49–57.

42.

Seidel

Shavelson

R. J.

(2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77(4), 454–499.

43.

Shavelson

R. J.

Webb

N. M.

(1991). Generalizability theory: A primer. SAGE.

44.

Shulman

(1998). Teacher portfolios: A theoretical activity. In Lyons

(Ed.), With Portfolio in Hand (pp. 23–37). Teachers College Press.

45.

Smith

P. S.

(2010, March 21–24). New tools for investigating the relationship between teacher content knowledge and student learning [Paper presentation]. Presented at the 2010 NARST Annual Conference, Philadelphia, PA, United States.

46.

Stecher

V.N.

Hamilton

Ryan

Robyn

Lockwood

J. R.

(2006). Using structured classroom vignettes to measure instructional practices in mathematics. Educational Evaluation and Policy Analysis, 28(2), 101–130.

47.

Stefani

Mason

Pegler

(2007). The educational potential of e-portfolios. Routledge.

48.

Strudler

Wetzel

(2011). Electronic portfolios in teacher education: Forging a middle ground. Journal of Research on Technology in Education, 44(2), 161–173.

49.

Tangmunarunkit

Hsieh

C. K.

Longstaff

Nolen

Jenkins

Ketcham

. . . Estrin

(2015). Ohmage: A general and extensible end-to-end participatory sensing platform. ACM Transactions on Intelligent Systems and Technology (TIST), 6(3), Article 38.

50.

University of California, Los Angeles. (2018). e-QIS (3.10.27) [Mobile application software]. https://play.google.com/store/apps/details?id=org.ohmage.x.eqis

51.

William

Grant Foundation, Spencer Foundation, & Bill & Melinda Gates Foundation. (2015). Measuring instruction in higher education: Summary of a convening. https://wtgrantfoundation.org/library/uploads/2015/11/Measuring-Instruction-in-Higher-Education.pdf

52.

Windschitl

Thompson

Braaten

Stroupe

(2012). Proposing a core set of instructional practices and tools for teachers of science. Science Education, 96(5), 878–903.

53.

Zee

Koomen

(2016). Teacher self-efficacy and its effects on classroom processes, student academic adjustment, and teacher well-being: A synthesis of 40 years of research. Review of Educational Research, 86(4), 981–1015.

54.

Zeichner

(2012). The turn once again toward practice-based teacher education. Journal of Teacher Education, 63(5), 376–382.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.61 MB