Organizational Research Methods

A cornerstone of the scientific ethos is that editorial board members should be selected based on their scholarly achievements, as demonstrated by publications in peer-reviewed journals and evidence that their work is of value to others in their disciplines. To discern if this reasoning is applied in practice, this study examined the scholarly records of the editorial boards of premier peer-reviewed journals sponsored by the leading professional associations in management and six related disciplines.

This commentary critically examines the issues raised in the article by Bedeian, Van Fleet, and Hyman from a functional perspective, one that might be used in decisions about who should be a member of a prototypic work group. It takes the position that, like a work group, a journal editorial board exists for a purpose, and that purpose (or function) should play a major role in determining the criteria to be used when staffing the board's membership. When contemporary thinking about the staffing of work groups is applied, the current practices of the journals of interest are found to be appropriate and defensible.

Bedeian, Van Fleet, and Hyman's article offers a critical examination of the scholarly achievements of editorial review board members for several scholarly journals in related disciplines, including the Academy of Management's primary scholarly journals, the Academy of Management Review and the Academy of Management Journal (AMJ). The author finds their critique to be interesting and thorough in many ways. However, this commentary examines several issues related to their critique, including the methodology, criteria for and process of selecting editorial review board members, the editor's role in the evaluation of work submitted to journals, and the performance of scholarly journals. Consideration of these issues allows a broader and more complete view of scholarly publishing, leading to some different conclusions.

Bedeian, Van Fleet, and Hyman offer some data that cast doubt on the qualifications of the reviewers of the flagship journals in our field. Using two alternative measures of authoring success, we found the reviewers of Academy of Management Journal are as a group more accomplished scholars than the authors are as a group. However, we suggest that a more relevant conversation should be about addressing the large gap between the demand for effective reviewers and the supply of individuals who are both successful authors and effective reviewers. We encourage a two-pronged approach involving reducing the demand for reviews on one hand and increasing the supply of effective reviewers on the other hand. We offer several procedural and structural alternatives to initiate a conversation that might help to move us toward some norms to encourage and reward effective reviewing in our field.

Commentators expressed contrasting views on the authors' examination of scientific achievement and editorial board membership in the management discipline. In response, the authors address key points with which they disagree and hold fast to their admonitory conclusions, neither compromising nor retreating from the recounting of base facts. If the authors' conclusions have prompted a measure of cognitive dissonance, they hope that any associated discomfort will lead to action on the part of all of the discipline's journals and their sponsoring organizations.

This article describes how a multilevel logistic regression (MLR) approach to assessing person fit can be used to test hypotheses concerning faking on personality assessments. Item difficulty and person trait estimates obtained from a two-parameter logistic item response theory model are used to predict the probability of endorsing an item in a MLR equation. The regression slope for item difficulty reflects the extent to which the probability of endorsement decreases as item difficulty increases. Less negative slopes may indicate faking, and slope variance may be modeled with person-level variables using MLR. Two examples are presented. Example 1 models faking on a personality assessment with dichotomous items. Example 2 extends the approach to scales using polytomous items.

Software developments increasingly facilitate inclusion of incomplete data, but relatively little research has examined the effects of incomplete data on statistical power. Seven steps needed to conduct power analyses with incomplete data for a variety of commonly tested hypotheses are illustrated, focusing on significance tests of individual parameters. The example extends a growth curve model simulation presented by Curran and Muthén (1999) to the incomplete data situation. How to estimate statistical power for a range of sample sizes from a single model, as well as how to calculate the sample size required to obtain a desired value of statistical power, is demonstrated. Effects of data being missing completely at random (MCAR) or missing at random (MAR) across a range from 0% (complete data) to 95% missing data are considered. SAS and LISREL syntax are provided in this paper with syntax for other software available from the authors.

The use of multilevel modeling to investigate organizational phenomena is rapidly increasing. Unfortunately, little advice is readily available for organizational researchers attempting to determine statistical power when using multilevel models or when determining sample sizes for each level that will maximize statistical power. This article presents an introduction to statistical power in multilevel models. The unique factors influencing power in multilevel models and calculations for estimating power for simple fixed effects, variance components, and cross-level interactions are presented. The results of simulation studies and the existing general rules of thumb are discussed, and the available power analysis software is reviewed.

Group-level constructs are often derived from individual-level data. This procedure requires a composition model, specifying how the lower level data can be combined to compose the higher level construct. Two common composition methods are direct consensus composition, where items refer to the individual, and referent-shift consensus composition, where items refer to the group. The use and selection of composition methods is subject to a number of problems, calling for more systematic work on the empirical properties of and distinction between constructs composed by different methods. To facilitate and encourage such work, the authors present a methodological framework for addressing the distinction between and the baseline psychometric quality of composed group constructs, illustrated by an empirical example in the group job-design domain. The framework primarily represents a developmental tool with applications in multilevel theory building and scale construction, but also in meta-analysis or secondary analysis, and more general, the validation of group constructs.