Examining the Dimensions of the Learning Organization Questionnaire

Abstract

Although the Dimensions of the Learning Organization Questionnaire (DLOQ) has been used internationally, especially in the field of Human Resource Development, during the last two decades, researchers have reported problems with multicollinearity and a lack of discriminant validity. The purpose of this study was to review the use of the DLOQ in published research and, in particular, from a construct validity perspective. Our aim was to better understand the deployment, outcomes, and utility of the instrument. Toward this purpose, we conducted a systematic literature review examining research using the DLOQ. Concurrently, we reviewed literature at the center of the DLOQ that led to the formation and early validation of the construct and items. These study findings may help researchers and theorists address the utility of the DLOQ and develop a more valid instrument to measure the learning organization culture.

Keywords

learning organization The Dimensions of the Learning Organization Questionnaire (DLOQ)multicollinearity construct validity discriminant validity convergent validity exploratory factor analysis (EFA)confirmatory factor analysis (CFA)

The 21st century can be defined as the era of globalization, technological revolution, demographic change, diversity, chaos, and turbulent change (Marquardt, Berger, & Loan, 2004). Organizations and work environments are changing more rapidly than ever before. Therefore, organizations and individuals must have the ability to adapt to the change and cope with new challenges that arise in novel situations (Senge, 1990; Werner, & DeSimone, 2009). With the emphasis on and potential in establishing the competitive learning culture, learning organization (LO) may well be a remedy pursued by many human resource development (HRD) professionals to enhance their organizational capacity and competitiveness that are required to survive.

Many organizations have set their sights toward becoming LOs. According to Marquardt et al. (2004), one of the critical success factors of FedEx, the world’s largest express transportation company, is the commitment to becoming a LO. However, scholars also have struggled with the identification and assessment of the factors that comprise and differentiate LOs from other firms. One of the best known of these efforts is the Dimensions of the Learning Organization Questionnaire (DLOQ), which was developed by Watkins, Yang, and Marsick (1997) and Yang, Watkins, and Marsick (1998, 2004). Inherent to the framing of the DLOQ is the multidimensionality (seven dimensions in all) of the central construct, including leadership for learning, system connection, embedded system, continuous learning, dialogue and inquiry, empowerment, and team learning.

In addition to the aforementioned “dimensions,” the DLOQ was developed based on a theoretical framework that integrates four overarching frames—organizational learning, workplace learning, learning climate, and learning structure perspective (Örtenblad, 2002). Although factor analytic information has been infrequently published, the validity of the DLOQ has been reported to be satisfactory in numerous studies (Hernandez & Watkins, 2003; Wang, 2005; Wang, Yang, & McLean, 2007; Yang et al., 2004). Accordingly, the DLOQ not only contributes to the framing and assessment of HRD-related practices, but the instrument has also contributed to LO-related research and theory building.

The DLOQ has been deployed internationally, primarily by researchers in the field of HRD, in 15 countries across Africa, Asia, Australia, Canada, Europe, the Middle East, South America, and the United States (Marsick, 2013; Song, Chermack, & Kim, 2013). As its use has expanded since its development, researchers have reported several statistical limitations regarding construct validity of the DLOQ (in both the long and reduced-item versions of the DLOQ). These limitations included multicollinearity (Egan, Yang, & Bartlett, 2004; Martinez, 2009), lack of discriminant validity (Yang et al., 2004), and single factor extraction (Benjamin, 2009; Martinez, 2009). Although not addressed in each published study, the aforementioned limitations have led some researchers to question both (a) the extent to which LO features are being assessed by the DLOQ via an individual “self-report” format and (b) whether the hypothesized dimensions are reflected in study participant responses.

Based on these concerns, the purpose of this study was to examine the DLOQ instrument with the aim of better understanding the strengths and weaknesses of this tool—within the context of both the espoused framework and constructs informing the instrument and established measurement standards. As emphasized by Nunnally and Bernstein (1994) and Strickland (2001), to establish validity, a survey instrument should link clearly to related theoretical assumptions. Therefore, Watkins and Marsick’s (1993), Marsick and Watkins’ (1999), and Yang et al.’s (2004) LO framework and previous research involving DLOQ validation are reviewed herein. The findings of the current study have the potential to inform the utility of the DLOQ and to provide elaboration regarding the validity of the DLOQ.

Research Questions and Method

One of the benefits of using the DLOQ is that it allows researchers to examine predictors of the LO construct (Mahseredjian, Karkoulian, & Messarra, 2011). Despite this, there is paucity of research that utilized the DLOQ as a measure of dependent variable (DV). Recently, Song et al. (2013) addressed this issue of positioning the DLOQ, but their findings were not convincing due to limited number of articles that they reviewed—the total number of articles was 35. To increase utility of the instrument, it is necessary to highlight the extent to which researchers have both positioned the DLOQ as a DV and investigated predictors of the LO construct, based on a more thorough, systematic, and extensive literature review. Since Watkins and Marsick (1993) contributed to the conceptualization of the LO and submitted the DLOQ “to rigorous critique for meaning” (Marsick & Watkins, 2003, p. 136), some researchers (Benjamin, 2009; Egan, 2002; Egan et al., 2004; Martinez, 2009; Yang et al., 2004) have reported that the DLOQ may lack construct validity. However, few researchers have examined the reasons why the DLOQ has been exposed to the issue of validity. The current study was guided by four research questions:

Research Question 1: How has the DLOQ been previously validated?

Research Question 2: To what extent has the DLOQ been used as a DV in empirical research, as compared with mediating/moderating and independent variables (IVs)?

Research Question 3: How have researchers reported the DLOQ data and study results in relation to the espoused seven DLOQ survey dimensions?

Research Question 4: How well does the DLOQ reflect the foundational constructs of the LO framework suggested by Watkins and Marsick?

To answer the research questions, we used systematic literature review method (Cho & Egan, 2009; Torraco, 2005) encompassing a specific literature search process and inclusion criteria.

Literature Search Process

To identify scholarly research aligned with our research questions, the following search engines were accessed online through a U.S. land grant institution library system, ABI/INFORM Complete (ProQuest), Academic Search Complete (EBSCO), Business Source Complete (EBSCO), ERIC (EBSCO), Education: A SAGE Full-Text Collection, Education Full Text (H. W. Wilson), Management & Organization Studies: A SAGE Full-Text Collection, Wilson Business Full Text (EBSCO), PsycINFO 1872-current (ProQuest), and ScienceDirect (Elsevier) were used as the primary databases. For doctoral dissertation resources, PsycINFO, ProQuest Dissertations and Theses (the United Kingdom and Ireland), LibCat (general catalog), Digital Library-Theses and Dissertations, and WorldCat (OCLC) were used as the primary databases. Two search terms, DLOQ and Dimensions of the Learning Organization Questionnaire, were used under the All Text option. All results containing one of these two search terms in their title, abstract, keywords, or body text were examined.

Inclusion Criteria

Four criteria were used to select the literature to be reviewed for this study, including (a) publication between January 1993 and August 2012; (b) publication in peer-reviewed scholarly/academic journals, unpublished doctoral dissertations, or conference papers; (c) articles published in English; and (d) quantitative research that used the full set of DLOQ items as established (i.e., 7-item, 21-item, or 43-item versions). Two studies using only a portion of the aforementioned seven dimensions of the DLOQ were excluded from our review. In the case of published dissertations, the published form of the research (i.e., usually a journal article) was chosen for review. The same approach was taken when there may have been duplication across a conference paper (where the author kept copyright privileges and published similar data in a copyrighted, refereed journal). The database search resulted in the identification of 90 studies that met the predetermined criteria: 65 journal articles, 22 dissertations, and 3 conference papers (see the appendix).

Literature Review

The LO Concept

The concept of the LO is most commonly associated with Senge’s (1990) work, The Fifth Discipline: The Art and Practice of the Learning Organization, in which he posited five components of a LO: systems thinking, personal mastery, mental model, shared vision, and team learning. According to Kofman and Senge (1993), the LO is viewed as a place where vision, patience, and courage are required and each individual in the organization is accepted as a “legitimate being” (p. 16). Garvin (1993) argued that many of the recommendations from Senge’s work are “far too abstract” (p. 79) and do not provide a guide for practical action. On the basis of this critical point of view, Garvin defined a LO as “an organization skilled at creating, acquiring, and transferring knowledge, and at modifying its behavior to reflect new knowledge and insight” (p. 80). In the United Kingdom, Pedler, Boydell, and Burgoyne (1989) suggested the learning company concept and defined it as “an organisation which facilitates the learning of all of its members and continuously transforms itself” (p. 2). Within this definition, learning and working are synonymous in the LO. In the United States, Marsick and Watkins (2003) suggested that organizational learning concepts have influenced the conceptual development of LO, which these authors defined as a living organism that uses learning to improve organizational performance. For the purpose of the current study, we adopted the definition of LO by Marsick and Watkins.

The DLOQ Framework of the LO

Organizations have attempted to become LOs to establish or maintain their competitive advantage. This strategy led to the formation and development of validated instruments to assess LO culture. Such diagnostic tools can be important, as organizational change necessitates the diagnosis of the current state of organizational culture at the first stage of the efforts (McLean, 2006). Researchers, scholars, and practitioners have tried to develop an instrument to measure LO culture. The best known of these efforts is the DLOQ. The DLOQ grew out of the LO framework suggested by Watkins and Marsick (1993) in their book, Sculpting the Learning Organization: Lessons in the Art and Science of Systemic Change, and in their later works.

Watkins and Marsick (1993) suggested that the design of LO requires six action imperatives: empowering people toward a collective vision, promoting inquiry and dialogue, encouraging collaboration and team learning, creating continuous learning opportunities, connecting the organization to the environment, and establishing systems to capture and share learning. According to Watkins and Marsick, these action imperatives are expected to occur at four different levels for an organization to have the capacity of continuous learning and change: individual (continuous learning and dialogue and inquiry), team (collaboration and team learning), organization (empowerment and systems), and societal levels (connection to environment).

In addition, Watkins and Marsick (1993) provided seven characteristics (7Cs)—continuous, collaborative, connected, collective, creative, captured and codified, and capacity-building—of an organization that has an enhanced ability to change as a result of the achievement of the six action imperatives. These 7Cs are the framework used to audit an organization’s present capacity while identifying the gap between the current and the desired state for a LO. Therefore, Watkins and Marsick (1993) provided selected items that were developed through integrating their framework of LO and a tool suggested by O’Brien (1992) to operationalize the 7Cs of the LO.

From a systems perspective, the six action imperatives are inputs to evolve into a LO, while the 7Cs are outputs which result from the inputs. According to systems theory, a system has inputs, throughputs, and outputs with continuous feedback loops to inputs and outputs (Burke, 2011; von Bertalanffy, 1968). An organization is viewed as a system and general applied systems theory, at the organization level, has been used to frame the LO concept. Thus, as shown in Table 1, the two concepts share a common ground. In Table 1, the framework for the LO endorsed by Watkins and Marsick’s (1993) model is presented.

Table 1.

The Framework for the Learning Organization.

Four levels of learning	Nature of learning	Six action imperatives	Learning outcomes (7Cs)
Individual	Change in behavior, knowledge, motivation, capacity to learn	Continuous learning opportunities	Continuous learning for continuous improvement
Individual		Inquiry and dialogue	Continuous learning for continuous improvement
Teams	Change in a group’s capacity for collaborative and synergistic work	Collaboration and team learning	Collaborative, connected, collective, creative
Organization	Change in organizational capacity for innovation and new knowledge	Systems to capture and share learning	Connected, captured and codified, capacity–building
Organization		Empowering people	Connected, captured and codified, capacity–building
Society	Change in overall capacity of community and society	Connection to environment	Connected by enhancing community’s capacity–building

Source. Adapted from Watkins and Marsick (1993, p. 263).

As presented in Table 1, at the heart of the framework for the LO are the six action imperatives that construct the six dimensions of the DLOQ other than leadership for learning. Although leadership for learning was originally ruled out from the action imperatives, the underlying roles of leaders were highlighted through the metaphor of “a sculptor” of the LO (Watkins & Marsick, 1993, p. 174). According to Watkins and Marsick (1993), a leader, as a sculptor, plays the pivotal role by empowering people with a vision and engaging in a dialogue that shapes the emerging product. Later, this emphasis on the critical role of leaders in the LO led to the inclusion of leadership into the action imperatives (Phillips, Watkins, & Marsick, 1996).

Leadership for learning was assumed to be an underlying feature of the LO, in that leaders should champion learning “at the individual, team, and organizational levels” (Phillips et al., 1996, p. 7). Marsick and Watkins (1999) reconceptualized their action imperatives based on the new LO framework. They suggested that “several core drivers of the learning organization” are culture, strategy for learning, processes and practices for learning, leadership for learning, and people practices (pp. 208-209). The action imperatives were thought of as the backdrop of the LO framework. They defined action imperatives as the categories of practices that need to change, while defining the drivers as things that people work with to enact the change. Leadership for learning was positioned at the center of the framework.

Development of the DLOQ

Based on their previous work (1993, 1996) and the resulting seven action imperatives, Watkins and Marsick (1997) published the DLOQ which was designed to measure the presumed seven dimensions of LO, including continuous learning, dialogue and inquiry, team learning, empowerment, system to capture learning, system connection, and leadership for learning. While some have suggested the underlying framing of the DLOQ is a theory (Swanson & Chermack, 2013), Watkins and Marsick have consistently indicated that the basis for the DLOQ is actually a model, emphasizing applied general systems theory (von Bertalanffy, 1968), and not a theory in-and-of-itself:

Some organizations systematically seek to capture and embed new learning in a manner that facilitates widespread dissemination of that learning both for current and future employees. Our model of the learning organization, which in turn serves as the basis for the DLOQ, grew out of this conception of organizational learning. It is built on the idea that change must occur at every level of learning—from individual to group to organizational to environmental—and that these changes must become new practices and routines that enable and support the ability to use learning to improve performance. (Marsick & Watkins, 2003, p. 135)

Their Likert-type instrument consisted of 42 items: 6 items for each dimension. Marsick and Watkins (2003) separated 1 item in the continuous learning scale into 2 items, resulting in a 43 item version of the DLOQ. They later presented two shorter forms of the instrument identified by Yang: the DLOQ-A with 21 items (3 items for each scale/dimension) and a 7-item form that reflects a single construct of a LO. In their effort to validate the DLOQ, Yang et al. (2004) recommended, on the basis of confirmatory factor analysis (CFA) results, a 21-item version rather than 43 to use for organizational study. However, an exploratory statistical approach that would have been used to reduce the 43 items into 21 was not exhibited. Instead, a confirmatory approach using “a model generating (MG) method” was taken on the basis of presuming the seven dimensions to obtain the 21-item version of the DLOQ (Yang et al., 2004, p. 39). MG is a method in which a researcher specifies a tentative initial model and modifies it until the model fits to the given data (Jöreskog, 1993). Therefore, some researchers began to question whether the DLOQ measured respondent perspectives regarding the LO dimensions and the extent to which the instrument was parsimonious and efficient.

Validation of the DLOQ in the Literature

Essential to establishing the strength of the DLOQ is its construct validity, which is defined as the extent to which observed variables measure the underlying attributes or latent variables that it is attempting to assess (Nunnally & Bernstein, 1994; Pedhazur & Schmelkin, 1991). Construct validity of an instrument is achieved by ensuring the adequate content validity, criterion-related validity, reliability, and parsimony (Hinkin, 2005) as well as convergent and discriminant validity (Pedhazur & Schmelkin, 1991; Raykov & Marcoulides, 2011). The concepts of convergent and discriminant validity have been defined differently among scholars. For instance, Pedhazur and Schmelkin (1991) adopted Campbell and Fiske’s (1959) seminal notion on the two concepts, viewing convergent validity as a convergence among different methods intended to measure the same construct; discriminant validity was referred to as the distinctiveness of constructs among different methods designed to measure different constructs. Because this viewpoint assumes problems and biases inherent in the use of a single method, an analysis of a multitrait-multimethod (MTMM) matrix is thought to be an appropriate method to study convergent and discriminant validity of measures (Nunnally & Bernstein, 1994; Pedhazur & Schmelkin, 1991). The MTMM matrix would be applicable to examine the validity of the DLOQ only if each dimension (i.e., multiple constructs or traits) of LO is simultaneously measured by both self-scoring DLOQ and other methods such as observation (i.e., multiple methods).

However, Hinkin (2005) and more recently Raykov and Marcoulides (2011) suggested that convergent validity is the extent to which scores on an instrument are related to measures of other constructs expected to be similar to one another; discriminant validity is regarded as the degree to which scores on an instrument do not correlate with other behavioral manifestations of different constructs on the basis of theoretical grounds. This viewpoint assumes that measures of a single method can be analyzed to test convergent and discriminant validity. Accordingly, CFA analyzing “covariance matrix” of measures is employed as one of the methods to examine the two kinds of construct validity (Raykov & Marcoulides, 2011, p. 207). We intend to put a special emphasis on this CFA approach to the convergent and discriminant validity because it would allow researchers to test the construct validity of their DLOQ data that were gathered through a single method (i.e., self-report). Given that the researchers have used the DLOQ as a self-scoring instrument, it is also important to recognize that the CFA approach could be combined with a meta-analysis to examine the validity of the DLOQ by analyzing the inter-factor covariance or correlation matrices (CMs) that were reported in previous literature.

Construct validation approaches

Factor analysis (FA) is one of the statistical methods used to validate an instrument by explaining the underlying structure that describes a set of variables (Mertler & Vannatta, 2010). In FA, observable variables are allocated into a smaller number of components to explore the underlying structure or confirm hypothetical relations among variables providing evidence of construct validity. Consequently, two major types of FAs are commonly used: exploratory and confirmatory (Kline, 2000; Pedhazur & Schmelkin, 1991).

Exploratory factor analysis (EFA) is used to uncover underlying factors in the early stage of developing an instrument that is intended to measure latent variables (Nunnally & Bernstein, 1994; Tabachnick & Fidell, 2007). EFA allows the reduction of a large set of items to a smaller set of items to create a scale with parsimony and simple structure, which are also obtained through retaining only those variables that clearly load onto a single factor. Although researchers interested in establishing the strength of their item sets may choose to skip EFA analysis altogether, EFA is a well-established, standard approach and is strongly recommended for researchers developing a new measure and for establishing instrument validity in new contexts or with new populations (Fabrigar, Wegener, MacCallum, & Strahan, 1999; Hinkin, 2005).

CFA is concerned with parameter estimation and tests of the hypothetical factor structure underlying the relations among variables (Brown, 2006; Pedhazur & Schmelkin, 1991). CFA is regarded as appropriate to use under two circumstances: when evaluating an existing instrument’s accuracy that has been retained from EFA (Brown, 2006; Hinkin, 2005), and when testing a measurement model that is based on “a well-developed underlying theory for hypothesized loading patterns” (Yang, 2005, p. 185). These two circumstances reflect an arguable question regarding the application of CFA: Can CFA be used without preliminary EFA if the scale under examination is based on theoretical grounds? Although the answer to the question is controversial, scholars agree that the combination of the two approaches would provide evidence for a more valid instrument for assessing sound attributes (Brown, 2006; Hinkin, 2005; Kline, 2000; Nunnally & Bernstein, 1994; Yang, 2005). The anticipated increase in validity assumes that the EFA and CFA are not conducted on the same data set. That is, a hold-out sample approach should be taken, where a sample is split into two groups that are used for the EFA and CFA, respectively. These steps allow the researcher to determine the predictive capability of a new model by assessing its capacity to predict the data from the hold-out sample.

DLOQ research and validation

Since Watkins and Marsick (1997) published the DLOQ, a number of researchers have used the instrument in their research. Based on the criteria to select the previous research using the DLOQ, 90 articles, dissertations, or conference papers were identified and reviewed—with particular attention to authors reporting the DLOQ construct validity. For Research Questions 1 and 2, we focused on IVs, mediators/moderators (M/M), DVs, validation methods such as EFA and CFA, CM, and mean of the each dimension (M). A summary of the review of the studies is presented in Table 2.

Table 2.

Use of the DLOQ and Validation in Identified Research.

DLOQ	IV	M/M	DV	NA	Total	%	EFA	CFA	CM	M
Seven dimensions	31	2	7	—	40	44	—	—	—	—
Two or three levels	9	0	0	—	9	10	—	—	—	—
Unitary dimension	17	2	3	—	22	24	—	—	—	—
NA	—	—	—	19	19	21	—	—	—	—
Total	57	4	10	19	90	100	8	27	27	60
%	63	4	11	21	100	—	9	30	30	67

Note. DLOQ = Dimensions of the Learning Organization Questionnaire; IVs = independent variables; M/M = mediator or moderator; DVs = dependent variables; NA = not applicable; EFA = exploratory factor analysis; CFA = confirmatory factor analysis; CM = correlation matrix; M = mean.

For Research Question 3, we used three different terms to signify the dimensionality that researchers of LO implicitly assumed about the DLOQ: Seven dimensions of LO were denoted for identified research that included all seven dimensions in the analysis; two or three levels of LO for the research that aggregated scores on each dimension into either people/structural levels or individual/team/organizational levels, respectively; and unitary LO for the research which either unified all scores into one dimension or used one item for each of the proposed seven dimensions of the DLOQ.

Of the final 90 studies, the DLOQ was used as an instrument to measure IVs in 57 (63%) studies, M/M in 4 (4%), and DVs in 10 (11%). Each of the seven dimensions of the DLOQ was utilized and analyzed in 40 (44%) studies; the scores on the DLOQ were aggregated into people/structural or individual/team/organizational levels in 9 (10%); and the scores on the DLOQ were unified into one dimension or one item reflecting each dimension (i.e., seven items in total) was used to treat organizational learning culture as a unidimensional construct (e.g., Joo, 2010) in 22 (24%). The detailed status of ways in which the DLOQ was used and validated in previous quantitative research is presented in Table 2.

With regard to validation of the DLOQ among identified published research, researchers of 35 studies (39%) tested construct validity of the DLOQ: 8 (9%) using EFA and 27 (30%) using CFA. In addition, the inter-factor CMs of the DLOQ were provided in 27 (30%) studies. The unitary or separate mean scores of the seven dimensions were provided in 60 (67%) studies with means ranging from 2.40 to 5.29 on a 6-point Likert-type scale. In the cases where a mean score was calculated from the measure based on a 5- or 7-point scale, we converted the mean into the value compatible to the 6-point Likert-type scale. Within the literature, the EFA results across the aforementioned studies revealed incoherent factor extraction, while that of CFAs verified adequate model fits to the data with acceptable fit statistics. In Table 3, the results of EFAs in the previous eight studies are exhibited.

Table 3.

EFA Results Reported in Identified Research.

Author	No. of items in DLOQ	Sample size (n)	No. of components extracted	Extraction/rotation methods	Year
Ismail	43	259	8 or 1 depending on the data analyzed	PCA/varimax	2005
Hawkins	21	460	2	PCA/varimax	2005
Dirani	43	298	8	PCA/varimax	2007
Murugiah	43	103	1	Not identified	2008
Hu	21	184	4	PCA/varimax	2008
Benjamin	21	194	1	PCA/varimax	2009
Martinez	43	284	1	Not identified/varimax	2009
Abbas et al.	43	44	6 including performance variables	PCA/not identified	2011

Note. EFA = exploratory factor analysis; DLOQ = Dimensions of the Learning Organization Questionnaire; PCA = principal components analysis.

Discussion

Current Status of the DLOQ Instrumentation and Validity

In quantitative research, research quality depends largely on the validity of the instrument that is used to measure constructs of interest (Nunnally & Bernstein, 1994; Pedhazur & Schmelkin, 1991). According to Nunnally and Bernstein (1994) and Hinkin (2005), psychometrically sound instruments are developed through a set of processes, whereby content validity, criterion-related validity, internal consistency, and parsimony are attained. Although it is possible that a shorter process of developing an instrument could result in a sound measure, utilization of the whole process recommended above is likely to produce a more valid measurement scale and/or determine the validity of extant measurement instruments (Hinkin, 2005; Kline, 2000; Nunnally & Bernstein, 1994; Yang, 2005).

However, our extensive review of the history of the DLOQ development revealed that the steps commonly recommended in the development of sound psychometric measures were not followed. As Watkins and Marsick (1993) published the book Sculpting the Learning Organization: Lessons in the Art And Science of Systemic Change, a number of studies using the DLOQ were initiated with no EFA having been utilized at the early stage of the DLOQ development. According to the recommendations to develop psychometrically sound measurement scales, EFA should be used prior to CFA to validate an instrument—as use of CFA alone, in instruments such as the DLOQ, may incorrectly affirm segmentation of survey responses (Kline, 2000; Nunnally & Bernstein, 1994). Use of EFA is especially important during the first and early stages of instrument development and validation. However, the use of EFA in DLOQ-related studies was not published until 2005—the results of which reported incoherent numbers of factors (see Table 3). Throughout the period from 1993 to 2012, we found only eight studies that conducted EFA to validate the DLOQ. The multicollinearity and constrained discriminant validity of the DLOQ might have been perpetuated through failure to utilize and/or report related EFA, which does not support measures involving high correlation among the underlying dimensions. In regard to this issue, Yang et al. (2004) suggested that developing a valid instrument was an ongoing process, and a more parsimonious assessment of the DLOQ should be conducted. In Table 4, the issues surrounding the history of the DLOQ development and validation are presented.

Table 4.

History of the DLOQ Development and Validation.

Year	Events and issues
1993	• Watkins and Marsick published the book Sculpting the Learning Organization: Lessons in the Art and Science of Systemic Change.
	• A foundational model of the LO that served as the basis for the DLOQ was presented in the book.
	• The six action imperatives in the book did not include leadership for learning.
1996	• Phillips, Watkins, and Marsick edited a case study book of American Society for Training and Development, In Action: Creating the Learning Organization.
1996	• The book suggested a framework for the LO with seven action imperatives, including leadership for learning.
1997	• Watkins and Marsick published the survey instrument, Dimensions of the Learning Organization Questionnaire that proposed the original 42-item version, including the dimension of leadership for learning.
	• Watkins, Yang, and Marsick published an article in the proceedings of the Academy of Human Resource Development (AHRD) conference, entitled “Measuring dimensions of the learning organization.”
	• One published study, but no EFA/CFA.
1998	• Yang, Watkins, and Marsick published an article in the proceedings of the AHRD conference, titled “Examining construct validity of the Dimensions of the Learning Organization Questionnaire.”
	• The article used CFA and included the dimension of leadership for learning.
	• Two studies, including one reporting CFA (the first to report factor analysis of any kind)
1999	• Marsick and Watkins published the book, Facilitating Learning Organizations: Making Learning Count.
	• The book distinguished the six action imperatives from leadership for learning that was viewed as the key driver of the action imperatives.
	• One study, but no EFA/CFA
2000	• One study, but no EFA/CFA
2002	• Two studies including one CFA
2003	• Marsick and Watkins published the conceptual paper, “Demonstrating the value of an organization’s learning culture: The dimensions of learning organization questionnaire,” proposing the 43-, 21-, and 7-item version of the DLOQ.
	• The dimension of continuous learning in the original 42-item version was elaborated to seven items “because it was determined that one item measured two critical concepts,” resulting in 43 items in total (Yang, Watkins, & Marsick, 2004, p. 36).
	• 10 studies, including 3 CFAs
2004	• Yang, Watkins, and Marsick used CFA to determine that the reduced 21-item version is a superior measurement model than the original 42-item model.
2004	• 6 studies, including 3 CFAs
2005	• 4 studies, including 2 EFAs and 1 CFA (the first validation study using EFA)
2005	• In the EFAs, one, two, or eight components were extracted depending on the methods and samples.
2006	• 5 studies, including 1 CFA
2007	• 9 studies, including 1 EFA and 3 CFAs
2007	• In the EFA, eight components were extracted.
2008	• 12 studies, including 2 EFAs and 4 CFAs
2008	• In the EFAs, one or four components were extracted, respectively.
2009	• 11 studies, including 2 EFAs and 4 CFAs
2009	• In the EFAs, one component was extracted.
2010	• 11 studies, including 3 CFAs
2011	• 10 studies including 1 EFA and 2 CFAs
2011	• In the EFA, six components, including the two factors of knowledge and financial performance, were extracted.
2012	• 5 studies, including 1 CFA

Note. DLOQ = Dimensions of the Learning Organization Questionnaire; LO = learning organization; EFA = exploratory factor analysis; CFA = confirmatory factor analysis.

One of the most controversial issues regarding EFA and CFA involved determination regarding when researchers should use either type of FA to validate an instrument. Yang (2005) suggested that “EFA is used to explore underlying factors when there is little or no prior theoretical guideline for a domain of interest, and CFA is used to confirm or disconfirm a hypothesized factor structure of interest” (p. 194). However, Hinkin (2005) argued that “the number of factors to be retained depends on both underlying theory and quantitative results” (p. 170). If researchers agree with Yang (as quoted above), they would not conduct EFA for the DLOQ because the DLOQ could be viewed as coming from a sound theoretical foundation (Marsick & Watkins, 1999; Phillips et al., 1996; Watkins & Marsick, 1993). If researchers lean toward Hinkin’s perspective (above), they would prefer the use of EFA over CFA at the early stage of validating the DLOQ because the complex nature of a seven-dimension item set should be established before stratifying the validation procedure using CFA. Given that many researchers have found benefit in the use of both approaches, especially during early validation of new tests/measures, it seems logical that the utilization of both EFA and CFA would produce more accurate measurement (Hinkin & Tracey, 1999; Kumar & Beyerlein, 1991; MacKenzie, Podsakoff, & Fetter, 1991). This may be especially true for the DLOQ as the underlying assumptions may be better described as a framework or “model,” rather than a theory (Marsick & Watkins, 2003, p. 135).

Another issue regarding the DLOQ is related to the shortening of the original instrument. In 2003, Marsick and Watkins provided the DLOQ-A version, a shorter form of the instrument where the number of items was reduced from the initial 42 to 21 items. Researchers assumed that the items of the DLOQ had been reduced through an extensive series of both EFA and CFA (e.g., Ellinger, Ellinger, Yang, & Howton, 2002). However, as shown in Tables 3 and 4, we could not locate any studies that conducted EFA to justify the shorter version of the DLOQ; we found that only one available CFA has been provided (with MG method) in the literature (Yang et al., 2004). Most commonly, because EFA allows researchers to eliminate irrelevant or redundant items for the construct of interest and thus result in a parsimonious measurement, instruments are validated and established using EFA (Hinkin, 2005; Kline, 2000; Mertler & Vannatta, 2010; Nunnally & Bernstein, 1994).

Correspondence of the DLOQ to the LO Framework

A more profound question about the DLOQ would be whether it reflects appropriately the authors’ framework associated with the LO. Watkins and Marsick (1993) suggested six action imperatives that are needed for an organization to evolve into a LO; leadership for learning was not included in the six action imperatives. Rather, leadership for learning was assumed as an underlying characteristic of the LO, which supports overall learning culture in the organization. In other words, leadership for learning is an overarching construct that may include the other six dimensions of the DLOQ. Although Phillips et al. (1996) placed leadership for learning in juxtaposition with the remaining six action imperatives, Marsick and Watkins (1999) again distinguished the former from the latter; leadership for learning was seen as the key driver of the remaining six action imperatives. That is, leaders promote continuous learning opportunities, dialogue and inquiry, empowerment, and team learning (Marsick & Watkins, 1999; Watkins & Marsick, 1993).

However, leadership for learning was included in the DLOQ simply as one of the seven dimensions, which appears to undermine “the learning organization framework” proposed by Marsick and Watkins (1999, p. 208). As was found in the four studies using EFA in which one component was extracted from the measures of the DLOQ, it appears that the DLOQ measures respondents’ unidimensional perceptions of their organizational learning culture. The debate concerning this issue is fueled by Marsick and Watkins (2003) who suggested that the seven-item form of the DLOQ “together creates a single scale of learning culture” (p. 146). Then, we face a critical question: Is the LO unidimensional or multidimensional? Given the multidimensionality stemming from the LO frameworks or theories that are espoused by major LO scholars and researchers (Moilanen, 2005; Örtenblad, 2013; Pedler, Burgoyne, & Boydell, 1991; Preskill & Torres, 1999; Senge, 1990; Watkins & Marsick, 1993), it seems prudent that LO should be a multidimensional construct. However, the current three forms of the DLOQ do not completely support multidimensionality (see Table 3) nor correspond to the theoretical framework of the LO, leading to concerns regarding multicollinearity and lack of discriminant validity. The evidence presented herein supports the DLOQ as a unidimensional measurment tool. Based on the number of items used, the DLOQ, is, at best, an inefficient measure of respondents’ perception of their organization’s general orientation to and support for learning.

Implications and Recommendations

There are several implications that emerge from this examination of the DLOQ. First, the review and critique of the DLOQ is not aimed at, nor does it fully discredit the findings of earlier studies or the overall framing of LO based on the work of Watkins and Marsick (and, later, Yang). Instead, based on available information from our systematic review of literature, we have raised key questions regarding the methods used to formulate and deploy the DLOQ and the extent to which the dimensionality (seven dimensions) of LO is measured using this instrument. Given the high reliability coefficients reported for the DLOQ in identified studies, it is clear that the DLOQ is consistently measuring respondents’ perceptions—but, likely, unidimensionally, rather than multidimensionally, considering the reported multicollinearity, lack of discriminant validity, and incongruity of the DLOQ to the LO framework by Marsick and Watkins (1999). Although our review and analysis is cause for considerable concern regarding the framing and validation of the DLOQ, several opportunities emerge from this exploration that can not only benefit the field but can also extend our understanding of the ways in which LO, and employees related perceptions, has and may be measured.

Based on our review, we have concluded that several next steps are warranted. These include the following recommendations for related research, theory, and practice.

Research

Five recommendations regarding research include the following. First, we suggest that researchers conduct a sequential EFA/CFA procedure to illustrate how a different set of items and factor structure might be obtained in lieu of a CFA MG procedure. As discussed earlier, EFA would be useful to explore the dimensionality of the LO concept measured by the DLOQ. Given the previous DLOQ research that showed mixed factor structures (see Table 3), this recommendation is especially challenging. CFA could be conducted with a hold-out sample to confirm the factor structure that results from the previous EFA. Although they lack more accurate information, such as a factor loading table, two recent studies (Dirani, 2013; Watkins & Dirani, 2013) followed the recommended procedure to validate the DLOQ. However, considering the limitation that, over the last two decades, the DLOQ had never been validated using EFA until 2013 (see Table 4), we believe that more empirical studies aiming at exploring the dimensionality of the DLOQ data should be accumulated, including in a diversity of international contexts. To reduce sampling error that might influence the EFA/CFA procedure, a careful sample design and large sample size should be ensured. Second, we encourage researchers to examine discriminant validity of the DLOQ, using meta-analysis of the R matrix and chi-square difference tests. A meta-analysis of the R matrix could be used to identify if the inter-factor correlation coefficients of the meta-analyzed matrix were .75 or above, which means that the dimensions are not distinct from one another (Nimon, Zigarmi, Houson, Witt, & Diehl, 2011). Meta-analysis would allow researchers to obtain a more generalizable finding of the relationship among the seven dimensions, because the method enables them to integrate the results across the studies that provided R matrix of the seven dimensions. Chi-square difference test estimates the χ² of model fitting both when the correlation coefficient of two dimensions is fixed as 1 and when the estimation is free (Hu, 2008). This method is recommended because researchers can investigate by observing the significance level of Δχ² whether or not the correlation coefficient of two dimensions in the meta-analyzed R matrix of the DLOQ is significantly different from 1 and thus the two dimensions are different from one another. Third, it might be interesting to see possible results stemming from a latent semantic analysis to explore the differences in meaning embedded in each item of the DLOQ. Latent semantic analysis could be utilized on the items as a means to examine manifest validity of the DLOQ by comparing the items in a dimension with those in another to identify the degree to which the seven dimensions have the same meaning to respondents (Nimon, Shuck, & Zigarmi, 2014). Fourth, should the investigation fail to yield a dimensional self-report assessment of LO, researchers should consider using the seven-item form of the DLOQ as an alternate and parsimonious approach to assessing LO-related perceptions. Upon rigorous examination, should the DLOQ be determined to not reflect seven dimensions as originally framed, researchers should answer the question “exactly what has the DLOQ, to date, been measuring?” Finally, it is recommended to use the DLOQ, or modified version, as an instrument to measure the DV (only 11% of previous studies used DLOQ as the DV, and mean scores were often low). Such examination may help researchers better understand factors associated with the increase or decrease in employees perceptions of their organization’s learning culture or LO-related features (e.g., Mahseredjian et al., 2011; Yahaya et al., 2011).

Theory

Two approaches to further theorizing related to the DLOQ can be described to build new theory or to adopt or adapt extant theory. As suggested in theory building literature and HRD practice literature (i.e., Theory Cycle; Swanson, 1997), ideally, research and theory are part of a virtuous cycle or exchange in which both are refined based on the LO framework and accumulated empirical-analytic evidence. According to Dubin (1976) and Lynham (2002), theory building in applied disciplines, such as HRD, undergoes ongoing refinement and development through the four phases of conceptual development, operationalization, confirmation or disconfirmation, and application. A core outcome of the conceptual development phase is a theoretical framework, which needs to be refined again through rigorous research, practice, and application in the real world. In this sense, several EFA results of research (see Table 3) invite researchers to theorize regarding the relationship among the seven action imperatives in Watkins and Marsick’s LO framework. For example, isn’t a leader one who empowers employees to take initiative for their learning? Following further examination, researchers and theorists may benefit from additional exploration and examination of the theoretical foundations underlying the DLOQ. Given the overt assumptions within Watkins and Marsick’s framework, multilevel theory building (Klein, Tosi, & Cannella, 1999; Upton & Egan, 2010) may be the most appropriate approach through which to further theorize about the LO. Multilevel theory development may better situate the interwoven assumptions associated with extant LO constructs and conceptualizations. Based on the research-to-practice-to-theory cycle, emphasized across the physical and social sciences and HRD, we should use the findings from the current study to advance LO-related theory as part of an ongoing, iterative process (Dubin, 1976).

Yet another angle regarding theorizing about employee attitudes and reactions related to the DLOQ, or employee perceptions of LO, is to explore what undergirds employees’ capacities to assess or react to LO-related factors. How might we come to better frame the underlying meaning of existing DLOQ results? A fruitful, post hoc, line of examination may be to consider current theories that could inform the historical DLOQ responses and results. For instance, employees simply may not be capable of estimating the differences between the intended dimensions as reflected in the extant survey. Instead, they may be simply reacting based on their general perceptions of their organization’s support for learning. This may explain why their responses across survey dimensions are consistently similar. Therefore, it may be worthwhile to examine specified aspects of organization support theory (Eisenberger, Huntington, Hutchison, & Sowa, 1986), or theories related to employees’ global perceptions about their organizational environment and support for learning. A possible theoretical alternative could combine current approaches to assessing organizational culture or environment with learning-related aspects.

Practice

Several scholars have differentiated research from evaluation (Mertens, 2010). What appears clear to us is that the DLOQ’s seven LO dimensions frame essential elements of an organization that can be systematically assessed and used to refine and extend an organization’s capacities as a LO. Although it appears clear that the DLOQ is measuring employee general reactions regarding learning in their organization, it is important that employee feedback be combined with other forms of assessment. Mixed-method evaluative and research approaches hold the potential for better understanding of employees’ individual reactions to the organizational learning environment, as well as detailed functional practices and outcomes related to the DLOQ framework.

In addition, there is some emerging evidence that the aforementioned emphasis by Watkins and Marsick on managerial support for learning and leader as “sculptor” of learning culture/subculture and practices within organizations is central to the LO concept (Egan, 2008). As learning-related results are the focus of most HRD practitioners, emphasizing support for LO as part of leadership development should be the central emphasis in managerial and supervisory training and development. Our early findings suggest if HRD practitioners use the DLOQ and the original seven dimensions as an evaluative tool—used by professional evaluators (or even internal HRD professionals) as an assessment tool to examine extant organizational practices, along with a refined employee self-report questionnaire—there will be many benefits in terms of better improving as well as understanding an organization’s learning-related infrastructure and culture.

Conclusion

As the strategic importance of learning in organizations increases, HRD professionals require a credible measurement instrument in their toolbox for research and evaluation. Unfortunately, based on the findings in the current study, we conclude that researchers have not been successful in providing rigorous empirical evidence to verify the seven dimensions of LO measured by the DLOQ. Thus, we invite researchers to conduct further DLOQ research to explore this dimensionality issue of the LO concept. Should further research confirm the DLOQ to be measuring a single construct or dimension, this would not invalidate previous studies entirely. Rather, the DLOQ would have to be reframed as a measure of respondents’ general perceptions, not regarding the specific aspects of their organization’s approach to learning. Instead of assessing the impact of the seven dimensions, the DLOQ may be measuring respondents’ general (or global) perceptions regarding organizational support for learning.

The credibility of inferences made by a quantitative research depends on the nature and quality of the accumulated evidence involving the construct under consideration (Pedhazur & Schmelkin, 1991). Before using the DLOQ, researchers need to pay attention to standards for psychometric research practices. Social science requires theory–research–practice exchange, a back and forth process of theorizing and exploration of support and empirical evidence. As Yang et al. (2004) suggested, researchers should start their investigation with the recognition that constructing a valid instrument is an ongoing process. We hope that this examination contributes to such a process and that HRD will benefit from further examination and consideration of the ways in which the concept of LO is framed, assessed, and measured.

Footnotes

Appendix

The link to the DLOQ research list (90 in total) is https://docs.google.com/open?id=0Bz3DYytftnMad0w5QlFJLXF2bUk

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Junhee Kim is a PhD candidate in the Educational Human Resource Development Program at Texas A&M University. His research interest revolves around leadership role and competencies, which are connected to other critical research topics such as learning organization, knowledge management, learning transfer, competency development, educational technology, mentoring, and scale development.

Toby Egan, PhD, is associate professor and senior fellow at the University of Maryland School of Public Policy and the Robert H. Smith School of Business. He is an experienced organization development practitioner and executive coach with over 20 years of experience in multinational, Fortune 500, government and nonprofit settings. Dr.Egan is an award winning researcher and advisor to doctoral students. His research explores organization learning and development, leadership, coaching/mentoring, and creativity/innovation.

Homer Tolson, PhD, is a senior professor in the Educational Human Resource Development program in the department of Educational Administration and Human Resource Development at Texas A&M University. His research interests are measurement and research design.

References

Abbas

R. Z.

Murad

H. S.

Ghaffari

A. G.

Siddiqi

A. F.

Ahmad

ur Rehman

Ashraf

(2011). Measuring the learning organization’s construct in Pakistan: A case study of public sector educational institutes. European Journal of Social Sciences, 18, 574-581.

Benjamin

(2009). The school as learning organization: Validation of the DLOQ with school staff (Unpublished doctoral dissertation). Alfred University, NY.

Brown

T. A.

(2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.

Burke

W. W.

(2011). Organization change: Theory and practice (3rd ed.). Thousand Oaks, CA: SAGE.

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Cho

Egan

T. M.

(2009). Action learning research: A systematic review and conceptual framework. Human Resource Development Review, 8, 431-462.

Dirani

K. M.

(2007). The relationship among learning organization culture, job satisfaction, and organizational commitment in the Lebanese banking sector and the effect of social patterns as moderator variables (Unpublished doctoral dissertation). University of Illinois at Urbana–Champaign.

Dirani

K. M.

(2013). Does theory travel? Dimensions of the learning organization culture relevant to the Lebanese culture. Advances in Developing Human Resources, 15, 177-192.

Dubin

(1976). Theory building in applied areas. In Dunnette

M. D.

(Ed.), Handbook of industrial and organizational psychology (pp. 17-39). Chicago, IL: Rand-McNally.

10.

Egan

T. M.

(2002). Learning organization dimensions and motivation to transfer learning in large firm information technology employees (Unpublished doctoral dissertation). University of Minnesota, Minneapolis, MN.

11.

Egan

T. M.

(2008). The relevance of organizational subculture for motivation to transfer learning. Human Resource Development Quarterly, 19, 299-322.

12.

Egan

T. M.

Yang

Bartlett

K. R.

(2004). The effects of organizational learning culture and job satisfaction on motivation to transfer learning and turnover intention. Human Resource Development Quarterly, 15, 279-301.

13.

Eisenberger

Huntington

Hutchison

Sowa

(1986). Perceived organizational support. Journal of Applied Psychology, 71, 500-507.

14.

Ellinger

A. D.

Ellinger

A. E.

Yang

Howton

S. W.

(2002). The relationship between the learning organization concept and firms’ financial performance: An empirical assessment. Human Resource Development Quarterly, 13, 5-21.

15.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299.

16.

Garvin

D. A.

(1993). Building a learning organization. Harvard Business Review, 71(4), 78-91.

17.

Hawkins

A. S.

(2005). The relationship of perceived learning culture to perceived organizational support and affective commitment (Unpublished doctoral dissertation). Alliant International University, San Diego, CA.

18.

Hernandez

Watkins

K. E.

(2003). Translation, validation, and adaptation of the Spanish version of the modified dimensions of the learning organization questionnaire. Human Resource Development International, 6, 187-196.

19.

Hinkin

T. R.

(2005). Scale development principles and practices. In Swanson

R. A.

Holton III

E. F.

(Eds.), Research in organizations: Foundations and method of inquiry (pp. 161-179). San Francisco, CA: Berrett-Koehler.

20.

Hinkin

T. R.

Tracey

J. B.

(1999). An analysis of variance approach to content validation. Organizational Research Methods, 2, 175-186.

21.

Y. C.

(2008). Dimension developing and structure testing of nonprofit learning organizations. In Zhang

Zhu

Han

(Eds.), Conference Proceedings of 2008 International Institute of Applied Statistics Studies: Recent Advance in Statistics Application and Related Areas, PTS 1 and 2 (pp. 1317-1322). Sydney, AU: Aussino Academic Publishing House.

22.

Ismail

(2005). Creative climate and learning organization factors: Their contribution towards innovation. Leadership & Organization Development Journal, 26, 639-654.

23.

Joo

B. K.

(2010). Organizational commitment for knowledge workers: The roles of perceived organizational learning culture, leader–member exchange quality, and turnover intention. Human Resource Development Quarterly, 21, 69-85.

24.

Jöreskog

K. G.

(1993). Testing structural equation models. In Bollen

K. A.

Long

J. S.

(Eds.), Testing structural equation models (pp. 294-316). Thousand Oaks, CA: SAGE.

25.

Klein

K. J.

Tosi

Cannella

A. A.

Jr. (1999). Multilevel theory building: Benefits, barriers, and new developments. Academy of Management Review, 24, 243-248.

26.

Kline

(2000). Handbook of psychological testing (2nd ed.). London, England: Routledge.

27.

Kofman

Senge

(1993). Communities of learning: The heart of learning organization. Organization Dynamics, 22, 5-23.

28.

Kumar

Beyerlein

(1991). Construction and validation of an instrument for measuring ingratiatory behaviors in organizational settings. Journal of Applied Psychology, 76, 619-627.

29.

Lynham

S. A.

(2002). The general method of theory building research in applied disciplines. Advances in Developing Human Resources, 4, 221-241.

30.

MacKenzie

S. B.

Podsakoff

P. M.

Fetter

(1991). Organizational citizenship behavior and objective productivity as determinants of managerial evaluations of salespersons’ performance. Organizational Behavior and Human Decision Processes, 50, 123-150.

31.

Mahseredjian

Karkoulian

Messarra

(2011). Leadership styles correlate of learning organization in a non-western culture. The Business Review, Cambridge, 17, 269-277.

32.

Marquardt

Berger

Loan

(2004). HRD in the age of globalization: A practical guide to workplace learning in the third millennium. New York, NY: Basic Books.

33.

Marsick

V. J.

(2013). The Dimensions of a Learning Organization Questionnaire (DLOQ): Introduction to the special issue examining DLOQ use over a decade. Advances in Developing Human Resources, 15, 127-132.

34.

Marsick

V. J.

Watkins

K. E.

(1999). Facilitating learning organizations: Making learning count. Brookfield, VT: Gower.

35.

Marsick

V. J.

Watkins

K. E.

(2003). Demonstrating the value of an organization’s learning culture: The dimensions of Learning Organizations Questionnaire. Advances in Developing Human Resources, 5, 132-151.

36.

Martinez

R. L.

(2009). Impact of the learning organization on financial performance (Unpublished doctoral dissertation). Capella University, Minneapolis, MN.

37.

McLean

G. N.

(2006). Organization development: Principles, processes, performance. San Francisco, CA: Berrett-Koehler.

38.

Mertens

D. M.

(2010). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods (3rd ed.). Thousand Oaks, CA: SAGE.

39.

Mertler

C. A.

Vannatta

R. A.

(2010). Advanced and multivariate statistical methods (4th ed.). Glendale, CA: Pyrczak.

40.

Moilanen

(2005). Diagnosing and measuring learning organizations. The Learning Organization, 12, 71-89.

41.

Murugiah

S. S.

(2008). Facilitating organizational learning to enhance capacity to manage emerging business challenges: Case study of safe finance in Malaysia (Unpublished doctoral dissertation). Columbia University, New York, NY.

42.

Nimon

Shuck

Zigarmi

(2014, February). Latent semantic analysis: Examining manifest validity of employee engagement measures. Paper presented at the 2014 Academy of Human Resource Development Conference, Houston, TX, USA. Retrieved from http://assets.conferencespot.org/fileserver/file/110980/filename/0329-000024.pdf

43.

Nimon

Zigarmi

Houson

Witt

Diehl

(2011). The work cognition inventory: Initial evidence of construct validity. Human Resource Development Quarterly, 22, 7-35.

44.

Nunnally

Bernstein

(1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.

45.

O’Brien

M. J.

(1992). The learning organization profile, version B. Milford, OH: O’Brien Learning Systems.

46.

Örtenblad

(2002). A typology of the idea of learning organization. Management learning, 33, 213-230.

47.

Örtenblad

(Ed.). (2013). Handbook of research on the learning organization: Adaptation and context. Cheltenham, UK: Edward Elgar.

48.

Pedhazur

E. J.

Schmelkin

L. P.

(1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Lawrence Erlbaum.

49.

Pedler

Boydell

Burgoyne

(1989). Towards the learning company. Management Education and Development, 20, 1-8.

50.

Pedler

Burgoyne

Boydell

(1991). The learning company. London, England: McGraw-Hill.

51.

Phillips

J. J.

(Series Ed.), Watkins

K. E.

Marsick

V. J.

(Eds.). (1996). In action, creating the learning organization (Vol. 1). Alexandria, VA: ASTD.

52.

Preskill

Torres

R. T.

(1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: SAGE.

53.

Raykov

Marcoulides

G. A.

(2011). Introduction to psychometric theory. New York, NY: Routledge.

54.

Senge

P. M.

(1990). The fifth discipline: The art and practice of the learning organization. New York, NY: Doubleday.

55.

Song

J. H.

Chermack

T. J.

Kim

(2013). An analysis and synthesis of DLOQ-based learning organization research. Advances in Developing Human Resources, 15, 222-239.

56.

Strickland

O. L.

(2001). An instrument’s conceptual base: Its link to theory. Journal of Nursing Measurement, 9, 3-4.

57.

Swanson

R. A.

(1997). HRD research: Don’t go to work without it! In Swanson

R. A.

Holton III

E. F.

(Eds.), Human resource development research handbook (pp. 3-20). San Francisco, CA: Berrett-Koehler.

58.

Swanson

R. A.

Chermack

(2013). Theory building in applied disciplines. San Francisco, CA: Berrett-Koehler.

59.

Tabachnick

B. G.

Fidell

L. S.

(2007). Using multivariate statistics (5th ed.). Boston, MA: Pearson Education.

60.

Torraco

R. J.

(2005). Writing integrative literature reviews. Human Resource Development Review, 4, 356-367.

61.

Upton

Egan

(2010). Three approaches to multilevel theory building. Human Resource Development Review, 9, 333-356.

62.

von Bertalanffy

K. L.

(1968). General systems theory: Foundations, development, applications. New York, NY: George Braziller.

63.

Wang

(2005). Relationships among organizational learning culture, job satisfaction, and organizational commitment in Chinese state-owned and privately owned enterprises (Unpublished doctoral dissertation). University of Minnesota, Minneapolis.

64.

Wang

Yang

McLean

G. N.

(2007). Influence of demographic factors and ownership type upon organizational learning culture in Chinese enterprises. International Journal of Training and Development, 11, 154-165.

65.

Watkins

K. E.

Dirani

K. M.

(2013). A meta-analysis of the dimensions of a Learning Organization Questionnaire looking across cultures, ranks, and industries. Advances in Developing Human Resources, 15, 148-162.

66.

Watkins

K. E.

Marsick

V. J.

(1993). Sculpting the learning organization: Lessons in the art and science of systemic change. San Francisco, CA: Jossey-Bass.

67.

Watkins

K. E.

Marsick

V. J.

(1997). Dimensions of the Learning Organization Questionnaire [Measurement instrument]. Warwick, RI: Partners for the Learning Organization. Retrieved from http://www.partnersforlearning.com/questions2.asp

68.

Watkins

K. E.

Yang

Marsick

V. J.

(1997). Measuring dimensions of the learning organization. In Torraco

R. J.

(Ed.), 1997 Conference Proceedings of the Academy of Human Resource Development (pp. 543-546). Baton Rouge, LA: The AHRD.

69.

Werner

J. M.

DeSimone

R. L.

(2009). Human resource development (5th ed.). Mason, OH: Thomson South-Western.

70.

Yahaya

Boon

Ramli

Baharudin

N. A.

Yahaya

Ismail

Shariff

(2011). The implications of Organizational Citizenship Behavior (OCB) towards the dimensions of Learning Organization (LO) in organizations in Southern Malaysia. African Journal of Business Management, 5, 5724-5737.

71.

Yang

(2005). Factor analysis methods. In Swanson

R. A.

Holton III

E. F.

(Eds.), Research in organizations: Foundations and method of inquiry (pp. 181-199). San Francisco, CA: Berrett-Koehler.

72.

Yang

Watkins

K. E.

Marsick

V. J.

(1998). Examining construct validity of the Dimensions of the Learning Organization Questionnaire. In Torraco

R. J.

(Ed.), 1998 Conference Proceedings of the Academy of Human Resource Development (pp. 83-90). Baton Rouge, LA: AHRD.

73.

Yang

Watkins

K. E.

Marsick

V. J.

(2004). The construct of the learning organization: Dimensions, measurement, and validation. Human Resource Development Quarterly, 15, 31-55.