Narrating in grey: An application to educational management information systems and accountability

Problem statement

In Chile, while the government has shown a growing interest to increase the Higher Education quality, the educational system as a whole still has a low level of accountability, thereby affecting the public confidence toward Higher Education Institutions (HEIs) (Dussaillant and Guzmán, 2014; Riquelme Silva et al., 2018; Rojas Ríos and López Stefoni, 2016; Fleet et al., 2014; López et al., 2015).

On the other hand, an extensive ICT use nationwide, mainly in urban areas (Stager Koller et al., 2017; Stäger Koller and Núñez Tissinetti, 2015), an increasing ICT use in Chilean higher education (Brun and Hinostroza, 2014; González Bravo and Valdivia Peralta, 2015) and a growing market of Higher Education ERPs in Latin-America (Cassidy, 2006; Abdellatif, 2014), are observed. The latter is probably due to these Educational Management Information Systems (EMIS) providing information in challenging environments for quality purposes, both within HEIs, and in governments and quality agencies (Haris et al., 2017).

The starting problem addressed in this research is the discrepancy between a growing EMIS use and a still low accountability level in Chilean HEIs. On the other hand, a growing number of scholars are recognizing the value of narrative reviews and ‘grey literature’, for generating new knowledge beyond peer-reviewed journals, incorporating practice and case reports, among many other sources (Bae, 2014; Paez, 2017; Paré et al., 2015; Rumrill and Fitzgerald, 2001)

Considering the cases using the already mentioned approach (Benzies et al., 2006), a mixed narrative review (NR) is carried out in order to demonstrate the value of narrative reviews, which include grey literature, in emerging higher education issues.

Narrative reviews

Narrative literature reviews or narrative reviews (NR) are focused to identify, analyze, assess and interpret the existing knowledge on a specific topic, depending on the final product to which it is aspired (Guirao Goris, 2015). They tend to focus on interpretation, following a path which could include multiple open-ended meanings (Ayala, 2018). In this sense, it is valuable when the researcher is interested in connecting many sources either for purposes of deepening understanding of a broad or complex issue (Thorne, 2018) or reinterpretation/interconnection pursuing theory building (Baumeister and Leary, 1997). It is possible to identify quantitative narrative reviews, qualitative narrative reviews, and mixed methods reviews, being the latter used when both quantitative and qualitative research are involved (Bae, 2014).

NR critics have argued that this is the simplest type of literature review, a subjective first approach (Rumrill and Fitzgerald, 2001), which does not seek generalization or cumulative knowledge, without systematic and comprehensive literature searching, detailed explanations and methods regarding the review process (Paré et al., 2015; Rumrill and Fitzgerald, 2001). It has even been pointed out that NR is less comprehensive, conducted by unidentified experts, without an explicit statement of methods and being in the last place of the hierarchy of scientific methods (Aguilera Eguía, 2014).

Answering these critics, several authors have exposed methods to improve the quality of a NR: detail databases and keywords used, dates and search strategy, how the terms used were combined, the number of sources found (Guirao Goris, 2015) or final report sections clearly identified (Ferrari, 2015; Rumrill and Fitzgerald, 2001). Even have been suggested guidelines for evaluating the quality of certain reviews in terms of rigor and relevance (Paré et al., 2015) or recommendations to improve the searching and extracting process, or the final report (Vom Brocke et al., 2009; Bandara et al., 2011)

In a deeper view, some critics towards NR come from traditional quantitative perspectives, which underestimates the value of practice, reflection or social inquiry (Ng et al., 2015). The problem is that in fact, science moves on when it is capable of generating original and creative hypotheses from multiple directions, some of them germinal, and not necessarily coming from studies published in peer-reviewed journals: for example, a health-related clinical case, or a specific school-practice (Thorne, 2018). In fact, reflection could contribute to the development of educational practice at a high level, providing a critical awareness about the theories, applications of epistemological positions (Ng et al., 2015) or adding dimensions of insight/application perspectives, and providing critical analyses (Rumrill and Fitzgerald, 2001). In conclusion, some NR offer much deeper or “richer” information than a classical empirical analysis or meta-analysis (Rumrill and Fitzgerald, 2001)

Narrative reviews and grey literature

The NRs include in their scope a wide variety of sources, it thus becoming necessary to analyze the concept of grey literature and its value for scientific research.

According to the ‘Luxembourg Definition’, grey literature refers to print or electronic literature produced by government, academia, business and industry, and not controlled by commercial publishers. This can include materials such as unpublished studies, conference abstracts, conference proceedings, book chapters, government and agency reports, unpublished doctoral dissertations (Gokhale, 1998; Godin et al., 2015) or PowerPoint presentations, evaluation reports, standards/best practice documents, guidelines, working papers (Benzies et al., 2006).

The inclusion of grey literature has a heuristic value for NR, expanding the scope, adding contextual information (Benzies et al., 2006; Mahood et al., 2014; Adams et al., 2016; Paez, 2017) and in fact is recommended when the phenomenon is complex with multiple components (Benzies et al., 2006). Combining grey literature along with peer-reviewed sources may therefore provide a more balanced view of the evidence (Mahood et al., 2014), reducing the impact of publication bias (Adams et al., 2016; Gracia Leiva et al., 2019; Bellefontaine and Lee, 2014). It is because of this reason, that scholars are increasingly recognizing the value of incorporating ‘grey literature’ (Adams et al., 2017)

Quality management in higher education

Chile was the first Latin American country that initiated quality assurance practices in higher education as a state policy in 1990, through the Higher Education Council (Inga Ipanaque and Velásquez Silva, 2005). In their early years, the procedures were focused over a compulsory licensing of new private HEIs. In 2003, accreditation of institutions started, and in 2006 the National System of Quality Assurance was created by law (Lemaitre et al., 2011). Institutional accreditation remained in the hands of the newly founded CNA (National Accreditation Commission), being a requirement for students who wished to access the Credit with State Guarantee (CAE) to pursue undergraduate studies in a HEI, and making accreditation an imperative for institutions to operate – survive – in the system (Fleet et al., 2014).Through the years, the concept of quality eventually changed from an abstract standard to an HEIs’ response to a set of externally established benchmarks in terms of legitimacy and information about the purpose and outcomes. As a result, citizens can make informed decisions considering these factors in programs or institutions (Kinser, 2014).

Applied to higher education, a quality assurance (QA) view has been defined as “set of mechanisms and processes aimed at controlling, guaranteeing and promoting the quality of higher education institutions” (CNA-Chile, 2015: 4). Gradually, the QA concept has been replaced by quality management (QM), which emphasizes continuous development and improvement, rather than just responding to external certifications (Cabrera Lanzo, 2018; Lemaitre et al., 2012). In HEIs, without quality information, it is impossible to manage quality (Beard and Humphrey, 2014; Mora et al., 2009) and to advance from QA to QM involves not only control over a deregulated system, but also to regain academic promotion of quality (Zapata and Torre, 2012a).

Managers and political authorities are rethinking higher education models, and adjust them to meet the environmental requirements. It must be guaranteed the existence of a sustainable and well-funded framework to support the HEIs’ efforts. It is for this reason that in many countries, quality and continuous improvement, accountability and qualification frameworks in the higher education sector are at the forefront of national political agendas (Pucciarelli and Kaplan, 2016).

Accountability

Accountability is a very widespread concept and a term that has many meanings: from the efficient functioning of institutions and efficient learning in students, to the system’s ability to stimulate the economy (Speziale, 2012). In the last three decades, this concept has been developed in the US and subsequently in the world, from the prevailing concept in the 1970s – design of government structures that adequately address the problem of autonomy versus the supervision of institutions – towards a vision that originated in the 1990s, focused on systemic results of HEIs (McLendon et al., 2006). Accountability and governance are necessary to preserve the pact between higher education and society; they are central issues for developing countries, and governments must be sure that HEIs are addressing the needs of society in terms of qualified graduates, and research to improve the quality of life of the citizens (Hendel and Lewis, 2005). There are multiple forces that promote accountability, from the governments that promote a scarce allocation of resources based on the results of the institutions (Kai, 2009), to the professional authority that promotes monitoring and continual improvement of processes and outcomes, for example through the use of management information systems (Shavelson, 2010).

Accountability, expressed through relevant public information and transparency, are key factors for citizenship when choosing a HEI and it also enhances the ability of citizen control over the state or the Higher Education market (United Nations Development Programme, 2006; Altbach, 2009). While Chile must build its achievements on the basis of an institutional framework in which the formal role of accountability is sufficiently valued, the country is not fully prepared for higher education quality systems based on trust and social capital (Dussaillant and Guzmán, 2014). This is due to an extremely deregulated educational market since 1981 (during the dictatorship of Augusto Pinochet), and it has taken twenty-five years to change to a more comprehensive regulation in a cultural context of an open market, an advanced use of technology, but where distrust in institutions persists.

Equity, quality and effectiveness and therefore accountability have been promoted as indicators of system success in Latin America (Castro et al., 2017), and the first way to provide quality information to achieve transparency, accountability and success of an HEI, is to have adequate management information systems (Rivera et al., 2009; Altbach, 2009).

EMIS in higher education

HEIs are purchasing or developing information systems to manage academic, non-academic and administrative processes. This is largely because their use makes services cheaper, easier, faster and more accurate (Gunawardhana and Perera, 2015), and the implementation of cutting-edge technology helps streamline HEIs and transform their processes to achieve outstanding results (Dlamini, 2015). These EMISs must provide information to directors that allows them to make key decisions for the organization, in the right direction considering environment and internal processes (Bubel et al., 2015; Martins et al., 2019). This information facilitates management at different levels (Toro, 2012): inputs for governance, academic records, human resources, physical resources inventory, financial systems. Most of the efficient performance of a HEI depends on having updated retention and dropout data of students, academic progression and enrolment, graduates employability rates and salaries, courses satisfaction reports, curriculum plans validity and teaching loads, among many other indicators. This implies an effective management of a large amount of information, and providing it in user friendly way, which often involves the use of ERP systems for several processes (Bubel et al., 2015) and even the daily work of all members of the organization (Stensaker, 2018).

The strategical use of data mining plays an important role in the sustainable growth, success, and quality assurance of universities (Goyal and Vohra, 2012), and has become relevant to assess and improve HEI quality. Through implementing strategic planning and effective/efficient allocation of resources, HEIs can be more effective and have more opportunities to make right investments/improvements, which accelerate their sustainable growth, success, and reputation (Al-Sabaawi, 2015; Sart et al., 2015, November; Zahid and Khan, 2016).

The use of information is an intangible resource, which ends up being related to organizational culture or management styles, that allows a university to develop in a competitive environment (Bubel et al., 2015). As Weiss says, in the context of the results of her research about leadership and the use of new technologies in HEIs “organizational quality factors…[are]…defined and a correlation determined between an organization’s organizational quality score and overall satisfaction with the CIO and IT department” (Weiss, 2010: 169).

However, as early as 1998, Abby Rubin (1998) indicated that in the case of developing countries, MISs have been implemented in the educational field, but that “…implementation and use of accountability measures, however, has been varied, not always living up to the ideal of what could be expected in terms of making the connection with quality improvement” (Abby Rubin, 1998: 287). Today, this situation has not changed substantively (Rojas Ríos and López Stefoni, 2016), despite the fact that educational organizations seek certain objectives in a transcendent way, objectives that mobilize them in different areas, for example, in the ICT incorporation for educational management, expecting a quality certification (Harrison et al., 2014)

Higher education institutions face pressures from the government and society as a whole (Bernasconi, 2012, Fall; Pedraja-Rejas and Rodríguez-Ponce, 2015), to demonstrate quality and effectiveness, which translates into processes of self-assessment and quality assurance (Garnett and Ecclesfield, 2008). These external demands can revolve around the concept of public value: the ethos and values of any public organization, service provider or profession must be assessed according to their creation of value, understood as better outcomes, services and trust. This allows the building of trust from the environment towards these organizations, making them more adaptive and stimulating collaborative relationships with society to the benefit of their students and the entire educational system (Garnett and Ecclesfield, 2008).

To recognize the importance of educational management information systems is relevant, considering first, the changing and challenging historical breakpoint in HE in terms of quality and accountability, and secondly, the significance that this issue has for the countries to design Higher Education public policies which considers the broader environment where institutions are inserted (Opazo et al., 2019). In Chile, the value of provision of information, transparency and accountability is particularly notable, given the advanced ICT use but distrust toward institutions.

Design and data collection

We conducted a narrative literature review (Bae, 2014; Baumeister and Leary, 1997; Cronin et al., 2008; Ferrari, 2015; Green et al., 2006; Rumrill and Fitzgerald, 2001) mainly using:

Databases EBSCO, Proquest and Redalyc.org from 1990 to 2018. 1990 was the year that Chile returned to democracy and the Constitutional Act on Education was issued (Ley Orgánica Constitucional de Enseñanza – LOCE).

The selection of relevant publications until December, 2018, was made based in four themes, operationalized in four keywords or key themes (Bearman and Dawson, 2013): Quality management/quality assurance (QM/QA), Chilean Higher Education, ICT Use and Information Systems. Sources that explicitly established a connection between the mentioned terms were selected. An exclusion criterion for the selection of relevant literature was not including articles exclusively focused on new technologies serving the teaching/learning process.

This process allowed the selection of 39 publications. Additionally, they were classified into two main categories: empirical and non-empirical. This general classification was utilized to allow the inclusion of public documents and books, not published in peer-reviewed journals.

Among the publications reviewed, eight were empirical, which used samples ranging between 10 accredited (Lemaitre et al., 2012) and 57 HEIs (López et al., 2015).

Data analysis

A narrative review was conducted, where different sources were analyzed according to key themes. The steps were the following:

Each source was read following the six questions suggested by Porritt et al. (2014), to determine relevancy and then again to collect information relevant to the research problem (Porritt et al., 2014). These six questions were related with a) time period covered b) language (English and Spanish) c) population/sample, d) phenomena stated in the main aim, e) study design and f) outcome measured (p. 48).

Quality management and ICT Use

In 2003, González reported that there were no systematic data regarding the computer resources implementation in Chilean HEIs, although “it could be said that all universities have computer resources for teachers and students…[and that]…a significant number has Intranet and technology classrooms that include access to teleconferencing” (González, 2003: 23). While there is now broad recognition of the value of EMIS for education, this development started towards the end of the 1960s and was closely linked to technological developments of the 1990s in the computing area, but had not yet reached its current expression at the beginning of the present century. In fact, as indicated by Marterer (2008), in 2002 it was possible to identify only two empirical studies in large universities worldwide about effects of ERP systems in higher education (United States and England).

Today, on the other hand, there is awareness in Chilean universities, particularly those that belong to the “Council of Rectors” (i.e., universities created before 1981), that ICTs and “Business Intelligence Mechanisms” are a fundamental part of their work, and the ongoing growth and complexity of HEIs, and the society’s needs and demands (Mora et al., 2009), have created new needs to be met from the management perspective (Fuentes Tapia and Valdivia Pinto, 2010; Mora et al., 2009; Busco et al., 2018), and that, complementary, information management is key in institutional accreditation (Fleet et al., 2014; Lemaitre et al., 2012).

As Lemaitre et al. (2012) pointed out, while improving the information management processes emerged as a response to external demands from accreditation processes, information had taken a leading role in quality management. This growing need has been answered from HEIs through incorporation of several technological tools, including ERPs (Enterprise Resource Planning) for academic management (Zapata and Torre, 2012a), being already a trend reported in other countries around the world.

Today, many Chilean universities utilize integrated ERPs for internal management (Zapata and Fleet, 2012): in most cases, these systems incorporate ‘academic management modules’ where managers can follow up on the students’ academic progress or to what extent professors and researchers are carrying out their professional activities. The institution’s complexity and maturity, will influence what use it will make of that information following the classification of intelligences listed by Terenzini (1993): Technical/Analytical, Issues or contextual.

Regardless of these specific differences, there has been growing recognition of EMIS value in the last decade, so that authors such as Bernasconi (2012, Fall) affirm that “not only in Chile, but also in Latin America, tools such as planning, business intelligence, market surveys and performance indicators have been strongly incorporated” (p. 13). It is very important that HEIs incorporate good information management practices that allow them to use quantitative information in a context of strategies that provide them with both meaning and purpose. This notion is directly related to the concept of e-maturity applied to the educational field – degree of provision, administration, and use of technology to support learning in the curriculum (Underwood, 2009) –, as a broader perspective (Bagozzi, 2007; Harrison et al., 2014; Nistor, 2014) that gives long term meaning to ICT use, as a way to achieve its objectives transcendently (for example, social or systemic determinants, quality or external certification; Harrison et al., 2014).

Quality management, ICT use and accountability

In the area of public visibility, Zapata and Fleet (2012) carried out an analysis of the web pages of 56 Chilean universities, and found that the institutions disseminate their information in a partial way, emphasizing aspects of academic offer rather than issues related to accountability (for example, qualification, financial aspects, etc.). In addition, they found that public and traditional private HEIs disseminate more information than new private universities. Something similar occurs when comparing accredited universities versus those that are not, being the former those that provide more information. The authors show how the quality of information is largely explanatory of institutional quality.

As an additional component, in Chile as in the whole world, since the 1990s the Institutional Analysis Offices (IAOs) have given a strong boost to the incorporation of EMIS. In Chile, they received an important boost with the quality assurance policies promoted by the state in the 2000s, following with some delay, the trends originated in United States, and followed by Europe and Asia (Opazo et al., 2019). It is important to mention that they are an important point of contact between universities and their environment, in two senses: IAOs provide official information to external bodies that require it, and provide the organization with information about the environment in which it operates (Rivera et al., 2009). Rivera et al. (2009) carried out a study in which they found that among 61 Chilean universities, 48 have a division in charge of information processing, support for decision-making and informing the environment (for example, state agencies). To the extent that authors recognize that the work of these units is based on collection, analysis and systematization of information, it is a problem not always to have online and updated managing data. Something similar happened until 2011, with national information systems: in that year the higher education public information started to be disseminated through websites such as Mifuturo.cl (Ferreyra et al., 2017), an initiative that provides information about degrees, enrollment values, length of careers, graduation rate, employability, among others (Espinoza and González, 2013). This is a clear example in Chile of how governments can use information and communication technologies to improve their systems, contributing to social development (Qureshi, 2009).

In any case, the problem regarding the significant and strategic use of information remains, and this is worrying as far this information finally feeds back institution quality in a process of continuous improvement. In the same line, it describes how the investment on technology should be guided by a project to lead the process along. In simpler words, HEIs should know what they are using technology for. Fukushi (2010) raises questions that organize socio-cognitive frames and place the organization regarding its technological reality. According to the author, the few Chilean HEIs using EMISs systematically do so motivated by the ‘know-what’ or ‘know-how’, but very few reach the higher stages, where they must ask about systemic understanding of growth or the “motivated creativity or ethics” (Fukushi, 2010: 14) with a strong social component in terms of social welfare. This is fully connected with theories like technological frames (Orlikowski and Gash, 1994) and with concepts expressed by Bagozzi (2007), in the sense that often ICT adoption models do not respond clearly to the question about the final purpose of the technology incorporation (Bagozzi, 2007: 245) which could even be social welfare in general (Bagozzi, 2007: 251)

This is also consistent with the findings of Aguayo et al. (2009), who reported that quality of strategic plans, management of material resources and information systems were among the most noticeable weaknesses of Chilean HEIs. Likewise, they indicate that a key factor that hinders optimal management control in the HEIs is the absence of an integrated management system (67%). That is consistent with managers’ interest in planning, developing information systems for such purpose (Cáceres, 2007), and with Solar et al. (2013), about the value of e-maturity in the field of strategic planning, delivery of information required by stakeholders and by law, among many other areas.

In short, although there is a greater recognition about the value of EMISs in Chilean HEIs, three types of problems are observed:

1) An insufficient number of universities have implemented adequate systems of institutional information management, with the subsequent low ICT use for management (Espinoza and González, 2013; Rojas Ríos and López Stefoni, 2016). For example, López et al. (2015) conducted a study with 48 accredited institutions until 2011, and found that when reviewing the results for accreditation, one of the difficulties experienced by HEIs is the limitation of information systems for decision-making, mainly in the planning area (López et al., 2015; Cáceres, 2007). It is interesting that the mentioned article by López et al. (2015), mainly aimed at evaluating the effects of accreditation in Chilean universities, ends up analyzing how these effects are connected with the society as a whole: as a way to balance the deregulated privatization of higher education.

This situation is due in part to the current paucity of national literature that explores this field and the diverse degrees of e-maturity among institutions. In those institutions that are just beginning a systematic incorporation of EMIS, it is sought to develop integrated systems of information to take decisions based on evidence. In more mature HEIs, a better management of indicators is carried out, which translates into concrete actions, for example, improvement of academic indicators (Lemaitre et al., 2012), being this consistent with the three tiers of institutional research mentioned by Terenzini (1993).

3) Complementing the aforementioned, long-term planning problems persist for many Chilean HEIs, and the ability of these long-term plans to guide an ICT use policy for management (Cáceres, 2007). In this line, Zapata and Torre (2012a) mention a study of 23 universities in seven countries showing greater progress in developing QA systems: Mexico, Costa Rica, Colombia, Chile, Argentina, Portugal and Spain. The authors reported that one positive result of QA processes, was the incorporation of more information for internal control processes and decision making (Zapata and Torre, 2012b), which implies the use of better information systems. The impact of this gradual transformation and valuation seems to be greater in Costa Rica, Colombia and Chile, and slightly less in Mexico, Portugal and Spain.(Zapata and Torre, 2012a). Even though these systems serve primarily the institutional and academic management, it is the governing authorities who perceive more clearly their benefits. Academic and professional associations, conversely, question their validity, reliability and usefulness (Zapata and Torre, 2012b).