Abstract
As the Internet makes secondary data increasingly more accessible and less costly to obtain, the need for methods to evaluate the reliability and validity of such data has grown accordingly. In efforts to harness the surge of data at the country level, researchers have created indexes to more efficiently track the performance of societies on a variety of dimensions. Large-scale global indexes like the Sustainable Society Index (SSI), published biannually since 2006 by the Sustainable Society Foundation, have the potential to provide valuable insights into important issues related to macromarketing, sustainability, and quality of life. The primary purpose of this article is to assess the value of a societal-level index for macromarketing research. By evaluating the reliability, internal validity and external validity of a societal-level index, such as the SSI, researchers will be better informed about the importance of such an index, but will also understand how the protocol used in this study can be applied to other indexes. In order to gauge the external validity of the SSI, the study also evaluates and utilizes three other established secondary data sources in conjunction with the SSI. These include: (1) the Legatum Prosperity Index; (2) Transparency International’s Corruption Perceptions Index; and (3) the Euromoney Country Risk Index.
Keywords
Introduction
The era of Big Data is dawning with vast amounts of information becoming available through the Internet on individuals, firms, NGOs, and governments (Davenport 2014). One study estimated that the world’s societies used 2.8 zettabytes of data in 2012, which is a staggering 2.8 trillion gigabytes (Gantz and Reinsel 2012). This annual amount of data-in-use is forecast to double every two years until 2020. However, less than half of one percent of this data-in-use is actually analyzed in any way.
With such a rapid expansion of the digital universe, it is not surprising that methods to convert this data into useful insights for researchers have lagged far behind advances in information technology. In short, a trust gap exists for many sources of secondary data. Researchers do not have sufficient methods developed to adequately evaluate data they can access.
While the study of secondary data sources and the indexes that can be created from this data is well established (see Fisher 1938; Gibbon 1971; Mullen 1989), recently researchers have taken a more concerted interest in these data sources. For example, Houston (2004, p. 161) argued that rigorous investigations of the construct validity of a secondary data proxy is feasible and that the resultant indicators can be used “to provide important initial or corroborating evidence to test theories.” Busse (2010) found that secondary databases are suitable for confirmatory research, and the use of a single database is appropriate and effective for both hypothesis testing and hypothesis generation. Nicholson and Bennett (2009, p. 423) cite various articles that “implore greater use of secondary data to augment the collection and analysis of primary data.” However, in their research they were surprised to find very few dissertations that actually made use of secondary data (Nicholson and Bennett 2009).
To date, little research has validated the secondary databases and indexes that have come into existence in recent years. Historically, researchers have treated secondary data with suspicion of lacking construct validity. However, macromarketing researchers have effectively evaluated and used secondary data in country-level analyses. For example, Peterson and Malhotra (1997) used International Living’s Quality of Life Index to perform structural equation modeling that disclosed how societies offer benefits and costs to citizens living in them. Hill and Dhanda (2004) used the Technology Achievement Index advanced by the United Nations Development Program and found evidence of a dramatic difference between developed and developing countries, with the least developed nations facing acute deficits in technological advancement. Given the societal-level focus of macromarketing researchers and the country-level focus of new indexes, macromarketers are well positioned to make valuable use of indexes using countries as the unit of analysis. What is now needed is the development of innovative methodologies to assess the validity, reliability, and usefulness of large-scale secondary data sources.
A likely hindrance to the wider adoption of indexes by macromarketers and other researchers is the lack of proven methodological protocols for rigorously evaluating the validity and reliability of these secondary-data sources. Before the rapid expansion of the internet, Malhotra, Peterson, and Kleiser (1998) presented an approach for examining and assessing the validity and reliability of a multiple-measure, longitudinal secondary data source. The purpose of the present study is to apply the approach of Malhtora, Peterson and Kleiser and see how it performs with 21st century data related to the sustainability of societies. The study will go beyond what has been done previously to evaluate multiple indexes that might provide a nomological network for the focal sustainability index.
To gauge the reliability and internal validity of such a societal-level index, the study will first concentrate on an important secondary-data source related to the sustainability of societies—the Sustainability Society Index (SSI). To better understand the generalizability of the protocol employed to assess indexes, the study next applies the protocol to another index developed that focuses on quality of life (the Legatum Prosperity Index). To put into perspective what the SSI represents, additional analyses, including cluster analyses and factor analyses, will be performed on the SSI and the Legatum Prosperity Index, as well as on societal-level indexes focused on corruption (Transparency International’s Corruption Perception Index) and country risk (the Euromoney Country Risk index). By applying the proposed methodological protocol to these indexes, researchers are offered a more complete view of what the SSI represents and how the protocol performs with societal-level indexes with foci on 1) quality of life, 2) corruption, and 3) country risk. By evaluating four societal-level indexes, this study offers researchers a richer understanding of how societal-level indexes widely used today compare with each other in terms of outputs for segmenting countries into meaningful groups. Additionally, by including the corruption perception index, the investigation goes beyond the two types of societal information (quality-of-life, and country risk) used by Malhotra, Peterson and Kleiser. A final purpose of this research is to extend the efforts of Malhotra and colleagues by addressing issues of sustainability, corruption, as well as advances in technology, the Internet, and other resources not available at the time of their work.
Background
Macromarketing as a sub-discipline of marketing is dedicated to examining the complex interplay among markets, marketing, and society (see Hunt 1981). Macromarketing scholars view marketing as a provisioning agent that facilitates exchange and benefits governments, businesses, societies, and individuals throughout the world (Fisk 1981). Macromarketing scholars also assert that marketing is particularly positioned to help improve the human condition (Layton 2009; Shultz 2007). Macromarketing, with its all-encompassing view of marketing, is therefore particularly well suited to develop innovative methodologies to assess the validity and utility of large scale secondary data sources like global indexes. These indexes provide macromarketers with the ability to attempt to understand secondary data on a worldwide scale with the intent to explain marketing phenomena while simultaneously promoting sustainability and improving quality of life.
Secondary data is an important source of information for marketing scholarship. It has several advantages over primary data. Secondary data is: (1) more readily available than primary data, (2) less expense to procure, (3) less time intensive, and (4) highly relevant. However, secondary data is only useful when it is proved to be reliable and valid (Malhotra, Peterson and Kleiser 1998).
Secondary data contains four categories of potential error in that could lead to negative consequences for macromarketers, practitioners, and policy makers. These include: (1) sampling and non-sampling errors, (2) errors that invalidate the data, (3) errors that require data reformulation, and (4) errors that reduce reliability (Rabianski, 2006). For example, sampling error could arise if the population under consideration is stratified but the sample represents only one or some of the strata (Tasic and Feruh 2012). Secondary data might be tainted because of actions or attitudes of the person(s) or the orientation of the organization collecting the data (Iacobucci and Churchill 2009). Data might reflect manipulation, contamination caused by inappropriateness, confusion or carelessness, or concept error (Tasic and Feruh 2012). These factors could potentially invalidate the data. Also, secondary data is sometimes not directly useful to the analyst because it does not adequately measure the concept being studied. These errors can result from: (1) changing circumstances, (2) inappropriate transformations, (3) inappropriate temporal extrapolations, and (4) inappropriate temporal recognition (Patzer 1995). Finally, the reliability of data is a function of the group that gathers, organizes, records, and publishes the secondary data (Tasic and Feruh 2012). Any errors (e.g. clerical, changes in collection procedures, and failure to use correct data) on the part of any of the organizations or individuals that collect the data can call into question the reliability and validity of secondary data sources.
Considering the potential negative consequences of using secondary data, macromarketing is uniquely positioned to examine these “big” data sources. Traditionally, data is big when it is “too much” for conventional systems to handle (Schultz 2014). Big data has typically been studied from the micro level (e.g. individual transactions) by companies like Google, Amazon, and Walmart. However, bigness is not just about size. Data may be big because there is too much of it (volume), because it is moving too fast (velocity), or because it is not structured in a usable way (variety) (Gobble 2013). Big data can also be “thick data,” data that has several layers to it (Schultz 2014). Big data is therefore multidimensional, and most advertising and marketing communication researchers simply do not know how to deal with it.
Big data is also “long and broad” (Schultz 2014). Most of the useful knowledge derived from big data comes from longitudinal data gathered and analyzed over time. Unless researchers look at data over time, they will find it hard to understand “when it started, where it has been, and where it is going” (Schultz 2014, p. 20). Marketing scholarship examines very few clear-cut relationships where researchers can statistically conclude that A leads to B and B leads to C, and so on, yet most of methodological and statistical tools are linear in nature (Schultz 2014). Large secondary data sources like global indexes are readily accessible, highly relevant, and (albeit unique) sources of big data. They are thick, multidimensional, long, and broad. Developing methodologies to establish the reliability and validity of these data sources is an important macromarketing endeavor.
Four Societal Level Indexes
To better understand what the SSI represents, this study will include an analysis of the SSI, as well as three other societal-level indexes: 1) the Corruption Perception Index (CPI), 2) the Legatum Prosperity Index (LPI), and 3) the Euromoney Country Risk (ECR) rankings. These three are important societal-level indexes whose sponsoring organizations have collectively been engaged in gathering, analyzing, and disseminating data and research for nearly 60 years. The findings from these institutions and indexes have served as the basis for numerous works and publications across a variety of disciplines and endeavors over the years (see Asiri and Hubail 2014; Kaivo-oja et al. 2014; O’Leary 2008; Saha et al. 2012). A basic understanding of these indexes is important in the context of this study. A description of each of these indexes serves to highlight the differences as well as the communalities shared by these databases. It also serves as a foundation from which we can evaluate the reliability and validity of the SSI. A short summary of these indexes follows below.
The Sustainability Foundation and the Sustainable Society Index (SSI)
The Sustainable Society Foundation is a non-profit organization established in 2006 as a private initiative by Geurt van de Kerk and Arthur Manuel with the objective of “stimulating and assisting societies in their development towards a more sustainable existence” (Kerk and Manuel 2013). The Foundation has developed the Sustainable Society Index which integrates constructs relating to human, environmental, and economic well-being into a framework that accounts for 99% of the world population and graphically illustrates the level of sustainability of 151 countries throughout the world (Kerk and Manuel 2013). The Foundation updates, publishes, and disseminate the results of the SSI every two years.
Sustainability and sustainability research are topics of increasing import in macromarketing (e.g. Meng 2015; Mittelstaedt et al. 2014; Viswanathan et al. 2014). The SSI is a global index particularly well suited to measure and study sustainability on a worldwide scale. It is one of the very few indexes to include all three wellbeing dimensions: human, environmental, and economic. The first two dimensions are important because they are goals to be achieved: full sustainability for human and environmental wellbeing. One cannot exist or progress without the other. On the other hand, economic wellbeing is not a goal. It is necessary to enable progress on the way towards sustainability. All three dimensions of wellbeing complement and complete each other in examining issues of sustainability. Validating this index as reliable and valid, therefore, is a beneficial exercise in macromarketing scholarship.
The SSI is comprised of four different levels including three overall well-being dimensions (human, environmental, and economic), and one overall sustainability rating. These three well-being dimensions are further subdivided into twenty-one indicators across eight categories. The levels and categories encompass a wide range of social, physical, environmental, and economic indicators and concerns. Input for the SSI comes from a wide range of global indexes and data sources (see Table 1).
The Sustainable Society Index Indicators and Data Sources.
Source: Kerk and Manuel (2013).
Public data sources are used to compute the SSI, and every effort is taken to use the most recent data available (Kerk and Manuel 2013). However, most data is from 2010 or earlier, and accordingly, the results of the SSI are continually lagging behind the actual global situation. The impact, for instance, of the worldwide financial and economic crisis will not become visible in the available data sources until the 2014 SSI (Kerk and Manuel 2013).
Additionally, the reliability of data from which the SSI is derived remains a concern (Kerk and Manuel 2013). It is assumed that the information and data collected from these sites and indexes are reliable and valid. However, this is certainly an optimistic evaluation of the realities and difficulties involved in collecting data for producing time series analyses. The problem appears to be decreasing over time as the importance of sound statistical data is generally recognized by government, industry, and academia alike (Moore, McCabe, and Craig 2012; Rice 2006).
In 2012, the Joint Research Centre of the European Commission (JRC) audited the Sustainable Society Index, and assessed the SSI with respect to (1) the conceptual coherence of the structure of SSI; (2) the statistical coherence of the structure of SSI; and (3) the impact of key modeling assumptions on the SSI scores and ranks over 2006-2012 (Kerk and Manuel 2013). The JRC presented a number of recommendations for the further improvement of the SSI, and all JRC’s recommendations were implemented in the 2012 SSI (Kerk and Manuel 2013). In addition, the rankings and ratings from 2006, 2008, and 2010 were updated to reflect the changes suggested by the JRC (Saisana and Philippas 2012). Appendix 1 presents the conclusions of the audit.
The Legatum Institute and the Legatum Prosperity Index (LPI)
Based in London, the Legatum Institute is an independent non-partisan public policy organization founded in 2007 whose research, publications, and programs advance ideas and policies in support of free and prosperous societies around the world (Legatum Institute 2014). The Institute undertakes original and collaborative research and publishes case studies and supplementary literature. Recent initiatives include the launch of the Legatum Prosperity Index. The 2013 Legatum Prosperity Index is based on 89 different variables across 142 nations throughout the world (Legatum Institute 2014). Source data come from variety of widely utilized international databases including (1) Gallup World Poll, (2) World Development Indicators, (3) International Telecommuncation Union, (4) Failed States Index, (5) Worldwide Governance Indicators, (6) Freedom House, (7) World Health Organization, (8) World Values Survey, (9) Amnesty International, and (10) Centre for Systemic Peace (Prosperity.com 2014).
The variables are grouped into eight sub-indexes, which are averaged using equal weights. They are (1) economy, (2) entrepreneurship and opportunity, (3) governance, (4) education, (5) health, (6) safety and security, (7) personal freedom, and (8) social capital. The Legatum Institute operates a transparent approach to its work on the Prosperity Index. In fact, the entire methodology of the Prosperity Index along, with the data used to create it, is available for free online (Prosperity.com 2014). The LPI is a model index with which to compare and contrast the SSI.
Transparency International and the Corruption Perceptions Index
Transparency International was founded in 1993 by Peter Eigen, a former regional director of the World Bank, with the mission to “stop corruption and promote transparency, accountability and integrity at all levels and across all sectors of society” (Transparency International 2014). Transparency International consists of over 100 locally established, independent national chapters as well as an international secretariat in Berlin, Germany. Each chapter addresses corruption in their respective country. These individual chapters also construct methods for measuring and reporting corruption that are relevant to their national context in order to bring about change. The secretariat provides support and cooperation among chapters, as well as collaborating with these chapters in order to address corruption on both a national and global scale (Transparency International 2014).
In 1995, Transparency International developed the Corruption Perceptions Index (CPI). The Corruption Perceptions Index ranks countries and territories based on how corrupt their public sector is perceived to be. It is a “composite index – a combination of polls – drawing on corruption-related data collected by a variety of reputable institutions” (Transparency International 2014). Additionally, the CPI reflects the views of observers from around the world. Data sources for the 2014 CPI include (1) African Development Bank Governance Ratings 2012, (2) Bertelsmann Foundation Sustainability Governance Indicators 2014, (3) Bertelsmann Foundation Transformation Index 2014, (4) Economist Intelligence Unit Country Risk Ratings, (5) Freedom House Nations in Transit 2013, (6) Global Insights Country Risk Ratings, (7) IMD World Competitiveness Yearbook 2013, (8) Political and Economic Risk Consultancy Asian Intelligence 2013, (9) Political Risk Services International Country Risk Guide, (10) Transparency International Bribe Payers Survey 2011, (11) World Bank Country Policy and Institutional Assessment 2012, (12) World Economic Forum Executive Opinion Survey (EOS) 2013, and (13) World Justice Project Rule of Law Index 2013 (Transparency International 2014)
The Corruption Perceptions Index has received some criticism over the years. The main issue stems from the complexity in accurately measuring corruption, which by definition happens behind the scenes (Cuervo-Cazurra 2008). The Corruption Perceptions Index therefore, needs to rely on third-party survey data that have been disparaged as potentially unreliable. However, as previously discussed, this issue has been of less import in recent years (Moore, McCabe, and Craig 2012; Rice 2006).
The second criticism is that data cannot be compared from year to year because Transparency International uses different methodologies and samples every year. Critics argue that this makes it difficult to evaluate the result of new policies as well as make longitudinal comparisons. However, supporters of Transparency International like Eric M. Uslaner (2008) argue that instruments like the Corruption Perceptions Index are meant to measure the perception of corruption and not necessarily the reality of the situation. He further contends that “perceptions matter in their own right, since… firms and individuals take actions based on perceptions” (Uslaner 2008, p. 13). In sum, the CPI is a useful and valid international data source. It is an index akin to the SSI and the LPI and as such, is well suited to compare and contrast with the SSI.
Euromoney Country Risk Rankings (ECR)
Euromoney was first published in 1969 by Sir Patrick Sergeant. It is part of Euromoney Institutional Investor, an international business-to-business media group focused primarily on the global finance industry. A public company since 1986, the group is listed on the London Stock Exchange as Euromoney Institutional Investor PLC (ERM). Euromoney Country Risk allows users to access live data from Euromoney’s country risk survey. Published semi-annually since 1993, the survey uses economists’ assessments of political, economic and structural risk to provide country rankings for 186 markets worldwide (Euromoney Country Risk 2014). Data includes bank stability, monetary policy/currency stability, corruption and institutional risk. The ECR scores are scaled from 0 to 100 (100 = no risk, 0 = maximum risk), and the scores are not designed to correspond to any other rating systems (Euromoney Country Risk 2014). ECR divides its countries into 5 tiers based on (1) economic characteristics, (2) political characteristics, (3) structural characteristics, (4) access to capital and credit ratings, and (5) debt indicators. Appendix 2 contains a description of how countries in those tiers are characterized.
Research Questions
The SSI has the potential to provide valuable insights into important issues related to marketing and sustainable business practices. Comparing the SSI with other indexes is a worthwhile undertaking, because such comparisons will lead to knowledge about how these indexes relate to each other. Currently, indexes are proliferating, but researchers do not know how an index devoted to sustainability, such as the SSI, performs in terms of its own reliability and validity. Additionally, researchers do not know how a sustainability index is related to other societal-level indexes. With such knowledge, researchers will be more confident in their use of these indexes. Researchers will also be more confident in methods developed before the rapid growth of the internet and how such methods can be extended to distinct dimensions of societies, such as corruption perceptions or sustainability.
Assessing the validity and reliability of the SSI through the use of established theoretical concepts and statistical analyses, is a valuable endeavor that both informs researchers’ understanding and adds much to the credibility of international secondary data sources. This study will proceed as follows. First, the SSI will be evaluated using a specific methodological protocol. This protocol includes an examination of the means, standard deviations, pairwise correlations, coefficients of variation, and cluster analyses. Second, the LPI will be subjected to the same methodology and a summary of this process will be presented. These two steps will help establish the construct validity of the two databases. Third, the Corruption Perceptions Index will be added and the three indexes and will be subjected to factor analyses to judge their similarities. This step will help to establish the content validity of these indexes. Finally, an exploratory factor analysis including a fourth index (the Euromoney Country Risk rankings) will be performed. This step will help demonstrate the criterion validity of these indexes. Based on the discussion above, this study’s primary research questions are:
Method
Assessing the Reliability and Internal Validity of the SSI
For this study, data has been compiled and analyzed from the 2006, 2008, 2010, and 2012 SSI’s. As previously mentioned, this study will employ the same two phase, six-step process employed by Malhotra, Peterson, and Kleiser (1998) in their examination and assessment of the validity and reliability over time of a similar multiple-measure, longitudinal secondary data source (see Table 2 and 2a). The cleaning of data from the SSI focused on identifying “inconsistent periods of data and observational outliers which deviate across periods of time” (Malhotra, Peterson, and Kleiser 1998, p. 192). Further measures were taken to assess reliability, validity, and convergent validity of the SSI. Additionally, both univariate and multivariate statistical techniques (e.g. cluster, stem-and-leaf plots, and factor analyses) were used to identify potentially “inconsistent cross-sections of data, inconsistent measures, and multivariate outliers” (Malhotra, Peterson, and Kleiser 1998, p. 195).
Phase 1: Qualitative Assessment.
Source: Malhotra, Peterson, and Kleiser (1998).
Phase 2: Quantitative Assessment-Reliability and Validity.
Source: Malhotra, Peterson, and Kleiser (1998).
Phase I
The SSI data were analyzed in two-phases beginning with a qualitative assessment. Joselyn (1977) developed a “go/no go” framework by which to assess the quality of secondary data. While this framework was developed for domestic marketing research using secondary data, it is also useful in evaluating secondary data gathered from large governmental and non-governmental entities such as is contained in the SSI (Malhotra, Peterson, and Kleiser 1998). As represented in Figure 4.3 of Joselyn’s text (p. 54), the evaluation procedure can be seen in Table 3.
Joselyn’s Evaluation Procedure.
Source: Joselyn 1977.
This evaluation procedure establishes the SSI to be credible and of significant worth to academics, practitioners, and policy makers alike. The dataset is extremely comprehensive and a number of research issues can be addressed by the SSI. The myriad research issues and questions that can be addressed by this single source combined with its great depth and breadth, lend further support to the quality of the SSI (Malhotra, Peterson, and Kleiser 1998).
The ease of accessibility and interpretation of the SSI further support the value of the index and its data. In addition, to complete and comprehensive coverage of countries of the world, SSF uses the highest quality data sources available. Past studies and analyses from reputable institutions (i.e. the European Union JRC) have also shown the data to be relevant in all the steps of Joselyn’s recommended sequence of secondary data evaluation. Further evaluation of data accuracy was undertaken during the Phase II quantitative assessments of reliability and validity in this study.
Phase II
This phase systematically and quantitatively assesses the reliability and construct validity of SSI’s 21 measures across the six-year period from 2006 to 2012. As a first task in this process, an examination of the descriptive statistics was completed. Tables 4 and 4a present the means and the standard deviations for the 21 SSI measures from 2006 to 2012.
Means For Measures in the 2006-2012 SSI.
Standard Deviations For Measures in the 2006-2012 SSI.
Visual examination of this table shows that the data appears to be consistent within categories and across years. Further statistical analyses will delve further into the correlation across these measures.
Table 5 measures the correlation between the measures and the years. No statistically significant differences were found among any of the categories and years except those in Table 5a. However, these correlations indicate positive trends as the means of these categories are increasing over the six year period in that the early year 2006 is significantly different than the later years. Of perhaps greater significance and interest are the categories “Clean Water” (M = 6.26, SD = 0), “Air Quality” (M = 4.32, SD = 0), and “Renewable Water Resources” (M = 8.14, SD = 0). As illustrated, these three categories show no change in the means from 2006 to 2012. This issue merits additional study and may prove useful as a topic of future research. However, taken in context, this issue does not negate the overall validity and utility of the SSI.
Pairwise Correlations between Years for Measures in the Sustainable Society Index.
Exceptions for Pairwise Correlations between Years for SSI Measures (n = 150).
Multivariate analysis can be used to identify questionable cross-sections of data as well as suspect measures in Phase II. Table 6 presents the communalities for each measure across the six years derived from common factor analysis using the maximum likelihood method of factor extraction with oblimin rotation (Mulaik 1972). The communalities represent the amount of variance shared with all the other years within each of the six-year series of measurements (Malhotra, Peterson, and Kleiser 1998). The low communalities of the categories “Public Debt,” “Income Distribution,” and “Employment” suggest that more information potentially could be tapped by using these measures with less redundancy. These findings suggest more research is needed to better understand how these measures move with respect to each other.
Communalities for Common Factor Analysis of 6-Year Series of Each SSI Measure.
Using an approach from manufacturing quality control (Aczel 1993), Figure 1 depicts an “s-chart” of coefficient of variations for the four years examined (i.e. 2006, 2008, 2010, 2012) for each of the twenty-one SSI measures. The coefficient of variation measures the variability of a series of numbers independently of the unit of measurement used for these numbers (Abdi 2010). In order to do so, the coefficient of variation eliminates the unit of measurement of the standard deviation of a series of numbers by dividing it by the mean of these numbers (Abdi 2010). The coefficient of variation is calculated and plotted in Figure 1 (Malhotra 1996). While repositioning or relative shifting should be expected to occur from year to year due to events occurring in the countries, a .5 threshold for the standard deviations of the four years limits a measure to have 68 percent of the cases reposition (relative to the mean) no more than one half of a standard deviation from one year to the next (Malhotra et al. 1998). Using .5 as a threshold value is done to retain methodological conservatism.
“S Chart” of standard deviations of SSI measures for 2006, 2008, 2010, and 2012.
As can be seen in Figure 1, the coefficient of variation of seven measures have standard deviations above .5 over the average of the years in question. These include (1) income distribution, (2) biodiversity, (3) renewable energy, (4) greenhouse gasses, (5) organic farming, (6) GDP, and (7) employment. Income distribution, greenhouse gasses, and employment are extremely close to the .5 threshold. The first two demonstrate downward trends possibly indicating an improvement in data collection or some other external event. The deviation in employment may also reflect externalities not accounted for in the data. Biodiversity, renewable energy, organic farming, and GDP provide additional opportunities for further research to investigate possible causes and implications of their significant deviations. While we have pointed out a number of statistical inconsistencies, it is beyond the scope of this study to investigate the exact nature and causes of these variations. Our task was to determine if these irregularities were enough to cast doubt about the overall reliability and validity of the SSI. The findings suggests that they do not.
While the methods of this study are grounded in statistical procedures, prudent judgment is required to adequately assess consistency. Statistical procedures must be judiciously used and tempered with judgment of the nature of the variable being evaluated (Malhotra 1996). Additional research into these areas may prove beneficial in further assessing these categories and their overall utility in the SSI.
Past research has warned of the potential influential effects of multivariate outliers (Mullen, Milne and Doney 1995). In this case, outliers refer to observations, or in the case of the SSI, countries that are statistically distinct from the majority of other cases. Using stem-and-leaf plots in SPSS several outliers were identified in various categories in the 2012 SSI. A summary of those outliers can be found in Table 7. Those outliers found to be highly above the rest of the observations in their categories are designated by an “A.” Conversely, a “B” designates those outliers found extremely below the rest. In examining these outliers, it appears that social, cultural, or economic reasons may explain their abnormalities, further indicating that the SSI does indeed measure what it purports to measure. Future research focusing on these outliers and the reasons behind there deviations could prove exceedingly beneficial and insightful.
Outlying Countries in the Sustainable Society Index.
Assessing the External Validity of the SSI
Similar to the way this study assessed the internal validity of the SSI, it also deploys a subjective assessment (done with clustering analysis) and then a quantitative one (featuring confirmatory factor analysis). A description of how the external validity of the SSI was gauged follows.
Subjective Assessment
Cluster analysis of the SSI
Cluster analysis has become a common tool for the marketing researcher. Both academics and industry rely on this technique for developing empirical groupings of persons, products, or occasions that may serve as the basis for further analysis (Punj and Stewart 1983). Cluster analysis groups individuals or objects into clusters so that objects in the same cluster are more similar to one another than they are to objects in other clusters (Hair et al. 2010). Although cluster analysis can be a very useful data-reduction technique, its application is as much an art as it is a science. The technique can consequently be easily abused or misapplied. Care must therefore be taken in interpretation of the results (Hair et al. 2010).
Clusters can be formed in numerous ways. Hierarchical clustering is one of the most straightforward methods. It can be either agglomerative or divisive (Hair et al. 2010). Agglomerative hierarchical clustering begins with every case being a cluster unto itself. At successive steps, similar clusters are merged. Divisive clustering starts with every unit in one cluster and ends up with every unit in individual clusters. This study employed the agglomerative method of hierarchical cluster analysis. Five clusters were chosen to represent the data. Less than five clusters did not adequately separate the countries into heterogeneous groups. More than five clusters included groupings that did not make sense according to the data. By converging on five clusters in a hierarchical cluster analysis in SPSS, several interesting groups of countries emerge (see Tables 8 and 9). Group 1 appears to be composed primarily of emerging markets. Group 2, on the other hand, contain many less developed or developing countries. Group 3 is composed primarily of developed countries of the world. Groups 4 and 5 contain many Middle Eastern and eastern European block nations. Group 5 appears to contain wealthier “desert island” countries, whereas group 4 appears to have countries that have experienced much turmoil, conflict, and political instability over the past few years.
Hierarchical Cluster Analysis of the 2006 Sustainable Society Index.
Hierarchical Cluster Analysis of the 2012 Sustainable Society Index.
In examining Table 10, one can observe the country movements between these five clusters between 2006 and 2012. In examining this table one can see that cluster 2 and 3 were extremely stable with only three countries moving out of these clusters during that time period. The movement of countries like Iraq, Jordan, Lebanon, and Pakistan out of cluster 1 and into the more war torn Middle Eastern countries (cluster 4) can also be seen. Additionally, with internal strife and political turmoil, Syria and Yemen also move out of the “desert island” cluster 5 and into cluster 4. South Korea and Taiwan and their movements from group 3 to group 1, and India in its movement from group 1 to 2 are examples of movements that may be counterintuitive and therefore merit further consideration and study.
Cluster Comparisons of the 2006 and 2012 Sustainable Society Index.
As there is a significant amount of consistency within these various clusters, movements between them appear due to external factors and to neither the reliability nor the validity of the SSI. Further investigation will prove to be useful in shedding additional insights into the possible causes and implications of these movements. In sum, the clusters generated by the SSI in 2006 and 2012 seem to accurately reflect the current global positions and standings of the vast majority of countries included in the SSI. Thus, the application of the protocol to the SSI was successful. This was evidenced by the high degree of reliability and validity found. This analysis was next extended to the Legatum Prosperity Index.
Analysis of the Legatum Prosperity Index
In order to better understand the ability of the methodological protocol to assess reliability and validity of societal-level indexes, the study next applied the protocol to the LPI. Doing this offers another test of the protocol, as well as knowledge about how another societal-level index performs. Importantly, results from analyzing the reliability and validity of the LPI were similar to those of the SSI. This provides further evidence for the usefulness of the protocol. Accordingly, researchers can be more confident in using not only the SSI in their research of societal issues, but also the LPI. Appendix 3 presents a brief summary of the analysis of the LPI.
Analyses of the Prosperity, Corruption, and Risk Indexes
The analysis was extended to include the Corruption Perceptions Index (CPI) and the Euromoney Country Risk (ECR) rankings in efforts to understand and evaluate the commonalities shared by these four indexes. Cluster analysis was used to judge the similarities between the SSI, CPI, and LPI that led to an exploratory factor analysis including the ECR that converged on single underlying factor.
Because of the positive correlation of dimensions for societal development, countries that rank high (low) on measures of sustainability (the SSI) and quality of life (the LPI) will rank similarly on measures of corruption (the CPI), and country risk (the ECR). Accordingly, the same method of cluster analysis used on the SSI was applied to both the CPI and the LPI. Tables 11 and 12 highlight the findings of these analyses (see Tables 11 and 12).
Sustainable Society Index and the Corruption Perception Index Cluster Similarities.
The Sustainable Society Index and the Legatum Prosperity Index Communalities.
Cluster’s 1, 2, and 3 are remarkably similar across the three indexes. In Table 11, Cluster 4 contains developed countries that correspond to cluster 3 in the SSI. Closer examination of the data reveals that the countries in cluster 4 rank slightly higher on the CPI than those in cluster 3. This cluster also contains Qatar and the UAE that correspond to the “desert islands” of cluster 5 of the SSI. Cluster 2, the cluster that corresponds to developing nations, in the CPI also contains Syria, Tajikistan, Yemen, and Uzbekistan. As previously mentioned, these countries correspond to the war-torn Middle Eastern nations in cluster 4 of the SSI.
In Table 12, cluster 4 of the LPI contains countries in cluster 2 of the SSI. Cluster 2 and 4 of the LPI contain the developing nations represented in cluster 2 of the SSI. Closer examination of the data reveals that countries in cluster 4 of the LPI score slightly higher than those in cluster 2. Cluster 5 in the LPI contains countries that are in cluster 3 of the SSI. Apparently, this group scored on average slightly less than the countries in cluster 3 of the LPI. Cluster 5 in the LPI also shares similarities with cluster 5 in the SSI.
Comparison of the four indexes using cluster and factor analysis
To compare the countries across the four databases, 16 countries needed to be eliminated. Fourteen of these countries were not in the 2013 LPI and the last two were not in the 2012 CPI (see Table 13). What is evident in Table 12 is the overall similarity of results using clustering analysis across the four societal-level indexes. The conclusion reached from this subjective assessment of the four indexes is that each of these indexes produces similar output when analyzed with clustering analysis. This is important because many researchers will want to know if these indexes can generate similar outcomes regarding the underlying factors and sub-group structures of societies of the world.
Countries Eliminated from the Four Database Analysis.
Quantitative Assessment
Confirmatory factor analysis
Confirmatory factor analysis (CFA) is a way of testing how well measured variables represent a smaller or underlying number of constructs (Hair et al. 2010). Furthermore, CFA statistics tell us how well our theoretical specification of the factors match reality (i.e. the actual data). Extant research has yet to employ CFA to societal level indexes in efforts to explore potential common underlying factors. Our research and findings are unique in this respect.
A total of 135 countries were used in the following analysis. The SSI score is a composite score ranked on a scale from 0 to 10. Both the CPI and the ECR are composite scores that rank the countries on a 0 to 100 scale. The LPI is computed as a weighted average of the eight sub-indices and range from a -3.194 to 3.534.
As previously mentioned, in order to develop the findings of a highly reliable construct in the realm of macromarketing, the SSI was compared with three other notable international secondary data sources. To evaluate convergent validity, a structural equation model using AMOS was developed and tested. First, an exploratory factor analysis was performed using maximum likelihood with the oblimin rotation to identify the underlying relationship between the four different indexes (Hair et al. 2010). The four indexes converged on one factor accounting for 75.02% of the total variance explained. The confirmatory factor analysis in Figure 2 in AMOS further validated these results (CFI = 1.000, RMSEA = .000).
Confirmatory factor analysis of the SSI, CPI, LPI, and ECR.
Both exploratory and confirmatory factor analyses suggest that the four sources of secondary data, while ostensibly measuring four related dimensions of societal development, are actually manifestations of the same underlying construct. Based on the results of this study, this single underlying construct can be characterized as representing the tendency for societies to order themselves and function well (poorly). This factor can be said to represent “societal development”.
Although the data that constitute these four different indexes overlap somewhat, they each aim to measure different fundamental concepts. The SSI measures countries on their level of sustainability. The CPI measures a country’s perceived level of corruption. The LPI measures quality of life, and the ECR measures political, structural, and economic risk. While fundamentally different, these four indexes show a remarkably high level of convergent validity. For this reason, it is quite likely that this agreement across the four indexes suggests these indexes are manifestations of the same fundamental underlying construct – societal development.
Examining the loadings of the four indexes on the underlying construct of societal development discloses that the SSI has about half of the value as the other indexes (.47 for the SSI compared to an average of .93 for the other three indexes). This suggests that while the SSI is a manifestation of societal development, it is less so than the other societal-level indexes. This is likely due to the components of the SSI that represent natural resource or environmentally-related aspects of countries, which might be inversely related to economic development. Further research about societal-level sustainability and the societal development construct is now in order.
Peterson (2006) proposed that societal development was “development” when discussing poor countries, and “sustainable development” when referring to wealthy countries. Furthermore, Peterson proposed that societal development could become the focus of macromarketing serving as a transdiscipline. Such a transdiscipline represents more than merely applying one discipline’s constructs and methodological techniques to another’s. Rather, a transdiscipline is where multiple disciplines intersect for combining their approaches in complementary and synergistic ways to create knowledge about a phenomenon of interest. Macromarketing could serve as an integrator for multi-disciplinary research in a transdiscipline focused upon societal development. Because of the decades of research taking a societal perspective using a systems lens, macromarketers are well-positioned to take the lead in engaging scientists of all types in a transdisciplinary effort to better understand how societies develop.
Implications and Conclusions
Secondary data can be a veritable treasure trove for researchers in macromarketing and sustainability research. As international secondary data are becoming more plentiful, and the quality of the data and data collecting methods are improving, academics and practitioners alike should take advantage of such beneficial information resources. However, despite the growing availability of secondary sources, careful examination of the data is imperative as the benefits derived from these data are inherently limited to the quality of the data (Malhotra 1996).
To address this concern, this article presented systematic evaluation procedures to gauge the value of the SSI both qualitatively and quantitatively. This evaluation of the SSI suggests that the SSI is a quality source of secondary data on the sustainability of societies today.
First, the systematic protocol presented in this study assessed the reliability and internal validity of the SSI using both qualitative and quantitative approaches. Confirming RQ1 and RQ2, the result of these assessments suggests that the SSI is a comprehensive, easily accessible, and interpretable data source that is both reliable and valid.
Second, this study went a step further and assessed the external validity of the SSI by comparing the results of the cluster analysis of the SSI with similar cluster analyses of both the CPI and the LPI. Finally, this study also presented a confirmatory factor analysis using structural equation modeling that incorporated the SSI as well as the three other societal-level indexes. Confirming RQ3 and RQ4, this study found substantial likeness between the four indexes. This demonstrates that there is a high degree of similarity between countries that are high (low) on measures of sustainability and countries that are high (low) on measures of corruption, risk, and quality of life.
As previously mentioned, sustainability research is a topic of increasing importance in macromarketing (e.g. Meng 2015; Mittelstaedt et al. 2014; Viswanathan et al. 2014). The SSI is a global index that is particularly well suited to measure and study sustainability on a global scale. It is one of the very few indexes to include the human, environmental, and economic facets of well-being. Each of these dimensions of well-being complement and complete each other in examining issues of sustainability. Validating the SSI as reliable and valid is an important contribution to macromarketing scholarship. In sum, this study found that the SSI can be a valuable tool for a variety of users, such as policymakers, government officials, educational institutions, NGO’s, industry, and others concerned with macromarketing and sustainability research.
In conclusion, this study assessed the value of a societal-level sustainability index for macromarketing researchers. The study applied a protocol to four different indexes addressing different societal phenomena, such as sustainability, prosperity, corruption, and country risk. Results suggest that the protocol is effective in gauging the reliability of twenty-first century societal indexes, as well as the internal and external validity of these indexes. By analyzing four different indexes, this study also captured a view of an emerging construct—societal development, which represents the ability (inability) of societies to organize themselves and function well (poorly). More research is now in order on societal development and the role of sustainability for countries of the world.
Footnotes
Acknowledgments
The authors would like to acknowledge and give a special thanks to Shikha Upadhyaya for her work on validating the Legatum Prosperity Index. Also thanks to Special Issue Editor Ben Wooliscroft, Editor-in-Chief Terrence Witkowski, as well as the three anonymous reviewers for their insightful and constructive feedback.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.