United Nations endorsement and support for human rights: An experiment on women’s rights in Pakistan

Abstract

The United Nations is one of the organizations charged with developing and promoting international human rights law. One of the primary ways that the United Nations tries to do that is by regularly reviewing the human rights practices of member states and then recommending new policies for that state to implement. Although this expends considerable resources, a number of obstacles have made it difficult to empirically assess whether the UN’s review process actually causes countries to improve their human rights practices. To study this topic, we conducted an experiment in Pakistan that tested whether respondents were more likely to support policies aimed at improving women’s rights when they learned that the reforms were proposed by the United Nations. Our results indicate that the respondents who were randomly informed of the United Nations endorsement not only expressed higher support for the policy reforms, but also were more likely to express willingness to ‘mobilize’ in ways that would help the reforms be implemented. Our treatment did not have any effect, however, on respondents that did not already have confidence in the United Nations. This suggests that the international human rights regime may only be able to aid domestic reformers when there is already faith in those institutions.

Keywords

human rights international law survey experiment United Nations women’s rights

Introduction

A web of international human rights agreements and institutions regulates how governments treat their citizens. Scholars vigorously debate, however, whether this web actually causes countries to improve their human rights practices. Critics of the system argue that, because of a number of shortcomings, international human rights law has failed to achieve its objectives (e.g. Posner, 2014; Hopgood, 2013; Moyn, 2010). For example, these scholars argue that weak enforcement mechanisms and the proliferation of rights have made the human rights regime ineffective. Defenders of the system argue that, despite its flaws, international human rights law has produced dividends in at least some circumstances (e.g. Goodman & Jinks, 2013; Sikkink, 2011; Simmons, 2009). For example, these scholars argue that domestic political actors have an easier time generating support for reforms when the international human rights system lends credibility to their efforts.

But scholars on both sides of this debate have agreed that direct pressure is unlikely to lead countries to change their behavior. Notably, international organizations cannot directly sanction countries for failing to improve their human rights practices. Instead, international organizations have to rely on indirect forms of pressure. For instance, the United Nations (UN) human rights committees review member states’ human rights practices and then issue reports with suggested policy reforms. Countries are not directly punished for failing to adopt the recommendations contained in the reports, but the hope is that the reports generate ideas and create pressure to reform.

Where there is disagreement, however, is whether these indirect forms of pressure actually change human rights practices. One possibility is that the reports issued by international organizations increase the probability that human rights reforms will be adopted, because the ideas contained within them will enjoy greater support when they are endorsed by prominent organizations like the UN. Notably, a line of research has suggested that domestic interest groups are able to leverage the ratification of international human rights treaties to gain support for reforms within their country (Simmons, 2009), and it is similarly possible that domestic organizations could leverage recommendations by the UN to generate support for reform. Moreover, research has also suggested that people may be more likely to support reforms when they are endorsed by the UN (Linos, 2011, 2013). This theory suggests that UN endorsement should increase public support for human rights reforms. But it is difficult to test this theory with observational data because the UN likely chooses which reforms to endorse based on their political viability. Given that reality, there is unsurprisingly little research on whether the UN’s investment in generating the reports has produced dividends.

We test the hypothesis that UN endorsement increases support for policies aimed at improving human rights. To do so, we conducted an experiment during interviews in Pakistan on support for women’s rights reforms. We choose Pakistan for our experiment because it is a developing country that is partially democratic and because there is a great deal of variance in views about human rights within the country. As a result, Pakistan is the kind of country in which previous research has suggested human rights law might be able to make a difference (Simmons, 2009). We choose to focus on women’s rights because the UN had recently issued a report suggesting specific policy reforms to improve women’s rights in Pakistan that we were able to leverage for our experiment, and because the status of women’s rights is a salient political topic in Pakistan. Moreover, previous research using observational data has suggested that the Convention for Elimination of Discrimination Against Women (CEDAW) may have helped improve women’s rights (Richards & Haglund, 2015; Cole, 2013; Hill, 2010; Simmons, 2009), but this finding has not been the subject of experimental research.

A team of six researchers administered our experiment during interviews with 614 respondents in Pakistan in December 2014 and January 2015. The respondents were all asked whether they agreed with four policy proposals aimed at improving the rights of women. The proposals were from a 2013 report produced by the UN CEDAW committee on the status of Pakistan’s compliance with its commitments to the rights of women. Half of the respondents were asked whether they support the proposals without information on where the proposals came from, while the other half of the respondents were told that the policies were proposed by the UN.

Our experiment found that UN endorsement had a substantively large and statistically significant effect on support for all four proposals. Moreover, the results of the experiment suggest that respondents that were told of the UN’s endorsement for reforms also expressed greater willingness to mobilize to support human rights. But our treatment had no effect on respondents that did not already have confidence in the UN. Although our experiment has clear limitations – most notably, we used a convenience sample that was not representative of the population of Pakistan – it does provide qualified evidence that the international human rights regime can aid domestic activists hoping to mobilize support for reform. However, those effects are likely to be small unless there is already faith in the relevant international institutions.

Background

The international human rights system

The international human rights system includes a number of treaties and institutions designed to protect people from having their rights violated. Many of these treaties and institutions have been developed and promoted by the UN. For instance, the UN has developed several core international human rights treaties. These treaties have been widely ratified, and they protect a large number of different rights. The ICCPR alone, for example, protects 58 different rights (Posner, 2014: 152–154).

The UN has also created several institutions charged with improving human rights protections. The UN Office of the High Commissioner of Human Rights promotes and coordinates the UN’s policies towards human rights; the Human Rights Council is a body of 47 member states that investigates and reviews alleged human rights violations; and the major international treaties all have their own committees comprising subject matter experts. For example, the CEDAW committee has 23 ‘independent experts’ that monitor countries’ implementation of the CEDAW agreement.

One of the primary ways that the treaty committees monitor countries is by issuing regular reports. Countries are obligated to file periodic reports to the committees on their implementation of the relevant treaty.¹ The treaty committee can then file responses to the country reports. These responses comment on developments within the country and recommend steps the country should take to improve implementation of the treaty. Often the recommendations are vague actions like improving women’s access to healthcare, but in other cases the recommendations are specific reforms like imposing a 33% quota on the number of seats reserved for women in the national parliament.

Public opinion and UN endorsement

Despite the considerable resources expended promoting the international human rights system, whether these treaties and institutions actually change countries’ behavior remains hotly debated. One thing that scholars on both sides of the debate largely agree on, however, are some of the mechanisms that are unlikely to cause countries to improve their human rights practices. For example, scholars on both sides of the debate have argued that the UN human rights system does not place direct pressure on countries to improve their rights practices (Posner, 2014: 95–104; Hafner-Burton, 2013: 44–66; Simmons, 2009: 121–125). Moreover, scholars on both sides of the debate have also argued that powerful countries rarely pressure other countries to improve their human rights practices (Posner, 2014: 80–81; Nielsen, 2013; Simmons, 2009: 116).

But some scholars have argued that one mechanism through which the international system can change the domestic political climate in ways that increase the probability that rights will be protected (Lupu, 2015; Hillebrecht, 2014; Conrad & Ritter, 2013; Lupu, 2013; Simmons, 2009; Dai, 2002).² One way international human rights law may alter domestic political climates is by making it easier to gain support for reforms (Hafner-Burton, LeVeck & Victor, 2016; Neumayer, 2005). In one prominent articulation of this theory, Simmons (2009) argues that international human rights law changes domestic politics because appeals to international standards make the public more sympathetic to demands for reforms. Simmons’s theory is specifically that treaty ratification can improve human rights practices by pre-committing leaders and emboldening activists, but it is also possible that UN commendation could be a powerful tool for domestic mobilizers. This corollary to Simmons’s theory is that the statements of international organizations may be used to make public opinion more supportive of a particular form, and, in turn, those changes in ‘public opinion [create] some pressure towards compliance with international law’ (Posner & Sykes, 2013: 27).

A number of studies have tested this mechanism using data from public opinion surveys (Davis, Murdie & Steinmetz, 2012; Hertel, Scruggs & Heidkamp, 2009) and survey experiments (McEntire, Leiby & Krain, 2015; Chilton, 2014; Wallace, 2013). These studies have largely found that information on international treaties can moderately increase support for the protection of human rights. What these studies have not tested, however, is whether the existence and activities of the entire international architecture designed to enforce these treaties – like the UN human rights organizations – actually changes public opinion. In other words, prior research has suggested that appeals to an international treaty may increase support for protecting a particular right, but they have not tested whether the specific policies proposed by international organizations are more likely to be supported than policies suggested without the authority of international human rights law.

But there is reason to believe that the endorsement of international organizations like the UN may increase support for human rights reforms. Notably, a body of research has tested the effects of endorsement on support for policies and found that individuals change their views based on the endorsement of political parties, elites, and even the UN (Bush & Jamal, 2015; Berinsky, 2009; Arceneaux, 2008; Druckman & Lupia, 2000). For example, Linos (2013, 2011) found that US respondents were more likely to support health reforms and maternity leave policies when they are informed that the policies are recommended by the UN. Linos (2011) also found that endorsement from the UN was more likely to change the views of US respondents when they already supported the UN.

Taken together, this suggests three clear hypotheses about the effect that UN endorsement may have on policies designed to improve human rights.

H1: People will be more supportive of human rights reforms when they learn they are endorsed by the UN.

H2: People will be more likely to express willingness to mobilize in support of policies that are endorsed by the UN.

H3: UN endorsement will have a greater effect on people that already have confidence in the UN.

Research design

Motivation

Reliable causal inference in the study of human rights is illusive. International human rights treaties are written with the preferences of (at least some) countries in mind (Downs, Rocke & Barsoom, 1996), and, after the treaties are written, countries are not randomly assigned to be signatories (von Stein, 2005). Additionally, variation is a basic requirement of causal inference, but most of the major international human rights agreements have been near universally ratified (Chilton & Tingley, 2013). Moreover, scholars have struggled to find unbiased measures of human rights practices (Fariss, 2014).

In response to these obstacles, researchers have used experimental methods to test whether information on international law increases support for human rights reforms (e.g. Putnam & Shapiro, 2017; Hafner-Burton, LeVeck & Victor, 2016; McEntire, Leiby & Krain, 2015; Ausderan, 2014; Chilton, 2014; Wallace, 2013; Tomz, 2008). The experimental research in this field, however, has several common issues that limit its ability to directly test existing theories on the effectiveness of international law. Our experiment was designed to address three of these issues.

First, these experiments have been conducted almost entirely in the United States.³ This is potentially problematic because respondents in the United States are likely to have idiosyncratic views on international law, and because previous research has also suggested that the international human rights agreements are unlikely to affect all countries equally (Conrad, 2014; Conrad & Ritter, 2013; Hafner-Burton & Tsutsui, 2007; Neumayer, 2005). Instead, research has suggested that international human rights law is most likely to change the behavior of ‘partial’ democracies – that is, states that are neither stable democracies nor stable autocracies (Simmons, 2009).

Second, the experiments have largely used vignettes to test whether references to international treaties change support for actions that violate human rights (e.g. Chilton, 2014; Wallace, 2014, 2013). But a primary mechanism through which the UN tries to encourage countries to improve their human rights practices is by issuing country-specific reports that outline policies it recommends that a specific country should adopt. If the expansive human rights apparatus that has been developed beyond the treaties does not improve human rights protections, it would call into question the value of a great number of the efforts that define the current international human rights system.

Third, these experiments have largely been administered to respondents recruited online. In many countries, however, policies proposed by human rights reformers remain controversial. Because the leading theory for why international human rights law may lead to improvements in domestic human rights is that it increases the likelihood that mobilization efforts will gain traction, simply testing whether respondents will express support for human rights reforms when completing an anonymous online survey may not accurately measure whether respondents would be willing to more publicly express support.

Case selection

We conducted our experiment in Pakistan for three reasons. First, research has suggested that the international human rights system is unlikely to improve rights protections in stable democracies or stable autocracies, but it may improve rights protection in countries that are neither completely democratic nor completely autocratic (Simmons, 2009). Pakistan is one of these countries. Not only was Pakistan identified as a transitional democracy by the methodology Simmons (2009) used to

Figure 1.

Articles that reference the United Nations (%)

classify countries,⁴ but Pakistan was rated as ‘partially free’ by Freedom House’s Freedom in the World Report in both 2014 and 2015.⁵ Second, there is a great deal of heterogeneity in public views towards human rights in Pakistan. If there were already widespread public support for human rights reforms within a country, it would be difficult to assess whether endorsement by the UN increases that support. Instead, an ideal country to study would have variance in public opinion on human rights issues. Pakistan is a country that is currently undergoing a great deal of debate about human rights generally, and about the role of women in society, specifically. Third, members of our research team had experience conducting research in Pakistan.

We decided to focus on women’s rights for several reasons. First, the UN completed a review of Pakistan’s women’s rights record in 2013. This provided us with timely, concrete proposals to evaluate that had not yet been adopted. Second, the condition of women in many parts of the world remains deplorable. For example, in Pakistan the practice of families ‘honor killing’ female family members for perceived slights remains widespread. Our decision to focus on women’s rights was thus in large part because it is important to research the effectiveness of efforts to improve the rights of women so that more can be done to change these practices. Third, empirical research using observational data has found more evidence suggesting that CEDAW has caused improvements in human rights than other agreements (Richards & Haglund, 2015; Cole, 2013; Hill, 2010). However, the small literature that has used experimental methods to study the effectiveness of international human rights (Putnam & Shapiro, 2017; McEntire, Leiby & Krain, 2015; Ausderan, 2014; Chilton, 2014; Wallace, 2014, 2013; Tomz, 2008) has not examined women’s rights.

A final consideration in favor of studying Pakistan is that debates, discussions, and reporting on human rights within Pakistan often reference the position of the UN. To illustrate this, Figure 1 reports the results of a content analysis we conducted of the relative frequency that the UN was referenced in newspaper stories on human rights in Pakistan.⁶ We specifically used the Factiva archive series to search for Pakistani newspaper articles mentioning ‘human rights’ or ‘women’s rights’ in each year between 2004 and 2014. We then evaluated the percentage of those articles that mentioned the United Nations. For comparison, we also conducted the same exercise for the United States.

Figure 1 shows that there are comparable trends in the number of news stories in Pakistan and the United States that reference the UN. In Pakistan, 24% (19,744 of 84,835) of news stories on ‘human rights’ mentioned the UN compared to 27% (60,124 of 221,516) of stories in the United States. Similarly, in Pakistan 23% (926 of 4,710) of stories on ‘women’s rights’ mentioned the UN compared to 13% (4,193 of 26,347) of stories in the United States. These results support the claim that the UN is often part of public discourse on human rights within Pakistan.

Subject recruitment

We administered our experiment to 614 respondents in Pakistan in December 2014 and January 2015.⁷ The experiment was administered during in-person interviews conducted by one of six academic researchers from Pakistan. The interviews took about 35 minutes to complete, and each respondent was paid 400 Pakistani Rupees (roughly $4.00 USD) for their time. The interviews were all conducted in the national language of Pakistan: Urdu. To be consistent with local customs, male researchers interviewed male respondents and female researchers interviewed female respondents.

Although it required considerable effort and expense to conduct our experiment through in-person interviews, we decided that it would be superior to conducting an online or telephone survey. This is because we believe that the important question is whether United Nation endorsement could increase Pakistanis’ willingness to publicly support reforms. Women’s rights are still a sensitive topic within Pakistan, and an online experiment revealing that UN endorsement influences private views may not demonstrate that the international human rights system improves the political climate for human rights reform. Thus, it was important to see if participants support reforms endorsed by the UN in face-to-face interactions.

But there are shortcomings associated with in-person interviews. Most notably, it is possible that the researchers conducting the interview could bias the results by changing their behavior to elicit the desired result. To guard against this concern, we informed the field team – with the exception of the leader – that the goal of our research was to study support for women’s rights generally, but we did not disclose that our primary interest was the effect of endorsement by the UN on support for those policies.

We primarily recruited university students and teachers in and around Islamabad. This recruitment strategy provided three practical benefits. First, because Islamabad is the federal center of Pakistan, these respondents were from all regions of the country. Second, the respondents offered the practical advantages of likely being receptive to participating in an academic survey. Finally, and most importantly, given the violence that has been directed to women’s rights activists in Pakistan, we decided that administering the survey on the grounds of secure universities would provide the greatest safety for our team.⁸

Although recruiting respondents at universities had these practical benefits, the drawback is that our sample is not representative of the population of Pakistan.⁹ The gender breakdown, age, household size, and income level of our sample is comparable to the population of Pakistan: our sample is 51% male and Pakistan is 52% male; our sample is 30 years old on average and Pakistan is 34 years old on average; our sample has an average household size of 6.8 people and Pakistan has an average household size of 6.8; and our sample reported having an income of 5.01 on a scale from 1 to 10 and the World Values Survey found an average response to the same question of 5.5 for Pakistan. However, our sample is less religious and dramatically more educated than the overall population of Pakistan. Specifically, 8% of our sample reports being not very religious when just 2% of Pakistan does; and, because we recruited respondents at universities, 99% of our sample had completed at least some college while just 8% of Pakistan has done so. Using this non-representative sample obviously reduces the generalizability of our results. We further discuss this issue below and explore how our results may have changed with a more representative sample.

Table I.

Policy questions for the endorsement experiment

	Question wording
1. Legislative quotas	…increasing the quotas allocated for women in the National and Provincial Assemblies and in the Senate from 17% to a minimum of 33%.
2. Honor crime pardons	…repealing all laws that allow perpetrators of so called honor crimes to negotiate a pardon from the victims’ families.
3. Marriage age	…raising the minimum age for marriage for girls from 16 to 18.
4. Election re-polling	…re-polling any location where women cast less that 10% of votes during an election.

Survey design

The survey had four parts. First, the subjects were asked a number of questions designed to establish their views on the rights of women, the role of religion in government, and the UN. Second, the subjects were asked if they supported four reforms aimed at improving the rights of women. Third, the subjects were asked a number of questions designed to test whether the experimental treatment increased stated willingness to mobilize in ways that may increase the likelihood of reforms being implemented. Fourth, the respondents were asked a series of standard demographic questions.

The second part of the survey contained the experiment. It asked respondents about four policies that were recommended by the UN Human Rights Committee on the Elimination of Discrimination Against Women’s in their 2013 review of Pakistan’s compliance with CEDAW. Although there are many recommendations in the 2013 report, our goal was to identify proposals that: (1) call for concrete reforms; (2) are easy to understand; (3) may generate differences in opinion; (4) cover different policy areas; and (5) have not yet been implemented. Table I lists the four proposals that we asked about.

All of the subjects that took our survey were asked how they feel about each of the four proposals. The experimental part of our survey was how the treatment was introduced. We started our questions to the ‘Control Group’ by saying that ‘A recent proposal calls for […]’, but we started our questions to the ‘UN Group’ by saying that ‘A recent proposal by the United Nations calls for […]’. In other words, the only difference between the two groups is that, for each policy, the UN Group was told that it was proposed by the UN.

After reading each proposal, researchers asked: ‘How do you feel about this proposal?’ The respondents were asked to assess their level of support on a five-point scale. Specifically, we asked respondents whether they: (5) strongly agree, (4) somewhat agree, (3) are indifferent, (2) somewhat disagree, or (1) strongly disagree.

Sample balance

The respondents were demographically balanced. Panel A of Figure 2 reports the standardized mean differences between the treatment and control groups (and associated 95% confidence intervals) for all our demographic variables. All variables are below the 0.2 rule of thumb guidance for imbalance. Additionally, we also used the balance test from Hansen & Bowers (2008) to evaluate imbalance across the post-treatment demographic variables. Using this test, the overall χ ² statistic is 10.6 and the associated overall p-value is 0.48. There is thus no evidence of demographic imbalance between the subjects in the two groups.

In addition to testing for demographic balance, we also tested to make sure the control and treatment groups held similar views on the subject of our study. To do so, we asked the respondents a series of pre-treatment questions designed to elicit their expressed beliefs on four topics: support for pro-women policies,¹⁰ support for anti-women policies,¹¹ belief that Islamic Law should be the basis for laws,¹² and confidence in the UN.¹³ With the exception of the question on the UN, these questions were based on a survey that examined women’s rights in Jordan developed by Bush & Jamal (2015).

Figure 2.

Balance for control and treatment group for pre-treatment questions

Figure 3.

Distribution of responses to policy questions

As Panel B of Figure 2 shows, the views of the Control Group and UN Group are fairly evenly balanced for the pre-treatment questions, and they also all have standardized differences below 0.2. We also evaluated the degree of balance for these measures using the Hansen & Bowers (2008) test. Using this test for the four measures in Panel B of Figure 2, the overall χ ² statistic is 6.06 and the associated overall p-value is 0.20. Additionally, we simultaneously tested for balance for the variables in Panel A along with the variables in Panel B of Figure 2. Across all 17 variables, the overall χ ² statistic is 19.3 and the associated overall p-value is 0.2. Taken together, these tests do not reveal any evidence of demographic or ideological imbalance.

Figure 4.

Mean responses to policy questions

Table II.

Primary results – ordered logit regressions

	(1) Quotas for legislators	(2) Honor crime pardons	(3) Marriage age	(4) Election re-polling

UN treatment	0.808**	1.257**	0.651**	0.837**
	(0.168)	(0.173)	(0.171)	(0.166)
Male	–1.755**	–1.033**	–1.411**	–0.905**
	(0.186)	(0.181)	(0.185)	(0.176)
Age	0.002	–0.001	–0.006	–0.003
	(0.010)	(0.010)	(0.010)	(0.010)
Household size	0.001	0.006	–0.006	–0.022
	(0.029)	(0.028)	(0.028)	(0.027)
Income	0.087	0.155*	0.106	0.090
	(0.064)	(0.065)	(0.067)	(0.065)
Education – High School	–0.722	–0.900	–1.025	–0.320
	(1.488)	(1.442)	(1.466)	(1.448)
Education – College	0.550**	0.061	0.507*	0.235
	(0.200)	(0.204)	(0.209)	(0.196)
Ideology – Conservative	–0.809*	–0.487	–0.658	–0.547
	(0.373)	(0.356)	(0.368)	(0.354)
Ideology – Moderate	–0.405	–0.075	–0.043	–0.156
	(0.219)	(0.217)	(0.234)	(0.216)
Religion – Extreme	–0.176	–1.153**	–0.693	–0.296
	(0.392)	(0.419)	(0.408)	(0.382)
Religion – Moderate	–0.098	–0.757*	–0.360	–0.305
	(0.339)	(0.369)	(0.358)	(0.327)

Observations	499	500	501	501
Model	O-Logit	O-Logit	O-Logit	O-Logit

Standard errors in parentheses. Omitted categories are Education – Graduate level, Ideology – Liberal, and Religion – Not very. Constant omitted. **p < 0.01, *p < 0.05.

Results

Primary results

Because the treatments were randomly assigned and there is no evidence of imbalance, we begin by evaluating the results without controlling for any other variables. Figure 3 presents the distribution of support for the four policy proposals for the Control Group and the UN Group. As Figure 3 shows, for all four questions, the respondents that received the UN treatment were more likely to express agreement with the proposal. For all four proposals, the UN Group had more respondents that strongly agreed with them and fewer respondents that strongly disagreed with them.

Figure 4 presents the mean responses and 95% confidence intervals for the same four questions. As Figure 4 shows, the UN Group not only expressed higher average levels of approval for each question, but the differences are all statistically significant at the 0.01 level or higher. Moreover, the treatment effects are also substantively large. The size of the differences ranges from 0.36 for the Quotas for Legislators question to 0.79 for the Honor Crime Pardons question. Overall, the average increase in support for the four policies was 0.56 on a scale of 1 to 5.

Table II further explores these results by reporting ordered logit regressions that estimate the treatment effects while controlling for demographic variables. Including these controls ensures that any treatment effects are not due to any lingering imbalance between the Control Group and UN Group. The results in Table II show that the UN Treatment was associated with an increase in support for all four proposals that is statistically significant at the 0.01 level. These results are robust to controlling for the pre-treatment belief questions introduced in Panel B of Figure 2 and using OLS regressions instead of ordered logit models.¹⁴ Additionally, our five-point response scale included ‘indifferent’ as the middle option, but it may be inappropriate to treat respondents that are indifferent as having moderate views (Kleinberg & Fordham, 2018; Mondak & Creel Davis, 2001). We thus estimated the regressions from Table II while dropping respondents that said they were indifferent to a given proposal. When doing so, the results were substantively similar to our primary results.

Mobilization results

As previously noted, the most prominent theory of how international human rights law might improve human rights outcomes is centered on mobilization (Simmons, 2009). The argument is that ratification of international agreements and participation in international institutions does not automatically translate into improved rights practices. After all, they do not have direct enforcement mechanisms, and domestic governments that wish to improve the protection of human rights are free to do so without prodding by the international community. Instead, the theory is that international agreements and institutions can make it easier for domestic groups to lobby for changes within their country. Our survey attempted to measure whether the UN treatment increased expressed willingness to mobilize in support of the proposals.

We specifically asked respondents how likely they would be to take four steps that could potentially lead to increased respect for women’s rights. First, to measure expressed willingness to Vote for Women we asked all of the respondents: ‘If a woman runs for election to the

Figure 5.

Mean responses to mobilization questions

National Assembly of Pakistan, how likely are you to consider voting for a woman in the election, all else equal?’ Second, to measure expressed willingness to Vote for Women-Friendly Parties we asked all of the respondents: ‘If a political party were to make improving women’s rights one of its main goals, how likely would you be to vote for the party in an election, all else equal?’ Third, to measure expressed willingness to Sign a Petition we asked all of the respondents: ‘If you were asked to sign a petition calling for the government to improve women’s rights in Pakistan, how likely would you be to sign it?’ Fourth, to measure expressed willingness to Support Increased Funding we asked all of the respondents: ‘Do you support the government providing funding to organizations aimed at improving women’s rights?’ The respondents were asked to respond to each of these four questions on a five-point scale.

Figure 5 presents the mean responses and 95% confidence intervals to these questions.¹⁵ For all four, the differences between the Control Group and the UN Group are substantively large and statistically significant at the 0.01 level. The average increase in claimed willingness to mobilize was 0.42 on the five-point scale. This suggests that the respondents were more likely to express willingness to mobilize after previously hearing that the UN had proposed reforms to women’s rights. This is perhaps surprising given that the only difference between the groups is that in a prior set of questions, the UN Group was told that four policies were proposed by the UN.

Figure 6.

Mean responses to policy questions by confidence in the UN

Figure 7.

Mean responses to mobilization questions by confidence in the UN

Results by confidence in the UN

Linos (2011) suggests that the effects of UN endorsement may be conditional on whether the respondents already have confidence in the UN. We thus subset the sample into respondents that said they either had a great deal of confidence or some confidence in the UN (Confident in the UN) and respondents that had none at all (Not confident in the UN).¹⁶ Figures 6 and 7 present our primary results and mobilization results for these two groups.

There are two things worth noting about the results in Figures 6 and 7. First, the UN treatment had a large and statistically significant effect on respondents that previously expressed confidence in the UN, but the treatment had little effect on the respondents that did not previously express confidence in the UN.¹⁷ In Figure 6, the respondents Not confident in the UN had an average response for the four questions of 3.41 for the Control Group and 3.42 for the UN Group. In Figure 7, there was a modest treatment effect: for the respondents Not confident in the UN, the average response was 3.49 for the Control Group and 3.70 for the UN Group.

Second, the respondents in the Control Group that expressed confidence in the UN were less likely to support the reforms and efforts to mobilize than the respondents in the Control Group that did not express confidence in the UN.¹⁸ This result is surprising, and further research is needed to explore why this is true. One possible explanation is that respondents that are Not confident in the UN may have doubts in the institution, in part, because they believe that human right reforms are necessary and that the UN has not done enough to provide them. These respondents thus may be more likely to support any proposals aimed at improving the rights of women.

Simulation results

A primary concern with our results is that the treatment effects may not materialize in a sample that was representative of the broader population. Notably, Figures 6 and 7 demonstrated that UN endorsement only had an effect for the respondents that already had confidence in the UN, and our sample had a higher share of respondents with confidence in the UN than the population of Pakistan. To assess the generalizability of our results, we thus conducted an exercise where we simulated what our treatment effects would be if our sample was weighted to reflect the rates of confidence in the UN found in the overall population of Pakistan.

To do so, we leveraged data from the World Values Survey (WVS).¹⁹ The most recently released wave of the

Table III.

Confidence in the UN in Pakistan (% of respondents)

	A great deal	Some		None at all	No answer /Didn’t know
A. Our survey	6.5	32.7		59.4	1.3
	A great deal	Quite a lot	Not very much	None at all	No answer /Didn’t know
B. WVS: Pakistan overall	2.7	20.4	41.9	28.4	6.6

C. WVS: Pakistan by education
No formal education	3.0	20.7	35.4	31.7	9.2
Primary education(incomplete or complete)	4.4	29.3	28.2	27.6	10.5
Secondary education(incomplete or complete)	2.6	16.8	48.9	26.8	4.8
University education(incomplete or complete)	3.4	14.8	47.7	34.1	0.0
D. WVS: World overall	9.8	30.4	28.0	21.1	10.6
E. WVS: World by regime type
Autocracies(Polity2 from –10 to 4)	3.0	20.7	35.4	31.7	9.2
Partial democracies(Polity2 from 5 to 7)	4.4	29.3	28.2	27.6	10.5
Democracies(Polity2 from 8 to 10)	2.6	16.8	48.9	26.8	4.8

WVS results are from Wave 6 of the World Values Survey. For the results by education, we aggregate results for respondents that have started or completed the given level of education. Regime types based on countries’ 2014 Polity scores.

WVS – conducted from 2010 to 2014 – asked respondents in 60 countries, including Pakistan, how much confidence they have in the UN. The WVS was specifically administered to 1,200 respondents in 2012 that were sampled to be representative of the population of Pakistan. Although the results are not perfectly comparable to ours because the WVS used a four-point response scale and we used a three-point response scale, the thrust of the question is identical.

Table III reports the results from our survey and several groups from the WVS. As Panel A reveals, 39.2% of our sample either had a great deal or some confidence in the UN. But Panel B reveals that only 23.1% of the WVS’s representative sample of Pakistan had a great deal or a quite a lot of confidence in the UN. This difference is not just driven by our respondents’ high levels of education. Panel C breaks down the WVS results by education level and shows that the most highly educated Pakistanis actually have less confidence in the UN. It is thus unlikely that we would have found as large a treatment effect as we did if we had administered our experiment to a more representative sample.

However, our treatment effects were sufficiently large that it is possible we would have still found a modest effect if we had administered our experiment to a representative sample. To assess this possibility, we conducted simulations where we used the same mean responses for each group, but where we randomly generated a sample that reflected the rates of confidence in the UN from the WVS.²⁰

For example, for the Quotas for Legislators question, in the Control Group, the respondents with confidence in the UN had a mean response of 2.80 (s.d. 1.53) and the respondents without confidence in the UN had a mean response of 3.17 (s.d. 1.35). In the UN Group, the respondents with confidence in the UN had a mean response of 4.17 (s.d. 0.99) and the respondents without confidence in the UN had a mean response of 2.90 (s.d. 1.31). For this question, each of our simulations created an artificial control group with 71 observations (307 respondents * 23.1% rate of confidence in UN taken from the WVS) with answers drawn from a distribution with a mean of 2.80 and a standard deviation of 1.53 and 265 observations with answers drawn from a distribution with a mean of 3.17 and a standard deviation of 1.35. An

Figure 8.

Simulation results

artificial treatment group was also created with 71 observations with answers drawn from a distribution with a mean of 4.17 and a standard deviation of 0.99 and 265 observations with answers drawn from a distribution with a mean of 2.90 and a standard deviation of 1.31. We then assessed the differences in means between the artificial control group and artificial treatment group. We ran 1,000 simulations for each question using this approach.

Figure 8 reports the simulated distribution of the standardized differences in mean responses for the treatment and control groups for each question. Proposals 2 and 4 had a mean standardized difference above the commonly used 0.2 threshold for a small, but non-negligible, effect (Cohen, 1988), and proposal 3 fell just short of the 0.2 threshold. Across all four questions, the mean standardized difference from the simulations was 0.25. Or, put another way, the treatment effects were statistically significant at the 0.05 level or lower in 16% of the simulations for proposal 1, in 100% of simulations for proposal 2, in 66% of simulations for proposal 3, and in 93% of simulations for proposal 4.

As a comparison, in our primary results, the standardized difference between the Control Group and UN Group was above 0.2 for all four questions, and across all four questions, the mean standardized difference was 0.48. Of course, these simulations make the admittedly very strong assumption that the only differences between our sample and the population of Pakistan that affect treatment sizes are differences in the rates of confidence in the UN. But they also reveal that it is at least possible that we would have found a small effect for UN endorsement on support for women’s rights reforms if our sample had much lower rates of confidence in the UN.

Finally, across all 60 countries, the WVS sample had levels of confidence in the UN that were comparable to our sample. In our sample, 39.2% had a great deal or some confidence in the UN, and 40.2% of the WVS overall had a great deal or quite a lot of confidence in the UN. In fact, Pakistan had lower levels of confidence in the UN than 49 of the countries in the WVS sample. Moreover, in the partial democracies that have been theorized as where international human rights law is most likely to have an effect, 42.2% of the respondents had a great deal or quite a lot of confidence in the UN. It is thus possible that the effects we find are larger than what we would find in a representative sample of Pakistan, but comparable to what we would find in countries with higher levels of support for the UN.

Conclusion

Considerable resources are expended every year operating the UN’s system of monitoring compliance with international human rights law. Our experiment tested whether policy recommendations produced through that process increase support for human rights reforms. We found that UN endorsement increased support for all of the policy reforms that we evaluated, and that respondents who were previously informed that the UN had proposed the reforms were more likely to express willingness to mobilize in ways that may improve the probability that they would be implemented. Finally, we also found that the UN endorsement had essentially no effect on respondents who did not previously have confidence in the UN.

However, our research design had several limitations that suggest the need for further research. First, the generalizability of our results is limited by our use of a convenience sample. Additionally, the population of Pakistan may not respond the same as people in other countries to UN endorsement of human rights reforms. Future research should thus try to explore whether our results hold for more representative samples and in other countries.

Second, our research design may not mimic real-world treatment effects. Our design is similar to endorsement designs that are typically used to indirectly measure levels of support for the endorser (Blair, Imai & Lyall, 2014). One interpretation of our results is thus that respondents that already have confidence in the UN are also likely to support the UN. That said, we did find results for the mobilization questions (e.g. stated willingness to sign a petition) we administered after our experimental treatment that included UN endorsement. This suggests that UN endorsement causes respondents to update their views about an issue, and thus that our results might do more than just measure latent support for the institution. But given this concern with our research design, future research should test for the influence of UN endorsement on support for human rights using alternative experimental designs.

Third, although endorsement from the UN might increase support for reform in certain situations, more popular organizations or political actors may be able to generate more support than the international human rights apparatus. Moreover, we tested the effect of endorsement from the UN in isolation. It might be the case that endorsements from other groups opposed to reforms – say religious leaders – might have larger effects in the opposite direction.

Fourth, even if UN endorsement results in modest increases in support for human rights reforms, it is not clear that UN reports actually translate into improved human rights practices. As previously noted, the UN may endorse issues based on their political viability in a given country, and the reports often contain dozens of proposals. Some proposals included in the reports thus may be likely to pass regardless of their inclusion in UN reports, and other proposals included in reports may be drowned out by other more important more salient reforms. It would be fruitful for future research to do in-depth qualitative investigations of whether domestic groups actually take proposals from UN reports and use the UN endorsement to get traction for their passage.

That said, our results still make several important contributions to our understandings of the international human rights system. Most notably, they provide micro-level support for the argument that local groups in transitioning democracies may have an easier time organizing when they can draw on the legitimacy of the international human rights regime.

Additionally, our results suggest that top-down strategies may still play a role in human rights improvements. For instance, Bracic (2016) found that participants in a series of trust games conducted in Croatia and Slovenia discriminated against Roma regardless of whether they live in towns where top-down efforts to reduce discrimination exist, but that participants discriminated less when they were in towns where inclusive NGOS are active. While Bracic’s result suggests that top-down strategies may not be effective, our results suggest that top down-strategies – like UN endorsement – may still be important if they make it easier for local groups to organize.

Finally, our results suggest that the UN ability to drive reform may be limited to situations where the UN already enjoys support. As the WVS data reveal, in some developing countries, like Algeria, a relatively small percentage of the population already has confidence in the UN. But in other developing countries, like Kyrgyzstan, confidence in the UN is high. This suggests that UN endorsement of human rights reforms is likely to have heterogenous effects across countries.

Footnotes

Acknowledgements

We are especially thankful to Mudassar Aziz and Gulfam Ahmad for their feedback on initial versions of our survey and field materials, their help in preparation of the Urdu version of the survey, and their endless support in data collection in Pakistan. We would also like to thank Tom Ginsburg, Sarah Kreps, Katerina Linos, Anup Malani, Martha Nussbaum, Sunita Parikh, and Eric Posner for helpful comments and suggestions. This project also benefited from presentations at the Cornell Law School, Maastricht University, University of California–Berkeley School of Law, University of Chicago Law School, University of Hamburg, and the Conference on Empirical Legal Studies.

Funding

We would like to thank the Coase-Sandor Institute at the University of Chicago Law School for financial support.

ORCID iD

Adam Chilton

Notes

References

Arceneaux

Kevin

(2008) Can partisan cues diminish democratic accountability? Political Behavior 30(2): 139–160.

Ausderan

Jacob

(2014) How naming and shaming affects human rights perceptions in the shamed country. Journal of Peace Research 51(1): 81–95.

Berinsky

Adam J

(2009) In Time of War: Understanding American Public Opinion from World War II to Iraq. Chicago, IL: University of Chicago Press.

Blair

Graeme

Imai

Kosuke

Lyall

Jason

(2014) Comparing and combining list and endorsement experiments: Evidence from Afghanistan. American Journal of Political Science 58(4): 1043–1063.

Bracic

Ana

(2016) Reaching the individual: EU Accession, NGOs, and human rights. American Political Science Review 110(3): 530–546.

Bush

Sarah Sunn

Jamal

Amaney A

(2015) Anti-Americanism, authoritarian politics, and attitudes about women’s representation: Evidence from a survey experiment in Jordan. International Studies Quarterly 59(1): 34–45.

Chapman

Terrence

Chaudoin

Stephen

(forthcoming) Public reactions to international legal institutions: The ICC in a developing democracy. Journal of Politics. https://doi.org/10.1086/708338.

Chilton

Adam

(2014) The influence of international human rights agreements on public opinion: An experimental study. Chicago Journal of International Law 15(1): 110–137.

Chilton

Adam

Tingley

Dustin

(2013) Why the study of international law needs experiments. Columbia Journal of Transnational Law 52(1): 173–237.

10.

Cohen

Jacob

(1988) Statistical Power Analysis for the Behavioral Sciences. New Jersey: Lawrence Erlbaum Associates.

11.

Cole

Wade M

(2013) Government respect for gendered rights: The effect of the Convention on the Elimination of Discrimination Against Women on women’s rights outcomes, 1981–2004. International Studies Quarterly 57(2): 233–249.

12.

Conrad

Courtenay R

(2014) Divergent incentives for dictators: Domestic institutions and (international promises not to) torture. Journal of Conflict Resolution 58(1): 34–67.

13.

Conrad

Courtenay R

Ritter

Emily Hencken

(2013) Treaties, tenure, and torture: The conflicting domestic effects of international law. Journal of Politics 75(2): 397–409.

14.

Creamer

Coosette

Simmons

Beth A

(2015) Ratification, reporting and rights: Quality of participation in the Convention Against Torture. Human Rights Quarterly 37(3): 579–608.

15.

Dai

Xinyuan

(2002) Information systems in treaty regimes. World Politics 54(4): 405–436.

16.

Dai

XInyuan

(2005) Why comply? The domestic constituency mechanism. International Organization 59(2): 363–398.

17.

Davis

David R

Murdie

Amanda

Steinmetz

Coty Garnett

(2012) Makers and shakers: Human rights INGOs and public opinion. Human Rights Quarterly 34(1): 199–224.

18.

Downs

George W

Rocke

David M

Barsoom

Peter N

(1996) Is the good news about compliance good news about cooperation? International Organization 50(3): 379–406.

19.

Druckman

James N

Lupia

Arthur

(2000) Preference formation. Annual Review of Political Science 3: 1–24.

20.

Dutton

Yvonne

Dancy

Geoff

Aloyo

Eamon

Alleblas

Tessa

(2017) Collective identity, memories of violence, and belief in a biased International Criminal Court: Evidence from Kenya. Working paper (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3014844).

21.

Fang

Songying

(2008) The informational role of international institutions and domestic politics. American Journal of Political Science 52(2): 304–321.

22.

Fariss

Christopher J

(2014) Respect for human rights has improved over time: Modeling the changing standard of accountability. American Political Science Review 108(2): 297–318.

23.

Goodman

Ryan

Jinks

Derek

(2013) Socializing States: Promoting Human Rights Through International Law. New York: Oxford University Press.

24.

Hafner-Burton

Emilie M

(2013) Making Human Rights a Reality. Princeton, NJ: Princeton University Press.

25.

Hafner-Burton

Emilie M

Tsutsui

Kiyoteru

(2007) Justice lost! The failure of international human rights law to matter where needed most. Journal of Peace Research 44(4): 407–425.

26.

Hafner-Burton

Emilie M

LeVeck

Brad L

Victor

David G

(2016) How activists perceive the utility of international law. Journal of Politics 78(1): 167–180.

27.

Hansen

Ben B

Bowers

Jake

(2008) Covariate balance in simple, stratified and clustered comparative studies. Statistical Science 23(2): 219–236.

28.

Hertel

Shareen

Scruggs

Lyle

Patrick Heidkamp

(2009) Human rights and public opinion: From attitude to action. Political Science Quarterly 124(3): 443–459.

29.

Hill

Daniel W

(2010) Estimating the effects of human rights treaties on state behavior. Journal of Politics 72(4): 1161–1174.

30.

Hillebrecht

Courtney

(2014) Domestic Politics and International Human Rights Tribunals: The Problem of Compliance. New York: Cambridge University Press.

31.

Hopgood

Stephen

(2013) The End Times of Human Rights. Ithaca, NY: Cornell University Press.

32.

Kleinberg

Katja B

Fordham

Benjamin O

(2018) Don’t know much about foreign policy: Assessing the impact of ‘don’t know’ and ‘no opinion’ responses on inference about foreign policy attitudes. Foreign Policy Analysis 13(1): 429–448.

33.

Kreps

Sarah

Wallace

Geoffrey PR

(2016) International law, military effectiveness, and public support for drone strikes. Journal of Peace Research 53(6): 830–844.

34.

Linos

Katerina

(2011) Diffusion through democracy. American Journal of Political Science 55(3): 678–695.

35.

Linos

Katerina

(2013) The Democratic Foundations of Policy Diffusion: How Health, Family and Employment Law Spread Across Countries. New York: Oxford University Press.

36.

Lupu

Yonatan

(2013) Best evidence: The role of information in domestic judicial enforcement of international human rights agreements. International Organization 67(3): 469–503.

37.

Lupu

Yonatan

(2015) Legislative veto players and the effects of international human rights agreements. American Journal of Political Science 59(3): 578–594.

38.

Lupu

Yonathan

Wallace

Geoffrey PR

(forthcoming) Violence, nonviolence, and the effects of international human rights law. American Journal of Political Science. https://doi.org/10.1111/ajps.12416.

39.

Mansfield

Edward D

Milner

Helen V

Peter Rosendorff

(2002) Why democracies cooperate more: Electoral control and international trade agreements. International Organization 56(3): 477–513.

40.

McEntire

Kayla Jo

Leiby

Michele

Krain

Matthew

(2015) Human rights organizations as agents of change: An experimental examination of framing and micromobilization. American Political Science Review 109(3): 407–426.

41.

Mondak

Jeffery J

Davis

Belinda Creel

(2001) Asked and answered: Knowledge levels when we will not take ‘don’t know’ for an answer. Political Behavior 23(3): 199–224.

42.

Moyn

Samuel

(2010) The Last Utopia: Human Rights in History. Cambridge, MA: Harvard University Press.

43.

Neumayer

Eric

(2005) Do international human rights treaties improve respect for human rights? Journal of Conflict Resolution 49(6): 925–953.

44.

Nielsen

Richard A

(2013) Rewarding human rights? Selective aid sanctions against repressive states. International Studies Quarterly 57(4): 791–803.

45.

Posner

Eric

(2014) The Twilight of International Human Rights Law. New York: Oxford University Press.

46.

Posner

Eric A

Sykes

Alan O

(2013) The Economic Foundations of International Law. Cambridge, MA: Harvard University Press.

47.

Putnam

Tonya L

Shapiro

Jacob N

(2017) International law and voter preferences: The case of foreign human rights violations. Human Rights Review 18(3): 243–262.

48.

Richards

David L

Haglund

Jillienne

(2015) Violence Against Women and the Law. New York: Routledge.

49.

Sikkink

Kathryn

(2011) The Justice Cascade: How Human Rights Prosecutions Are Changing World Politics. New York: WW Norton.

50.

Simmons

Beth A

(2009) Mobilizing for Human Rights: International Law in Domestic Politics. New York: Cambridge University Press.

51.

Tomz

Michael

(2008) Reputation and the effect of international law on preferences and beliefs. Working paper (http://web.stanford.edu/∼tomz/working/Tomz-IntlLaw-2008-02-11a.pdf).

52.

von Stein

Jana

(2005) Do treaties constrain or screen? Selection bias and treaty compliance. American Political Science Review 99(4): 611–622.

53.

Wallace

Geoffrey PR

(2013) International law and public attitudes toward torture: An experimental study. International Organization 67(1): 105–140.

54.

Wallace

Geoffrey PR

(2014) Martial law? Military experience, international law, and support for torture. International Studies Quarterly 58(3): 501–514.