Demand-side policy: Global evidence base and implementation patterns

Introduction

Demand-side management (DSM) refers to actions and programmes on the demand-side of energy meters that seek to manage energy demand in order to meet various policy objectives, such as carbon emissions reduction, energy security and fuel poverty reduction. ¹ DSM includes energy efficiency, demand response, on-site generation and on-site storage on the demand-side of meters, ¹ thus excluding actions undertaken at the distribution, transmission or generation levels. Although the policy side of DSM has received attention in the academic literature, the volume of studies is masked by the much greater focus on (non-government-stimulated) energy utility programmes, technological trials and modelling studies of the future potential of specific aspects of the demand-side, such as demand response or energy efficiency. Much of the research has focussed on North American experiences since the energy crises of the 1970s and European experiences since the 1990s when the environmental agenda became more prominent. More recently in the 2000s and 2010s, there has been an increase in studies from east-Asia, particularly China.

A global meta-evaluation of the evidence base for DSM policy is lacking in the literature, and this paper presents some of the results of a four-year project to systematically review the global evidence base for DSM policy. This paper focuses on answering the following research question in order to contribute towards filling the research gap:

What DSM policies have been implemented around the world with high-quality documented evaluations?

Policy process

Policy research can be divided into examining the policy process, policy design, policy implementation and policy evaluation.^2,3 In political science, there is a vast literature on policy evaluation and in the Stages Model of the policy process, outlined in Hogwood and Gunn ⁴ (p. 4), policy evaluation is the eighth stage in a nine-stage process, as shown below:

Deciding to decide

However, some authors, such as Hill ² (p. 143) and John ⁵ (p. 196) have criticised the Stages Model for being overly simplistic of how policy works in practice. Hill ² describes evaluating policy as a function of a controlled trial method (with a control group) or reaching desired states (such as a reduction in air pollution) (pp. 279–280). Nevertheless, in contrast to Hill, ² Nagel ³ defines policy evaluation as: ‘…evaluating alternative public policies, as contrasted to describing them or explaining why they exist’ (p. 133). Nagel ³ highlights a number of criteria for evaluating government policy: equity, validity, importance, usefulness, originality and feasibility (p. 134–136). He extends this to include the following broad terms: effectiveness, efficiency, public participation, predictable and procedural due process, and political feasibility (p. 92).

What is common in all three of the examples^2,4,5 is that there is an inherent acceptance that policy evaluation is an important part of the policy process. However, what they do not acknowledge is that policy evaluation is not as common in practice (particularly in some fields, such as energy policy) as would be expected in theory. As this paper demonstrates, thousands of DSM policies have been implemented around the world, but only 690 high-quality evaluations have been produced, the majority of which were conducted by industry rather than governments or academia.

Governments have a number of ‘tools’ at their disposal for the implementation of policies: ‘top-down’, which includes law and regulation; ‘internal to the state’, which includes bureaucracy and public management; and ‘non-standard’, which includes information, persuasion, deliberation, networks and governance (John, ⁶ p. 10). Historically, much of the focus in energy policy implementation has been on the role of tools that are top-down, such as regulations, or internal to the state, such as the use of existing and new institutions. ⁶ However, there has been a growing interest in the role that non-standard tools, such as persuasion and deliberation, can play in addition to traditional methods. Despite the usefulness of John’s ⁶ conceptual framework of policy ‘tools’, one criticism of its application to energy policy is through the use of the label ‘non-standard’ for some ‘tools’ that have been used as ‘standard’ policy approaches in the past, such as information campaigns and stakeholder networks.

Increased attention is being given to the role of information, persuasion and deliberation in policy implementation. For example, the UK Government set up a Behavioural Insights Team within its Cabinet Office in 2010 with the aim of applying the ‘nudge’ theory to areas of policy, such as energy efficiency, health, tax, consumer data access and organ donation. ⁷ The UK was the first country to set up a dedicated team within Government to apply behavioural theories to public policy (however, the Behavioural Insights Team was privatised in 2014 to become independent of government). The ‘nudge’ theory, developed by Thaler and Sunstein, ⁸ argues against the traditional neo-classical economics view that people always make decisions rationally under conditions of being fully informed through access to perfect information. Instead, it states that people make predictable mistakes based on their experiences (heuristics) and social interactions. Dawnay and Shah ⁹ expand this notion by listing key influences on people’s decisions:

Other people’s behaviour matters

The influence of peers, experiences, habits, social norms and the availability of (and ability to process) information are key characteristics of human behaviour ¹⁰ that need to be considered in the design and implementation of policies to encourage consumer engagement with DSM. As Gellings and Chamberlin ¹¹ note: ‘Research indicates energy-use behaviour and belief are resistant to change. Successful approaches to the consumer seem to be personal, possibly emotional, specific and narrow with concrete suggestions’ (p. 340). Hence, the traditional incorporation of rational choice theory in policy development is beginning to be replaced with a holistic policy framework that incorporates the importance of information, persuasion and deliberation. These arguments have particular relevance to the residential sector, but can similarly be applied to the non-residential sector. For example, competitors may represent credible ‘peers’ and hence an organisation is more likely to adopt certain measures if their competitors have done so (Gellings and Chamberlin, ¹¹ p. 334). A comprehensive review by Sorrell ¹² highlights that governments are increasingly using ideas from behavioural economics and social psychology to inform policy design for energy efficiency, though he argues that an effective policy approach would draw upon all disciplinary perspectives.

Hogwood and Gunn’s ⁴ Stages Model is useful in breaking down the policy process into sub-stages beyond simply design, implementation and evaluation. However, there are three important criticisms of the conceptual model: firstly, evaluation should take place at all stages of the policy process; secondly, post-policy evaluations should be used to inform the design of future policies; and thirdly, there is an inherent assumption that post-policy evaluations are actually conducted in practice. The next section applies these discussions to DSM policy.

Demand-side policy

DSM policy refers to all types of demand-side policies. However, in practice, much of the literature has focused on energy efficiency policy rather than policies for demand response, on-site generation or on-site storage. Figure 1 summarises the main types of demand-side policy, broken down into those that are market-based, regulatory, information-based or fiscal. This draws parallels to other categorisations in the literature, such as Grubb, ¹³ which categorises policies into: information-based tools, regulatory standards and financial incentives (p. 165). However, such categories have focused specifically on energy efficiency rather than the wider concept of DSM. A temporal critique of DSM definitions and categorisations since the 1970s is provided in Warren. ¹ Thus, Figure 1 aims to provide a broad, but comprehensive, overview of the range of demand-side policy types that have been implemented by governments in the past. As such, it is not a definition of DSM but a categorisation of DSM policy types. The figure is developed from the separate ‘Systematic Review’ reference list provided at the end of the paper, which is discussed in Methodology section.

OPEN IN VIEWER

Figure 1.

The main categories of demand-side management (DSM) policy.

Each of the policies in Figure 1 can be broken down further into specific policies. For example, incentive payment-based demand response (IPBDR) refers to incentives, such as direct financial payments, given to consumers to reduce energy consumption during peak periods. Specific types of IPBDR include the provision of regulatory and market frameworks to allow direct load control, interruptible/curtailable programmes, demand bidding, ancillary services market provision, emergency demand response and relieving network constraints in specific locations. ¹⁴ However, due to the global scale of this research project, the level of IPBDR is examined rather than specific policy sub-types, as the aim was to be as comprehensive as possible across all types of DSM within the resource constraints of the research. As discussed in the last section, further research with greater resources should utilise the same methodological approach (outlined in Methodology section) and examine the specific policy level, either within a specific category of DSM (energy efficiency, demand response or on-site generation/storage) or across the range of categories.

Evaluations of DSM policy are typically ex-ante, ex-post or a combination of both types. Ex-ante appraisals look at the expected effects of a policy (for example, ‘deemed’ energy savings and engineering estimates), whereas ex-post evaluations look at the empirical results of a policy (for example, measuring the actual impacts from monitoring studies). ¹⁵ Ex-ante approaches can only estimate in advance the potential effectiveness of a policy but are much less resource-intensive to carry out. In contrast, ex-post approaches require the measurement and monitoring of policy impacts, which increases the reliability of the estimates of the impacts but are more resource-intensive to carry out. Combined approaches are useful where ex-post evaluations feed into improving the accuracy of modelling tools in ex-ante appraisals. ¹⁶ However, as Stern and Vantzis ¹⁷ show, evaluators in North America tend to undertake ex-post approaches, whereas in Europe, evaluators more commonly conduct ex-ante approaches. Stern and Vantzis ¹⁷ argue that Europe can learn from the North American experiences in conducting ex-post evaluations to improve the reliability of policy evaluations.

In addition to the type of evaluation, who evaluates the policy is similarly important. Ex-post evaluations undertaken by governments can be politically damaging if they show a policy to have failed, and it is possible that biases could exist in such evaluations. As a result, there is arguably a role for independent third parties to undertake evaluations of DSM policies using a combination of ex-ante and ex-post methods.

In calculating energy savings in all types of DSM policy evaluation, it is crucial to consider the factors summarised in Figure 2, which represents the theoretically ‘perfect’ evaluation under conditions of abundant available resources. In practice, due to resource constraints, trade-offs are made as to what aspects of policy evaluation are examined. As with Figure 1, Figure 2 is developed from the separate ‘Systematic Review’ reference list provided at the end of the paper, which is discussed in Methodology section. The figure is broken down into: calculating the energy consumption baseline, making gross savings adjustments, attributing energy savings and monitoring and verification.

OPEN IN VIEWER

Figure 2.

Important considerations in demand-side policy evaluation.

The energy consumption baseline refers to how the consumption that would have occurred without the programme (the baseline) was calculated. Making gross savings adjustments looks at how adjustments to energy savings were made, such as the lifetime of the measures installed or the degree of rebound effects, where financial savings from reduced energy consumption can be used to increase energy use overall, either directly (through the same activities) or indirectly (through other energy-consuming activities).^18,19 The attribution of energy savings refers to, for example, whether or not the programmes resulted in additional savings to what would have occurred had they not been implemented (additionality), or the degree of free ridership, where consumers would have still engaged with DSM in the absence of the programmes. Monitoring and verification looks at the degree of monitoring to assess the actual impacts of the programmes and to ensure that energy savings are not double counted.

A key part of the research was to fill an important gap on the quality of the evidence base for DSM policy evaluation. In order to do this, the research applied systematic review techniques to assess evidence quality, as outlined in the next section. Furthermore, a comprehensive meta-evaluation of the global patterns of DSM policy implementation and evaluation is lacking in the literature. As discussed in Introduction section, much of the previous research on DSM has focused on (non-government-stimulated) energy utility programmes, technological trials and modelling studies of the future potential of specific categories of DSM, such as demand response, energy efficiency or on-site generation/storage, rather than DSM policy implementation and evaluation.

Methodology

To explore the global evidence base, the research employed a systematic review to extract the vast majority of the relevant, high-quality ex-post evaluations that have been conducted on demand-side policies. The spatial and temporal breadth of the systematic review were not pre-defined, only the data sources that would be used for the search. A literature review was first conducted to identify the main databases and websites that publish evaluations of DSM policy.

The research undertook a pragmatic, inductive approach to science and utilised a mixed-methods type of systematic review based on the principles of realist synthesis (see Pawson and Tilley ²⁰ and Pawson ²¹ for an overview of realist synthesis). Systematic reviews involve collating and synthesising all of the work that has been done on a particular intervention, trial or programme to better understand what works and what does not work. ²² Systematic reviews are a type of evidence review that is commonly confused with literature reviews. Evidence reviews are methods that aim to collect data and evidence for analysis using systematic techniques (such as detailed search strategies, inclusion criteria and quality assessments). In contrast, literature reviews aim to collate relevant studies on a particular topic using non-systematic techniques and appraise them in order to draw conclusions. ²³ A central aspect of this process is the identification of the key arguments and research gaps, and thus the purpose of literature reviews is not to collect data for analysis. ²³ Detailed comparisons of evidence reviews, literature reviews and different types of evidence reviews are provided in Warren. ²⁴

Systematic reviews have had limited attention in the energy and climate policy field in contrast to other disciplines, such as the medical sciences and some social science disciplines (e.g. education and social welfare), where they are an established method. Systematic reviews involve eight main stages as shown below:

Review questions

For the search strategy in stage three, a number of search terms were pilot tested to identify those that comprehensively returned the most relevant hits. In systematic reviews, a trade-off is usually made between the number of databases and sources included and the number of search terms used under resource and time constraints. The pilot tests identified that the breadth of sources was more important than an increased number of search terms in the context of the research question. One of the primary reasons for this is due to the sporadic distribution of the limited evidence base on ex-post DSM policy evaluation. A detailed discussion and justification of the search terms is provided in Warren. ²³

In stage four, it is crucial to identify what is considered relevant for answering the review question(s) determined in stage one. It also allows the review boundaries to be established based on the ability to access sources, the size of the review team, the time available to conduct the review and the budget dedicated to data collection. As such, the most commonly adopted sampling strategy for evidence reviews in the social sciences is purposive, non-probability sampling. This research employed the inclusion criteria shown in Table 1.

OPEN IN VIEWER

Table 1.

The systematic review inclusion and exclusion criteria.

Inclusion criteria

Documents that answer the research questions

Documents that discuss government-stimulated policies and programmes

Documents that are written in English

Documents that are ex-post policy evaluations

Documents that are freely accessible and downloadable from the internet

Exclusion criteria

Documents that look at DSM policy but not policy mechanisms and impacts

Documents that discuss utility-stimulated DSM programmes

Documents that discuss trials, pilots and small-scale R&D programmes

Documents that model the future potential of DSM

Documents that discuss theoretical aspects of DSM policy

Documents that are not written in English

Documents from hand searching

Documents from referrals

Documents from bibliographies and ‘snow-balling’

Stage five is the most important stage in the systematic review process, as it entails the assessment of the quality of the documents that are deemed relevant after the search has been conducted. There are numerous study quality assessment scales used in the medical sciences, where systematic reviews have had the most attention, but many are based on quality criteria that are less relevant for energy policy evaluations. For example, the Jadad Scale ²⁵ scores documents based on whether or not the studies were randomised, blinded and withdrawal rates were stated. In energy policy evaluations, studies rarely involve standardised randomised control trials or equivalent and are often undertaken in different ways using different methods. ²⁶

Systematic reviews have now become established in some social science disciplines, such as education and social welfare, as developed by the Campbell Collaboration ²⁷ and University-based research centres, such as the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) ²⁸ at University College London (UCL). However, the method has had limited attention in the energy policy field, though some collaborations, such as the UK Energy Research Centre (UKERC)’s Technology and Policy Assessment (TPA) theme ²⁹ and the Collaboration for Environmental Evidence (CEE), ³⁰ are trying to fill this gap.

In the social sciences that use systematic reviews, there are number of study quality assessment scales that have been developed and tailored to answer specific research questions. For example, the TAPUPAS framework, as developed by Pawson et al., ³¹ was developed primarily to address research questions in social care. The framework focuses on transparency (are the reasons for the study clear?), accuracy (is the study honestly based on relevant evidence?), Purposivity (is the method used suitable for the aims of the study?), utility (does the study provide answers to the questions it set?), propriety (is the study legal and ethical?), accessibility (can you understand the study?), and specificity (does the study meet the quality standards already used for this type of knowledge?).

However, the TAPUPAS framework has arguably wider applications than social care, as the quality criteria are broad enough to be relevant to other social science disciplines. Despite this, this research developed a new, more tailored scale that can be used to evaluate the quality of evidence in policy evaluations. By tailoring the scale to policy evaluation, a deeper understanding of the quality of the evidence base for ex-post demand-side policy evaluations could be obtained. The Warren scale was discussed and justified in Warren, ²⁶ but the scale is shown in full in Figure 3.

OPEN IN VIEWER

Figure 3.

The Warren scale for assessing quality in energy policy evaluations.

Documents must score at least half of the available points (i.e. 7/14 points) to be included in the final sample. It is common in the systematic review literature to have scales with thresholds of half or two-thirds of the total number of points. An additional criteria is that at least one point must come from either of the two most heavily weighted parts of the scale (the processes for implementation and evaluation) and at least one point must come from another part of the scale.

Once the final sample of high-quality documents has been established, stage six involves the extraction of relevant data and information from the documents. A summary of the quantitative metrics that were extracted in order to answer the research question is shown below:

Evidence quality:

These metrics form the basis of the discussions in the next section. All extracted data were stored in a standardised spreadsheet-based analytical database. The full review protocol outlined in this section was pilot tested with two other researchers using two search terms in two databases (discussed further in Warren ²³ ). This ensured that the review protocol was robust and could be replicated.

The next section presents and discusses the results from the systematic review. Discussions centre around four main areas of analysis in order to determine the global evidence base and implementation patterns for demand-side policy: the overall patterns of evidence (in terms of data sources, DSM policies and countries/states), policy packages, temporal patterns and DSM policy objectives.

Results

Overall patterns

Four stages of document filtering were conducted: initial hits in the data sources, hits after title skimming, hits after abstract skimming and applying the inclusion/exclusion criteria, and the final sample after the study quality assessment was applied. The data sources that were consulted are listed in Table 2. These sources were determined in the literature review that was conducted prior to the systematic review (as discussed in Methodology section) as the databases and websites that are more prominent in publishing evaluations of DSM. The number of initial hits and the number of documents in the final sample are given for each data source.

OPEN IN VIEWER

Table 2.

The databases and sources included in the systematic review.

Database name	Final sample	Initial hits
Energy Policy	1	549
Energy	5	194
Energy Efficiency	8	44
Energy Economics	1	35
The Electricity Journal	8	308
The Energy Journal	0	0
Electric Power Systems Research	0	18
Energy and Buildings	1	35
Resource and Energy Economics	1	20
Open Grey	0	1
Google (Literature Review)	5	58
American Council for an Energy-Efficient Economy (ACEEE)	12	199
European Council for an Energy-Efficient Economy (ECEEE)	14	61
IEA’s Demand-Side Management Programme	4	159
Association for the Conservation of Energy (ACE)	1	7
National Grid	0	3
International Energy Program Evaluation Conference (IEPEC)	21	102
International Partnership for Energy Efficiency Cooperation (IPEEC)	0	0
British Association for Energy Economics (BIEE)	1	2
International Association for Energy Economics (IAEE)	1	120
US Department of Energy (DoE)	0	46
US DoE Energy Citations Database	8	332
US Federal Energy Regulatory Commission (FERC)	0	34
US Energy Information Administration (EIA)	0	83
UK Department of Energy and Climate Change (DECC)	4	664
UK Office of Gas and Electricity Markets (Ofgem)	0	3
UK National Audit Office (NAO)	0	1
UK Public Accounts Committee for the House of Commons (PAC)	0	7
China National Development and Reform Commission (NDRC)	0	0
Australia Energy Regulator (AER)	0	600
Australia Department of Industry	0	7
California Public Utilities Commission (CPUC)	1	401
European Commission Department of Energy	1	150
Thomas Reuters Web of Science	4	79
Across 34 databases	102	4322

What is clear is that the sources that appear to produce the most initial hits are not necessarily the same sources that produce the most number of high-quality ex-post DSM policy evaluations. The databases and websites that produce ≥200 initial hits are: DECC, AER, Energy Policy, CPUC, DoE Energy Citations Database and The Electricity Journal. However, the sources that have the most number of documents in the final sample are: IEPEC, ECEEE, ACEEE, DoE Energy Citations Database, The Electricity Journal and Energy Efficiency. Thus, only two sources (DoE Energy Citations Database and The Electricity Journal ) overlap between the groups. These six sources represent the databases with the highest overall evidence quality for ex-post DSM policy evaluations.

The total final sample size was 102 high-quality documents, which covered 690 ex-post evaluations (from the initial total hits of 4322 documents). The number of documents did not equal the number of evaluations due to some documents including evaluations of more than one policy and country/state. Sub-national states and provinces also appeared in the sample, as some state-level governments have implemented demand-side policies independent of national governments (such as in the USA, China, Australia and Canada). There is a large body of evidence at the sub-national state and provincial level in some countries (particularly in the USA), and excluding their experiences would limit not only the sample size but also the accuracy and comprehensiveness of the global synthesis.

The evidence base is dominated by industry (59/102 documents), followed by academia (29/102 documents) and then government (14/102 documents). An important finding here is that globally, few high-quality ex-post evaluations of DSM policies have been conducted by governments. As discussed in Policy process section, a number of academic discussions on the policy process make inherent assumptions that post-policy evaluation is an active part of the policy process for governments. In practice, the picture is much more mixed with regards to DSM policy. For example, the DoE Energy Citations Database includes primarily documents written by industry and academia, and high-quality ex-post evaluations produced by the US Department of Energy (DoE) appear limited. However, relative to other countries, DoE and the former UK Department of Energy and Climate Change (DECC) (now the Department for Business, Energy and Industrial Strategy (BEIS)) have been the more prominent government departments in evaluating demand-side policies.

In terms of DSM policy types, alternative utility business models (such as the experiences in the USA at a state-level with decoupling policies and performance targets) (122/690 evaluations), information campaigns (including energy auditing) (118/690 evaluations), loans and subsidies (including tax incentives) (100/690 evaluations), utility obligations (89/690 evaluations) and performance standards (for appliances, equipment and buildings) (81/690 evaluations) have been the most frequently implemented and evaluated policies. In contrast, voluntary programmes (12/690 evaluations) and infrastructure rollouts (primarily smart meter rollouts) (4/690 evaluations) had the least number of policy evaluations. Despite this, the evidence base for infrastructure rollouts is likely to increase post-2020 following the implementation of the European Union (EU)’s Smart Meter Rollout Directive (Directive 2009/72/EC), ³² which mandates that all countries in the EU must reach at least an 80% rollout of smart meters to small energy consumers by 2020.

The evidence base is greatest in the USA (25/102), the UK (22/102) and the US state of California (20/102). France (13/102), China (12/102), Denmark (10/102), Italy (10/102), the US state of New York (9/102) and Germany (8/102) also perform well by the frequency of high-quality evaluations.

Overall, 30 countries and 36 sub-national states have implemented at least one DSM policy and produced at least one high-quality ex-post evaluation. At a state-level, US states dominate the sample with 103/690 evaluations, 18/690 evaluations cover Chinese provinces, 13/690 evaluations cover Australian states, 4/690 evaluations cover a Belgian region (Flanders), 3/690 evaluations cover Canadian provinces and 1/690 evaluations covers an Indian state (Orissa).

Policy objectives

DSM policies are implemented for a variety of reasons, such as to ensure energy security, to enhance economic productivity (through energy efficiency and new business opportunities), to reduce carbon emissions and to reduce consumer energy bills. The systematic review extracted the primary policy objectives from each policy evaluation (as stated by the evaluators) and examined their frequency of discussion across DSM policies and countries/states in the sample. Figure 4 presents the overall results.

OPEN IN VIEWER

Figure 4.

The primary objectives for DSM policy implementation.

Where there were a number of evaluations within each document, policy objectives for each policy or country/state were noted (where stated by the evaluators). Overall, the top four reasons for countries/states to implement DSM policies are: to reduce carbon emissions (47 evaluations), to ensure energy security (41 evaluations), to increase the uptake of energy efficient technologies (36 evaluations) and to create new resources and markets (33 evaluations).

The first policy objective refers to all aspects of environmental improvement except for local air pollution reduction. However, all 47 evaluations referred to the reduction in carbon emissions in some form. The second policy objective refers to the balance of supply and demand primarily at a national or state-level, though in one evaluation, it also referred to improving on-site energy security in buildings. Peak load reduction was commonly mentioned in the 41 evaluations and is considered part of the ‘energy security’ policy objective rather than the ‘reduce energy use’ policy objective. This is due to the latter referring to an overall reduction in energy use rather than load shifting. The third policy objective refers to a range of specific goals, such as to increase the uptake of certain technologies by consumers, to improve the national/state building fabric and value through retrofitting, and to improve the efficient use of energy across the country/state. The fourth policy objective refers to the creation of new markets for DSM, to introduce the demand-side as a resource on an equal basis to supply-side options and to stimulate investment in DSM.

Although carbon emissions reduction and energy security appear to be the main policy objectives for DSM in most countries/states, there is some spatial diversity between continents. In North America, enhanced productivity through energy efficiency and new markets has been a strong driver in recent decades. For example, the American Recovery and Reinvestment Act (ARRA) of 2009 devoted significant financial support to DSM, which led to rapid job creation and retention as well as a reduction in energy use and greenhouse gas emissions. ³⁹

In Europe, carbon emissions reduction and reducing consumer energy bills have become crucial drivers for DSM. For example, a number of the utility obligations in Europe (such as in the UK, Italy, France and Denmark) are aimed at reducing carbon emissions by meeting energy savings targets and have sub-targets for low-income groups to contribute to reducing fuel poverty.^40–42 Fuel poverty refers to consumers living on a low income in a home that cannot be kept warm at a reasonable cost, as defined in the UK’s Warm Homes and Energy Conservation Act 2000 ⁴³ and reviewed in the Hills’ Fuel Poverty Review. ⁴⁴ In Asia, particularly China, reducing local air pollution and preventing black-outs during crises have greatly expanded the development of DSM in recent years. The systematic review shows that much of the rapid growth in DSM has come from China (for example, see Bin and Jun ⁴⁵ ; Pengcheng et al. ⁴⁶ ; and Zheng et al. ⁴⁷ ). The three documents that mentioned reducing local air pollution as a key policy objective were based on China. Preventing black-outs falls into the ‘energy security’ policy objective in Figure 4 and was mentioned in the majority of the evaluations that focussed on China, as DSM is viewed as an important contributor to meeting the rapid growth in energy demand in the country.

Conclusion

The paper presented some of the results of a four-year project to systematically review the global evidence base for demand-side management (DSM) policy. A comprehensive meta-evaluation of the evidence base across the full range of DSM policy types is lacking in the literature and the paper aimed to contribute to filling this gap by answering the following research question: What DSM policies have been implemented around the world with high-quality documented evaluations? The research employed a systematic review of the evidence base and identified 102 high-quality documents covering 690 ex-post policy evaluations from 34 databases and sources (from an initial sample of 4322 documents). The documents covered 30 countries and 36 sub-national states across six continents, and twelve individual types of DSM policy and nine DSM policy packages. Spatially, the USA, the UK, California, France and China have the greatest number of high-quality documents in the final sample.

The findings show that the top-five most frequently implemented and evaluated DSM policies are alternative utility business models (such as decoupling policies and performance targets), information campaigns, loans and subsidies, utility obligations, and performance standards. The primary policy objective for DSM is to reduce carbon emissions, followed by ensuring energy security, increasing the uptake of energy efficient technologies, and creating new resources and markets. It is clear that individual DSM policies are more commonly implemented and evaluated than DSM policy packages, as just 45/690 evaluations focussed on policy packages. Of these, information campaigns in combination with loans and subsidies, and performance standards in combination with labelling, are the more commonly implemented and evaluated policy packages. Although temporal patterns were not the central focus of the analysis due to the potential digitisation bias for evaluations, the results show that the number of high-quality ex-post evaluations have increased over time since the energy crises of the 1970s. Overall, this paper argues that demand-side policy will play an increasingly important role as a complement to low carbon activities on the supply-side in the transition to a more environmentally sustainable energy system.

With greater time and resources, the systematic review could be extended to include non-English documents (translators would be required), hand searching for non-electronic documents (access to specified libraries and organisations would be required), snowballing and referrals. In particular, this would allow more detailed temporal analysis to be conducted. Furthermore, future research with greater resources should utilise the same methodological approach and examine the specific policy level (for example, energy audits rather than information campaigns), either within a specific category of DSM (energy efficiency, demand response or on-site generation/storage) or across the range of categories.

From the analysis, it is clear that the evidence base for high-quality ex-post demand-side policy evaluations is limited with most evaluations being produced by industry rather than governments or academia. Thus, there is a need to encourage evaluation practices in governments to ensure that enough resources are dedicated to post-policy evaluation in order to improve the design and implementation of future DSM policies. Such an approach should aim to establish institutional learning beyond lessons learned to use robust ex-post evaluation methods to inform future ex-ante appraisals and pre-policy impact assessments. Despite this, the potential political bias of governments not wanting to publicly evaluate and declare policies that did not perform as well as originally anticipated remains an important challenge that warrants further attention. Overall, the evaluation practices published in sources such as the International Energy Program Evaluation Conference (IEPEC), the European Council for an Energy-Efficient Economy (ECEEE), the American Council for an Energy-Efficient Economy (ACEEE), the US Department of Energy (DoE)’s Energy Citations Database, The Electricity Journal and Energy Efficiency are good role models for the production of high-quality DSM evaluations.