Evaluation

The last five years have witnessed the increasing use of evaluated pilot programmes as a way of developing health services policy in the UK. Total purchasing was the first major quasi-market development in the National Health Service to be independently evaluated from the outset. The initiative allowed local, volunteer pilots considerable freedom to implement extensions to existing arrangements for general practitioner budget holding for specialist services in the National Health Service. The experience indicates that future evaluations of similarly complex innovations should give attention to developing explanatory frameworks (i.e. theories), which include consideration of the impact of the context in which interventions are introduced on their potential outcomes. Such an approach should help in ensuring the generalizability of evaluations through theory building and thus increase their relevance for policy development. In addition, evaluations must be designed to be capable of accommodating the needs of changing policy imperatives if they are to have long-term usefulness.

The National Agency for Higher Education in Sweden carried out an evaluation, during 1995–1996, of all higher education programmes in care and nursing. The aim was to see if the programmes reached standards appropriate for higher education. Several care and nursing schools and programmes were found to be below the necessary standards, i.e. the criteria set by evaluators engaged in the national evaluation. According to those responsible for the programmes, the evaluation and its criteria were used at the local level to set out plans for development, even though the evaluation was criticized. Therefore in this article it is argued that national evaluations can work as national governing instruments, to support national policy. Two different theoretical frameworks are used to explain how actors at the local level are ‘governed’. Through this, it is believed that a more comprehensive understanding can be gained of how evaluation is related to governance.

It has been suggested that most evaluations only look at achievement of pre-determined goals. The Comprehensive Community-oriented Evaluation model discussed in this article attempts to broaden evaluation methodology making it more comprehensive. The model consists of three main components – design, process and impact. In order for results to be truly comprehensive, five evaluation strategy elements are necessary. These are: a mixed method, stakeholder involvement, feedback, audit and results consensus. The model was tested on an HIV/AIDS project in Zimbabwe, the conclusion being that the model is workable and has the potential to assist all stakeholders to produce more comprehensive evaluation results.

The focus of this article is on efforts undertaken within the United States government, especially the US Agency for International Development, to develop a coherent approach to determining the effectiveness of its democracy support programming. The main four priority areas are to support: civil society development; competitive political processes; the rule of law; and governance.

The article begins by asking the question of why this issue is important. It then provides context, and briefly assesses varying perspectives and motivations of different stakeholders. It examines and critiques current approaches being taken to ‘measure performance’ and concludes by suggesting some potentially useful policy and research directions. These include instituting structural reforms to enhance the independence of the evaluation process, disaggregating approaches to assessing the impact of programming, focussing on priority countries, and expanding use of integrative methodologies.

In a short pilot study we systematically involved healthcare consumers in identifying research topics, and prioritizing, commissioning and reporting evaluations of health technologies. We analysed records of consumers' suggestions for their content and, wherever possible, compared them with suggestions from academics and staff of the UK National Health Service. Consumers tended to highlight issues about patients' views, social contexts, information and support needs, long-term outcomes, and dissemination of research findings to consumers. These issues were usually not addressed by professionals who tended to focus more on technical and economic aspects of the work.

Consumers made unique contributions and brought fresh ideas and skills, which were readily incorporated, into the programme. Their direct involvement promises to better satisfy the information needs of those who use the NHS, and similar approaches may be applied to evaluating other public services.

Evaluators in the health sector struggle to develop effective mechanisms with which to evaluate healthcare reform programmes. Evaluation of these programmes is difficult because they are entangled in complex, inherently political processes and often shift away from their original aims and objectives as a result of policy changes within the health system. This article describes the evaluation of a healthcare reform process occurring in South Australia and discusses some of the methodological challenges encountered in evaluating a case study of continuous change with shifting policy objectives. It describes how the evaluators have attempted to address these challenges through a flexible and dynamic action-research approach. The article concludes by discussing the role of the evaluator, as intelligent observer, in reflecting on and documenting change in the highly political and complex field of healthcare reform.