Abstract
Ajmal Baig, CEO of Semiotics Consultants, a development consulting firm based in Islamabad, Pakistan is in the midst of heated debates with his team members on the evolution and performance of their currently ongoing project on education in the province of Punjab. The project, ‘Collaborative Education Project (CEP)’, is an GDA funded project for which a grant of Euro 10 million was allocated. The project was launched in 2003 and is now in its mid-term stages. During the life of the project, its Logical Framework Analysis (LFA) has been revised four times in the light of feedback from stakeholders and experience on the ground. The project team at Semiotics is reviewing and debating this journey. A robust and well thought out LFA is the starting point for any good program design and it also serves as its monitoring and evaluation tool. However, learning organisations expect that changes will be made based on ground realities and changing circumstances leading to revisions in the LFA. The key question is how much deviation from the original design is kosher. Too little and too much flexibility both have the potential to adversely impact the project outcomes.
Discussion Questions
How have development interventions evolved in developing countries?
How do you like the original design of CEP as depicted in the first LFA? What elements do you agree or disagree with? Why?
How and why has the LFA of the project evolved? What are your observations on the changes? Have these changes helped or hurt the project?
How much change in an LFA is kosher?
What lessons can be learnt for the design and implementation of projects from the CEP experience?
On 12 January 2007, Ajmal Baig, CEO of Semiotics Consultants (henceforth called Semiotics), a development consulting firm based in Islamabad, Pakistan, was in the midst of heated debates with his team members on the scope and progress of their ongoing project on education in the province of Punjab. This project, called the Collaborative Education Project (CEP), was launched in 2003 in a collaboration between the Government of Pakistan (GoP) and the Global Development Agency (GDA), headquartered in Vienna. Semiotics was the local executing agency of the project. Ajmal and his team had worked hard with managers from GDA as well as stakeholders from the GoP and various Pakistani educational institutions to make the project a success. They had faced many obstacles and obtained interesting insights along the way, which had resulted in various changes in the project design and delivery. This had led to the development of four logical frameworks (LFA) for the project over some years.
The debate was triggered when Ajmal and his team were informed that a senior manager from GDA was arriving in late January to review the project.
Ajmal moaned:
The tragedy is that this manager is new to the job. Consequently, she will revisit issues of scope, scale and budgets that had been debated so many times with earlier managers over the last four years. Each manager has to develop their own understanding and comfort level of the project.
The argument was regarding the frequent revisions of the LFA and their budgetary and utility impacts. Tahira Mughal, Monitoring and Evaluation Manager at Semiotics was apprehensive. She was accountable to donors for raising any red flags well in time.
She interjected:
If we use the first LFA to assess our results so far, then we seem to fall far short. On the other hand, if we use the final LFA we would come out as star performers. While I do not mind being a star, I am just wondering whether we are all fooling ourselves. We can always reconstruct any project’s LFA to completely match it to the obtained results. This will allow us to show fantastic results all the time. What is the point of wasting time in the development of a so-called ‘rigorous’ project design with detailed budgets if we keep on changing the goal posts?
Ajmal laughed:
You are being too philosophical. You need to spend more time in the field, Tahira. The real world gives you many surprises and the best of plans need to be changed in the face of new data. Frankly speaking, what is more important at the end of the day, the delivery of a good project or staying stuck with the original LFA?
Semiotics Consultants (Pvt.) Limited
Semiotics Consultants (Pvt.) Limited was established in 1988. It was an Islamabad-based research and consultancy firm which specialised in socio-economic development solutions, capacity-building, institutional development and strategic management. Its areas of activities included rural development, environmental planning and impact assessment, evaluation of public sector policies and projects, education and training, women in development, health, population welfare, research planning and management and private sector development. By 2007, it had successfully completed over 200 projects. Semiotics had worked with an impressive range of international clients, such as the Asian Development Bank (ADB), International Labour Organization (ILO), Canadian International Development Agency (CIDA) and the Department for International Development (DFID). Among local clients were various Pakistani government departments at federal/provincial/district levels and local NGOs.
Ajmal Baig, CEO of Semiotics Consultants had worked in the social and development sector of Pakistan for almost 25 years. He had a Masters in Development Economics. Semiotics had a core group of 10 professionals and had access to many professionals and organizations all over Pakistan, who worked as subcontractors on various projects.
Development Interventions in the Third World Countries
The period from 1945, the end of the Second World War, to the late seventies, witnessed significant changes in the politico-economic scene of the world. Many countries obtained independence in Asia and Africa. However, they were still dependent on their colonial masters for establishing and running governance structures and systems. In addition, financial, human and technical support was badly needed in many of these poor countries. The Cold War between the Soviet Union and the United States of America led to a race to enlist as many countries as possible within ones’ umbrella of influence. Provision of development assistance to under-developed or developing countries became an important instrument in this scramble. The United States of America and its rich allies like Germany, Japan, Canada and the United Kingdom poured in large sums of money in the shape of foreign aid into the poorer countries for humanitarian assistance or various socio-economic development initiatives. Multilateral agencies like the World Bank, Asian Development Bank, and UNDP were supplemented by large donor agencies, such as USAID, CIDA and DFID to channelize financial and technical assistance in support of socio-economic development.
By the mid-eighties of the last millennium, political leaders in the developed world, such as Ronald Reagan and Margaret Thatcher, challenged the existing economic paradigms and strongly advocated neoliberalism. This resulted in severely reduced government spending in the developed world itself, and greater reliance on free-market economic worldview. By this time, development practitioners had also got disillusioned by the results obtained from various development interventions undertaken over the many decades. While billions of dollars were poured into poor countries in Asia, Africa and South America, many countries, especially those in Africa, showed limited impact. Rather, many of these countries seemed to be becoming even more dependent on foreign assistance. On the other hand, newly industrialized countries like Korea, Malaysia and Singapore had demonstrated a different route to dev-elopment and self-sufficiency.
Development experts and political leaders demanded better transparency, accountability and results from development project managers and donors, alike. There was strong criticism that many development initiatives failed as they were conceptualized and designed by experts who had little understanding of the ground realities, they were not owned by the recipient country managers or target beneficiaries and focused too much on activities rather than results. As a result of all this, experience development world was introduced to the theory of change framework, which clearly outlined the underlying theories behind any development intervention linking explicit causes, actions to desired effects. Tools like Results-Based Management or Logical Framework Analysis were introduced in order to improve the design and monitoring of development projects.
A different kind of criticism of these development interventions was from opinion leaders who believed that these were simply tools to continue the dominance of the developed world. The donors were seen as peddling their own agendas for ulterior motives. Critics would point out that while many donors were pushing for democracy, human rights, and improving the economic and social conditions of the poor, the same countries were fully supporting ruthless dictatorships in many countries and leading globalization agendas of corporations which would ultimately lead to further deprivation of the poor.
The efforts to improve aid effectiveness culminated in a meeting in Paris in 2005 where over one hundred participants from developing countries and donors agreed on a declaration (called Paris Accord). Following five key principles were agreed upon:
Managing for Results: Partner countries would establish result-oriented reporting and assessment frameworks to monitor and evaluate national and sector development strategies. Mutual Accountability: Partner countries would reinforce participatory approaches by systematically involving a broad range of development partners when formulating and assessing progress in implementing national development strategies. Harmonization: Donors would implement, where feasible, common arrangements for monitoring and evaluation. Alignment: Donors would use a country’s own institutions and systems, where these provide assurance that aid will be used for agreed purposes, in order to increase aid effectiveness. Ownership: Partner countries would exercise leadership in developing and implementing their national development strategies and donors would respect partner country’s leadership and help strengthen their capacity to exercise it.
This new thinking and approach towards development assistance were also shaping how development work was being carried out in Pakistan.
Primary and Secondary Education in Pakistan
Pakistan’s performance in the education sector, in general, and primary and secondary education, in particular, had not been up to the mark. Lack of policymakers’ motivation leading to low levels of public sector investment in education, typically hovering around 2% of GDP, was primarily blamed for poor educational outcomes. Pakistan’s relative performance in the provision of basic education when compared with those of similar countries was clearly deficient (Figure 1). Many studies on the status of education reported problems of access, quality and equity. They also revealed gender and geographical disparities. Thus, the figures for enrolment and completion rates for girls, as compared to those of boys, were significantly lower in all age brackets, all over Pakistan. Similarly, performance on a variety of education indicators was poorer in provinces like Baluchistan and Sindh than those of Punjab. Even within Punjab, there were disparities among more developed and less developed districts. Overall, Pakistan had the highest population of out-of-school children in the South Asian region.
The Government of Pakistan had, over the years, made several commitments and plans for improving the educational outcomes. Commitments were made for Millennium Development Goals and in ‘Education For All’ in 2000 at the World Education Forum. National Education Policy 1998–2010 envisaged improvement in the quality of administration and teaching in the schools. An Education Sector Reforms Action Plan was launched in 2001 in various provinces with the technical and financial assistance of different donors and development agencies. This programme addressed the issues of educational attainments, lack of access and gender- and location-based inequities in the country.
There had been some changes in the public sector management of education on account of constitutional reforms in the early decade of the new millennium. According to the 1973 Constitution of Pakistan, the Federal Government was responsible for policy, planning and promotion of educational facilities in the federating units. While education was a provincial subject, the Federal Ministry of Education was responsible for curriculum and textbook development and for tertiary education. The devolution plans of General Musharraf and the consequent Local Government Ordinance in 2001 attempted to devolve planning and management of primary and secondary education to the districts within provinces. Its effective implementation was facing administrative challenges and was in progress in 2007.
Global Development Agency (GDA)’s Plans for Education
The Global Development Agency (GDA) was a consortium of leading developed countries established with the idea of better harmonization among these countries in administering foreign aid for development assistance. In 2000, the GDA decided to focus on a fewer group of countries for development assistance. Consequently, a major portion of GDA resources was dedicated to 20 countries of focus, which were chosen based on their needs and their capacity to benefit from aid. Pakistan was one of the selected countries.
The GDA had been involved in development assistance in Pakistan for many years. It had extended financial and technical assistance in many areas such as health, poverty reduction and education. In education, the GDA had completed various projects aimed at all, primary to tertiary levels of education. However, top management at GDA felt that a focus on improving accessibility, quality and affordability of primary education was critical for the development of Pakistan. Consequently, in 2003 an initiative for the improvement of primary education was launched under the banner of Collaborative Education Project (CEP). This project was expected to dovetail with the Education Sector Reforms Action Plan. The preliminary work on the project conceptualization was done by domain experts flown in from Vienna, in consultation with local sector specialists of GDA and a few Pakistani experts. CEP was budgeted at Euro 10 million and was expected to be delivered over five years.
Request for Proposal for CEP and the First Logical Framework (LFA)
A GDA advertisement in Pakistani newspapers called for services of Pakistani consultants (firm, agency, institution, NGO or consortium) with expertise in the social sector, specifically in education, to submit proposals for a project for basic education in Pakistan. The goal of the project was to ‘improve the quality and delivery of education in Pakistan, especially for the female population, and to increase access to education by the poor’. The Request for Proposal (RFP) for CEP outlined some basic parameters of the project, including a proposed theory of change as depicted in a tentative logical framework. However, the selected consultants were allowed flexibility to refine the scope and design of the project based on local realities and inputs from key stakeholders. The RFP suggested Punjab (the most populated province of Pakistan) as an important target for intervention.
According to the preliminary logical frameworks (LFA), the project had four major components, namely policy development, institutional development, teacher-training and community mobilization (Table 1). Specific activities expected results and indicators within these components were to be defined by the selected consultant in co-operation with local partners such as the Ministry of Education in the Federal Government, the Department of Education in the Punjab Government and the district officials in selected districts. The tentative budgets were in line with the scope and scale of the work envisaged.
Logical Framework 1
The project was to proceed in two stages under a phased contract with a (1) design and appraisal phase and (2) an implementation phase. The hired firm was to carry out a detailed design for the project. If the design was accepted by the GDA, then the selected firm could also be requested to manage the implementation of the project.
For Ajmal and Semiotics, this was an exciting opportunity. They had not undertaken a project of such a huge scope and budgets so far. To be able to successfully complete such a project would be a major achievement and stride forward for his firm. Ajmal realized that he would be facing a tough competition from other, larger firms who would be bidding. He understood that Semiotics’ proposal had to be very good.
28 organizations from across Pakistan submitted their proposals and ultimately Semiotics won the contract for the first phase of the project. Semiotics was required to collaborate with their local partners to clarify and revise if necessary, any aspects of the preliminary design. This phase had to be completed within 4 months, at the end of which GDA required a project implementation plan (PIP), and an appraisal report.
Development of the Second LFA
With the contract in hand, Ajmal quickly put together a team of specialists to begin work on the design of the project. His design team included experts from the fields of capacity-building, teacher-training, gender integration and community mobilization. Team members had the experience of working in the field for a number of years. Detailed and frequent meetings were held with managers from GDA to understand and clarify various objectives and activities. The first red flag raised by the Semiotics design team was on the overall budget and the scope (both activities-wise and geographical) of the project. They felt that the scope of the project was too ambitious for the limited budget. It was unrealistic to cover 8–10 districts (as envisaged by GDA managers) of Punjab. They suggested narrowing the geographical scope down to only two districts.
The next issue was picking the targeted districts for intervention. The basic idea was to pair a district historically weak in educational outcomes with a strong district. The first two names discussed were Kasur and Lodhran. Kasur was located in the north of Punjab, while Lodhran was in the south. Kasur, in close proximity of Lahore, was a strong district in many areas of development. In addition, the GDA already had some presence in Kasur with their devolution support programmes. Lodhran was among the districts with the worst educational outcomes. The team felt that this selection would, not only, enable learning in two educationally different environments, but also reduce any political criticism of lack of equity in the project. Further discussions within the team; however, led to the dropping of Kasur in favour of Multan. Multan was picked due to its close proximity to Lodhran. Multan had all the characteristics of a strong district and its selection allowed more effective and less expensive administration of the project. Additionally, it had a teacher-training college where students from nearby districts of Bahawalpur and Lodhran obtained their training.
Under the component of teacher-training in the preliminary LFA, one of the outputs was ‘Enhanced processes for gender-sensitive curriculum review, development and implementation within teacher-training facilities’. The GDA was of the view that teacher-training curriculum was weak, gender-in‑ sen-sitive and not appropriately linked with the school curriculum. Hence, there was an urgent need for improved curriculum design and development. Semiotics design team knew quite well that the undertaking of any curriculum reform in Pakistan was complicated and politically extremely sensitive. The Education Ministry at the Federal Government had attempted curriculum reforms a couple of times in recent past but had to back down both times in the face of severe opposition, especially from the more conservative strata of society, including media, civil society organizations and political parties. The team informed the GDA that the Governmental institutions would not be comfortable in supporting such an initiative especially one that is being funded through an international agency. The team proposed that instead of getting into the job of redesigning the curriculum, the CEP could invest in strengthening the capacity of the key institution that has the responsibility of developing curriculum; that is, the curriculum wing in the Ministry of Education, Islamabad.
Another area that needed major rethinking was again within the component of teacher-training. The preliminary LFA proposed ‘Improved competency of teachers in primary curriculum content area’. During initial investigations with other stakeholders, Semiotics met with managers in the Department of Education in the Government of Punjab. They found out that the Punjab Government had a contract with Agha Khan Education Network (a famous NGO) under which government teachers of Punjab were already receiving training for improved teaching of mathematics and science. The education officials suggested that they would be more interested in obtaining GDA’s assistance for a more challenging area; that is, English language teaching (ELT). Semiotics decided to incorporate this new development into their design.
As discussions with stakeholders and specialists progressed, a few other changes (i.e., role of school management committees) were suggested in order to make the project viable and useful. Ajmal was getting very apprehensive during all these changes as he was unsure about the GDA’s reaction. While all the changes were being discussed with the GDA staff and explanations offered, Ajmal was worried that GDA might make a significant cut in the budgets and timelines in the face of drastically reduced district coverage and other changes. Too much divergence could lead to a negative reaction from GDA specialists in Vienna.
All this work, which was very extensive, also took its toll on timelines. Semiotics could only complete the revised design tasks in nine months. The final Semiotics proposed LFA was submitted to GDA in mid-2004 for approval (Table 2).
Logical Framework 2
Development of the Third LFA
The second LFA was incomplete in the sense that it did not have detailed outlines of all the expected activities and related performance indicators but was focused primarily on defining the scope of the project. It was Semiotics’ version of what the project should be about; given the ground realities and feedback from local partners. Fortunately, for Ajmal, GDA agreed with all of the suggestions including the reduction of district coverage to only two, Multan and Lodhran. He was relieved to find out that no budget and time revisions were made. Rather, he felt that the GDA management had a very open and participatory attitude and was not keen on dictating their vision.
The next step was to chalk out the details of the activities in the agreed upon components of the pro-ject in order to achieve the desired results. Teams from the GDA and Semiotics engaged in extensive second rounds of discussions to flesh out the project. Gap analysis and need assessments of targeted institutions were completed in order to design appropriate interventions. Detailed plans were made for each component of the project and they were fervently discussed and argued upon.
The first component was ‘institutional development’, and both parties agreed on the following tasks:
Strengthening of Curriculum Wing, Federal Ministry of Education Strengthening of Directorate of Staff Development (DSD), and Education Department of Government of Punjab. Support for the University of Education (UoE), Punjab in developing a pre-service teacher education programme Strengthening of District and Tehsil Level Education Administration of Multan and Lodhran.
GDA relented to not getting involved in curriculum design and development and instead moved over to strengthening the capacity of the curriculum wing. Similarly, it gave up on its plan for actual teacher-training and decided instead to help build the capacity of the teacher-training institutes, that is, DSD and UoE. It was felt that this will bring about a more sustainable approach to improvements in teaching. Semiotics had proposed an ELT intervention but the GDA was hesitant to invest in English language training. The GDA’s position was that in the face of pretty dismal literacy rates in most developing countries, putting scarce money behind the teaching of a ‘second’ language did not make a lot of sense. This was difficult to sell to donor governments.
The second component of the project was ‘teacher education’ and the final agreed upon tasks were as follows:
Strengthening of University Colleges of Education (UCEs) in Multan and Federal College of Education (FCE) in Islamabad. Improvement of practicum in UCEs and FCE. Strengthening of UoE Internship Program.
The remaining two components, that is, ‘policy development’ and ‘Community Mobilization’ created a major rethink in the project. After a lot of heated debates, both components of the project were taken out of the project altogether. The Semiotics’ team had spent a lot of effort in fleshing out details of community mobilization. They came up with an ambitious plan that included mobile libraries, annual education forums and meetings, revamping and reforming school management committees and community education policy working groups. Similarly, a detailed plan was developed for policy development. However, the more the teams spent time on these details the more it became obvious that these were unwieldy domains, which had the potential of derailing the project. Ajmal felt that the designs of these two components were over-ambitious and it would be hard to show results as multiple factors were involved and not all of them in the control of Semiotics. Most interventions projects required impact assessments at the end. Ajmal’s experience told him that if they went ahead with this over-ambitious design, the impact assessment would raise more questions than answers. In addition, Ajmal felt that these designs would adversely impact budget and time estimates. There were major disagreements within Semiotics team on both these components. It was decided that they would let GDA experts react to the designs.
Although the GDA had proposed both of these components in their preliminary design, they also got concerned when they read the details. They felt that the community development component was so detailed and ambitious that it seemed like a separate full-fledged community development programme and not a sub-component of the collaborative education project. It was difficult to link it with the pro-ject’s educational goals. Another concern was that it would be difficult to gauge the results of this component within the limited time frame of the project. Community mobilization had elements of behavioural change in the targeted communities which required more time than the life of this project. The final decision was to take out this component from the project. In a similar vein, the ‘policy development’ component was also taken out as a separate head; however, a few of its activities were incorporated into the institutional development component.
Upon finalization of this third LFA (Table 3), Semiotics completed its contract with the GDA by submitting a PIP and an appraisal report. The GDA was so pleased with the final design of the project that they decided to retain Semiotics as the implementation partner as well. Semiotics got a go-ahead to launch the project in November 2005. However, due to the massive earthquake that struck northern parts of Pakistan in October 2005, the project was put on hold as all donor funding was redirected towards relief and rehabilitation of earthquake victims.
Logical Framework 3
Development of the Fourth LFA
It was only in March 2006 that Semiotics was contacted by GDA to start implementing the project and their contract was amended from design and appraisal to the implementation phase. Project implementation started in April 2006.
The first semi-annual report of the project was due in September 2006. Ajmal and his team had created detailed Annual Work Plans (AWP) for the project and the team had a clear set of actions to implement within a schema of components, activities, tasks and sub-tasks. They used this annual plan as the basis for their report by simply describing tasks completed for each of the activities planned for the first and second quarters of the first AWP.
The GDA was not happy with the report. They felt that it was more of an activity report rather than the result-based report that was required by GDA top bosses. This triggered a new debate on the LFA of the project. Result-based reporting was presenting some confusion at Semiotics as many indicators labelled as outputs could also be labelled as outcomes. Then again, the time needed for the intervention to make an impact, coupled with the requirement of quarterly progress reports, meant that some more immediate measures, like activities, be seen as proxy indicators for outputs or outcomes. There was pressure from GDA management that performance indicators specified in the LFA should be more precise and less ambitious so as not to create confusion a few years later during any impact assessment studies. GDA policies entailed impact assessment studies of selected projects in order to learn and improve future interventions. For GDA managers, the CEP was an important project and they wanted clear readings of the outputs and outcomes in order to replicate it in other parts of Pakistan or even the rest of the developing world.
The GDA decided to send an monitoring and evaluation expert from Vienna to work with Semiotics to make a more workable and measurable LFA. The new rounds of discussions with the experts led them to revisit many activities, outputs and indicators. These careful rewordings of the different elements of the LFA led to the development of the fourth and final LFA of the project (Table 4).
Logical Framework 4
The Debate
The Semiotics team was satisfied with its performance so far but, there were some internal debates about the frequent revisions to the project. While the journey from LFA 1 to LFA 4 made a lot of sense, there were some questions that bothered the team. Was the original design, with its scope of work and consequent budgets, not properly thought through or was a grand plan needed to sell the project both internally to GDA bosses and externally to GoP? The scope of the CEP had changed significantly, whereas the project budgets had remained the same. Was the original project idea massively underfunded or did the implementers’ waste precious development grant? How much flexibility should one allow in a project without making the original LFA practically irrelevant as a monitoring and evaluation instrument? While GDA managers were always kept in the loop and while many changes were even initiated by GDA, there was the concern of how the senior managers at the donor would look at the ongoing project. Semiotics was very keen to bid for even more ambitious projects of GDA now that they had the CEP partly under their belt. However, the final verdict had yet to come from the GDA team arriving in a week’s time to Islamabad for monitoring of the progress made so far.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this case.
Funding
The authors received no financial support for the research, authorship and/or publication of this case from Semiotics or GDA.