Journal of Social Development in Africa (1986), I, 89-99 Evaluating Social Development 'How much' or 'How good'?* P OAKLEY ABSTRACT A contrast is made between economic types of rural development projects, which tend to be quantitative, and social development projects, which are qualitative. A number of evaluation models were studied on particular social development projects. A before-and-after analysis of particular social development group projects generated five indicators by which such development could be quantified: activities, action, changes in group behaviour, nature of intervention, and relationship with other groups. Continual monitoring of these aspects is suggested through a number of methods. The importance of relevance in evaluation procedures for small scale projects is emphasised. Introduction In the past decade or so, and largely as a result of disillusionment with the more economically oriented type of small rural development project, there has been an increasing emergence of small rural projects whose objectives can be defined as social development. These social development projects have largely been the initiative of non-government or other local voluntary organisations. This is not to say that government development programmes are not concerned for social development, but the kinds of small rural development projects to which this article refers are more commonly associated with non-government development programmes. There is, however, little common agreement to date on a precise understanding of the term social development, although there appears to be a consensus that it concerns more than the provision of social services and amenities to rural areas, ie education, housing, water supply or health services. Rural development projects with economically determined objectives stress * A paper first published in Reading Rural Development Communications Bulletin No 14,1982. + Agricultural Extension and Rural Development Centre. University of Reading. London Road. Reading, UK. 90 P Oakley results which are quantitative, social development projects emphasise processes which are qualitative. It can be argued that much of the effort directed towards improving the lives of the rural poor is frustrated by the fact that a majority of these people are marginalised from the development process. Social development projects are directed towards preparing and allowing the rural people (or, more precisely, small fanners, tenant farmers, or sharecroppers) to participate more fully in development so that they become an inevitable part of the process. Social development evaluation The more conventional and more commonly employed approach to project evaluation sees evaluation as based upon measurement and the establishing of performance criteria which can be quantified. However, whilst small rural development projects will contain objectives which are conceptually easy to quantify (ie crop yields, net returns, wells dug, roads built), they may also contain less specific objectives (such as local participation, institution building, changing of attitudes) which are conceptually difficult to quantify. Statements taken from the objectives of two such social development projects demonstrate the qualitative nature of the objectives and, as such they are unable to be adequately monitored by conventional evaluational techniques: "... to help them (small farmers) choose their own destiny and handle their own development' and '. . . to encourage greater self-determination and greater awareness of the possibilities for development'. It is now evident, however, that an increasing number of small rural development projects are being undertaken in which the latter type of objective is dominant, and, consequently, conventional evaluation techniques are inappropriate. In such projects the emphasis must be placed upon a qualitative understanding of the process resulting from the activities of the project, rather than on the collection of quantitative data. It is critical to understand such processes, which involve a series of non-material values, if one wishes to judge whether the project has achieved its intended effect. The difficulty, however, is to establish satisfactory criteria by which such processes can be better understood and, secondly, to determine the phenomena by which these criteria could be qualitatively expressed. At this point it may be useful to indicate briefly the nature and magnitude of the difficulty. A review of a number of social development projects and other literature identifies the following as the kinds of implicit and explicit objectives of such projects: Evaluating Social Development: 'How much' or 'Howgood'? 91 — participation — organisation/solidarity (of client group) — reduced dependence — changing social relationships — initiative/motivation/leadership — creation of awareness It is the above kinds of objectives which we would like to understand better in terms of their achievement. Characteristically these are objectives associated with projects directed towards the rural poor- small farmer, tenants and share croppers, for example - and the projects argue that such objectives are indispensible prerequisites to such groups benefitting from the more economic or material aspects of rural development. As the above kind of project is a relatively recent phenomenon in rural development, the amount of empirical research into the understanding of the processes involved in such projects is extremely small. However, the importance of non-economic objectives of small rural development projects and, consequently, the inappropriateness of economic criteria for measuring the effects of such projects is beginning to be recognised. The need for more appropriate evaluation design and technique for evaluating social development programmes is quite clear, although the amount of empirical work to date is extremely limited. In this respect the work of Wahidul Haque and his colleagues is of particular use. Haque's (1977) model for project evaluation identifies the following broad area of criteria: 1. The Economic base which is understood to be economic benefit, distribution equity, collective accumulation, horizontal accumulation, both expanding the size of local participation and of multiplicity in other areas, and developing social and institutional linkages with other similar self-reliant efforts. 2. Attitudinal criteria which are a sense of solidarity, democratic values, a spirit of co-operation, a collective spirit and collective self-reliance, 3. Self-Administration and momentum which can be categorised as experience in economic and social administration, generation of internal cadres (the 'spread agents' who will be responsible for mobilising the group for different economic and administrative tasks), indigenous momentum in material, institutional, psychological and leadership terms to develop the selfreliant basis of the project. Haque's model offers a framework for trying to understand the kinds of criteria which might be valid for evaluating social development projects but only the criteria for evaluation. The next stage is to determine how such criteria might be identified, observed and recorded. In this respect the work of Parlet and Hamilton is of use. They suggest an approach to evaluation which 92 P Oakley abandons direct measurement in favour of intensive study of the project as a whole. This approach, therefore, is more concerned with description and interpretation than with measurement and prediction. Similarly it is suggested that the evaluation of social development projects should be a participatory exercise, with the project group itself contributing to the description and interpretation. It is in the context of the above that a colleague and I undertook some research to examine the concept of social development and the difficulties associated with its evaluation. This research will by virtue of its focus be long term, but sufficient material has emerged to justify an initial examination. One aspect of the research was to understand how social development project staff perceived and practised evaluation in their work. Interestingly, in our fieldwork in two continents we discovered that project staff generally identified a common problem: that is, how to observe better, understand and monitor the non-quantifiable aspects of their work. Project agents believed that their project groups (ie small farmers, sharecroppers, landless labourers), were 'developing' but they lacked a framework on which to structure and explain this development. They lacked the means to explain effectively the progress of their work. In this respect we can confirm the usefulness of Haque's model of project evaluation. We used this model neither slavishly nor deliberately for structuring purposes. Yet spontaneously, when articulating their understanding of the evaluation of social development, project staff explained their understanding in concepts similar to those employed by Haque. On each project we visited, we examined the staff, their understanding of monitoring and evaluation and the practices currently adopted. Just as we discovered a wide range of interpretations and practices of social development, so we identified great differences in evalution methodology and the degrees of importance attached to evaluation and monitoring procedures. In some cases we found interesting and innovative systems of self-evaluation being used, but no documentation of results. In these cases priority was given to examining and improving the project's approach to group development. Latin America provide us with some of the most interesting evaluation models. There were three more easily identified models. The first, from Mexico, was a curved model of evaluation, which rejected a linear understanding of the project's development in favour of incorporating both the expected and unexpected consequences. The second, from Brazil, could be described as Initial Situation - Interference - Actual Situation - Designed Situation. The third from Guatemala, was a dynamic system of evaluation based on the process of reflection action and conducted at different levels: Evaluating Social Development: 'How much' or 'Howgood'? 93 communication work plan geographical basis population strata. More frequently however, we found the projects had no tradition of critically appraising their work in a systematic way and allocated little time or effort to this activity. Most projects were handicapped in attempting to evaluate their work by lack of understanding of the social development process and an absence of clearly identified objectives. Without these, evaluation became subjective assessment. This can be seen in a number of projects, where the project declared that change has occurred, but does not explain the basis for such a statement. Examples of this are drawn from India and Latin America: 'We had achieved our goals partially, that is to develop people to collectively seek solutions to their own problems . . .' (India) 'The experience of communal work and communal reflection, in order to achieve an advance in the consciousness of the group, has been largely successful' (Mexico). The need for some kind of framework or better methodological understanding of social development projects, which would help structure staff observations and information on the projects, is clearly indicated. Given the length of this article, it is not possible here to explain in detail our research findings from the main geographical areas visited — the Andean Region, NE Brazil and India. We would refer readers to our two research reports for this detail. While the contexts were different, there were a considerable number of similarities in terms of approaches to project evaluation in each area, perhaps the greatest being how agents attempted to monitor, and thus evaluate, their project work. There were three main elements in this activity common to all areas: — Project evaluation meetings: regular and systematic meetings of project staff when work is reviewed and future plans determined. — Project progress reports: a continual process for reporting upon each individual project. These reports were both for internal use and also to meet the demands of funding agencies. While, however, the writing of project progress reports was a common activity, their nature and content varied widely. — Agents' diaries: the keeping of a daily/weekly record of project events and activities, comments and analysis- and forward plans was a particular feature of the Indian projects. In several instances these diaries were compelling reading for their vivid portrayal of the day to day problems which the 'weaker sections' in India's rural areas face and their attempt to overcome them. 94 P Oakley Similarly, a common feature of each of the projects was the identification and development of project groups. The groups took different forms, but their development as autonomous bodies able to direct their own members's development was the central objective of the social development work. Any evaluation exercise, therefore, would be concerned with identifying the criteria and the means by which the development of the project groups could be understood. Criteria and indicators Central to the development of a system for evaluating the impact of social development projects is the identification of appropriate criteria and indicators against which projects could be measured. During our visits to projects we asked project staff to identify possible criteria which could be used to judge the degree of social change taking place. Most projects found it useful to think of group development in terms of its acquisition of desirable characteristics. With project agents we built up a composite list of characteristics (or criteria) shared by marginal groups of small farmers, landless labourers and sharecroppers before the initiation of social development or grass roots non-formal education work. We then asked agents subjectively to characterise the project groups after an undetermined period of time. Through a 'before' and 'after' description of group development, we wanted to see whether we could identify those factors relevant to evaluating social development work. Below we present two composite lists drawn from suggested 'before' and 'after' characteristics of project groups in each of the three research areas, which reflects the changes which agents believe had occurred in the project group over time. These changes, when characterised, could constitute the criteria for evaluation. The characteristics of group members that emerged before the process of social development were summarised as follows: a sense of individualism reflected in a predominance of private property, little interest in community projects, low participation in decision-making; a lack of critical analysis of their situation and inability to identify the causes of structural problems and work out solutions to them; economic, social and political dependence on and exploitation by patrones; lack of confidence in their own ability to change the situation; a lack of organisations which effectively represent their group interests; lack of co-operation; and finally ignorance, suspicion and isolation, which meant people were afraid to talk, discuss and become involved. The characteristics of groups after a period of social development were assessed as internal cohesion; a sense of solidarity; a critical consciousness/ critical faculty, an active and critical participation; reduced dependence and Evaluating Social Development: 'How much' or 'How good'? 95 increased self-confidence; self-management, self-sufficiency and group autonomy; collective resources with capital reserves; project management capability; democratisation of power and collective responsibility; articulation with other institutions controlled by campesinos; involvement in the creation of other similar groups; and lastly an ability to deal with government officials. In setting out the above characteristics of an organised and effective group—the basic objective of the social development process—we are not suggesting that each group will manifest all these characteristics. The lists are presented to indicate the kinds of change which agents felt had resulted from their intervention. Such changes involve complex social and organisational process and are subject to a wise range of internal and external factors. It is difficult to isolate the effect of the project agent's intervention on these changes, or to say that such a characteristic has been the direct result of this intervention. Nor is it rare to find groups which, in the agent's view have achieved a number of the above characteristics but which regress temporarily, permanently, dr even dissolve. If we accept the above characteristics as manifestations of the changes within the groups which the agents have observed over a period of time and if we use these changes as criteria by which to evaluate the effect of the agent's work, then we face the problem of how to describe or explain these criteria. We need to identify indicators which can be used to help illustrate the changes taking place. The issue is how can agents monitor their work in order to be better able to both understand and explain this work. As a result of our analysis of this issue with agents, we arrived at the following indicators which agents felt could be used to observe the changes taking place in the project groups: 1. Activities: in which the group participates, ie group farming, construction works, organisations. 2. Action: specific action taken by the group, with or without the agent's assistance. Examples of such action which were suggested include: — action to obtain social welfare benefits — action to demand implementation of existing legislation. 3. Changes in group behaviour: nature of meetings, confusion/order, use of language, abilities to rationalise and explain, nature of participation. 4. Nature of intervention: changes in the relationship of the agent to the group as a result of the group assuming responsibility for its own development, ie — direct — indirect — withdrawal 5. Relationships with other groups: nature of relationship with other groups, joint action and evidence of inter-group enterprises and organisations. 96 P Oakley The above five indicators could, therefore, constitute the means by which social development might be observed and objectively evaluated. These indicators provide a framework which can help structure agents' observations of the groups' development and also assist them in assessing the impact of their social development work. The order of their presentation does not indicate any order of importance. We stress that the above indicators were suggested by the project agents themselves. Operational aspects The next stage is to consider the operational aspects of monitoring and evaluating social development work. In this respect we would argue that the operational arrangements should be designed by each project team in light of the scale of their resources and project activities. The emphasis would be on project's social development work. The advantage of this approach, as opposed to one reliant on periodic external evaluation, is that the evaluation process is itself an educational process for the agents and project groups in which they could develop their understanding of the complexities of social change and assess the outcome of their work. The approach we are suggesting has a lot in common with 'process oriented qualitative research' (Weiss and Rein in Rossi, 1972). These authors describe that approach as one in which sensitive observers monitor the unfolding of a programme as it is going on, noting particularly those events which are critical, collecting documents and sensitively observing the effects of the programme on institutions and individuals. We suggest, therefore, that the monitoring of social development projects should be based upon the continual collection, recording and observing of phenomena associated with the above five indicators. Such a continual process would essentially be descriptive and would, at established intervals, be interpreted in terms of the criteria by which the project's progress is to be judged. In the absence of any empirical support, we are unable to establish a time scale for the above process, although in our continuing research we are initially monitoring a number of project groups over a period of 15 months. Indeed in our work we have not been able to give much positive guidance to project agents. The difficulties which they themselves highlighted include the inclusion of a continual recording process into an already heavy work load; the different abilities of project staff for the skills required for continual recording (ie writing, identification of critical issues), the subjective involvement of agents in the process they are expected to record, and the differing levels of 'development' of different groups. Evaluating Social Development: 'How much' or 'How good'? 97 The agent, therefore, faces the problem of how to collect the information necessary to support a continual monitoring process. We have already suggested the five areas of indicators around which information will be gathered. Some information might in fact be collected from secondary sources, but the greatest part will need to be gathered by the project agents. The following methods could be employed: - group profile or survey: in order to establish the situation as the social development work begins. Such a profile would probably illustrate the kinds of characteristics already outlined and would serve as the basis for determining the extent and nature of future change. - collection of quantifiable data: where appropriate such data can provide an important dimension to changes which have taken place. Such data might include: numbers of group meetings and participants, numbers involved in particular activities and basic figures on any economic activities undertaken. - continual descriptive accounts and observations on each project group: this would involve the keeping of a monthly record, based on the five areas of indicators, of each group. In this method agents would be encouraged to write freely and subjectively with less concern for structured comment or formalised reporting. The emphasis would be on the agent recording the changes as he or she sees them. The record would then be subject to periodic interpretation. - the minutes or other forms of recording of agents' meetings: such meetings are used to reflect upon the results of work to date and often contain useful information which can assist the monitoring exercise. The above methods will, of course, lead to the collection of information. The next critical stage is interpretation. Indeed the interpretation is the key to the success of the monitoring exercise. It will serve little purpose merely to collect information and record observation; when such information and observations are interpreted in terms of the social development process, we shall have little idea of what, if any, changes have taken place. The information collected will have to be related to the initial situation which was described at the beginning of the social development work. The interpretation will assess the information in terms of this initial situation and suggest the changes which are taking place. The original analysis of the group's situation will have resulted in a number of objectives for the social development work. The interpretation, therefore, will be in terms of these specific objectives. The final important question is, how to establish an internal monitoring procedure which is limited in its demands on agents' time and is not overbureaucratic. The whole exercise will have lost its purpose if it results in deskbound and clerical agents. The essential work of the agents is with the project 98 P Oakley groups, and yet we must build into this work means by which the work can be more closely monitored. Perhaps a number of basic principles might help in establishing this internal procedure: — the emphasis should be on brief but continual reporting/observing. — the agent should be encouraged to write/record freely and not be too concerned with producing an impeccable text. — the neccessary standardised recording forms/sheets must be available in sufficient quantities. — periodic (ie every six months) reviews of information collected on each group, so as to understand the overall process taking place. In an exercise which we conducted with a number of projects and project groups in NE Brazil we found that, over a six month period, agents estimated that they had spent on average three hours per month on the continual monitoring exercise. The exercise was an experiment and somewhat limited but it did indicate the demands it might make on an agent's time. Similarly we found that understandably in the early stages agents were less sure of what to record. The advice, therefore, was to record spontaneously with little concern, in the first instance, for the utility of the information recorded. However, as the recording developed, agents began to have a clearer picture of the nature of the change taking place within the group and began to discriminate and deliberately observe those aspects of the group's activities which they felt were more important for monitoring purposes. Conclusion Our researches have revealed that project agents are conscious of the complexities of social development and of the difficulties associated with evaluation. As we have seen, social development is concerned with nonmaterial processes, and such processes are more difficult to understand than, for example, a programme to increase the yield of a particular crop. And yet at this moment there is little in the conventional literature which can be of help. The emphasis on the economic evaluation of development programmes, and the paucity of evaluation guidance available to agents at the grass roots level who are working directly with the rural poor is indicative of research priorities. And yet we would argue that, on strictly numerical terms of rural families reached, this social development effort is more widespread than the effort of the larger, economically oriented development programme. The majority of academics and other researchers, who would have the responsibility for developing appropriate monitoring and evaluation procedures, are largely separated from social development work. There is a vast body of literature on project evaluation which is written and presented in a way which is totally Evaluating Social Development: 'How much' or 'Howgood'? 99 inappropriate for grass roots project agents. But this is where the effort is needed; in the design and testing of appropriate monitoring and evaluation procedures at the small project level. Bibliography HAQUEWefa/ 1977 Towards a Theory of Rural Development. Development Dialogue, No 1 GALJART B 1981 "Counter-development" Community Development Journal, Vol 16 CERNEA M & TEPPING B 1977 A System for Monitoring and Evaluating Agricultural Extension Projects, World Bank. OAKLEY P AND WINDER D 1979 An Exploratory Review of Rural Social Development Projects and the Question of their Evaluation, Research Report. 1980 A Review of Social Development Work in Rural India, Research Report.