Does all our hard work make a difference? Congratulations on gaining a Lloyds TSB grant- it is a real recognition of the hard work you are putting into your community! Most grant holders are quite small community, self help or voluntary organisations, run without paid staff, and some of the questions funders ask can seem like just more things to take up your time and get in the way of the real work. This leaflet is written by a grant holder, The Evaluation Trust, to help you to make monitoring and evaluation useful to your organisation, interesting- and even fun! Evaluation is about seeking to find the answers to questions about how your organisation is working and what difference it is really making. Evaluation also involves comparisons- the past with present, 'before and after' like the hair advertisements! 'Programme evaluation is the systematic collection of information about the activities, characteristics, and outcomes ( or results) of programmes •to make judgments about the programme, •improve programme effectiveness, and/or •inform decisions about future programming. ' M.Q. Patton How did you answer the question on the grant application form 'How will this aspect of your work be evaluated?' Your answer should have grown out of the answers to earlier questions on the application- 'The details of the work for which you applied for funds', 'How many people will benefit and in what way?' However, in practice, it is often find it easier to plan activities and go ahead and do them and not think to much about really knowing the results or differences they will produce- it seems enough that you work hard and care a lot and you just hope you are achieving results. But don't worry- its never too late for you to think about the results you want from the work and how you will discover you are achieving them! Evaluation can seem a bit of a mystery and has a reputation for being technical and difficult- in practice it is part of every day life. You often think about a meal you have just cooked or chat to other people about some experience you have shared.We can make judgments about how far a meal or an activity meets our needs, and what could be changed. It is not difficult for us to develop criteria or indicators for our judgments. Our judgments involve our values and experiences. BUT, what counts as valuable, a positive change, a real difference may vary depending on whether we are a service user, a volunteer, a staff member or a managing body member. This is evaluation- just not as planned or as systematic a collection of evidence as we would do in a project. Some indicators or pointers to success or effects can be quite simple–..an all volunteer black women's group got a little funding to run a healthy eating course. At the end of the course as well as having a group discussion led by someone from outside the group to review the training, they also had a party. Some of the food was deliberately ' unhealthy', but the course members 'told off' the party organisers! The real test would of course be a follow up after 6 months to see if attitudes and eating patterns had changed. A parents group that wanted to encourage members to give each other support, looked at who people turned to in time of crisis, and the friendships and contacts outside the group. A local community group had been going for 30 years but its building was not being used a great deal. With some outside help, the new committee talked to local people about what they wanted, and thought the needs were . Not only did the committee discover their views, but people started to get involved again, and new funding was raised. In voluntary and community groups people work hard, have little money, are very committed, and some how the results must be good.- BUT at end of the day what difference does all that hard work make? . How do you know what is being achieved - the results, if any of the work? . How will you know you are succeeding? Are you effective? . Did you do what you said you would do? If not why not? . What is working- and why? . What is not working - and why? . What could you do differently? . Is the project meeting needsŒ whose needs? . What do users/members think about the work? . How does this project compare to other similar projects? . What lessons have been learnt? . What needs to change to improve the programme? The answers to these questions are discovered through evaluation.There are therefore different kinds of evaluation, asking and answering different sorts of questions- . about the needs and issues that organisations are trying to respond to . about processes- how an organisation works, how things are done . about performance- the outputs or activities and costings ... . about outcomes- the results of the work; what real difference it has made. Getting the questions you want your evaluation work to answer really clear is a tough task- we often want to ask too many and do not focus! You will notice that we have not talked about aims and objectives– Firstly, as an organisation you need: •a clear purpose- the WHY you exist; •well defined aims or goals - the WHAT you are there to achieve; the difference you are trying to make and it helps to have: •specific objectives with targets- the HOW you are doing it or the practical steps to achieve the aims with numbers and dates because with these issues clarified it is certainly easier to evaluate whether you are achieving your aims, by looking for evidence of results. BUT not all community and voluntary groups are in this situation and sometimes too, looking at whether you have achieved your aims is not asking the right questions. It may be more important to look at the effects on and benefits for the users rather than a strict evaluation against objectives. What you do is not just about numbers of activities and people involved and the costs, though these really matter- it is also about finding and giving value to your ways of working with people, and what you are achieving, though it is much harder to do! Ideally all evaluation work . identifies the successes- what you do well. . assesses the weaknesses- what can be done differently. . clarifies what needs to be changed or strengthened, and the reasons why this needs to happen . is built into the life of the organisation and helps with future development- this is because the whole process will have built up the organisation and the people involved A small charity sending arts materials into prisons decided to survey the people who it had worked with and the prison officers who had referred them, to identify improvements in the service and see what difference their work made. They developed a simple quite visual feedback form and were delighted at the results. Now they not only have useful information to help with funding, they also have some really good ideas for improving the service. Why bother- what are the gains for your organisation? Evaluation work in your organisation can: - get feedback from users and others - develop a better project through knowing the local needs - Identify areas for improvement in your service to users - Help share learning and experience across the organisation - Improve accountability to users, members and funders - Give greater work satisfaction for everyone involved - Celebrate progress and achievement - Identify changes or new directions - Make the case for new resources A local organisation supporting elderly and disabled people to live at home decided to do their own evaluation and got everyone together who had a stake in the organisatioon- its users, their families and those that cared for them, the workers and management committee, using some outside help. The participants jointly decided the questions the evaluation work should answer and the methods, and developed the tools used. Members of the committee helped with some of the interviewing of users themselves, and a working group including users, reflected on what the evidence was telling them and identified changes needed.. The work not only brought major changes to the service, but improved communication and participation in the organisation, changed the attitudes to users and carers, and created a partnership with users and carers to action the learning, and build it into the organisation's life. Who needs to be involved in evaluation in a voluntary or community organisation? Who has a stake in the service? •Who are part of the jigsaw of interests? Who has something to offer? •Experiences, needs, views, information, knowledge, questions, skills, time, resources.... Who would have something to lose from an evaluation? •an essential service; a job; power.... Are some of these the Key Stakeholders in your organisation? Users/participants/members Potential users Carers and friends Volunteers Managing body members Local or interest community Funders Workers Steps to useful evaluation work -Preparing the ground- answering the question why plan and evaluate and getting all the parts of the organisation 'on board'. -Bringing people together in a group - and negotiating people's involvement ASKING QUESTIONS What are the key evaluative questions to which you want the answers? What criteria or indicators can you identify for success? ANSWERING QUESTIONS How will the answers be gained? What methods will the evaluation use? Are they credible and interesting? What monitoring data do you already have? What informal feedback do you have? DRAWING CONCLUSIONS What can be learnt from the information? What else helps you understand the data? MAKING NECESSARY CHANGES How will you change what you do as a result of what you have learned? Should you redefine your aims or objectives in any way? What do you need to learn? How can this be best communicated and to whom? What about the methods and tools? There are a wide range of tools and methods available. People tend to worry about methods- but getting the questions right, and thinking who you need to involve are more important. Only collect information you will use and act upon! You are using everyday skills of making contact, getting people to speak, and listening and taking notes- only in a more planned and rigorous way. There are both qualitative and quantitative methods they have different costs and benefits, and reach to different parts of the jigsaw- -Quantitative methods of collecting information involving measuring, counting, and collecting numbers -Qualitative methods of collecting information involving feelings, experiences, processes, understandings and events. Some ideas for building up useful evaluation– . Keep simple monitoring or collection and recording of information about a piece of work to keep track of day to day activities and operations tracking records about a piece of work or project and use them. Its purpose is to provide regular feedback on how things are going and help the organisation make decisions. Ask yourselves, who is/is not using the project? How are they using it? What are the costs in comparison to the activities? How are things changing from earlier years? . Spend time at the end of an activity in reflecting on the value of what you have done. . Build your monitoring and evaluation work into your planning- think about what the results you want from the work and how you will discover you are achieving them. . Find safe interesting ways to take feedback from your users, members or beneficiaries about their needs and experiences, the improvements and changes they want, and what difference the programme makes. Be imaginative, using photos, drawings, sentence completion, and group discussions as well as short visual questionnaires. Test all your tools before using them and value qualitative research methods that find and make visible the work. . Keep a file or notebook for all feedback however informal- thank you letters, complaints, the 'chat' in the minibus on the way home- these will be useful too for any report you write. . Find trusted outsiders as resources to evaluation work in the organisation (like CVS development workers, voluntary sector trainers and consultants, university staff or students or Board members). They can help you develop good tools and undertake sensitive interviews with, for example, fearful users, and help you be impartial. . Be very careful to maintain confidentiality . When getting involved in some else's area of work, be sensitive to how threatening and exposing it can feel even if you are doing a good job. Work in partnership with them so that it is a joint discovery rather than the Inquisition! . Actively encourage workers and volunteers to look at pieces of work critically and identify learning, improvements and changes. Regular 'away days' to stop and think help to build a learning team help, maybe involving Board members too. Use any supervision as a tool of joint learning and reflection as well as control and accountability for actions. . Consider building a partnership with another community or voluntary group which is similar in approach, but not in direct competition for the same funds. Meet to compare projects, perhaps developing visiting teams involving users, to take user feedback and interview staff and board members on behalf of the other organisation. . Act on any learning from self evaluation work or it will not be worth the time involved. A self-help group of parents running an integrated playgroup used a student as a participant observer in the playgroup to observe in-depth what was going on in the group. She also did some comparative observation in other groups and interviewed all key stake holders. Out of this work came solid evidence of the positive gains for the children, issues to work on and material for training and development. Remember evaluation can be * really useful * low cost and not too time consuming * interesting, exciting Œ and even fun ! Would your organisation find it useful to get some help? The Lloyds TSB Foundation has given the Evaluation Trust funding to extend our presentwork with small groups in the South West and adjoining areas offering: .free advice, training and direct help to set up monitoring and self evaluation systems. .free consultancy help to undertake a piece of evaluation work, that will directly improve its effectiveness. For more information please contact: Sarah del Tufo 12 Bulmershe Road, Reading RG1 5RJ phone/fax: 0118 966 4864 e mail: evaluationtrust@maurice.u-net.com Another useful contact: Charities Evaluation Services ( CES) 0207 713 5722; cesuk@compuserve.com Other useful reading: BBC Children in Need guide to self evaluation 1998 Finding Out: an informal guide to self-evaluation for Family Centres by Willem van der Eyken et al (1997) - useful for many organisations. Monitoring Ourselves. Anne Connor 1993 ( CES )Wadsworth, Yoland. Everyday Evaluation on the Run, Allen and Unwin 1997.