Overview

An Overview of Evaluation Methods

Counting

• Cost/activity ratio; relationship of overheads to activity (unit costs)
• Monitoring data including financial
• Cost comparisons - Cost effectiveness /Cost benefit analysis
• Statistical analysis ( occasionally )
• time usage
• census and poverty/ deprivation data

Observation

• Observation of individuals
• Observation of objects
• Observation of groups
• Observation of neighbourhoods
• Observation using fixed checklists, eg behaviour or interaction
• Shadowing
• Mystery customers

Looking at documents/ records

• Diaries or logs
• Monitoring data
• Project comparisons
• Quality standards work

Interviewing

• Checklists
• Opinion surveys/exit polls
• Surveys
• Oral Histories
• Individual interviews- face to face, telephone, e mail, texting
• Group Interviews/discussions- face to face, chatroom; telephone

Case studies

• Oral Histories
• Story telling / Story dialogue method
• Portfolios
• Critical incident analysis

Group exercises and review meetings

• Group, project and organisational reviews;
• prioritising, visioning and planning exercises
• quality circles
• User panels
• peer group assessments / benchmarking visits
• Timelines- highs and lows
• Body ( mind, heart) with the carrier bag and the dustbin exercise
• Story telling
• Quality standards work
• Organisational personal history charts
• Role Plays
• A song/ skit/ mime that shows how things are (give the group 10 minutes to symbolise their experience and learning in one of these ways)
• Physical Positioning re issues or organisations
• Power/Priority Circles ( eg where power lies)
• H Form and Pair Ranking

Questionnaires

• Questionnaires - open/closed questions; Structured/semi structured questionnaires
• Rating/self rating methods
• Testing - knowledge, skills, attitudes , performance

Visual and Projective techniques

• Visual tools
• triggers; vignettes; visual card sorts; sentence completion
• photos
• drawings- maps of facilities with people in them; this is how I see X; this is where I am.. etc
• tapes
• Psychodrama/story telling
• Graffiti walls

An organisation has choices about how to find resources for an evaluation:

An organisation may do all the work internally, perhaps using a managing body member or trusted outsider to take member/ client/ user/ funder feedback

• An organisation may link with another similar organisation and undertake a peer evaluation, perhaps using a team of 1 or 2 workers/ volunteers, managing body members, users/ beneficiaries from one organisation to visit/ interview/ take feedback/ read documents from another; and then put together its own team for a return visit. This pattern is particularly useful for networks.
• An organisation may look for a trainer, evaluator researcher or consultant who will help them set up an evaluation, get the questions right, develop tools like interview guides, and help with the analysis and writing up. This will be useful if you are seeking to involve a number of different stakeholders. Students with good support can also be helpful.
• An organisation may seek an independent evaluator.

Most smaller organisations cannot simply go on a training course and then set up evaluation systems without additional consultancy help in clarifying the key questions, and developing the tools.

How do we find the right consultant?...

The needs that lead organisations to seek a consultant are varied- from help in developing a future options strategy, to resolving conflict over aims between workers and their managing body. And therefore involve a variety of roles and tasks.

Who is the ‘right’ consultant?

Obviously they need appropriate knowledge, skills and experience, but also someone with the approach, ways of working and values that fit your organisation. Ideally before you start looking for a consultant you will be clear about what is the focus and purpose of the consultancy, and what the organisation wants it to achieve- an agreement that involves at least some of the key groups with a stake in a voluntary or community organisation, including the managing body, the workers, the volunteers, and the users or members.

I say ‘ideally’ because sometimes there is not this agreement and it is not possible to negotiate a clear brief, and that becomes part of the work of the consultancy. However, the clearer and more honest the brief about the task and any issues, and the results you are seeking, along with your budget and timescale, the more likely you are to get the right consultant. What is essential is some openness in the organisation to change in response to what comes out of the consultancy and without it you are wasting your money.

You can develop a short list of potential consultants through asking local development agencies like Councils of Voluntary Service and other local organisations who have used consultants about their experiences; and through directories and lists linked to national organisations. Funding may be available through Community Champions or CAF.

Having helped a number of smaller organisations recruit and use consultants we have gained some useful learning:

• The need to prepare for the discussion, carefully working out what questions you want to ask after the consultant’s presentation; a supportive outsider, like a local development worker, may help you.
• The involvement of service users or beneficiaries is very valuable and can crucially influence the decision, but they may need support, in order to take part in a real way.
• The real value in asking the consultants for contacts with a couple of other organisations they have worked for, and telephoning them beforehand. Unlike many job references this can be a good source of information especially about the values and approaches of the consultant, what it was like working with them and how useful their work has been..
• The need to ask the consultant about their approach and the extent they are prepared to negotiate this with you- a telling question put to me was asking the criteria I would use to evaluate the consultancy.
• The value of quite a long relaxed dialogue as it is in part an exploration of openness, trust and confidence on both sides.
• The requirement of a simple written contract covering tasks and timescales, mutual expectations, confidentiality, access, and ownership of data as a protection for both parties.
• The value of the organisation setting up a small steering group covering all the stakeholder groups to manage the consultancy and get the organisation involved.

Methods

Collecting Information on the profile of service users

• Explain why you need the information.
• Assure people of confidentiality.
• If people are reluctant to give you profile information try and find out why. They may, for example, be concerned that you might pass the information onto other agencies.
• If possible, allow a relationship of trust to develop with the user before asking for profile information.
• Never guess profile information - it should always be self-reported.
• Where possible allow people to describe their characteristics themselves rather than asking them to say which of your categories they fit into.
• Where appropriate take a sample.

Jayne Weaver
The Evaluation Trust

Tips for Effective Record Keeping

• Look for the most ‘natural’ time and place to collect the information you need. This will be as close as possible to the source of the information. For example, information on the number of people who attend a workshop would most naturally be collected during the workshop itself and not the next day based on the memory of the facilitator.
• Make sure that everyone who is collecting the information understands what the information is used for. If people understand why they are collecting information they are more likely to collect it properly.
• Give feedback on the results to the people who have been collecting the information.
• Make sure that everyone who is collecting the information has been given clear and consistent guidelines on how to go about it.
• Write down what forms are used, when and how they should be filled in, and what the information is used for. This means that knowledge does not get lost in one person’s head.
• Store your information safely.
• Check that you are not collecting the same piece of information more than once.

When is observation a particularly appropriate tool?

Usually observation is a qualitative research method, and therefore has the previously identified strengths and weaknesses. It can be used quantitatively- eg when counting the number of times a participants talks in a group; or the amount of time a caller has to wait in the reception area before seeing an advisor. It might involve being part of a group doing something and seeing what happens; or sitting in a Voluntary Organisation's waiting area.

Observation can be both a diagnostic tool- to help understand what is going well and what is going wrong- and an illuminative tool, when it is particularly useful in helping discover what individual organisations do and don't do, rather that what they say they are doing. It is particularly relevant in organisations where the work is based on relationships rather than definite services and is therefore difficult to specify. Observation can produce description at such a depth and detail that it allows the reader almost to enter the experience. Though we are dealing with short observations, they can produce material that is very rich and emphases meanings and experiences.

The principal difficulties in using observation in evaluation come in the very careful negotiations that need to take place beforehand about the role of the observer, and the large amount of data which it produces. As with all qualitative research methods, it is worth linking it with other methods, rather than using it on its own.

Different types of observation

There are different types of observation which focus both on:

how the observation is done:

ie open / unfocussed observation structured observation

and how the observer behaves:

being external to what is observed being a participant in what is observed

Your place on the participant observer continuum is likely to move and change all the time.

You also have choices about whether your observation is overt or covert- though you would be unwise for it to be covert and not negotiated. Having said that watching what is going on whilst visiting can be very useful, and does not need anyone's agreement.

How to undertake observation

It is just a more careful and recorded version of the skills we already have in the NGO sector.

1. Obviously it is worth noting something about the:
   • Building/space/layout
   • Actors/roles - age, sex, ethnicity, appearance, groupings.....
   • Time
   • Activities
   • Objects, furniture
   • Actions - who makes decisions?
   • Events and their sequence
   • Time
   • Aims
   • Your feelings and reactions.

   Also drawing and photographing can be helpful.

2. In relation to activities, the kind of questions I have found helpful are:
   • Who is present?
   • How is an activity introduced?
   • What are people trying to achieve?
   • What`s said?
   • How did participants respond?
   • What happens next? - especially unplanned action and informal interaction
   • What does not happen?
   • Who participates? And how? Who does not?
   • What are people doing? What are their bodies ‘saying‘?
   • What is the atmosphere?

   Your feelings/reactions as an observer are part of the data and need noting.

Analysing, Reflecting upon and Judging the Evidence

This is a really difficult part of the work, and one that many voluntary and community organisations find so difficult that they put the evaluation work away and never complete it! Sometimes you get to feel overwhelmed by the data- working with someone else or going away from it for a few days can help.

It is important to look at every piece of evidence, preferably with people from the key group of organisation's stakeholders, jointly analysing and making sense of information collected; and in light of the learning, the group making decisions about change needed in the organisation. Often you have to do the analysis on your own but really early on check out themes and conclusions with key informants- have I got this right? You may be met with anger and defensiveness but persist as this is part of the learning process. You may have got it wrong and have to be willing to make changes.

You start this stage with looking at every piece of evidence:

e.g. a summary of questionnaire responses, summaries of notes of interviews from users with all the things that could identify individuals removed, some drawings, some monitoring data, a piece of observation, a summary of telephone interviews with funders or Board members........

and asking yourselves:

• What is this telling us?
• What are the key points?
• What are the range of issues raised?
• How does what people in this group say vary across the group?
• How does what this group is saying "fit" or conflict with what others are saying? What issues recur? What are special to this group?
• How important are the different issues?
• What kinds of patterns, or themes are emerging?
• Are there contradictory views about the organisations‘s purpose, processes or outcomes/ results?

• Search for explanations: why do things occur or why are they done the way they are?
• Look for surprises: was there a 'dog who did not bark in the night?'
• Search/ look for what needs changing
• Search/ look FOR WHAT CAN BE CHANGED

This is detective work- Raymond Chandler rather than Agatha Christie!

It involves a cycle of discovery, reflection and review.

Some suggestions about how to move from initial analysis to further analysis and developing your report.

There is not much written about this, but this is how I and others tackle the job:

• Get really familiar with the data- read it, read it aloud, listen to it if on tape.
• Identify recurrent themes or issues as you see them.
• Index ( using numbers or coloured pens) the material according to themes - OR photocopy your notes/ summaries etc and cut them up put into theme piles and stick them on sheets OR type onto a computer, cutting and pasting into themes.
• Chart out the material in each theme and pull out core elements; search for patterns and relationships in the data ( You may find some themes getting too big and subdividing, and other themes becoming so small that you realise it is your theme rather than what the evidence is telling you! ).

This is where your interpretations can be different from others: you need to be able to justify how you are coming to conclusions, constantly checking out "Have I the evidence to draw this conclusion? Does my story fit the data?" and being quite systematic and rigorous in your approach. We return again to ethical issues- being aware of our bias and prejudices, and unwilling for example to suppress data that does not fit our conclusions.

It can be useful to draw diagrams or trees to see how themes relate or connect. I nearly always find I make many changes in the order and development of themes as I go along; I collapse headings and then re order.

It can be useful to count instances ( For example the number of people from different groups who told me that a particular project had been set up to fail.) On the other hand just because only one person told us something that does not mean we discount it: it may be very important when seen against other pieces of evidence. ( eg a parent told me about a particular member of staff throwing a child across the room: taken with other evidence about the fear of this staff member, and the supervisor‘s concerns, a picture began to emerge).

Use people‘s quotes - they are much more powerful evidence than our assertions

Tie in qualitative and quantitative data

Writing the report- or identifying other ways of communicating the learning.

If the members of the stakeholder group do not write up the report (and usually they do not) the report writer needs to work through with them a number of drafts( or videos, drawings, cartoons...) consulting wider group of participants; the stakeholder group then needs to disseminate the learning widely, including feedback to all participants.

Some suggestions about how to write your report.

• Be very clear about your audience. What do they want/ need to know? What do you want to tell them?
• Either make it brief so people will read it - put information you do not want to loose into appendices - or have a good summary and use that as your main report with other materials as back up.
• Consider producing more than one report- internal and external, produced in the appropriate formats.
• Consider producing a very simple feedback summary for users and other participants.
• Present findings and recommendations clearly so people will understand it - beware of jargon!
• Use visuals- graphs, pie charts, bar charts, diagrams, stories in boxes. photos, cartoons.......
• Avoid criticism or praise of individuals where possible, and deal with sensitive or controversial issues with 'appropriate honesty and tact'.

For suggested Headings, after title and contents pages, consider:

1. Summary: including recommendations and acknowledgments- mainly bullet points ( One/ two pages )
2. Introduction- the Context:
   • The issues and national and local context in which the organisations works.
   • The organisations' purpose, values, work/ activities, funding, organisational ‘map‘ and a short history.
   • ( Ask yourselves- would someone not knowing anything about the organisations have enough of a picture from this section of the report in order to understand the rest?)
3. The Evaluation:
   • The purpose of the evaluation- WHY did we do it?
   • The focus of the evaluation- WHAT questions did it need to answer?
   • WHO was involved in the evaluation?
   • Values and Methods- HOW was it done?
4. The Findings
5. Conclusions and recommendations
6. Appendices inlcuding
   • List of people interviewed ( excluding users ) and other sources of evidence.
   • Topic guide or interview questions.
   • reports and publications consulted etc

Tips for Interviewing Techniques

Be clear about the purpose of the questions - what questions do you want it to answer. What are the issues in the groups you are contacting - there may be different reactions in different cultures to asking questions - who will people talk to?

The interview questions

• These can be a set of questions or themes to follow through.
• They can be open or closed.
• Avoid leading questions - e.g. “Would you say that X organisation offered a high quality service?”
• Test out the questions for meaning. Will they mean to others what they mean to you? Think about the sort of replies a question will generate - can you categorise them to make them quantitative? Saying them out loud can help.
• Are there more ways than one of answering the question? Look critically at questions and get others to do the same.
• Always pilot a questionnaire before you use it even if it is only one or two people - this always throws up the problems with the questions. Identify those questions which will not generate meaningful data.

Some guidelines for Interviewing- face to face and on the telephone

1. Introduce yourself : (Carry an identification):
   • Explain the purpose of the interview (i.e. to find out their views about XYZ in order to help it develop further).
   • If you are phoning, ask if this is a convenient time, and agree another time if it is not.
   • Reassure that though you will be taking notes, what they say will remain anonymous and confidential. Only you will know who is the person making the comments and you guarantee that anything that could identify them will be taken out of the summaries.
   • Emphasise that we need their help and encourage them to speak openly. " We want to know what you really think!", "Warts and all!" .
   • Refusal to take a call can often happen at the beginning of a telephone interview- it is worth phoning back.
2. Confidentiality:
   • If anonymity is promised, it must be respected- if you tell a colleague of the dreadful discussion you had with Stephan Miartus of the X organisation it soon gets back; but of course you can talk generally without identifying the person involved.
   • Offer them copies of your notes if they want them.
3. The interview:
   • You are using the everyday skills you already have of making contact, getting people to speak and listening; only more planned, rigorous, and recorded! You may want to practice phoning or interviewing first with someone you know.
   • Try and check as you go along what you think you are hearing them saying. and probe further if they give you vague answers.
   • The main aim is to obtain information, not to offer your own views, though sometimes people want to know them. You need to be as encouraging/ neutral as possible, even if their views are not what you would wish. Do not let your tone betray your views. You can share your views at the end of the conversation, if you wish. It is always acceptable to offer information. Be sensitive to communication problems or fear of expressing views.
   • Do not forget posture, gesture, facial expressions if you are with them- you need to communicate warmth, interest, encouragement and caring- rather than coldness, disapproval and indifference.
   • You can interview people face to face if you wish - you are likely to get more material, but it will take longer than a telephone interview.
4. Recording:
   • You need to try and record people‘s words as far as possible- this kind of qualitative data is much more useful and real.
   • Keep writing as you talk, and if face to face keep as much eye contact as possible. If on the ‘phone then use encouraging/ listening noises or phrases.
5. The end:
   • At the end thank them for their time, explain what happens next, how they can see the report etc, and affirm the confidentiality.

Focus group meetings

1. Focus groups are group interviews of 7-12 people with a relatively neutral moderator helping the group talk to each other:
   • They have been called a steered conversation. They usually last 45-75 minutes.
   • It helps to have a second moderator to observe and take notes, or to use a tape recorder with permission. Part of the moderator‘s task is to make sure that all participate, and that the conversation stays approximately on the topic.
   • It is quite a democratic method as people can take more control than in an individual interview.
   • The group can be pre- existing, or selected by you.
   • Do not forget refreshments and a comfortable room.
   • It is particularly useful for exploring views and attitudes, with the group process producing more often, than individual interviews, as people build on each others ideas or challenge them.
   • It is not a good method for discussing sensitive issues.
   • You are using the everyday skills many people already have- getting people to speak in a group and listening; only more planned, rigorous, and recorded!
2. Suggestions for running groups:
   • Introduce yourself. Explain the purpose of the interview (i.e. to find out their views about XYZ in order to help it develop further).
   • Go around room - names/how did they get involved with XYZ...?
   • Emphasise that we need their help and encourage them to speak openly. " We want to know what you really think!", "Warts and all!" "I am here to learn from your experience - you are the experts here." "I want to hear as many different experiences and views as possible."
   • Reassure that though you will be taking notes, what they say will remain anonymous and confidential, and comments will not be attributed.
3. The interview questions:
   • Usually you are exploring topics and interested in getting them to share their opinions and views, and tell us how they see the world - use the questions as conversation prompt.
   • Start with easy general questions to try and get their interest- leave specific and sensitive questions to the end.
   • Often the discussion will take off and your well prepared topic guide is not used!
   • Try and check as you go along what you think you are hearing them saying- especially if you are finishing off a long discussion on a single topic, ask‘ Am I hearing what you are saying correctly?‘
   • Probe if they give you vague answers.
   • If a lot of people are silent break into 2/3s and get them to feed back- or ask people direct questions.
   • The main aim is to obtain information, not to offer your own views, though sometimes people want to know them. You need to be as encouraging/ neutral as possible, even if their views are not what you would wish. Do not let your tone betray your views. You can share your views at the end of the conversation, if you wish. It is always acceptable to offer information. Be sensitive to communication problems or fear of expressing views.
   • Do not forget posture, gesture, facial expressions if you are with them- you need to communicate warmth, interest, encouragement and caring- rather than coldness, disapproval and indifference.
4. Confidentiality:
   • If anonymity is promised, it must be respected- if you tell a colleague of the dreadful discussion you had with Stephan Miartus of the X organisation it soon gets back; but of course you can talk generally without identifying the person involved.
5. Recording:
   • You need to try and record people‘s words as far as possible- this kind of qualitative data is much more useful and real.
   • Keep writing as you talk, and if face to face keep as much eye contact as possible.
6. At the end
   • Thank them for their time, explain what happens next, and affirm the confidentiality.

The head, heart, carrier bag and dustbin exercise

Large piece of paper with body drawn on it with a head, big heart, carrier bag and a dustbin set away from the figure (could be on the wall or floor).

Post-its/stickers for everyone- write something on 4 post its-

One post-it for the Head - something I've learnt from being part of this project...

One post-it for the Heart - something I've felt / experienced from being part of this project...

One post-it for the Carrier bag - something I'll take away from being part of this project...

One post-it for the Dustbin - anything I want to forget or that was not so good about being part of this project...

The Story-Dialogue Method

Introduction

Storytelling (or the story dialogue technique) was refined in Canada by Labonte and Featherstone (1997) but is actually a much older idea which builds on traditional, oral communication and learning techniques. Labonte developed the method as a means of recognising and respecting the expertise that people have in their own lives in relation to community development and health issues. The storytelling takes place in a supportive group setting, although it could be adapted for use in a one-to-one situation for those with less confidence. The process is structured so that valuable personal experiences are used to draw out important themes and issues affecting the community and then action can be planned around these insights.

Materials Used

Paper, pens, coloured cardboard, felt-tip pens.

Method

Storytelling uses a mixture of story and structured dialogue based on four types of question : "what?" (description), "why?" (explanation), "so what?" (synthesis), and "now what?" (action). Open questions are asked of the storyteller by the other members of the group (about six people) and this generates dialogue, but with a particular set of objectives in mind : to move from personal experience to more generalised knowledge (insights) and action. The whole process, once the story has been written, should not take longer than about 60-90 minutes. A skilled facilitator is central to this method.

• Introduction.

  The facilitator should spend some time explaining the process to the group and providing everyone with paper and pens. The facilitator must also keep a check on the timing of the different stages and move the group on appropriately.

• The story (5-10 mins).

  The storyteller has to spend time before the session writing their story. The story is based around their experience of a particular issue/theme and examples should include a description of the event and their feelings about what happened and how it affected them. While the story is being told, it is important for listeners not to interrupt, to note down details of the story and ideas for questions and to respect confidentiality.

• Reflection Circle (10 mins).

  The listeners then quickly jot down their immediate reflections on the story : how is this story also my story? ; how similar/different is the story to my experience? Then they share their reflections within the group, one at a time with no interruptions (people can opt to pass).

• Structured Dialogue (25-45 mins).

  This is not an interrogation of the storyteller and it is important to respect different views and to use active listening skills. Several people in the group should be asked to make notes of additional information gained during this dialogue. This part of the process is based around four types of question :

  • "what?" – description questions (What were the problems / issues / needs? Who identified them / how did they arise? What did you do? What were the successes / difficulties? How did it turn out?)
  • "why?" – explanation questions (Why do you think it happened? Why did you/they react as you/they did? Why did you do what you did (the strategies or actions)? Why do you think it worked/didn't work?)
  • "so what?" – synthesis questions (What have we learned? What remains confusing? How did people or relationships change? What unexpected outcomes occurred?)
  • "now what?" – action questions (What will we do differently next time? What will be our next set of actions? What are the key lessons? What power do we have to do things more effectively in the future and how can we increase this power?)

• Review Story Records (5 mins)

  Each person shares their notes with the group. If only one story is used, for example in problem solving, generating a written record of the dialogue may not be important : a discussion around the notes followed by more dialogue around the story may be enough. In other situations however, for example in research or planning, recording insights for further reflection is very important. In these uses, the group will be listening to and discussing two or more stories on the same theme in order to see which insights are similar or different, and creating Insight Cards.

• Create Insight Cards (15-20 mins)

  The group creates 2-4 Insight Cards for each of the four types of question, or about 8-16 Cards altogether. This is not a fixed number, and some questions will produce more insights than others. Insights could include useful lessons/tips or questions/challenges that are still left : the main thing is that they represent something important and worth sharing with others outside the story group. Each insight is written on a separate piece of coloured card and should include enough detail so that it is understandable to people outside the group. The Insight Cards from each story group can then be arranged into common themes.

Strengths and Weaknesses

Storytelling is an empowering process which values the unique personal experience of members of a community. However it is essential to maintain confidentiality since without this, storytellers may be unwilling to reveal in enough detail the nature of their experience. It is an especially powerful tool when several stories are told by members of a community around the same theme. In this way the insights generated can share much in common and produce a practical action plan which the whole community can commit to. The storytelling process may appeal more strongly to women than men.

Although the method should take about 60 - 90 minutes altogether, it can take longer than this if sensitive issues are raised which the group finds more difficult to discuss.

The Story Dialogue method involves considerable commitment from all those taking part, especially the storyteller, but can yield a wealth of local expertise and information which can then be taken forward and used to challenge issues which communities feel strongly about.

Questionnaires and Survey Design

• Do know the purpose of the survey.
• Do keep the questionnaire brief and concise. Only ask the questions where you know why you need the answers and what you will do with them. If you are only collecting the information because funders want it and you do not need it, go and ask them what they will do with the answers. Often they withdraw the request.
• Involve representatives of target group for questionnaire in identifying issues/ structuring questions and designing questionnaire – helps with clarity and ensures wording is appropriate
• Do pilot- i.e. get feedback on your initial list of questions. Try it out on friends, colleagues, and family and unless it is completely impossible, on some of the group whose views you are seeking.
• Do make questions simple and short- use ordinary English as far as possible. Include only one topic per question and avoid doubling up- e.g. do not ask about whether people are happy with the amount AND kind of care offered.
• Do start with easy questions and put any sensitive or personal questions at the end of the questionnaire. Most people give up if they find questions off putting. If you really need to know sensitive things like age or income offer bands and tick boxes.
• Use tick boxes where possible but also ask open questions e.g. identifying benefits perhaps in boxes but also asking open questions like ‘ what difference, if any, did getting the service make for you? If you give a tick box list make sure it offers all reasonable choices and offer ‘ Other’ and a space to name it.
• Do give people the chance to give negative as well as positive responses, especially if you are using ratings, otherwise people tend to tick all the positive boxes without really thinking. E.g. How much do you agree or disagree with the following statements? - and have both positive and negative statements.

Consider offering pictures to circle and or sentence completions and/or feeling/ experience words to circle. ‘Did you find the advice you were given .....
Of no use
Quite useful
Very useful
Not applicable
OR How did doing X make you feel?

• If you are using ratings people often get confused about scores- i.e. is 1 high and 5 low or vice versa. Sometimes words are better-
  E.g. Seldom or never, Occasionally, Frequently
  OR Very Poor, Not Very Good, OK, Good, Very Good
• Even better, specify what you mean- "Do you read the Programme Newsletter regularly, that is, at least three out of every four issues?"
• Always go for an odd no of choices- 3 0r 5 and make sure they are spaced and laid out symmetrically. Sometimes YES/ NO or 1) Agree 2) Disagree is enough. Only go for mutually exclusive options e.g. NOT 1-2; 2-3 etc
• DO give clear instructions like: Please tick the right box or please circle the right answer - Or you get confused responses. I tend to put instructions in italics. Also ‘Please turn over’- or they do not.
• Be aware of culture, language and class in writing questions- e.g. Was your dinner delivered at the time agreed? Means different things to different people.
• Avoid asking people to rank responses. Ranking questions may be framed as follows:
• Following are three options for development of our service: 1. Start a bathing service. 2. Extend weekend cover. 3. Offer home based welfare rights help
  
  • Which is your top option? _____
  • Which is your second best option? _____
  • Which option do you think is least needed? (useful to include if the list is longer than 3 - not otherwise.)
• Avoid leading questions: Try not to give respondents the impression that a question has a "right" answer. "Most experts believe that youth need structured after-school activities. Do you agree?" Such leading questions may prejudice the results of the survey.
• Only ask people about areas they know about. I.e. instruct to go to Question X if your answer was ‘NO’.
• Grouping together questions on the same topic also makes the questionnaire easier to answer i.e do not wander around different topics but have a logical order.
• Explain at top of form the reason for the survey and how it will be used. Also any confidentiality issues. E.g. needs to find out your views about its work so it can improve its service. This questionnaire is anonymous and your replies are completely confidential, though if you want to sign it you can.
• Add a final question around: Is there anything else you would like to tell us? Often get brilliant answers
• Quality layout really improves responses rates- putting questions in boxes, greying options etc. See layout for last census. Use font size at least 12 and of 13 or 14.
• As well as SAE or Freepost envelope think about offering a prize draw. You may need a tear off strip to preserve anonymity. Always put a return address as often the envelope gets lost.

Resources

The Evaluation Trust Picture Gallery

The Evaluation Trust Picture Gallery is still being collected.

"I don't know much about Evaluation, but it sounds like fun!!"
© Andy Ellis

Community and Voluntary Organisations Tree
© Evaluation Trust

The Action-Reflection Learning Circle
© Sarah del Tufo

Bibliography: Resources on Impact Assessment and Evaluation

Resources for Funders

Short review articles:

• The impact of NGO development Projects
  ODI Briefing paper 1996 (2) http://www.oneworld.org/odi/
  FREE
• Participatory monitoring and evaluation: Learning from Change.
  IDS Policy Briefing Issue 12 Nov. 1998 http://www.ids.ac.uk/ids
• Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review
  Marisol Estrella and John Gaventa IDS Working Paper 70 1998
  £9.00
• Impact assessment methodologies for micro-finance: a review.
  (Prof.) David Hulme, at the Univ. of Manchester.
  E-mail Cath Baker at Cath.Baker@man.ac.uk
• Mand E news: An NGO news service re monitoring and evaluation:
  http://www.mande.co.uk/news.html

Books and Publications

• Participatory Impact Assessment: H. Goyder et al Action Aid 1998 £4.00
• Outcomes and impact: Evaluating change in Social Development 1998 P. Oakley and B. Pratt INTRAC
• Assessing the impact of NGOs, Women and Empowerment. Dawson, E. OXFAM 1996
• Linking costs and benefits in NGO projects. Riddell R. et al. 1996 ODI
• NGOs and Rural Poverty Alleviation. Riddell R. et al. 1995 Oxford University Press and ODI
• Impact Assessment: a review of best practice J. Dawson, ITDG, 1996
• Social Development Policies, Results and Learning: a multi agency review. Thin, N. et al 1998 Dept of social anthropology, Edinburgh University. Copies from s-hovell@dfid.gtnet.gov.uk
• Searching for impact methods: NGO Evaluation Syntheseis Study. Riddell, R, et al 1997 OECD ( DAC) Expert group on Evaluation. http://www.valt.helsinki.fi/ids/contents.htm
• Best practice in evaluation of Humanitarian Assistance programmes.Hallam, A. ( working draft) OECD ( DAC) Expert group on Evaluation a.hallam@odi.org.uk
• Is Social Auditing good for us? Oxfam takes a look at a growing phenomenon.Dawson, E. 1998 aburgess@oxfam.org.uk
• Other Social audit material from The New Economics Foundationhttp://sosig.ac.uk/NewEconomics/newecon.htm
• Methods for the Evaluation of poverty oriented aid interventions. 1995 DANIDA.
• NOVIB and OXFAM UK/I Impact assessment Research Programme- various studies.
• The impact of NGO development Projects.Briefing paper. ODI
• Evaluating the effectiveness of Poverty Alleviation Programmes -the OECD, the UN, and the World Bank have agreed to focus on a series of key goals in partnership with developing countries... This site explains the core set of indicators. http://www.oecd.org/dac/indicators/

Resources for Community and Voluntary Organisation grant holders

Books and Publications

• Everyday Evaluation on the RunWadsworth, Yoland. 2nd Edition , Allen & Unwin 1997. (115pp)
• Monitoring & evaluation made easyAnne Connor. HMSO 1993 £12.95
• Monitoring OurselvesAnne Connor 1993 Charities Evaluation Services ( part of a series)
• Partners in EvaluationFuerstein, M.T. 1987 TALC £4.50
• Self evaluation- a combined handbook & toolkit. NCH 1993 £20.00
• Introducing Evaluation- W. van der Eyken 1992 Bernard van Leer Foundation ( also available in Spanish, Arabic, Hebrew and Portuguese)
• Evaluating Social Development ProjectsMarsden, D. & Oakley, P. (eds)1990 Oxfam
• Toolkits- a practical guide to assessment monitoring review and Evaluation Gosling et al 1995 SCF
• A Basic Guide to Evaluation for Development WorkersFrances Rubin 1995 (Oxfam) GBP4.95
• Guide to Social Analysis for Projects in Developing Countries ODA 1995 (HMSO) GBP19.95
• Monitoring and Evaluating Small Business Projects: A step by step guide for private development organizations SEEP , 1987 (PACT Publications) GBP25.00

One of the best sources in USA and UK of these kinds of publications is the ITDG publications- they have bookshops, a catalogue , a postal service and are on line. ( All available at ITDG Bookshop or by post - tel: 00 44 207 436 9781 ; fax: 00 44 207 436 2013 itpubs@itpubs.org.uk)

Publications

In some cases we provide PDF versions of publications. To download Adobe Acrobat Reader, visit The Adobe Website. Visually impaired people may wish to visit the "Access Adobe" Website.