MRC Elsie Widdowson Laboratory

Evaluation tools

Evaluation can be useful:

  • to ensure the project has clear aims and objectives from the outset
  • to provide information on the outcomes of an event along with suggestions for improvement
  • to find out who attended the event
  • to use meaningful results to demonstrate that all efforts have been worthwhile

The method of evaluation you choose will depend on the questions that you would like to answer. Possible questions include:

  • Do I want to capture change over time or is this a one-off exercise?
  • Do I want to establish a set of targets and then measure whether we’ve completed them?
  • Do I want to compare what we are doing with what others are doing?
  • Do I need external verification, or can this be an internal exercise?
  • Do we need to measure what the whole institution is doing?
  • Do we want to understand what is happening at the individual project level?
  • Are we interested in finding out how individual faculty members and their community partners best collaborate for mutual benefit?
  • Do we want to measure engagement from a community perspective?

We have put together a number of different evaluation tools that may be of use to you when running your own event, project or activity.


1. Evaluation assessment checklist

Can be used when initially planning an evaluation in order to create an evaluation plan and to help decide which tools or techniques will be most appropriate to generate the most meaningful results depending on the target audience.

WHO     Target – know your audience. Who is the evaluation for?

WHAT    Area – what are you evaluating? (process, outcome, impact)

WHEN    Timing – will the findings have any effect? (cost:benefit analysis)

HOW      Tools & techniques – what is most appropriate?


2. Evaluation Matrix

Can help you to choose the most appropriate tool (horizontal side of the matrix) to answer each of the questions identified (vertical side of the matrix) in your project proposal.

Evaluation Questions Wordle Radar diagram 5 people Blob tree Focus group Questionnaire Star ratings
a) What were the top 5 comments from the audience? X X X X
b) Would it be worth running this event again? X X X X X X X
c) What attitudes were formed by the audience? X X X X X X
d) Were the staff/volunteers helpful and knowledgeable? X X X X

3. Wordle

A “word cloud” that can be used to illustrate how frequently a word is used within an evaluation to describe an event. The clouds give greater prominence to words that appear more frequently.
For example if you ask the audience to describe the event using only 3 words, you can then put all of the words into the wordle (available at http://www.wordle.net/)
The larger the word produced, the more often it appeared in the source text. The wordle below shows that most of the audience thought this example event was fun, interesting, exciting, creative and enjoyable, with a very low incidence of negative words.

 


4. Radar diagram

Draw the outline of a radar diagram representing a range of experiences from the event, and ask the audience to complete this appropriately to reflect their feelings (see example below). The closer the point is to the experience, the higher the audience rated that experience.


5. Five people

After the event, think of 5 very different people and describe how you would explain the event to each of them, e.g.

  1. Young child aged 5-7 years
  2. Elderly person aged >70 years
  3. Youth worker
  4. Middle-aged person of low socioeconomic status
  5. Young mum

This can not only help you to better understand the purpose of the event, but can also help you to find any errors and/or determine what you might change if you were to re-run the event, even if that person or age group is not within your target age range.


6. Five words

After the event, ask a variety of different people with different levels of involvement in the event to sum it up for you using 5 separate words.

These people could be:

  1. The project organiser (i.e. yourself)
  2. A senior member of staff from your organisation
  3. A member of staff volunteering at the event
  4. A non-staff member volunteering at the event
  5. One member of the audience/public

7. Design a flyer

Write an advertisement, design a flyer or create a poster for your event after it has occurred. Use appropriate text and images and use an appropriate writing style for the target audience.

– Does this differ from your original advertisement, flyer or poster?

If so, how? E.g. is it more or less scientific, is the later version more colourful, does it use more or less images, have you used a different font, etc?

– This my help you to determine a true target audience for this type of event.


8. Round robin evaluation

  • Each person in the audience/group writes 2 positive and 2 negative comments on one side of card, and on the other side they write 2 things they liked and 2 things they didn’t like.
  • Cards are then all passed round and everyone rates each comment on a scale of 1 to 5 (1 being strongly disagree and 5 being strongly agree).
  • Statements and total average scores are collated and then acted upon to improve future events.
  • This method determines how representative each individuals opinions are.

9. Confidence log

Teachers, staff and/or volunteers could complete a confidence log to see if their confidence in teaching a subject has changed since attending the event.

e.g. How confident are you in teaching this subject area/running this event yourself? (Please tick the appropriate option)

[  ] Very confident
[  ] Confident
[  ] Some confidence
[  ] Little confidence
[  ] No confidence

Results could then be transferred into a table for collation.

Very confident Confident Some confidence Little confidence No confidence
Pre-event 11 9 28 14 8
Post-event 20 17 22 9 2

10. Blob tree

Developed by Pip Wilson to show how people feel about taking part in a certain task or event.

These may be handed out at the beginning and end of a session/event to see if the audience’s feelings/confidence about the topic have changed.

e.g. before the event takes place an individual may know very little about the topic and feel that they are at the bottom of the tree or swinging uncontrollably from the tree. However, after the event (if successful!) that individual may feel as though they are standing at the top of the tree or they are in the middle helping others to understand.


11. Observation framework

Allows the observer to record any situations that may arise during an event and its specific time of occurrence such as relevant incidents, questions, interruptions to the event, which may help to improve later events.

This should be completed during the event and any comments should be passed to the project organiser/speaker as soon as possible after the event for them to consider.

The observer should try to capture the moods and feelings of the audience and write down as much as possible to include in the evaluation, including:

  • Event name and date
  • Observer’s name
  • Activity/session
  • Level of interaction
  • Head count
  • Male:female ratio
  • Average age
  • Enthusiasm
  • Relevance to aims
  • Other comments

12. Focus group

  • The group should meet for ~90 minutes no more than 2 weeks after the event has taken place.
  • The group should contain no more than 15 and no fewer than 10 members.
  • Group members should be selected randomly from the audience, trying to select individuals across a broad age range (if the event is open to the general public).
  • Before the session the group should be made to understand that any personal information or data that they provide will remain confidential, and the purpose for collecting their comments and where/how they will be recorded should be explained.
  • Unique questions should be asked of the group in order to find out how much the group learnt from the event.
  • Two evaluators should be present during the group session, one to ask the questions and start the discussions, and a second to record the conversations and observations relating to the group’s behaviour.

13. Interview

  • Audience members could be interviewed on a 1:1 basis no longer than 2 weeks after the event.
  • Interview questions will be specific for each project or event.
  • Individuals to take part should be selected randomly from the audience
  • Before the session the individual should be made to understand that any personal information or data that they provide will remain confidential, and the purpose for collecting their comments and where/how they will be recorded should be explained.
  • The interview should last no longer than 30 minutes and should be kept informal so as to keep the interviewee relaxed.
  • The interviewer should record any comments and any visible signs from body language during the interview.
  • A dictaphone could also be used to record the interview so that all dialogue is captured effectively and used within the evaluation report to improve future events.

14. Project notebook

  • Keep a project notebook or diary.
  • This should be kept from the very beginning of the thought process that is undertaken before a project is started and continuously updated throughout the whole phase of the project, finally coming to an end when the project has been signed off and the full evaluation report has been completed.
  • All of the project organiser’s thoughts, queries and questions should be recorded.
  • It is important to record all dates and times that entries are made so that specific issues/errors/problems can be avoided in future, and if similar problems arise in future projects it can be clear how they were resolved and the timescale needed to get the project back on track.
  • All of the audience reactions, questions and feedback could also be recorded here.
  • The outcome or result that was carried out as a consequence of the feedback provided by the audience should also be noted.

15. Implementation log

  • Useful to record any changes that were made during each stage of the project.
  • Can help to see who was responsible for the change, if the project improved as a result of the change, and if that change would help the project to be more successful in the future.
Session Original plan Actual delivery
1 Completion of first questionnaire and hand out 3-day food diaries Questionnaires were taken home to complete and handed in at the next session as time was limited
2 LGC staff asked us not to attend so that they could see how the cooking schedule would work within the time allowed No HNR attendance
3 Salt and sugar Salt and sugar worksheets
4 Snack swaps Snack swaps
5 Vitamins and minerals Staff from the main LGC programme attended for filming purposes and requested no HNR attendance
6 Final evaluation, 3-day food diary and overview of the LGC programme No food diaries were given out as these were meant to be used as a comparison to see if the pupils’ diets had changed at all over the 6-week period but this was unrealistic and the children found the diaries difficult to complete with a very small amount actually handed back in from the first session

16. Questionnaire

  • Questionnaires can take many forms including checklists, multiple choice questions, open and closed questions, etc.
  • Successful questionnaires tend to include a combination of formats, with the opportunity to answer questions in a variety of styles.
  • Should be simple to understand and not too long (2 sides of A4 maximum).
  • The questionnaire should be collated so that it answers the information required without requiring the completer to write too much or for too long.
  • Pictures, visuals, tick boxes and multiple choice questions can be useful to save time and get the required information.

17. Star ratings

  • A list of different areas of the project/event with the opportunity to give a rating for each.
  • Ratings can be provided on a scale of 1 to 5, where 1 is strongly disagree and 5 is strongly agree.
  • Enables the project reviewer to see which aspects were of most interest to the target audience, and to see where any improvements could be made.


18. Reactionnaire

  • Used for the first 2-3 times that a new project is run to enable fine tuning and identify any problem areas.
  • Allows you to gain reactions from the audience immediately after the event.


19. Multivoting

  • Narrows a large list of possibilities down to a smaller list of the top priorities or to a final selection.
  • Preferable to normal voting as it allows an item that is favoured by all, but not necessarily the top choice of any, to rise to the top.

HOW TO:

  1. Display a numbered list of options;
  2. Decide how many items must be on the final reduced list, and also how many choices each person can vote for (usually 5);
  3. Each person chooses their top 5 items that they think are most important, and then they rank these choices in order of priority (1st choice ranks highest);
  4.  Tally votes;
  5. For each item, the rankings are totalled next to the individual rankings to show a clear picture of the top priorities or most popular choices.

20. Affinity diagram

  • Organises a large number of ideas into their natural relationships
  • Can be used when you are confronted with many disordered facts/ideas, when issues seem too complex to understand or when group consensus is necessary

HOW TO:

  1. Each person should write down each idea on a separate piece of card;
  2. Related ideas should be placed next to each other;
  3. Repeat until all ideas are grouped (if an idea seems to fit into 2 groups, make a second note of it);
  4. Everyone should now discuss the shape of the chart, any surprising patterns, and any reasons that people found to move an idea from one group to another (a few more changes may be made after these discussions if appropriate);
  5. When final groups have been selected, choose a heading for each group;
  6. Combine groups into supergroups if appropriate.

21. SWOT analysis

  • SWOT stands for strengths, weaknesses, opportunities and threats
  • Strengths and weaknesses are internal factors
  • Opportunities and threats are external factors
  • Can be used in the first stage of planning to focus on the key issues of the project/event and what it needs to achieve in order to be novel and unique
  • Can be very subjective
  • TOWS analysis can also be used which simply looks at the negative factors first in order to turn them into positive factors
  • Keep your SWOT short, simple and specific


22. Decision matrix

Evaluates and prioritises a list of options

HOW TO:

  1. Brainstorm the determining factors for evaluation that are appropriate to the project or event;
  2. Discuss and refine the list of factors to those that are most important;
  3. Draw a matrix. Write the factors as labels along one edge and the list of options along the other edge. Evaluate each choice against the determining factors using a rating scale;
  4. The winner is the highest number in the totals column.
Safety Cost:benefit analysis Fun Educational Fits with dept. objectives Total
Exhibition
Stand
Interactive display
Workshop
Talk/seminar

In conclusion…

  • Each method of evaluation will have its own combination of costs and benefits
  • Plan effective evaluation at the outset of the project;
  • Evaluation is a way to evaluate if a project is running well, if not, you can make changes as you go along;
  • Match your methods to what you need to find out;
  • Plan a project timeline to track the progress of a project and amend as you go to use for a re-run of that event.