In the context of our grants program, we use the term ‘evaluation’ to describe two broad evaluation purposes.
The first is the evaluation that goes on (or should go on) at key steps as your project unfolds, to make sure that the final product is as good as it could be.
So, for example, if you are developing a plain language information brochure about the law, all those things that go into developing the brochure, such as user testing draft(s) of the publication, getting legal and plain language experts to check and improve the content, are part of this meaning of evaluation. Similarly, if you are working on a research report, you would send drafts to colleagues with expertise to critically comment on drafts, prior to completion of the published version
This type of evaluation is all about the formation of the product, and most of us probably do it without even recognising that we are. Nevertheless it is a very important form of evaluation, and in the interests of supporting projects of high quality, the Foundation pays close attention to the this type of evaluation in the application.
The second context where we use the term evaluation is to assess whether at the end, the project has achieved what it set out to achieve (i.e. the outcome). This is what most of us probably think of when we hear the term ‘evaluation’.
Evaluation of the outcome aims to measure the project’s actual outcome against the original intended outcome. For example, if you ran a seminar to increase the knowledge of a target audience about a certain issue, you might survey participants before the seminar, and then again some time after, to assess whether there was an increase in their knowledge.
Two ways to evaluate the outcome
Sometimes, it may be too difficult or too expensive to truly measure the actual impact of a project. For example, your project may have developed a radio program on legal issues that is played on community radio stations across the state, with the aim of improving community awareness of those issues. While there are some measures of ‘outcome’ (such as measuring before and after legal inquiries to certain services, etc), to really evaluate the project’s impact you would probably need to conduct some form of expensive state-wide population survey (which is almost certainly far more expensive that the project itself.)
In such cases you may need to look for ‘proxies’ to be measured instead of real indicators of impact. Quite often the most useful proxies relate to the process you used to do the project. So in this case I might evaluate the process I used to develop my radio program, and then how I ensured it reached a wide audience by asking:
As we have just seen, evaluation does at least two important things. It can help you ensure that the product of the project is as good as it can be, and it helps you know if your project has achieved its aim, which will in turn help you to answer a range of other questions - such as is this the most appropriate strategy to meet this need, do I need to implement this project to a wider audience?
Of course, in answering these two questions, many others might be answered as well, such as how the product or project can be improved.
Evaluating your project
Most Foundation-funded projects will require an appropriate evaluation strategy, usually with components that contribute to the formation of the product (to make the product the best possible on current knowledge/practice), and that assess the outcomes (either by evaluating the outcome or, by proxy, evaluating the process to achieve the outcome).
What is ‘appropriate’ will depend on the type and nature of the project. For example:
Examples
Below are examples of questions to ask and information that can be gathered to evaluate plain language law, community legal education and awareness raising projects.
For each example, there is a broad evaluation question that represents the main question you would like answered. Recognising that sometimes it is not feasible to directly answer this main question, we suggest some more specific questions which can be more easily answered, but still provide useful information.
It may be possible to answer this question through an evaluation of the outcome, if the target audience is well defined and accessible (eg prisoners, people in nursing homes etc). This may be possible by gathering information through a survey or interviews to assess the target group’s levels of knowledge on the topic prior to their accessing the resource. At an appropriate time after the resource has been distributed to the target group, you may wish to re-test their levels of knowledge on the topic.
However, if your target audience is, for example ‘all tenants in NSW’, then a comprehensive evaluation of whether you did reach all tenants in NSW may be beyond the scope of the project. In this case, your outcome evaluation could focus on the overall process you used to do the project. An alternative may be to focus on the overall process of producing the product (i.e. following best practice principles in project management and in the production, marketing and distribution of the product), and whether it was done well (eg was it user tested, checked by an expert in the field, etc). This is how the formation evaluation stage can contribute to the outcome evaluation.
You may ask other questions more directly related to the development and distribution of the product which might tell you about the conditions in which knowledge is most likely to have been increased such as the following:
| Is the content accurate? | Use legal specialists to review the content |
| Was the content useful? | Carry out user testing during the development of the resource. Once the publication is complete or before a new version is produced, have a focus group of users to discuss whether the publication was useful and usable. |
| Who uses or used the resource? | Keep records of who requested copies of the resource. However please note that distribution lists can be a misleading indicator of use as resources can be sent to organisations and never used. Actual requests are a more accurate indicator of demand and possible use. Carry out a survey of users three months after the production of the resource. |
| Was the resource produced according to best practice guidelines? | Check your resource against the Publishing Checklist guidelines |
Depending on the budget and timeframes of the project, the question “did the workshop increase the knowledge of the target group?” may be able to be answered through conducting an evaluation of the outcome. When gathering information to answer this question, one method may be to assess the target group’s levels of knowledge on the topic prior to undertaking the workshop.
This could be done by asking participants to complete a short survey at the beginning of the workshop or telephone interviews with the target group prior to the workshop. At an appropriate time after the target group have participated in the workshop, re-test their levels of knowledge on the topic through a phone or mail out survey or through telephone or face to face interviews.
However, if this question is too difficult to answer within the timeframe and budget, an evaluation of the content and process of developing and conducting the workshop and whether it was done well may be a feasible alternative.
You may ask other questions more directly related to the development and delivery of the workshops which might tell you about the conditions in which knowledge is most likely to have been increased such as the following:
| Is the content accurate? | Use legal specialists to prepare the content. |
| Was the workshop attended by the target group? | Was the workshop promoted through the appropriate channels used by the target group? Was the workshop held in a venue accessible to the target group?How many people came and where were they from? |
| Was the content useful to the people attending? | Was the content user tested with representatives of the target group during development? Observe participants during the workshop, e.g. were they engaged, did they ask relevant questions etc? Use a questionnaire to test participant knowledge before and after the seminar to see if their legal knowledge has increased. |
| Did the target group use the information afterwards? | For example, telephone a sample of workshop participants six weeks after the workshop and ask them if they can identify any changes to their practice arising from the workshop. |
In undertaking an awareness-raising project, the aim may be to change the behaviour of a certain target group.
Depending on the budget and timeframes of the project this question “did people change their behaviour after the campaign?” may be able to be answered by conducting an evaluation of the outcomes if the target group is specific and accessible.
This may be possible by contacting a sample of the target group prior to the campaign and asking them what they are currently doing in relation to the issue, through a survey or interviews. Perhaps then follow up with a post campaign survey or interview to find out if the sample of the target group has changed their behaviour in relation to the issue.
However, this question may be hard to answer within the timeframe and budget of the project. If that is the case, an evaluation of the content and style of the campaign (i.e. user testing of key messages and design) as well as an evaluation of the overall process of developing and running the campaign and whether it was done well, may be a feasible alternative.
You may ask other questions more directly related to the development and delivery of the awareness raising campaign which might tell you about the conditions in which people may change their behaviour such as the following:
| Are the campaign materials appropriate? | User testing during the development of the materials. |
| Was awareness raised during the campaign? | Use a questionnaire with a sample of the target audience to test knowledge before and after the campaign. |
| Was there a change in the referral patterns to services? Is there a lasting effect in changes to referral patterns? | Monitoring service referral statistics before, during and after the project. |
| Aim | Description/strategy | Evaluation questions | Data Source |
| To improve the knowledge of community workers in rural NSW about their legal responsibilities in relation to children at risk of abuse or neglect. | The project will develop and distribute a self-paced DVD that assists community services staff in rural NSW to learn about their mandatory child protection reporting responsibilities. | Did you increase the knowledge of community workers? | Survey or interview community workers to assess their levels of knowledge on the topic prior to their accessing the resource. Re-test their levels of knowledge on the topic at an appropriate time after the resource has been distributed. |
 | | Was the content accurate? | The content will be developed to be consistent with the Child Protection Interagency Guidelines. An expert reference panel will be established that will include child protection specialists and legal specialists. The expert panel will review the draft content and final content for accuracy. |
| | Was the content useful to learners? | a) User testing of draft content with a group of potential users b) Survey a sample of users after the product has been distributed. Questions could include: was it useful, did you like the content and layout/format, what would you do differently? |
| | Who uses the resource? | Survey sample of people the resource was distributed to three months after distribution to determine whether they used it. |
| Aim | Description/strategy | Evaluation questions | Data Source |
| To improve the knowledge of Aboriginal tenants in NSW about their legal rights and obligations when renting. | Aboriginal tenants in NSW lack knowledge about their rights and obligations when renting. This project aims to address this by producing a wallet card with key information about NSW tenancy laws and by running community legal education sessions for community leaders. | Was the content accurate? | The content of both the workshop and wallet card will be reviewed by legal specialists in tenancy. |
| | Was the content useful to users? | Focus groups for user testing during development of both the wallet card and community legal education sessions. A survey of those who attended the community legal education session, three months later will also be undertaken. |
| | Who uses the wallet card? | This will be answered through following up with a sample of those who have used it three months on. Questions could include: was it useful, did you like the content and layout/format, what would you do differently, etc. |
| | a) Was the wallet card developed according to best practice guidelines? | a) Production process conforms with Foundation best practice guidelines for plain language legal publications. |
| | b) Were the community legal education sessions run according to best practice guidelines? | b) Community legal education conforms to NACLC CLE best practice guidelines. |
| | Was the community legal education session attended by the target audience? | Participant lists |
