Download paper: Justice Issues 16: Effectiveness of public legal assistance services: a discussion paper
Legal assistance services and strategies are designed to help people to resolve their legal problems. Those funding and providing services intend that these efforts actually make that difference, and do so efficiently given their limited resources. Given this aim, what does it mean to be ‘effective’ and how can this be demonstrated? As this paper will explain, measuring effectiveness is about measuring whether or not the program achieved its aim or intended outcome.
Recent interest in effectiveness is evident in many policy and strategic planning documents which include recommendations that legal assistance agencies’ activities should be evaluated in terms of the outcomes that they achieve and their associated costs. The National Partnership Agreement on Legal Assistance Services
(COAG 2010) is a recent example.
There are significant benefits to be gained by evaluating activities in terms of whether they achieve intended outcomes and these are outlined in this paper. However, demonstrating that a strategy actually caused
a desired outcome—that is, that the strategy rather than any other factor has actually made the difference—can be an ambitious task. Evaluating effectiveness in causal
terms requires careful planning, dedicated resources, and particular expertise and methodology. This is especially the case in sectors such as the legal assistance sector, which have not, as yet, widely used this approach.
Evaluation research can also be used to examine issues other than effectiveness and cost. For example, it can inform questions of process, such as why an activity succeeded or failed, and how was it perceived and received by key stakeholders. While these questions are also important, the focus of this paper is on the evaluation of the outcomes
of activities, that is, their effectiveness
In terms of scope, this paper provides an introduction to evaluating effectiveness rather than a ‘how to’ guide. A list of resources is included at the end of this paper for readers interested in comprehensive guides to conducting outcome-based evaluation.
A list of resources is included at the end of this paper for readers interested in comprehensive guides to conducting outcome-based evaluation.
The aim of this discussion paper is to encourage and support the evaluation of effectiveness in the legal assistance sector by:
- explaining the meaning of key terms such as effectiveness, cost-effectiveness, efficiency and evidence-based practice
- demonstrating how evaluating activities in terms of their outcomes can challenge commonly held assumptions and provide valuable evidence to inform practice and further strategy development
- indicating how outcomes can be identified for measurement
- introducing relevant evaluation methods, particularly in terms of their ability to accurately measure the effectiveness (and potentially cost-effectiveness and cost-benefit) of services.
The writing of this paper was greatly assisted by input from Maria Karras and Suzie Forell, document analysis and comments on earlier versions by Megan Laufer, and by comments and/or searching for relevant source documents by Abigail Gray, Maureen Ward and Anna Russell. Thanks also to Terry Beed, Jenny Lovric and Geoff Mulherin for comments on an earlier draft of the paper.