ContentJust Search pageLJF site navigationLeft navigation links
LJF Logo
Publications sectionJustice Awards sectionResearch sectionGrants sectionPlain language law sectionNetworks section
Just Search
 
Research Report: Justice made to measure: NSW legal needs survey in disadvantaged areas
cover image

Justice made to measure: NSW legal needs survey in disadvantaged areas  ( 2006 )  Cite this report

Ch 1. Introduction



Print chapter
Search or view whole report
View PDF

Methodology of legal needs studies


There has been a lack of harmonisation in the survey methodologies employed to measure legal needs around the world. This lack of international harmonisation severely limits the ability to compare the findings from different surveys. For example, it is difficult to reach firm conclusions about the relative incidence and resolution rates of legal issues in different countries, jurisdictions and populations. The types of methodological differences across survey studies of legal needs and their consequences for interpreting survey findings are discussed below.

Jurisdiction

The conduct of legal needs surveys in different countries means that legal needs are being assessed within different systems of law and different networks of legal services. The number and type of legal issues that people typically encounter in a given jurisdiction is likely to be influenced by the adequacy of that jurisdiction’s law in relation to different legal rights.

Similarly, people’s responses to legal issues, and the outcome of legal issues, are likely to be influenced by the mechanisms available for legal resolution. There are differences between jurisdictions in the types of legal services that are available (e.g. legal advice services, legal aid, courts, tribunals, dispute resolution agencies), and in the efficiency and effectiveness with which these services are able to resolve legal issues. There are also differences within jurisdictions in the extent to which effective non-legal resolution mechanisms are available.

The likely impact of jurisdictional differences on the incidence and handling of legal needs highlights the importance of measuring legal need within the context of each jurisdiction of interest—including within the Australian context.

Population type

The study of legal needs in different countries also means that inherent differences between populations may come into play. For example, there may be significant demographic, geographical, cultural and attitudinal differences that reflect different life circumstances and impact on the propensity to experience particular types of legal problems or to respond in particular ways to legal issues. Such differences between the populations studied reduce the comparability of survey findings.

A major difference between survey studies of legal needs is that some assess legal needs in general populations, whereas others address legal needs in disadvantaged populations. As will be detailed later, there is accumulating evidence that socioeconomic disadvantage is associated with an increased vulnerability to legal problems (e.g. Genn & Paterson 2001; Pleasence et al. 2004b), so higher incidence rates might be expected from studies examining disadvantaged rather than general population samples. Furthermore, disadvantage has been argued to affect access to justice once legal problems are experienced, through a variety of mechanisms, including direct exclusion, direct and indirect discrimination, non-equivalent legal rights, inability to enforce legal rights, ignorance of the law and legal processes, lack of communication skills, and lack of appropriate advocacy and support services (Schetzer, Mullins & Buonamano 2002). Thus, differences between general and disadvantaged populations might also be expected in the resolution of legal events.

Definition and measurement of disadvantage

It cannot be assumed that all studies examining the legal needs of disadvantaged populations are assessing similar populations. A variety of sociodemographic groups have been identified in the literature as being relatively more vulnerable to suffering disadvantage. These groups include women, young people, older people, Indigenous people, people from a non-English speaking or different cultural background, people with a disability, people with low or no income, people with a low level of education or literacy, people living in institutions (e.g. prisons, juvenile correction centres, immigration detention centres, nursing homes), and homosexual and transgender individuals (see Schetzer et al. 2002). The type and level of disadvantage experienced by these various sociodemographic groups may be different (Australian Bureau of Statistics (ABS) 2004c), and legal needs survey studies of disadvantaged populations do not necessarily assess the same mix of these groups.

Furthermore, there is a lack of consensus concerning the definition and measurement of socioeconomic disadvantage (see ABS 2004c; Ainley, Graetz, Long & Batten 1995; Harding, Lloyd & Greenwell 2001; Marks, McMillan, Jones & Ainley 2000). This lack of consensus restricts the comparability of legal needs surveys in different disadvantaged samples. In broad terms, socioeconomic disadvantage is construed as some sort of deprivation, hardship or inequality with regard to a person’s standard of living, well-being or other life opportunities or outcomes, resulting from the person’s social or economic status (ABS 2004c; Ainley et al. 1995; Marks et al. 2000).

Socioeconomic status and socioeconomic disadvantage are usually portrayed as multidimensional concepts. However, there is little consensus about the definitive set of indicators that should be used to measure the different dimensions of socioeconomic status or the various outcomes that indicate some aspect of socioeconomic disadvantage (see ABS 2004c; Ainley et al. 1995; Marks et al. 2000).1 While income, educational attainment and occupational status are usually seen as key components of socioeconomic status, and low levels of these variables are usually viewed as indicating socioeconomic disadvantage, there is little agreement about what other variables should be considered (ABS 2003b; Ainley et al. 1995). A wide range of variables have been inconsistently used across studies to measure socioeconomic disadvantage, including poor language proficiency, belonging to a minority ethnic group, Indigenous background, single-parent or blended family structure, family breakdown, poor health, poor housing, disadvantageous geographical location, residential mobility, crime victimisation and transport difficulties (see ABS 2003b, 2004c; Ainley et al. 1995; Marks et al. 2000; Vinson 1999).

Furthermore, consensus is also lacking about what levels of the key indicators of socioeconomic status constitute disadvantage. In fact, there has even been protracted disagreement about the measurement of the single most widely used indicator of socioeconomic disadvantage, namely poverty (Harding et al. 2001; Saunders 2003). For example, the poverty line has been drawn at varying proportions of average or median incomes (Harding et al. 2001). In addition, poverty measures have been based on different units of income including personal, family and household income, and on different indicators of standard of living, including disposable income and disposable income after housing costs have been met (Harding et al. 2001). It has been readily demonstrated that estimates of the number of people in poverty are notoriously sensitive to the specific index used to measure poverty (Harding et al. 2001; Saunders 2003).

It has also been argued that the concept of poverty is too narrow as a framework for disadvantage, as exemplified by the frequent lack of overlap between measures of income poverty and a range of other indicators of social inequality (Arthurson & Jacobs 2003; Saunders 2003).

In recent years, there has been growing interest in the concept of social exclusion as a broader framework for understanding socioeconomic disadvantage (ABS 2004c; Saunders 2003). In general terms, social exclusion refers to a lack of participation in mainstream societal activities and a lack of access to standards of living, rights, goods and services enjoyed by the majority of society (Arthurson & Jacobs 2003). Although a precise, universally accepted definition of social exclusion has also remained elusive, it has been argued that the concept of social exclusion has considerable theoretical import (Arthurson & Jacobs 2003; Saunders 2003). Social exclusion is purported to highlight the multidimensional nature of disadvantage, including the range of causes of disadvantage, the different dimensions of society where disadvantage is manifest (e.g. social, economic, legal/political and cultural/moral dimensions) and the variety of outcomes that constitute exclusion. Social exclusion is also argued to focus attention on the dynamic nature of disadvantage, suggesting that disadvantage is not a static, permanent nor pervasive condition, but that people can move between inclusion and exclusion at different points in time and with respect to different aspects of their lives (Arthurson & Jacobs 2003; Saunders 2003). Furthermore, the notion of social exclusion has stimulated considerable debate about the extent to which disadvantaged individuals are responsible for their predicament versus the extent to which outside agents or societal structures shape the experience of inequality (Arthurson & Jacobs 2003; Saunders 2003).

Many past legal needs surveys conducted in disadvantaged samples have relied on some type of income-based measure of poverty or financial hardship as the single criterion for selecting interviewees (e.g. ABA 1994; Dale 2000; Rush 1999; Schulman, Ronca & Bucuvalas Inc. 2003; Spangenberg Group 1989; Task Force on Civil Equal Justice Funding (Task Force) 2003). However, as already noted, low income does not always correlate highly with other measures of disadvantage. Other studies have used multiple indicators to select disadvantaged areas from which to draw their sample (e.g. Cass & Sackville 1975), while still others have examined specific disadvantaged groups such as the homeless or those in temporary housing accommodation (e.g. Dale 2000; Pleasence et al. 2004b). The differences in the measurement of disadvantage need to be kept in mind when comparing the results of legal needs surveys conducted in different disadvantaged populations.

Measurement of legal needs

Differences in the measurement of legal needs across studies are also likely to influence the types of legal needs identified, the incidence rates obtained, the reported reactions to legal needs and the resolution rates obtained. For example, studies assessing legal needs have measured the incidence of legal events over reference periods of different lengths, used different types of survey methods, used different definitions of legal needs and measured various key variables in different ways. These are discussed in turn below.

Reference period

The reference periods used to measure the incidence and resolution of legal needs have varied from 12 months in some of the recent surveys in the United States (e.g. ABA 1994; Schulman et al. 2003) to five and a half years in the United Kingdom (Genn 1999; Genn & Paterson 2001) to the respondent’s lifetime (e.g. Curran 1977). Clearly, the longer the time period, the greater the opportunity for legal events to be experienced, and the greater the opportunity for events that are experienced to be resolved. Thus, other things being equal, longer reference periods tend to result in higher incidence and resolution rates.

It is also worth noting that there are different advantages and disadvantages associated with using reference periods of different lengths. A disadvantage of a short time period is that there is only limited opportunity to detect infrequently or seasonally occurring legal problem types (Pleasence et al. 2004b). However, there are also disadvantages to using long reference periods in retrospective surveys where individuals are asked to recall past events, such as past legal events. For example, longer reference periods for recall are associated with increased risk of inaccurate or decayed memory of events (e.g. Biemer, Groves, Lyberg, Mathiowetz & Sudman 1991; Rubin 1982; Sudman & Bradburn 1973).2

Survey method

The advantage of using the survey method to measure legal events is that it has the potential to measure both expressed and unexpressed legal need. That is, the survey method can be used to provide information not only about legal events that result in the use of legal services (expressed need), but also about legal events that do not result in legal service provision (unexpressed or latent need). For example, surveys can provide information on legal needs that are ignored and legal needs that are resolved outside the traditional sphere of legal services. As a result, compared with data on legal service use (e.g. Scott, Eyland, Gray, Zhou & Coumarelos 2004), survey data have the potential to provide a more comprehensive picture of legal need.

However, it is worth noting that surveys are subject to a variety of measurement errors and limitations which should be kept in mind when considering survey findings. For example, as already mentioned, retrospective surveys are subject to a variety of recall errors (Biemer et al. 1991; Sudman & Bradburn 1973). Surveys are also subject to various forms of response bias, such as the tendency to offer socially desirable answers to threatening or sensitive questions (e.g. Biemer et al. 1991; Oppenheim 1992; Presser, Rothgeb, Couper, Lessler, Martin, Martin & Singer 2004). Furthermore, the usefulness of survey findings depends on the appropriateness of factors such as survey design, sampling strategies and data collection strategies (e.g. Biemer et al. 1991; Oppenheim 1992; Scheaffer, Mendenhall & Ott 1986).

In terms of sampling strategies, it is worth noting that not all the surveys of legal needs used random probability sampling. The advantage of large-scale random sampling is that the findings obtained for the sample are generalisable to the population from which the sample was drawn. Findings of surveys using non-random sampling techniques should be treated more tentatively. It should be noted, however, that even if a sample is randomly drawn, a poor response rate can result in a biased sample. Some survey studies of legal needs report low response rates of below 50 per cent (e.g. Schulman et al. 2003; Task Force 2003), while others fail to report response rates (e.g. Cass & Sackville 1975; Dale 2000; National Consumer Council (NCC) 1995; Rush 1999; Spangenberg Group 1989). Furthermore, given that there is no standard way of computing response rate (Biemer & Lyberg 2003; Groves 1989), it is sometimes difficult to compare response rates across studies, particularly given that some studies fail to adequately explain how the reported response rates were calculated (e.g. Curran 1977).

In addition, past studies have used different modes of data collection. Some studies used telephone interviews while others used face-to-face interviews. Face-to-face interviews tend to be more conducive to establishing trust and rapport, and tend to provide greater opportunity for longer and more in-depth probing (Biemer et al. 1991). As a result, face-to-face interviews are likely to be more conducive to assessing a large number of legal needs. Telephone interviews have the advantage of being less labour intensive, and hence, less expensive (Biemer et al. 1991). It has also been suggested that the greater anonymity of the telephone interview compared with the face-to-face interview may improve reporting on sensitive topics (Biemer et al. 1991; Oppenheim 1992).

The unit of measurement has also varied across studies, with some studies examining the legal issues of individuals (e.g. Genn 1999; Genn & Paterson 2001; Pleasence et al. 2004b) and other studies examining legal issues within households (e.g. ABA 1994; Schulman et al. 2003). Examining problems faced by all members within a household rather than problems faced only by the individual being surveyed is likely to increase reported incidence rates.

Concept of legal needs

It is worth noting that the study of legal needs has largely been isolated from the study of the broader array of human needs that drive human behaviour. The broader concept of human needs, and the forces affecting the emergence and ultimate satisfaction of human needs, are well charted in other disciplines and guide much of the psychosocial literature on consumer behaviour, consumer satisfaction, services marketing and communications theory (e.g. Hanna 1980; Maslow 1970; Sheth, Mittal & Newman 1999). Although examining the concept of legal needs within the broader psychosocial context has the potential to deepen our understanding of legal needs and legal service provision, such an exercise is beyond the scope of the present study.

Within the socio-legal literature, there appears to be no consensus about the precise meaning of the term ‘legal needs’. This term is often used without explicit definition and is often used interchangeably with the term ‘access to justice’ (Schetzer et al. 2002). However, there is a tendency towards using ‘legal needs’ to refer to rights within the existing legal system, and using ‘access to justice’ more broadly to refer to the mechanisms for obtaining justice, either within the existing legal system or via reform to the existing legal system (Schetzer et al. 2002).

While there is general agreement that actively seeking a legal resolution to a problem reflects the existence of some form of legal need, there is little consensus about what else indicates the existence of legal need. Some of the earlier studies used the narrow approach of focusing on legal problems typically taken to legal practitioners (e.g. Griffiths 1977; Royal Commission on Legal Services 1979). However, such studies have been criticised for being unable to provide information on legal issues that may be dealt with in other ways or that remain unresolved (Genn 1999). Furthermore, such a narrow approach does not acknowledge that, in some instances, failing to seek a legal resolution does not necessarily imply the absence of legal need, but may merely indicate:

  • the individual’s lack of awareness that the issue is a legal one or that the issue has a potential legal resolution
  • failings in the legal system (e.g. barriers to efficient, inexpensive legal resolution)
  • the existence of other, non-legal means for resolution (Schetzer et al. 2002).

Genn (1999) adopts a more sophisticated approach that does not rely on respondents’ legal knowledge for the identification of legal problems. Genn’s (1999) study measures ‘justiciable events’ where a justiciable event is:

    a matter … which raised legal issues, whether or not it was recognised by the respondent as being ‘legal’ and whether or not any action taken … involved … the civil justice system (p. 12).

The advantage of this approach is that it minimises the under-reporting of legal issues by allowing for the inclusion of legal issues that are handled outside the justice system, legal issues that are not always recognised as such, and legal issues that are ignored or remain unresolved.

Studies have also differed in terms of the number of legal problems they attempt to measure. Some studies assessed fewer than 30 legal needs or events (e.g. Cass & Sackville 1975; Curran 1977; Fishwick 1992), whereas others assessed 60 or more (e.g. ABA 1994; Genn 1999; Genn & Paterson 2001; Pleasence et al. 2004b). Not only do the number and type of legal issues examined vary, but the definitions of a given legal issue are not always the same. Furthermore, the types of legal issues included under categories with similar names are sometimes different.

Rush (1999) argued that many studies also tend to focus too narrowly on so-called civil legal issues and exclude criminal and family issues. They argue that civil, criminal and family legal issues should be measured and differentiated from one another because they tend to be related to different demographic profiles.

Clearly, the reported incidence and resolution rates are likely to be affected by the number, definition, type and range of legal needs examined. For example, reported incidence rates are likely to be increased by the inclusion of a broad rather than narrow range of legal needs, and by the inclusion of frequently occurring rather than relatively rare legal events. The inclusion of legal problems that are complicated or are difficult or time-consuming to resolve is likely to decrease reported resolution rates.

Measurement of other key variables

There also appears to be little consensus about the definition and measurement of other key variables that are often assessed by survey studies of legal needs, such as the response to legal needs, the outcome of legal needs, satisfaction with legal services and satisfaction with the outcome of legal needs. Again, such differences need to be considered when interpreting the findings of different studies.

While some studies focus on legal issues where individuals consult a legal practitioner or a particular type of legal service agency (e.g. Cass & Sackville 1975; Curran 1977; Dale 2000; Fishwick 1992; Rush 1999; Schulman et al. 2003), other studies more broadly examine both legal and non-legal sources of advice in response to legal issues (e.g. ABA 1994; Genn 1999; Genn & Paterson 2001; Law and Justice Foundation of NSW (LJF) 2003; Maxwell et al. 1999; Pleasence et al. 2004b).

Studies also differ in the particular outcomes of legal issues that they measure. For example, resolution via agreement between the parties, adjudication, legal proceedings, or abandonment have been inconsistently assessed across different studies (see ABA 1994; Genn 1999; LJF 2003; Maxwell et al. 1999; NCC 1995; Pleasence et al. 2004b).

There have also been differences between studies in how they measure individuals’ perceptions of legal assistance and outcomes. For example, while some studies have focused on the perceived satisfaction with the outcome of events (ABA 1994; Dale 2000; LJF 2003; Schulman et al. 2003; Task Force 2003), others have focused on the achievement of various objectives (Genn 1999; Genn & Paterson 2001; Pleasence et al. 2004b). The concept of satisfaction has been the focus of considerable attention in the psychosocial literature, where a diversity of definitions and measurements of satisfaction have emerged. While there is a lack of consensus about the precise meaning and measurement of satisfaction, there is some agreement that satisfaction is a complex response which is shaped by the fulfillment of individual needs as well as by the fulfillment of expectations about quality and fairness (Oliver 1997). It should be kept in mind that obtained satisfaction rates are likely to be influenced to some extent by the precise definitions and measurements of satisfaction used.

Summary of legal needs survey studies

Table 1.1 summarises the methodological features of the major recent legal needs survey studies, including whether a general or disadvantaged population sample was surveyed, the type of survey used, the number of legal issues assessed and the length of the reference period used. The main findings from these survey studies are reviewed below. The findings on the incidence of legal needs are presented first, followed by the findings on response to legal needs and then the findings on the outcome of legal needs.

Table 1.1: An overview of major legal needs surveys

StudyPlace and time of surveySample descriptionData collection methodLegal issues and categoriesReference period
United States
Curran (1977)US
1973–1974
w general population random sample
w 2064 adults 18+ years
interviews29 problemslifetime
Spangenberg Group (1989)
(pilot for ABA 1994)
US (except Alaska, Hawaii) 1988w disadvantaged population random sample
w 500 low-income households
telephone interviews34 problems in
10 categories
1 year
American Bar Association
(ABA 1994)
US
1993
w general population random sample and disadvantaged population sample
w 3087 low- and moderate-income households
telephone and
in-person nterviews
67 questions in
17 categories
1 year
Dale (2000)Oregon
1999–2000
w disadvantaged population quota sample
w 1011 low- and moderate-income households
in-person interviews97 questions in
19 categories
1 year
Schulman et al. (2003)Massachusetts
2002
w 2000 low-income householdstelephone interviews104 situations in
14 categories
1 year
Task Force on Civil Equal Justice Funding (Task Force 2003)Washington state 2002–2003w disadvantaged population quota sample of 1333 low-income households in 15 demographic groups and
w disadvantaged population random sample of 810 low- & moderate-income households
in-person interviews

telephone interviews
56 civil legal issues
in 17 categories
1 year
United Kingdom
National Consumer Council (NCC 1995)England and Wales
1995
w general population random sample
w 8358 people aged 16+ years
in-person interviews13 types of serious
civil dispute
3 years
Genn (1999)England and Wales
1997–1998
w general population random sample
w 4125 adults 18+ years
in-person interviews60 events in
13 categories
5.5 years
Genn and Paterson (2001)Scotland
1998
w general population random sample
w 2684 adults 18+ years
in-person interviews60 events in
13 categories
5.5 years
Pleasence et al. (2004b)England and Wales
2001
w general population random sample
w 5611 adults 18+ years
in-person interviews78 problems in
18 categories
3.5 years
New Zealand
Maxwell et al. (1999)New Zealand
1997
w general population multi-stage stratified random probability sample
w 5431 people 15+ years
in-person interviews27 problems3 years
Australia
Cass and Sackville (1975)Sydney
1973
w disadvantaged sample from 3 Sydney areas with high poverty indicators
w 548 adults 18+ years
in-person interviews24 civil and family issues
in 6 categories
5 years
Fishwick (1992)NSW
1990
w general population random sample
w approx. 0.5% of NSW population 15+ years (0.5% of 4 480 800 persons)
in-person interviews11 legal events1 year
Rush Social Research
Agency (Rush 1999)
Australia
1998
w disadvantaged population quota sample
w 2229 adults 18+ years from low-income households
telephone interviewsCommonwealth law matters2–3 years
Law and Justice Foundation of NSW (LJF 2003)
(pilot for present study)
Bega Valley LGA, NSW 2002w disadvantaged population random sample
w 306 people 15+ years from Bega Valley LGA
in-person and telephone interviews84 events in
17 categories
1 year


In addition, although some authors appropriately distinguish between indicators of socioeconomic status and indicators of socioeconomic disadvantage (e.g. ABS 2003b), this distinction is not always apparent (Ainley et al. 1995). There has also been considerable debate about the merits of using single, multiple or composite indices to measure socioeconomic status and socioeconomic disadvantage (see Ainley et al. 1995, Marks et al. 2000).
Note that the use of extremely short recall reference periods can result in telescoping errors whereby events outside the reference period are inaccurately recalled as having occurred in the reference period (e.g. Biemer et al. 1991). Apart from the length of the reference period, other factors that can affect the accurate recall of an event include the salience of the event, the occurrence of other similar events and the type of question used to elicit recall (e.g. Biemer et al. 1991).

 In addition, although some authors appropriately distinguish between indicators of socioeconomic status and indicators of socioeconomic disadvantage (e.g. ABS 2003b), this distinction is not always apparent (Ainley et al. 1995). There has also been considerable debate about the merits of using single, multiple or composite indices to measure socioeconomic status and socioeconomic disadvantage (see Ainley et al. 1995, Marks et al. 2000).
 Note that the use of extremely short recall reference periods can result in telescoping errors whereby events outside the reference period are inaccurately recalled as having occurred in the reference period (e.g. Biemer et al. 1991). Apart from the length of the reference period, other factors that can affect the accurate recall of an event include the salience of the event, the occurrence of other similar events and the type of question used to elicit recall (e.g. Biemer et al. 1991).


CLOSE
Coumarelos, C, Wei , Z & Zhou, AH 2006, Justice made to measure: NSW legal needs survey in disadvantaged areas, Law and Justice Foundation of NSW, Sydney