Methodology of legal needs surveys
Genn’s (1999) justiciable problem approach in the Paths to justice study in the UK has been applied to the more recent, ongoing English and Welsh Civil and Social Justice Survey (CSJS; Pleasence 2006; Pleasence et al. 2004c, 2010). This approach has also been adopted around the world, including in:
- Australia (Coumarelos et al. 2006)
- Bulgaria (Gramatikov 2008)
- Canada (Currie 2005, 2007b)
- China (Michelson 2007a, 2007b)
- Germany (Hommerich & Kilian 2007)(5)
- Hong Kong (Hong Kong Department of Justice (HKDOJ) 2008)
- Japan (Murayama 2007, 2008)
- the Netherlands (van Velthoven & Klein Haarhuis 2010; van Velthoven & ter Voert 2004)
- New Zealand (Ignite Research 2006; Maxwell, Smith, Shepherd & Morris 1999)
- Northern Ireland (Dignan 2006)
- Slovakia (cited in Hadfield 2010).(6)
In the US, state surveys broadly based on the national survey by the ABA (1994) have been conducted in 16 of the 50 states. The US surveys have tended to retain a heavier focus on access to lawyers and courts than have the surveys following Genn’s (1999) approach. Nonetheless, the US surveys conducted in the following states still canvassed some actions apart from using lawyers:
- Arizona (Arizona Foundation for Legal Services & Education (AFLSE) 2007)
- Massachusetts (Schulman Ronca Bucuvalas Inc. (Schulman) 2003)
- Nevada (Gene Kroupa & Associates (GKA) 2008)
- New Jersey (Legal Services of New Jersey (LSNJ) 2009; Miller & Srivastava 2002)
- Tennessee (Tennessee Alliance for Legal Services (TALS) 2004)
- Vermont (Committee on Equal Access to Legal Services (CEALS) 2001)
- Virginia (Schulman 2007)
- Washington (Task Force on Civil Equal Justice Funding (Task Force) 2003)
- Wisconsin (GKA 2006).
The remaining US surveys focused virtually exclusively on the use of legal advisers:
- Alabama (Alabama Access to Justice Commission (AAJC) 2009)
- Connecticut (Center for Survey and Research Analysis (CSRA) 2003)
- Georgia (Dale 2009)
- Illinois (Legal Aid Safety Net Steering Committee (LASNSC) 2005)
- Montana (Dale 2005)
- Oregon (Dale 2000)
- Utah (Dale 2007).
In addition to varying in their conceptualisations of legal need and socioeconomic disadvantage, legal needs surveys have differed in their methodology. This lack of methodological harmonsiation extends beyond the differences associated with the US- and Genn-based traditions and limits the comparability of surveys (Coumarelos et al. 2006). Some of the main methodological differences and their impact on interpreting findings are discussed below.
Legal needs surveys have been conducted in different jurisdictions, resulting in the assessment of legal needs within different systems of law, networks of legal services and infrastructures for legal remedy. Such jurisdictional variation is likely to affect the nature of legal need and resolution (Coumarelos et al. 2006). For example, the adequacy of the law in terms of legal rights may influence the legal problems experienced, the resolution strategies adopted and the outcomes achieved. The available network of legal services, including the adequacy of public legal services, may be critical in facilitating resolution and preventing the escalation of problems. The institutions of legal remedy, such as courts, tribunals and dispute resolution bodies, are also likely to affect the experience and handling of legal problems. For instance, differences between the US and the UK in legal infrastructure and practices have been argued to produce different problem-solving strategies in the two countries (Hadfield 2010; Sandefur 2009). In addition, it has been argued that the legal infrastructure can differentially affect various subgroups within a jurisdiction and can thus produce inequality in access to justice (Sandefur 2009).
There are also jurisdictional differences in the non-legal mechanisms available for resolving legal problems and in the broader social services available for dealing with any issues that may compound or exacerbate legal problems (Coumarelos et al. 2006; Forell, McCarron & Schetzer 2005). For example, it has been argued that major legal, social, economic and political transformations within countries can impact on the experience and resolution of legal problems and can complicate cross-jurisdictional comparisons (Gramatikov 2008; Hadfield 2010).
The likely impact of jurisdictional differences on the experience, handling and outcome of legal problems highlights the importance of measuring legal need within each jurisdiction of interest (Coumarelos et al. 2006).
Inherent differences between the populations studied, such as demographic, social, cultural, attitudinal and geographical differences, may reduce the comparability of surveys (Coumarelos et al. 2006). Such factors may influence life circumstances and may affect vulnerability to particular types of legal problems and the resolution strategies adopted.
One major difference between legal needs surveys is whether the sample is drawn from the general population or from a disadvantaged section of the population (Coumarelos et al. 2006). The US surveys all used disadvantaged samples comprising low-income households eligible for legal aid, whereas most of the surveys based on Genn’s (1999) approach assessed legal needs throughout the general population. Exceptions include the NSWLNS by Coumarelos et al. (2006) and the earlier of the two Canadian surveys (Currie 2005), which were based on Genn’s approach but used disadvantaged samples. The studies involving disadvantaged samples also used different criteria to select their samples, including single measures of low income (e.g. ABA 1994; Dale 2000; Rush 1999; Schulman 2003; Spangenberg Group 1989; Task Force 2003), multiple or composite measures of disadvantage within certain geographical areas (e.g. Cass & Sackville 1975; Coumarelos et al. 2006) and the investigation of specific disadvantaged groups, such as those with poor housing (e.g. Dale 2000; Pleasence et al. 2004c).
Given the relationship between socioeconomic disadvantage and legal need that has been observed within individual studies, it would be expected, other things being equal, that the incidence of legal problems would tend to be higher in the studies involving more disadvantaged samples (see Coumarelos et al. 2006; Currie 2007b; Genn 1999; Genn & Paterson 2001; Pleasence et al. 2004c).
Social, cultural and attitudinal dissimilarities between populations may also impact on the direct comparability of surveys conducted in different countries. For example, Genn and Paterson (2001) argued that attitudinal differences may have been responsible for their finding of a lower incidence of legal problems in Scotland than in England and Wales, despite the use of identical methodology. They suggested that the Scots were less likely to construe situations as problematic and, hence, were more likely to underreport problems due to their more fatalistic, community-oriented and self-assured attitudes. However, Pleasence (2006) contended that the lower incidence in Scotland may in part reflect real differences in life experience as a result of various other dissimilarities between the populations, such as geographical, demographic and jurisdictional dissimilarities.
In China, Michelson (2007a, 2007b) found that political connections and regional area affected the experience of legal problems and the lodging of official complaints. He suggested that historical, economic and social contexts affect disputing behaviour. Similarly, Murayama (2007) noted that cultural and institutional factors have been proposed to explain the lower litigation rate in Japan compared to Western countries.
Coverage of legal problems
The legal needs surveys adopting Genn’s (1999) approach have used survey instruments that differ in a number of ways. Similarly, the US surveys have used non-identical instruments. The use of non-identical instruments can impact on the legal problems captured and the responses and outcomes that predominate in the results. Some major sources of variation between survey instruments are described below.
Number and types of legal problems
The types of problems canvassed by legal needs surveys have varied considerably, ranging from fewer than 30 to more than 100 (see Coumarelos et al. 2006, pp. 14–15). Rush (1999) suggested that civil, criminal and family law issues should all be measured, as they tend to be related to different demographic profiles. However, some surveys have focused solely on civil issues, and others have examined only a restricted set of civil issues. For example, in New Zealand, Ignite Research (2006) focused only on civil issues for which grants of legal aid were available.
Furthermore, the definition and wording of each type of legal problem and the grouping of legal problems for reporting purposes have varied. Legal issues with the same name across surveys are not necessarily identical, and problem categories with the same name do not necessarily comprise the same set of specific problems. The results of different surveys are likely to be considerably affected by the number, type, range and definition of legal problems examined (Coumarelos et al. 2006). In particular, incidence rates are likely to increase with broader coverage of legal problems and with wider capture of commonly occurring problem types. The differential coverage of legal problems across studies is also likely to impact on the strategies, resolution rates and outcomes reported. For example, surveys have shown that the type of problem is a strong predictor of the response adopted, the duration of the problem and the nature of the outcome (e.g. Coumarelos et al. 2006; Currie 2007b; Dignan 2006; Pleasence 2006; van Velthoven & ter Voert 2004).
Legal needs surveys have typically attempted to canvass problems that involve a certain minimum level of legal need. The US studies sought to identify problems that can be remedied through the justice system and used threshold language to ‘rule out situations unlikely to produce legal need’ (Consortium 1994, p. 5). Many of the surveys following Genn’s (1999) approach purposely filtered out ‘trivial’ problems (e.g. Currie 2007b; Dignan 2006; Genn 1999; Genn & Paterson 2001; Gramatikov 2008; HKDOJ 2008; Ignite Research 2006; Pleasence 2006; Pleasence et al. 2010). Genn’s (1999) ‘triviality threshold’ involved focusing on problems that were ‘difficult to solve’ and deemed important enough to ‘warrant action’. This type of threshold may underestimate the incidence of legal problems. It focuses too heavily on whether respondents can accurately judge the severity of problems, the likely consequences of taking action and the likely barriers to resolution. Thus, this type of threshold may sometimes fail to capture serious problems because these problems were incorrectly judged to be either trivial or unsolvable, or because they were handled easily.
A few studies have included less serious problems in order to provide a more accurate estimate of legal problem prevalence (e.g. Coumarelos et al. 2006; Murayama 2007). The inclusion of less serious legal problems does not preclude the measurement of the level of severity of each problem. Thus, comparisons between severe and trivial problems can be undertaken. In addition to including less serious problems, Coumarelos et al. (2006) included a few events that were not ‘problematic’ but had legal implications (e.g. buying or selling a house; making a will). The inclusion of ‘non-problematic’ legal events is likely to overestimate the prevalence of legal problems, since these events may not constitute legal need.
The different thresholds for inclusion of legal problems are likely to influence the proportion of serious problems captured across studies. Given that problem severity affects the response adopted and the outcome achieved (Pleasence 2006; Pleasence et al. 2004c), surveys that cover more intractable problems are thus also likely to find lower rates of successful resolution.
Reference period and memory effects
Legal needs surveys have assessed legal problems over variable periods of time, typically ranging between one and five years (Coumarelos et al. 2006). The recent Australian, New Zealand and US surveys employed one-year reference periods, while the UK, other European, Canadian, Chinese and Japanese surveys employed reference periods of between two and five years.(7)
Variable reference periods can complicate cross-study comparisons by affecting the capture, observed impact and observed resolution of legal problems. Longer reference periods provide more opportunity for capturing infrequent legal problems, examining the long-term impacts of problems and reaching resolution (cf. Pleasence et al. 2004c). However, as detailed below, longer reference periods are more likely to involve inaccurate recall.
Memory effects are not specific to legal needs surveys but are a potential limitation of all social surveys. Both ‘memory decay’ and ‘forward telescoping’ can influence recall accuracy (see Biemer, Groves, Lyberg, Mathiowetz & Sudman 1991; Gottfredson & Hindelang 1977; Huttenlocher, Hedges & Prohaska 1988; Lynn, Buck, Burton, Jäckle & Laurie 2005; Neter & Waksberg 1964; Rubin 1982; Rubin & Baddeley 1989; Sudman & Bradburn 1973; Thompson, Skowronski & Lee 1988; Tourangeau, Rips & Rasinski 2000). Memory decay involves completely forgetting past events or forgetting details about those events, such as when they occurred. It is well established that memory decay is worse for longer time periods and for less significant events (Lynn et al. 2005; Sudman & Bradburn 1973; Tourangeau et al. 2000). Pleasence, Balmer and Tam (2009) specifically examined the recall of legal problems with the CSJS. Consistent with the broader literature, they found substantial memory decay, estimating that at least three-quarters of legal problems went unreported. They also found better recall for serious than for minor problems. In addition, although the largest drop in recall occurred within the first year, recall continued to decline over time, with the worst recall for the least recent problems.
Forward telescoping involves inaccurate recall of the recency of events, with events that occurred before the reference period being incorrectly reported as having occurred during it (Lynn et al. 2005; Neter & Waksberg 1964; Rubin & Baddeley 1989; Sudman & Bradburn 1973; Tourangeau et al. 2000). There is some evidence that telescoping is more likely with more salient events and with longer reference periods (see Lynn et al. 2005; Neter & Waksberg 1964; Tourangeau et al. 2000).
Thus, longer reference periods are likely to capture a higher proportion of serious legal problems when compared to shorter periods. This expectation is due to the greater tendency both to forget minor problems over longer periods and to telescope serious problems into longer reference periods. The optimal reference period for legal needs surveys has not been examined. However, a reference period of 12 months has been proposed to be acceptable for optimising recall of crime victimisation events, given memory effects (United Nations Office on Drugs and Crime 2010).
Unit of measurement
Legal needs surveys have differed in terms of whether they measured the legal problems experienced by individuals or households (Coumarelos et al. 2006). While the US surveys have typically measured prevalence within households, the surveys conducted elsewhere have tended to use the individual as the unit of measurement. Assessing the problems faced by all household members rather than only one household member is likely to yield higher incidence rates.
Measurement of other key variables
Legal needs surveys have also varied in their definition and measurement of a number of other key variables. Such variables include the demographic factors underlying the experience of legal problems, the types of advisers and resolution strategies used, the adverse impacts of legal problems, the finalisation of legal problems, and the satisfactory outcome of legal problems. To give but one example, ‘disability’ has been defined and measured inconsistently. Generally, disability has been defined expansively to include a broad range of both physical and mental illnesses or conditions that are long-term and impair functioning or participation in society (see Coumarelos et al. 2006; Currie 2007b; Pleasence 2006; Pleasence et al. 2004c). However, the conditions subsumed within ‘disability’ are not always clearly delineated and do not always include mental illness (see Balmer, Buck, Patel, Denvir & Pleasence 2010).(8)
Again, such differences in the measurement of key variables need to be considered in cross-study comparisons.
Generalisability of findings
The usefulness of social surveys depends on the extent to which their findings provide an accurate, unbiased or representative picture of the broader population. The ‘generalisability’ of the sample results to the population hinges on the adequacy of a number of methodological factors, as outlined below.
Large-scale random probability sampling is the blue ribbon approach for selecting an unbiased sample that is representative of the population. Random sampling means that all members of a population have equal chance of selection, and, thus, that the sample profile is likely to accurately mirror that of the population. Small sample sizes can limit generalisability, even with random sampling, because they make it more difficult to capture a broad cross-section of the population. Most legal needs surveys have used random sampling. However, non-random sampling has been used occasionally, such as selecting low-income earners by approaching places they frequent (e.g. Dale 2007), selecting survey sites to maximise regional and economic variation (e.g. Michelson 2007a) and using non-random opt-in internet panels (e.g. van Velthoven & ter Voert 2004). The results of such studies may not accurately reflect legal need and resolution in the broader population.
Poor response rates can reduce the generalisability of survey results, particularly if there are systematic differences between the people who participate and the people who refuse (American Association for Public Opinion Research (AAPOR) 2009; Groves, Cialdini & Couper 1992). In the case of legal needs surveys, obtained incidence rates could be inaccurate if the demographic groups that are especially vulnerable to experiencing legal problems are either substantially overrepresented or underrepresented in the sample due to systematic non-response.
There are several methods for calculating response rate, and response rate estimates can vary dramatically depending on the particular method used (Biemer & Lyberg 2003; Groves 1989). So that survey quality is open to scrutiny, it is critical that legal needs surveys detail the response rate and its calculation. It is also crucial that the demographic profile of the sample is comparable to that of the population. Some legal needs studies have provided very limited information on response rate or sample profile (e.g. Cass & Sackville 1975; Curran 1977; Dale 2000; Rush 1999; Spangenberg Group 1989; Winfield 1995).
Response bias and mode of data collection
Legal needs surveys share the limitations associated with all social surveys. As already noted, recall errors are inherent features of retrospective surveys (Sudman & Bradburn 1973). In addition, social surveys are subject to certain types of response biases that may produce inaccurate answers (Beatty, Herrmann, Puskar & Kerwin 1998). Respondents sometimes lie because of concerns about confidentiality or being viewed in a socially undesirable light. Interviewer–interviewee rapport, anonymity and topic sensitivity can also affect the accuracy of answers (Oppenheim 1992; Presser, Rothgeb, Couper, Lessler, Martin, Martin & Singer 2004; Weisberg 2005).
The mode of data collection is one factor that can influence response bias. Face-to-face interviews may be more conducive to establishing rapport, particularly in relation to personal or sensitive issues, and may provide greater opportunity for in-depth probing (see Biemer et al. 1991).(9)
Telephone interviews, however, provide greater anonymity than face-to-face interviews, which may increase respondents’ trust in confidentiality and improve the accuracy of reporting on sensitive topics (see Biemer et al. 1991; Oppenheim 1992; Weisberg 2005).(10)
Face-to-face interviewing was typically used in the UK, other European and Asian surveys (Genn 1999; Genn & Paterson 2001; Gramatikov 2008; HKDOJ 2008; Michelson 2007a; Murayama 2007; Pleasence 2006; Pleasence et al. 2004c, 2010). Telephone interviewing was used in most of the US surveys and in the recent Australian, Canadian and New Zealand surveys (Coumarelos et al. 2006; Currie 2007b; Ignite Research 2006; Legal Services Corporation (LSC) 2007, 2009). A few studies used both telephone and face-to-face interviewing (ABA 1994; CEALS 2001; Dale 2009; Schulman 2003; TALS 2004). The Dutch studies used internet surveys (van Velthoven & ter Voert 2004) or a combination of internet and face-to-face interviews (van Velthoven & Klein Haarhuis 2010).
Summary of legal needs survey methodology
Recent legal needs surveys have generally followed the Genn (1999) or ABA (1994) traditions. They have produced broadly consistent key findings, despite methodological differences. Their main common findings are reviewed below.
5. This is a German-language report, and, as a result, the findings of this study have not been reviewed.
6. The surveys in China, Germany and Slovakia were more loosely based on Genn's (1999) approach.
7. The recent Hong Kong survey (HKDOJ 2008) used multiple reference periods.
8. Throughout this report, for convenience, 'disability' is used to refer to a variety of terms used by other authors to broadly cover both physical and mental illnesses or conditions, such as 'chronic illness or disability' (Coumarelos et al. 2006) and 'long-standing ill-health and disability' (Pleasence 2006). Where authors report on mental illness separately from other disabilities, this is noted. The specific definition of disability used by the LAW Survey, which includes both physical and mental conditions, is detailed in Appendix A2, 'Comparison of sample and population profile: Disability status' section.
9. Murayama (2007), for example, used face-to-face interviews because of a perceived reluctance among the Japanese to discuss personal matters over the phone. In contrast, van Velthoven and ter Voert (2004) used an internet questionnaire because of a low response rate in the Netherlands for face-to-face interviews.
10. In addition, telephone interviews have the practical advantage of being less labour intensive and less expensive (see Biemer et al. 1991) and may therefore be more conducive to larger sample sizes, particularly over extensive geographical areas.