Using Administrative Data to

Examine the Efficacy of Welfare-to-Work Programs

Peter Mueser

Sharon Ryan

Melinda Thielbar

Department of Economics

University of Missouri-Columbia

September 1998

Address correspondence to: Peter Mueser/Sharon Ryan, Department of Economics, University of Missouri, Columbia, MO 65211

E-mail: mueser@econ.missouri.edu ryan@econ.missouri.edu

Using Administrative Data to

Examine the Efficacy of Welfare-to-Work Programs

Peter Mueser, Sharon Ryan, and Melinda Thielbar

Department of Economics

University of Missouri-Columbia

This paper investigates the feasibility of measuring the efficacy of programs designed to move welfare recipients into jobs using administrative records on individual program participation linked with state records on individual earnings.

Like many states, Missouri collects quarterly earnings for all individuals covered by its unemployment insurance system. We have linked this information to administrative records listing participants in the FUTURES program, Missouri's welfare-to-work program developed under federal JOBS legislation. We are able to distinguish those forced to participate in the program from those who chose to participate. In addition, since a waiting list was used to allocate individuals to the program during the period of our study, those on the waiting list who were not chosen to receive services may serve as a control group.

Our analysis indicates that volunteers and required participants differ in ways consistent with administrative rules. The process by which individuals were selected from the waiting list also reflected such rules. Selection did not appear to favor those whose characteristics would suggest employment success.

Estimates of program impact on participant earnings underscore the importance of controlling for a variety of characteristics, in particular prior earnings and employment experience, and of distinguishing volunteers from those required to participate. Ordinary least squares estimates of program impact are modest but positive and statistically significant, and they are generally consistent with expectations. Instrumental variables estimates are similar, suggesting that individual differences in willingness to remain on the waiting list are not an important source of bias in estimated impact.

Recent years have seen a flurry of studies evaluating government training programs. In large part, this work has been fed by a substantial body of data based on experimental designs. Earlier attempts to evaluate programs had been criticized because they failed to recognize the critical role that self selection could play in biasing estimated effects. In contrast, an experimental design, with subjects randomly allocated to receive the program "treatment," can eliminate the bias caused by selection of program participants.⁽¹⁾

Unfortunately, randomization is often not feasible. One common approach is to find a group with similar characteristics to serve as a control, possibly selecting from a large population using individual demographic characteristics and labor market experience to match those who have participated in the program (Johnson and Stromsdorfer, 1990). Such an approach suffers from the difficulty that even when many individual characteristics are available, those who choose to participate in the program may differ in unmeasured ways from the selected group. LaLonde (1986) concluded that program impact estimates based on nonexperimental comparison groups did not correspond with those based on random assignment designs. Although an extensive literature has attempted to develop models and matching methods that can be used to obtain program impact estimates using nonexperimental data (Bassi, 1984; Ashenfelter and Card, 1985; Heckman and Robb, 1985; Heckman and Hotz, 1989; Heckman, Ichimura and Todd, 1997), no definitive solution has emerged.

Bell et al. (1995) suggest a nonexperimental approach based on using as a comparison group applicants to a program who have been denied access by staff. Such "screened out" individuals differ in systematic ways from those provided access, reflecting staff judgments about the likely success applicants will experience following participation. While the screened out sample is clearly inappropriate as a simple control group, if all individual factors used in the screening process are available to the investigator, including these in the prediction of the outcome can allow consistent estimates of program effect.

The analysis here investigates whether waiting lists used to allocate potential participants to a job readiness and training program may be used to identify the impact of the program. The approach is similar to that of Bell et al. in that it creates an internal control group whose labor market performance is, in effect, compared with program participants. It shares with that approach the important advantage that motivational factors that draw individuals to a program will be similar for those on the waiting list and those who are chosen from the waiting list. On the other hand, whereas Bell et al. had available a rating assigned by staff indicating expected applicant success, we have no such rating in our data. We do, however, have a variety of demographic characteristics, as well as measures of labor market and welfare experience, that would have been available to staff in selecting applicants from the waiting list.

Our use of a comparison group based on a waiting list removes two important sources of bias that have been identified in the literature, those due to geographical mismatch and lack of comparability between data sources ( Friedlander and Robins, 1995; Heckman, Ichimura and Todd, 1997). Those on the waiting list who are not offered services are resident in the same geographical areas within the state as those chosen from the waiting list, eliminating labor market differences. Also, our comparison group data are obtained from the same sources as our treatment group, eliminating inconsistencies due to variation in survey instruments or data processing. More generally, the vagaries of selection from the waiting list assure that there is substantial overlap in the characteristics of those offered program services and those left on the waiting list. In contrast, in many natural environments, those who do not participate differ dramatically in their characteristics from those in the program, which makes statistical correction for such differences difficult (Heckman, Ichimura, Smith and Todd, 1996).

Our study is unusual in that it includes both those who volunteer to participate in a job readiness and training program and those who are required to participate. Theory suggests that the impact of a job assistance program may differ for these groups. Those AFDC recipients who are required to participate face a reduction in benefits or related sanctions if they fail to undertake specified training activities. If these requirements constitute a burden, some individuals may take actions that remove this requirement, possibly discontinuing both AFDC participation and training activities. The result may be either higher or lower labor market involvement than in the absence of the program, but, in either case, this impact will be due, in part, to incentives created by program demands and sanctions, not the value of training or related services (Friedlander, Greenberg, and Robins, 1997). The impact of the program for volunteers and required participants may also differ because of unmeasured factors tracing to differential selection into the two groups.

It must be recognized that our methods share the shortcomings implicit in most evaluations of training programs. For both voluntary and mandatory participants, the measure of impact we obtain is for the policy that makes the program available or required for certain individuals, that is, we examine the impact of an offer of services. As Heckman, Hohmann, Khoo, and Smith (1997) note, such policy impact is not a clean measure of the efficacy of training or other services a program provides. Some of the individuals who are offered services never actually respond to the offer, and some who initially respond drop out before they receive any meaningful services. Equally important, those who are denied program services may receive similar or related training through other channels. This reduces differences between those who receive an offer of service and those who do not in terms of the services actually received, with the result that our measure of impact will understate the impact of services received. This understatement will be particularly large where there are good available substitutes for services offered by the program.⁽²⁾

Data Structure and Basic Definitions

Our focus will be on Missouri's FUTURES program, adopted under federal JOBS legislation, which provides various services designed to aid and encourage employment of individuals receiving AFDC.⁽³⁾ Although the program first began operation in 1990, it was not available on a statewide basis until 1992. Until the mid 1990s, scarce training slots were allocated based on waiting lists, which were generated through the AFDC program and passed on to FUTURES staff. An AFDC recipient was either exempt (because of a physical inability to work, the presence of young children in the home, or other factors) or mandatory (formally required to participate in FUTURES). Any AFDC recipient could also volunteer to participate in the program. FUTURES staff chose individuals from two lists, a "mandatory" and a "voluntary" list. Those who were mandatory and had volunteered appeared on both lists. When a slot became available, an individual could be chosen from either list, with the criteria for the choice between lists, and among those on a list, being idiosyncratic across various state offices. While the number of individuals on waiting lists, in aggregate, exceeded those in the program during this period, a substantial number of individuals spent less than one month on a waiting list.

Our data are in the form of a file for each month indicating whether an individual was on a waiting list, was receiving FUTURES services, or had been called by program staff and offered FUTURES services but had not yet responded. The file also indicates whether the individual was designated as voluntary, mandatory, or both, during that month. A small proportion of those on our files were receiving services but were not identified as either voluntary or mandatory. For the most part, these were individuals in special programs whose placement into FUTURES differed from that of other participants; our analysis will omit them.

The analysis focuses on women who had a date of entry into the FUTURES system October 1992-September 1995.⁽⁴⁾ We coded date of entry of an individual into FUTURES based on month of first appearance on our files, omitting any individual who appeared on the file for September 1992 in order to avoid including those whose contact extended from an earlier period.⁽⁵⁾ We coded individuals into three categories on the basis of how they were classified the first time they appeared on our files: voluntary only, mandatory only, or both voluntary and mandatory. Those cases for which the first appearance on the file indicated that they were being offered services were coded as having spent zero time on the relevant waiting list. For those individuals that we first observed on a waiting list (i.e., who had not yet offered services), we counted time on the waiting list as the number of months they were on that list until they were offered services (meaning they were either called by program staff or were observed receiving services) or they ceased to appear on that waiting list.⁽⁶⁾

In addition to sex, we have available from FUTURES or AFDC files the standard demographic characteristics: age, education and race. We have information about the family, including the number of children in the household and the age of the youngest child. We also know the number of months the applicant had been on AFDC at the time of first appearance on our file. Finally, we have employer reports of total quarterly earnings for all employees covered by Missouri's unemployment insurance program. For each individual, we have calculated total earnings indicated in these reports in the four quarters prior to the quarter in which the individual appeared on our files. Earnings data are available also for the follow-up period. While these earnings data have the obvious shortcoming of omitting earnings from employment not covered by the unemployment insurance system⁽⁷⁾ or earnings from employment outside Missouri, the earnings data are otherwise quite reliable.

Voluntary and Mandatory: The Interaction of Bureaucratic Rules and Participant Choice

As noted above, individuals on our files either volunteered to participate in the FUTURES program or were required to participate. At the time of our study, regulations classified AFDC recipients as mandatory FUTURES participants unless they met exemption criteria, the most important being the presence of a child age three or less, or a child under age six if appropriate child care was not available. For both volunteers and mandatory individuals, most spent some time on the waiting list, and only a minority in either group were ultimately offered FUTURES services.

Table 1 provides statistics which allow us to observe how volunteers differ from those who were mandatory. The sample consists of all individuals who volunteered or were first classified as mandatory for FUTURES in the period October 1992-September 1995. The most important difference is in the period of time for which the two groups had been receiving AFDC. Those listed as mandatory were in large part new AFDC applicants. Rows 5 and 6 show that three quarters of those in the mandatory classification had received AFDC for one month or less, while only about a quarter of those listed as voluntary were in this category. In large part, this may reflect how we selected individuals from the FUTURES files. Those who had received AFDC for extended periods may have been classified as mandatory participants prior to October 1992 and were therefore omitted from our study by construction. Among long-time AFDC recipients, only those who experienced a change in circumstances during our study period--e.g., the aging of a child--would appear on our file as mandatory FUTURES participants. In contrast, any new AFDC applicant who failed to meet exemption criteria would be classified as mandatory.

The selection criterion explains why volunteers differ so dramatically in the age of the youngest child. Rows 8 and 9 show that among mandatory participants, fewer than 7 percent had a youngest child under age two, and only 22 percent had a child under age four. On the other hand, among volunteers, 44 percent had a child under age two, while more than three quarters had a child under age four. This points up an interesting contrast between the standards imposed for required participation and the characteristics of those who volunteered: A substantial number of those who were exempt because they had young children chose to volunteer for services. Statistics for those who were required to participate and who also volunteered (right columns) are very similar to those who were required to participate and did not volunteer.

The volunteers were much more likely to be nonwhite⁽⁸⁾ than were required participants (row 2). On the other hand, those mandatory participants who also volunteered were less likely to be nonwhite than other mandatory participants, suggesting that it is the application of the mandatory criterion that causes the racial difference between the mandatory and voluntary samples. Nonwhites are likely to have longer spells on AFDC than whites, so that the low proportion of nonwhites among mandatory participants may well reflect the fact that our mandatory sample is largely drawn among new AFDC applicants.

One measure that summarizes important differences between samples is the average prior earnings, which is the sum of earnings in the four quarters prior to appearance on our waiting list or participation in the program, based on employer reports to the Missouri unemployment insurance system. Row 15 shows that volunteers and mandatory participants were about equally likely to have earnings but row 14 indicates that average earnings for volunteers were only half of that for mandatory individuals. This difference, which clearly results primarily from the fact that volunteers are much more likely to have extensive AFDC histories, suggests that volunteers do not necessarily have more promising employment prospects than mandatory individuals.

Finally, row 16 shows that nearly half of all voluntary cases in our file were ultimately offered FUTURES services, whereas fewer than one-fifth of those who were only on the mandatory list were offered services. Apparently, a large number of individuals were formally required to participate in the FUTURES program but remained on the waiting list without ever receiving an offer of service. About a third of the individuals who were required to participate but also volunteered were ultimately offered services.⁽⁹⁾

The differences in the probability of being offered services for those on the voluntary and mandatory lists could be partly due to federal restrictions on the use of FUTURES program funds. According to federal JOBS legislation, in order to be eligible for continued federal funding, 55 percent of FUTURES funds must be spent on AFDC recipients who fall into one of three federal target groups:

1) Households that have received AFDC 36 of the last 60 months.

2) Heads of household under 24 without a high school degree, GED, or with little labor market experience.

3) Households that will lose AFDC eligibility in two years because the youngest child is 16 years or older.

In our sample, the fact that volunteers are more likely to be long-term AFDC recipients and are more likely to be chosen for FUTURES is not surprising given the constraints placed on program staff.

Federal regulations also require that 20 percent of the FUTURES mandatory population must be served. The actual proportion of the mandatory participants in our study who were offered services was about 23 percent, suggesting that this constraint may have been binding. Conditions of the offer of service are explored more fully in the next section.

Access to Service: The Waiting List and Determinants of Service Offer

Table 2 presents the proportions of individuals in our sample who were offered FUTURES services over the period in question, classified by the length of time on the waiting list. For those who were offered services, length of time on the waiting list varied substantially. About one in five of those who were ultimately offered services received the offer without any time on the waiting list. A substantial portion of those who were offered services waited several months. The fourth column indicates the proportion of individuals who were offered services after waiting more than a year, while the fifth column identifies those who were offered services after some interruption in their time on the waiting list. The sixth column identifies those who did not receive an offer of service during the period of our study. Of those not offered services, some remained on a waiting list for extended periods, while others dropped off after relatively short periods. In tabulations not listed here, we found that the proportion receiving an offer of services was higher in later years of our study, reflecting expansion in the program relative to the population eligible to receive services.

The waiting list serves the function of providing access to the program for some individuals while limiting the access of others. Tables 3 and 4 provide means and standard deviations of several variables, dividing the sample by whether the individual received an offer of service, separately for the voluntary and mandatory populations. There are numerous differences between those receiving services and those not, but, in general, they are smaller than differences between volunteers and mandatory participants. For both voluntary and mandatory cases, those receiving an offer of service have, on average, spent longer on AFDC and have lower prior earnings.

Observed differences reflect both self-selection by individuals and selection off the waiting list by agency employees. Among those who are not selected immediately to receive services, a substantial portion left the waiting list after some period of time. For both volunteers and mandatory individuals, the most common reason to drop off the waiting list was leaving AFDC. Mandatory individuals could also leave the list if their status changed so that they qualified for an exemption (e.g., birth of a child), or if they got part time jobs. Since those who remained longer on the waiting list had a greater chance of being offered services, those offered services are likely to differ for reasons of self-selection.

In order to identify the impact of agency selection, as distinct from individual self-selection, we have run analyses that predict selection from the waiting list, controlling for time on the waiting list at that point. As noted above, the process by which individuals were chosen from the waiting list was not fully specified, and undoubtedly varied across local offices. A simple “first-come-first-serve” standard was frequently applied, but there was substantial uncertainty about how an initial ordering was established, especially among individuals placed on the waiting list in a given month. Much of the variation in chance of selection from the waiting list was due to differences in the number of available slots, relative to the demand, across offices.

Tables 5 and 6 present coefficients for logistic regressions that show how selection from the waiting list occurred for volunteers and mandatory participants. Coefficient estimates suggest that FUTURES program staff selected heavily on individual characteristics for the first and second months on the waiting list but less for those who had been on the waiting list for longer periods. It appears that FUTURES workers were looking for certain criteria for first selection into the program, and, after the pool of individuals with those characteristics has been exhausted, chose more-or-less at random.

The voluntary list shows this pattern most strongly. As the reader can see from Table 5, region within the state and year of entry are statistically significant through all four initial months on the waiting list. This is not surprising since program resources relative to the AFDC population are different across the state. However, the likelihood of being chosen as a volunteer without spending any time on a waiting list is heavily dependant on individual characteristics such as the age of the youngest child, education, and age of the AFDC recipient. Given that young AFDC recipients with little labor market experience fall into one of the three federal target groups, it is not surprising that age and education would be considered when admitting applicants. This is also true for length of AFDC receipt. Another characteristic that significantly affects the likelihood of being chosen into FUTURES is school attendance or involvement in a training program upon entering AFDC (In School). One possibility is that these recipients may only apply for FUTURES supportive services or for FUTURES to pay tuition or other training-related costs. If this is true, these recipients would be among the easiest and least expensive to serve, and an AFDC or FUTURES caseworker would want to make an effort to encourage a self-initiated effort at training.⁽¹⁰⁾

After one month on the voluntary waiting list, the only characteristics (other than region of service) that are consistently significant are the measures of time spent on AFDC prior to volunteering. This is probably because FUTURES staff continued to look at these characteristics in order to meet federal guidelines.

Examination of the pseudo-R² for the selection from the mandatory list (Table 6) suggests that, especially in the early months on the list, selection is much better predicted than is selection from the voluntary list (Table 5). In addition, the probability of selection appears less dependant on region within the state and more dependant on individual characteristics, even for longer periods of waiting, than is selection from the voluntary list. The difference between mandatory and voluntary selection may be due, in part, to FUTURES workers being more selective when they must "force" unwilling participants into job training programs because of the federal requirement that FUTURES serve 20 percent of the mandatory population. Those characteristics that are most consistently significant are race and age of the youngest child.

Program Impact: Ordinary Least Squares Regression Estimates

The outcome measure that we will consider is annualized earnings in the first half of 1997. Since the period of our study is from October 1992 through September 1995, those offered services would have had from one to four years to complete the program. The simplest approach to examining the impact of the FUTURES program would compare earnings of those who were offered services with those who were not. The first regression reported in Table 7 provides estimates based on this comparison. Dummy variables identify individuals who were on the mandatory list, and those on both mandatory and voluntary lists; the omitted category is volunteers. Three dummy variables indicate whether the individual received an offer of services for each waiting list category.

The intercept indicates that the average earnings for volunteers who were not mandatory and received no offer of services was $2474. Coefficients for the dummies identifying other listings indicate that individuals who were on the mandatory waiting list or on both lists and were not offered services received slightly higher earnings than did volunteers, but the differences are not statistically significant. Volunteers who received an offer of services obtained earnings that were $144 higher than those who did not, a difference that is statistically significant. Coefficients for the variables indicating an offer of service for the other waiting list categories show that those who were required to participate in futures earned $149 less if they were offered services, which is also statistically significant. Those who were on both lists earned $127 more if they were offered services, but this difference is not statistically significant.

These coefficients combine program impact with any differences that are due to composition effects associated with receipt of an offer of services. In the remaining equations presented in Table 7, we examine the impact of controlling for additional factors. One obvious question is whether some of the observed differences in outcomes are due to geographic variation in labor market opportunities that are associated with an offer of services. In the second equation, we have controlled for the seven service regions, and, within these, we have identified additional geographic divisions, distinguishing county groups by population density and similarity of socioeconomic structure. Altogether, the state is divided into 14 regions, identified by 13 dummies. These controls for region alter estimates of program impact substantially. Volunteers who are also required to participate now display a positive and significant effect of a service offer, while the estimated negative impact of an offer on mandatory participants is no longer statistically significant.

We anticipate that earnings will very likely differ according to the year in which individuals appeared on the waiting list. By design, individuals are receiving AFDC when they appear on the waiting list, and, for many, this identifies a period of temporary economic distress. With time, many of these individuals would obtain employment in the natural course of events. This means that a greater gap between first appearance on our file and post-program measured earnings will be associated with higher average earnings. Since the chance of selection into the program differs by year, controls for year are necessary to determine program impact. Equation (3) introduces controls for year of entry (1992 is the omitted category). As expected, those who entered the list in more recent years have substantially lower wages in 1997. Estimates of program effect change very little, however.

The fourth equation presented in Table 7 adds controls for age, race and educational attainment. These measures have estimated coefficients that are as expected. The program impact for volunteers declines by about a quarter, while other changes in estimates of program are smaller. The next estimation equation controls for number of children in the household, the age of the youngest child, and the marital circumstance of the individual at the time she first began receiving AFDC. As expected, those with more children and younger children are expected to earn less, and those who were coded as never having lived with the father of their children (in contrast to those whose marriages dissolved) earn less. Changes in estimates of program impact are minimal.

Equation (6) adds controls for time on AFDC as well as measures of labor market experience. Those with no time or one month on AFDC appear to have lower earnings than those with greater time on AFDC. Although additional months on AFDC are associated with a small decline in earnings, the estimated impact is not statistically significant. The "work experience" variable indicates the number of months the individual worked in the year prior to beginning AFDC, while a dummy variable identifies the large proportion who did not work at all. As expected, more work is associated with higher post-program earnings. Also included is reported earnings in the year prior to entry onto the list, which is entered as a quadratic and as a dummy variable identifying those with no earnings. The measures of prior earnings clearly capture factors that are critical in predicting post-program earnings.

These labor market controls increase estimated program impacts substantially. In equation (6), the effect of an offer of service is statistically significant in all cases, varying from $120 for mandatory participants to over $300 for mandatory participants who also volunteered. Changes in estimates reflect the fact that those offered services generally had less labor market experience and lower earnings than those who were not offered services. In large part, this is due to the agency selection by time on AFDC: Offers of service were much more likely to go to long-term AFDC recipients, whose labor market histories were less promising.

Tests for interactions in the impacts of independent variables. The analysis of Table 7 has assumed that the control variables have similar effects on volunteers and mandatory participants, and that controls have the same impacts for every year and across all regions of the state. Table 8 presents estimation equations that include interactions to examine whether estimates of program impact are sensitive to these assumptions.

Equation (2) enters interactions that allow every variable controlled in the previous analysis to have different impacts for the three waiting list classifications. An F-test shows that the additional 72 variables improve the fit significantly, but changes in measures of program effect are modest. Compared to equation (1), which repeats the final equation in the previous table, we see that the estimated impact of the program for voluntary recipients increases by almost 20 percent, while impacts for the other programs decline by less than 10 percent. Program effects are essentially the same in equation (3), which presents estimates for an equation which controls only the subset of interactions that are statistically significant.

Equation (4) presents program effect estimates controlling for interactions involving the seven main service delivery regions with other variables in the equation, while equation (5) controls only those interactions among these that are statistically significant. Changes in estimated effects due to inclusion of these interactions are small.

Finally, equations (6) and (7) include interactions involving dummies for year of entry into the program. We might anticipate that measured variables would have different effects depending on year of entry, especially given that the time between variable measurement and post-program earnings differs by year. While some interaction effects are, in fact, statistically significant, estimates of program impact are not substantially affected.

In sum, it appears that interaction effects involving independent variables have little substantive impact on estimates of program impact. None of the estimates in equations that control for interactions differs by as much as a standard error from those that ignore such effects.

Program Impact: Instrumental Variables

Although we have controlled for all measured characteristics that are available, if individuals who are offered services differ in systematic but unmeasured ways that would help to predict their ultimate earnings, estimates based on ordinary least squares will be inconsistent. As we have noted, one advantage of our approach is that the administrative selection and self-selection implicit in the voluntary/mandatory classification are controlled in our analysis. However, the offer of service clearly is not random within waiting list classification. The likelihood of ultimately receiving an offer of service can be traced to two types of factors.

The first is the selection made by FUTURES staff from waiting lists. Our analysis above shows that this selection is strongly associated with several measured characteristics, reflecting various constraints imposed by federal regulations. The second source of variation is an individual's choice of how long to remain on the waiting list. Most of those who appear on our files but were not offered services dropped off the waiting list after relatively short periods of time, generally just a few months. Those who remained on the waiting list for longer periods would be more likely to receive services.

While both of these sources of variation in the likelihood of an offer of service will cause difficulties for estimation, we hypothesize that staff choices are less tied to unmeasured personal characteristics. If this is the case, selection from the waiting list may usefully serve as an instrument for program participation. Heckman (1997) has argued that, in the presence of heterogeneity in the benefits individuals anticipate from participation, instrumental variable methods may not work, even where the instrument appears exogenous. In our case, however, we show that using a measure based on the waiting list as an instrumental variable does provide meaningful estimates if it is independent of unmeasured factors predicting post-program earnings.

For individual i specify that Z_i is a binary variable indicating whether the individual is chosen from the waiting list after spending t months on the list. Condition the analysis on the individual being available to be selected, meaning that she has been on the waiting list at least t months. Assume Z_i is independent of individuals' unmeasured characteristics. Among those not chosen (Z_i=0), some ultimately obtain an offer of services, while others do not. Although a delay in treatment could have an effect on its impact, in our sample the additional wait is usually just a few months. We therefore make the assumption that the expected outcome is the same whether the individual is selected to be offered treatment at time t or after a more extended time on the waiting list.

Given these assumptions, it is possible to show that the treatment effect for individual i with characteristics X, who is offered services at time t but would not otherwise be offered services, can be written as:⁽¹¹⁾

The estimate described above is what Imbens and Angrist (1994) label a local average treatment effect (LATE). It must be stressed that this estimate of program impact is for individuals who received an offer of service because they were selected at time t but who would not otherwise have received an offer. The estimate does not identify the program effect for the inframarginal participant, one who would have received an offer of service after time t, reflecting, in part, willingness to remain on the waiting list additional periods.

It is clear that our data provide multiple waiting list variables, each of which can be used to obtain a conceptually distinct measure of program impact, corresponding to populations spending various times on the waiting list. Table 9 presents estimates of program impact using this method. The equation used to obtain estimates reported in each column corresponds to the final estimation equation reported in Table 7, but with the three variables indicating an offer of service treated as endogenous. Variables indicating selection from the waiting list at the specified time serve as instruments that just identify the estimated impact. For example in the first equation, the three variables that identify selection without any time on the waiting list (for volunteers, mandatory participants, and mandatory participants who volunteered) are the instrumental variables that identify program impacts.

Each column therefore presents program impact estimated for a different subpopulation. In contrast to ordinary least squares estimates, which, in effect, draw on a comparison of all individuals who are offered FUTURES services with observationally similar individuals who are not, each of these estimates is based on the small number of individuals selected from the waiting list at a given point in time who would not otherwise have been offered services.

For volunteers, the estimated program impact based on these estimates is positive in each case, and is larger than the comparable ordinary least squares estimate in four out of the five cases. It is statistically significant in only one case, although it approaches statistical significance in another. For mandatory participants, the estimated impact is also generally larger than that of the ordinary least squares estimate, but again is only significant in one case. A similar generalization applies for the third group, mandatory participants who also volunteered, although for these individuals, one of the estimates is much larger than the ordinary least squares estimates, and one of the estimates is negative although small in absolute value. While differences between estimates reported in the five columns are substantial, they are clearly dwarfed by sampling error.⁽¹²⁾

While it is clear that sampling error precludes any strong inferences, we may conclude that these estimates provide little basis to infer that the ordinary least squares estimates are seriously misleading. If the instruments we have constructed are appropriate--meaning that selection from the waiting list is independent of unmeasured factors that influence earnings--these estimates support the conclusion that those who receive a service offer experience a modest increment in their post-program earnings.

Comparison with Experimental Estimates of Program Impact

Beginning in the 1980s, federal waivers allowed states to design more complex welfare-work programs. Federal rules, however, generally required an evaluation protocol as well, resulting in a number of evaluations involving random assignment of a control group. Data from these experiments have been evaluated by a large number of researchers during the past twenty years. A review of this literature shows that our results are in line with what other researchers have found using experimental methods, supporting the view that our method of choosing a comparison group from those on the waiting list for FUTURES services is successful.

Most evaluations find that program participants experience modest but statistically significant gains in employment (see for example Bane and Ellwood, 1994; Blank, 1997; Couch, 1992; Gueron 1996; Friedlander and Hamilton 1996; and Friedlander and Robins, 1995). As Blank (1994) reports, "Results (from program evaluations involving random assignment) showed clear positive results in terms of employment gains for program participants. The cost-benefit evaluations indicated that states more than covered costs of the programs through lower AFDC costs" (p. 186). In general, the modest increases in annual earnings were due to increased hours worked by program participants rather than to any increase in wage rate. The impact on wage rates has been found to be negative as often as they are positive (O'Neill and O'Neill, 1997; Couch, 1992).

For the purposes of comparing our results with prior studies, we focus on estimates of annual earnings gains. In each case, we report estimates based on studies employing experimental designs. In all the studies, many of those in the treatment group did not receive services, often because they dropped out early in the program. Hence, these estimates, like ours, identify the impact of an initial offer of program services. Since the distinction between mandatory and voluntary participation may be important, we consider estimates for the two kinds of programs separately.

Gueron and Pauly (1991) review evaluations of a variety of mandatory programs which were undertaken as demonstrations in the 1980s. If we examine the seven "broad-coverage" programs which provided services similar to Missouri FUTURES, we find an average annual gain in earnings, adjusted for inflation, running from $15 to $738, with a mean of $396. However, four of these programs provided less intensive services, primarily aid in job search, than the FUTURES program. If we consider studies of the three programs that provided both job search assistance and more extensive training, the range of effects is from $317 to $738, with a mean of $597.

Since these are demonstration programs, we might expect impacts to differ from those of the federally mandated programs set up under JOBS legislation. Friedlander, Greenberg and Robins (1998) report on four studies which obtained experimental estimates of JOBS program effects. Impacts on annual earnings range from $91 to $1183, with a mean of $459. Our estimate, indicating a program effect of $120 for mandatory FUTURES participants, is somewhat lower than the average of the experimental studies, although it is in the range of observed estimates.

We know of no experimental evaluation that focuses on voluntary participants in JOBS programs, but evaluations have been done of several voluntary demonstration programs designed to move AFDC participants into the labor market. Friedlander, Greenberg and Robins (1998) report on four programs with estimated impacts on annual earnings varying from $215 to $3873, with a mean effect across the four programs of $1372. These numbers are generally higher than our estimates, which are $152 for FUTURES volunteers who were not mandatory and $301 for those on both the mandatory and voluntary list. Each of these programs differed in structure, and none offered the same combination of services as FUTURES. Based on cost per participant, three of these programs were substantially more expensive than FUTURES, suggesting that the greater impacts may reflect greater service impact.

Conclusion

The analysis here attempts to determine whether administrative data on participants in Missouri's welfare-to-work program, linked with individual earnings data provided by employers, can be used to estimate program impact. In addition to providing information on individuals who were offered services, the data include information on individuals who were on waiting lists but who never received an offer of service. Since those receiving an offer and those not receiving an offer are similar in important respects--both comprise AFDC recipients who were placed on the list in anticipation of participation in the program--it is natural to consider program impact by comparing these groups. The data also distinguish individuals by whether they volunteered to participate in the program or whether they were required to participate, allowing examination of the role this distinction may play.

We have first focused on the process by which individuals became identified as volunteers or required participants, and how they were selected from the waiting list to receive an offer of service. Our analysis shows that differences in individual characteristics observed across these groups are explicable in terms of the constraints faced by program administrators. For example, long-term AFDC recipients were much more likely to be chosen to receive services, consistent with federal requirements that such individuals be serviced. Overall, the selection criteria did not appear to favor those whose characteristics would suggest employment success.

Estimates of program impact underscore the importance of controlling for a variety of characteristics, but there is no evidence that interactions among year, waiting list classification, or region substantially influence estimated impact. Ordinary least squares estimates of program impact were modest but positive, and broadly consistent with impact estimates in studies of similar programs based on experimental design. Instrumental variables estimates were similar, suggesting that individual differences in willingness to remain on the waiting list were not an importance source of bias in estimated impact.

In summary, our analysis shows that where administrative data include waiting list information, it is feasible to develop estimates of the impact of welfare-to-work programs.

Ashenfelter, Orley and Card, David. "Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs." Review of Economics and Statistics, 1985, 60(3), pp. 648-60.

Bane, Mary Jo and Ellwood, David T. Welfare Realities: From rhetoric to reform. Cambridge, Mass.: Harvard University Press, 1994.

Bassi, Laurie J. "Estimating the Effect of Training Programs With Non-Random Selection." Review of Economics and Statistics, February 1984, 66(1), pp. 36-43.

Bell, Stephen H.; Orr, Larry L.; Blomquist, John D. and Cain, Glen G. Program applicants as a comparison group in evaluating training programs. Kalamazoo, Mich.: W.E. Upjohn Institute, 1995.

Blank, Rebecca M. "The Employment Strategy: Public Policies to Increase Work and Earnings," in Sheldon H. Danziger, Gary D. Sandefur and Daniel H. Weinberg, eds., Confronting poverty: Prescriptions for change. New York: Russell Sage Foundation, 1994, pp. 168-204.

Blank, Rebecca M. It takes a nation: A new agenda for fighting poverty. New York: Russell Sage and Princeton University Press, 1997.

Bloom, Howard S.; Orr, Larry L.; Bell, Stephen H.; Cave, George; Doolittle, Fred; Lin, Winston and Box, Johannes M. "The Benefits and Costs of JTPA Title II-A Programs: Key Findings from the National Job Training Partnership Act Study." Journal of Human Resources, Summer 1997, 32(3), pp. 549-676.

Couch, Kenneth A. "New Evidence On the Long-Term Effects of Employment Training Programs." Journal of Labor Economics, October 1992, 10(4), pp. 380-88.

Friedlander, Daniel and Hamilton, Gayle. "The Impact of a Continuous Participation Obligation in a Welfare Employment Program." Journal of Human Resources, Fall 1996, 31(4), pp. 734-56.

Friedlander, Daniel and Robins, Philip K. "Evaluating Program Evaluations: New Evidence On Commonly Used Nonexperimental Methods." American Economic Review, September 1995, 85(4), pp. 923-37.

Friedlander, Daniel; Greenberg, David H.; Robins, Philip K. and Caldwell, Bruce. "Evaluating Government Training Programs for the Economically Disadvantaged." Journal of Economic Literature, December 1997, 35(4), pp. 1809-55.

Gueron, Judith M. "A Research Context for Welfare Reform." Journal of Policy Analysis and Management, Fall 1996, 15(4), pp. 547-61.

Gueron, Judith M. and Pauly, Edward. From welfare to work. New York: Russell Sage, 1991.

Heckman, James J. "Instrumental Variables: A Study of Implicit Behavioral Assumptions in One Widely Used Estimator." Journal of Human Resources, Summer 1997, 32(3), pp. 441-62.

Heckman, James J. and Hotz, V. Joseph. "Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs." Journal of the American Statistical Association, December 1989, 84, pp. 862-74.

Heckman, James J.; Ichimura, Hidehiko and Todd, Petra E. "Matching As an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme." Review of Economic Studies, 1997, 64, pp. 605-54.

Heckman, James J. and Robb, Richard Jr. "Alternative Methods for Evaluating the Impact of Interventions: An Overview." Journal of Econometrics, October/November 1985, 30(1/2), pp. 239-67.

Heckman, James J.; Ichimura, Hidehiko; Smith, Jeffrey and Todd, Petra. "Sources of Selection Bias in Evaluating Social Programs: An Interpretation of Conventional Measures and Evidence On the Effectiveness of Matching As a Program Evaluation Method." Proceedings of the National Academy of Sciences, November 1996, 93, pp. 13416-20.

Heckman, James J.; Hohmann, Neil; Khoo, Michael and Smith, Jeffrey. "Substitution and Drop Out Bias in Social Experiments: A Study of an Influential Experiment." Unpublished paper, 1997.

Imbens, Guido and Angrist, Joshua. "Identification and Estimation of Local Average Treatment Effects." Econometrica, March 1994, 62(4), pp. 467-76.

Johnson, Terry R. and Stromsdorfer, Ernest. "Evaluating Net Program Impact," in Ann Bonar Blalock, ed., Evaluating social programs at the state and local level: The JTPA evluation design project. Kalamazoo, Michigan: W.E. Upjohn Institute, 1990.

Koon, Richard L. Welfare reform: Helping the least fortunate become less dependent. New York: Garland, 1997.

LaLonde, Robert J. "Evaluating the Econometric Evaluations of Training Programs With Experimental Data." American Economic Review, September 1986, 76(4), pp. 604-20.

O'Neill, Dave M. and O'Neill, June Ellenoff. Lessons for welfare reform: An analysis of the AFDC caseload and past welfare-to-work programs. Kalamazoo, Mich.: W.E. Upjohn Institute, 1997.

We show here that our measure of selection from the waiting list can be used as an instrument if it is independent of unmeasured factors influencing post-program earnings. The estimate we obtain has been called a local average treatment effect (LATE) by Imbens and Angrist (1994).

Specify that the outcome of interest is a function of program participation and individual characteristics, where we allow the impact of characteristics to depend on participation:

The measure of obvious interest is

Heckman (1997) has argued that terms used as instruments seldom satisfy these formal requirements, even when it might appear that they are fully exogenous. For example, he argues that use of the draft lottery as an instrument to predict military service produces inconsistent estimates of the effects of military service on later earnings. The reason is that those who decide to join the military despite little risk of draft (i.e., those with high numbers in the lottery) will do so because their expected benefits from military service are larger than their observed characteristics would imply, inducing a dependence between risk of being drafted and expected gain among those who enter the military. This difficulty would appear to preclude the use of most instruments, even those that appear exogenous; the exception would be the case where a proportion of all applicants were randomly denied access to the program.

Our waiting list variable is subject to Heckman's criticism. Our maintained assumption is that the decision at one point in time to offer services to an individual is essentially random, conditional on measured individual characteristics. Those who are not offered the service at such a point, however, may remain on the waiting list, and many ultimately do receive an offer of service. Among this latter group, those who are offered services will be those who were most willing to remain on the waiting list, a group for whom the service is most valuable. This implies that Z_i will not be independent of U_i¹-U_i⁰, suggesting that Z_i is not a good instrument. We show next that this conclusion is somewhat misleading. The waiting list variable can be used to identify effects of the program we examine, where those effects must be understood to apply to a specific group, not all those offered services.

Let us define the waiting list variable we use in greater detail. For individual i, Z_i is a binary variable indicating whether the individual is chosen from the waiting list after spending t months on the list. Condition the analysis on the individual being available to be selected, meaning that she has been on the waiting list at least t months. Among those not chosen, some ultimately obtain services, while others do not. These two groups are defined as:

Designate k(X) as the proportion of individuals in the population with characteristic X who would be offered the service if no offers were made at time t. If we take the expected difference between the group offered services at time t, and those who are not, we can write

Research reported here was undertaken in connection with a University of Missouri-Columbia project funded by the State of Missouri Coordinating Board for Higher Education, the Departments of Economic Development, Elementary and Secondary Education, Labor and Industrial Relations, and Social Services, under the direction of the Missouri Training and Employment Council. We wish to thank Richard L. Koon, Robert LaLonde, David Mandy, Seth Sanders, Ken Troske for helpful comments. The opinions and interpretations expressed in this paper are those of the authors and do not necessarily represent those of any agency of the State of Missouri.

Key: See data appendix.

Key: See data appendix.

Key: See data appendix.

Key: See data appendix.

Key: See data appendix.

¹Interactions of seven region dummies with Age, Race, Years of Education, Number of Children, Months on AFDC, Work Experience Prior to AFDC, and Earnings in Previous Year.

²Interactions of Year of Entry dummies with Age, Race, Years of Education, Number of Children, Months on AFDC, Work Experience Prior to AFDC, and Earnings in Previous Year.

Table 9: Instrumental Variables Estimates of Program Impact Based on Equations Predicting Post-Program Earnings Controlling for Individual Characteristics:

1. See Friedlander, Greenberg and Robins (1997) for a review of studies evaluating training programs.

2. Bloom et al. (1997) point out that if receipt of service can be fully measured and it can be assumed that dropouts obtain no benefits from the program, adjustments can be made to determine service impact. Our data allow us to determine whether individuals offered services actually participated during the three years of our study. However, some of those who were denied access to the program in this period undoubtedly obtained the same or similar services though other channels or obtained services through the program after the period of our study. Given this difficulty, we have not attempted any adjustment of our estimates.

3. See Koon (1997) for an examination of Missouri's FUTURES program focusing on the first two years of implementation.

4. Women made up four-fifths of the individuals on our files for whom the voluntary/mandatory classification was available.

5. This means that a small number of individuals who first appear in our files after September 1992 may have had contact with the FUTURES system prior to that time.

6. Most of those who ceased to appear on the waiting list disappeared from our FUTURES file, and therefore had no further contact with FUTURES in our study period. A small portion shifted status (e.g., an individual originally coded as a volunteer only was reclassified as mandatory), and so could have been offered services later.

7. Coverage is similar to that in other states. Among the most important omissions are U.S. Postal Service workers, employees in religious and some nonprofit organizations, railroad employees, college students working for their colleges, military employees, and the self-employed.

8. Approximately 97 percent of nonwhites in our sample are identified as black, with the remainder in other racial groups or not identified.

9. The rates, in part, reflect the high levels of turnover in the AFDC population. See Gueron and Pauly (1991) for a discussion of factors affect participation rates in employment programs.

10. Of course, such selection, insofar as it is not captured in measured variables, would bias estimated program impacts if selected individuals were likely to have promising employment prospects.

11. The detailed derivation is shown in the appendix.

12. We also estimated effects using separate equations for each year of entry and each waiting list classification. Estimates vary substantially among subgroups, but, given the importance of sampling error, no patterns emerge.

13. Note that e_i depends in part on whether the individual is selected at a future time.