Abstract
Background:
This article studies the effect of PIPOL, an integrated program of active labor policies launched by the Friuli Venezia Giulia, an Italian region, in 2014.
Objectives:
To understand the impact of training in a classroom setting (off-the-job) and work-related training (on-the-job) on employment integration of benefit recipients.
Research design:
We adopt a counterfactual approach by comparing a target group (treated) against a control group (19,899) extracted by means of propensity score matching and Mahalanobis distance matching among subjects who, while registered in the program over the years 2014–2016, had never benefited from it. The selection of about 7,175 recipients in the program and in each type of intervention was random. Subjects: About 30,000 job seekers made up of 3,911 interns, 2,945 trainees, and 319 recipients of training and internship within PIPOL. Target: Young people, Not in Education Employment or Trainings, and over 30s.
Measures:
We look at different outcomes: employment tout court and employment in open-ended contracts.
Results:
The overall net impact of PIPOL was equal to +5 pp on average. Specifically, impact results were classroom training none, internship sizable (+14.1 pp), and training combined with an internship, quite sizable (+9.6 pp). Furthermore, training to gain a qualification was the most effective (+6.4 pp) among those receiving combined training and internship. Internship also increased the chance to find permanent employment (+3 pp). Among recipients, women, immigrants, and low-skilled recipients registered the most sizable impact on finding employment and training in manufacturing and construction was more effective than elsewhere.
Conclusions:
Italian young people have ever-increasing academic attainment but, due to the sequential nature of the education system, little work-related competences. This could explain the greater success of internships on classroom training.
Keywords
Introduction
This article studies the effect of PIPOL, 1 a program of active labor policies launched by the Friuli Venezia Giulia, an Italian region, in 2014. The program was funded with different financial sources, including the European Social Fund (ESF) and the Youth Employment Initiative (YEI), and was aimed at smoothing the school- or university-to-work transition by providing participants with a set of new competences to (re)enter the labor market.
The evaluation exercise focused on the first stage of PIPOL which had been completed by the end of 2016. The aim is to understand the impact of off- and on-the-job training on the employment integration of benefit recipients. To address the issue, we employed a counterfactual approach: A control group was extracted by means of propensity score matching (PSM) among those who, though registered in the program over the years 2014–2016, had never benefited from it due to a number of different reasons which will be discussed in the Data section but which were essentially random. Ours is the only published paper on Italy that drew the control sample from the same pool of registered individuals from which the target group was chosen, as Angrist (1998) suggested, to minimize omitted heterogeneity issues.
As a robustness check, we also used the Mahalanobis distance matching procedure to select the control group. The latter approach is more rigorous as it selects only individuals in the control group who are identical in terms of observed characteristics to the target group.
The study regards 7,175 people, of which 3,911 attended off-the-job training, 2,945 attended on-the-job training, and 319 both types of intervention. The control group is almost three times bigger (19,889). We based our analysis on the monitoring data of the program and administrative data on compulsory communications (Comunicazioni obbligatorie or COBs) that employers send to employment offices whenever a labor contract is started, discontinued, or modified. The latter provided information on outcome variables.
Overall, this is an excellent and uncommon data set for Italy and, we believe, also beyond Italy. First, it is constituted of an unusually large number of individuals for both the target and control groups. Second, it contains a wealth of information not only on the individual characteristics of the beneficiaries but also on the type of program implemented, of services provided, and outcomes. Furthermore, we also obtained information on previous employment experience for both the treated and control groups. This is uncommon in other data sets and proved to be particularly interesting when looking at the performance of young people. According to Caliendo et al. (2017), work experience before entering the program is able to capture unobserved differences among target and control groups. Hence, this study represents an important addition to the emerging Italian literature on program evaluation, considering the small number of previous studies, a lacuna also noted in the meta-analysis of the international literature authored by Card et al. (2010). Furthermore, this study is one of the first analyses of the effects of measures implemented within the YEI. To our knowledge, previous studies are limited to a paper assessing the impact of YEI in Latvia (Bratti et al., 2018) and the evaluation of the Italian YEI (Anpal, 2018; Isfol, 2016).
The study shows positive net effects of the PIPOL program on the employment status of recipients, mainly attributable to on-the-job training.
The remainder of this article is organized as follows. The second section describes the main characteristics of the PIPOL program. The third section reviews the previous literature on active labor policy evaluation. The following two sections discuss methodology and data. The sixth section presents the descriptive analyses, and the seventh section presents the results of the impact analysis. The eighth section contains some further robustness checks. Some concluding remarks follow.
The Main Features of PIPOL
Friuli Venezia Giulia, a region in the North East of Italy, adopted PIPOL by integrating several financial resources, among which are the ESF and YEI, with the aim of bringing together several types of active labor market measures under a unique tool. The first phase of PIPOL, which is our focus, lasted from May 2014 to December 2016. The program’s budget was around EUR 71.5 million, of which EUR 30.6 million were funded by ESF.
PIPOL was targeted at different groups of people with different needs: Group 1 included young people aged 15–19 at risk of dropping out of school, 2 Group 2 included young Not in Education Employment or Training (NEETs) under the age of 30, Group 3 included under-30 youngsters with a high school diploma or a professional qualification attained within the preceding 12 months, Group 4 included young people under 30 with a university degree obtained at least 12 months earlier, and Group 5 included over-30 people who are unemployed or at risk of unemployment.
The participation in the PIPOL program was structured in three phases. Phase 1 was registration: People eligible can register online or by accessing a public employment service (PES) or other institutions for some groups. 3 During Phase 2, registered people received orientation services and were profiled according to their needs band; this service had to be offered to people within 60 days from the registration to the PESs. An individual action plan (Piano di azione individuale [PAI]) was established, showing the type of active policies to be received. Phase 3 is the implementation of active measures, such as on-the-job training, classroom training, labor incentives, and support for business creation. These active measures had to be delivered within 4 months from the beginning of the PAI. It is worth noting that the groups of people under analysis (from two to five) were all equally eligible for the measures analyzed in this article, off-the-job and on-the-job training programs.
Literature Review
International literature does not indicate clear evidence of the impact of active labor policies. Card et al. (2010), for instance, analyzed 97 studies implemented worldwide from 1995 to 2007 and found that training programs showed statistically insignificant or even negative results in the short run (up to 1 year from the interventions) but generated more positive impacts than other proactive measures in the long run.
More recently, the same authors have updated the 2010 research. They analyzed more than 200 studies implemented from 2007 onward (Card et al., 2018). In addition to confirming previous results, though with more robust effects of training programs in the medium-long run, the study showed that the impact tends to be different according to the target group considered, with larger effects for women and the long-term unemployed. Less clear-cut results were found for programs addressing young people. Moreover, according to Card et al. (2018), proactive schemes are more effective in times of economic crisis than in periods of economic expansion, especially the programs of training and skill development.
Another recent meta-analysis by Vooren et al. (2018) based on 55 studies implemented with experimental or quasi-experimental methods from 1990 to 2015 founded that the most effective measures both in the short and in the long run were incentives and hiring subsidies in the private sector, immediately followed by training interventions.
Kluve et al. (2019) reviewed 113 evaluation studies on specific programs relative to young people, implemented in both industrialized and developing countries. Overall, only slightly more than a third of the evaluations showed positive results of youth programs as measured in terms of employment insertion or of impact on wages. According to Kluve et al. (2019), in more industrialized countries, the types of interventions implemented were less important than the context and the way in which they are implemented. The authors also underlined that interventions aimed at integrating different types of programs had a greater chance of success.
The relevant literature relative to Italy has appeared only recently. Irpet (2011) studied training programs financed by Regional Operational Program ESF 2000–2006 in Tuscany between 2007 and 2008. Using the PSM and a rich set of control variables collected through a survey of treated and untreated people, they found a positive effect on the employment probability 3 years from the end of the program, with a positive differential with respect to the control group equal to 10 pp for the unemployed and 20 pp for young people entering the labor market for the first time. However, only for the latter, the impact on the probability to find stable employment was greater than zero, while for the unemployed, training did not add value to employability in permanent jobs. Moreover, among unemployed men over the age of 30, and with a low level of education, the effect was greater (for women the effect was positive but not statistically significant). Among the inactive group, a positive impact was found for under 20-year-olds with a high secondary school diploma. For both groups, the impact is greater when the recipient enters the training program as a long-term unemployed.
Ismeri Europa (2011) studied the impact of Dote lavoro e formazione (voucher for work and training; DLF) in Lombardy in 2009–2010. The study, relying on PSM and conventional control variables and identifying the control group with applicants who did not participate, found a null impact on short-term (less than a year) employment probability but a positive empowerment impact. Recipients had 8 pp greater probability to become active job seekers than the control group.
Pastore (2020) found, in the DLF, the typical problems of quasi-markets (lion’s share of private organizations, cherry picking, gaming, and asymmetric information). However, different expedients were devised in the program to minimize these shortcomings and keep them at a physiological level. The DLF has indeed inspired the recent Jobs Act reform of the PES in 2015.
A similar approach was used in a study on on-the-job training programs which were implemented in the Marche region with the ESF in 2007–2013 (Fondazione Brodolini, 2013). The analysis showed an immediate positive impact in the months following the experience. This, however, tended to disappear after a year. On-the-job training also positively affected the length of subsequent employment by about 40–50 days in 12 months as compared to the control group.
Mazzolini and Orlando (2014) studied training programs in an Italian region over the years 2010–2011 financed with the ESF, using applicants not admitted as the control group and applying the PSM on a conventional set of variables; they were unable to find a statistically significant impact on the employment probability, both after the course’s end and in the medium term. This result was true only for training courses finalized to favor job reentry, while the postdiploma training showed positive results 18 months from the end of the program 16 pp higher than the control group.
Severati (2015) analyzed post-diploma training courses for 20- to 29-year-olds in different Italian regions at the end of 2000. They applied the PSM approach and used as control variables, the date of enrolment at PES and the previous labor experiences (24 months before the participation), besides the conventional variables on the characteristics of participants (such as age, education, and gender); their control group is composed of people younger than 29 years enrolled at PES offices as unemployed available for work. They found a positive impact in Piedmont (5 pp), starting from 12 months from the beginning of the courses but a negative impact on the probability to find stable employment. The youngest recipients showed a higher impact, women did not. 4
In the first report on the Italian YEI, Isfol (2016) assessed the impact on employment of the interventions from May 2014 to September 2015. The study focused in particular on the employment status at the end of the program. The control group consisted of young people enrolled in the program and awaiting treatment or being taken on in the program and young people under 30 not registered in the program, and matching methods and inverse probability weighting were applied. The results showed an increased employment probability of the treatment group of about 7.8% on average and 8.9% for on-the-job training. In the second report on the Italia YEI, Anpal (2018) studied a group of approximately 218,000 young people, of which approximately 136,000 treated individuals and found a net impact of the interventions on the probability of employment after 18 months of approximately 7.6% more than the control group. The study relied on covariate-matching methods based on the lowest distance between treated and control groups in terms of distance in profiling scores, age, and month of taking charge of individuals.
Bazzoli et al. (2018) assessed the impact of various active policy programs implemented in Trentino under the aegis of the ESF or with regional funds, applying the PSM (based on the blocking with regression adjustment estimator) and a rich set of control variables, such as employment status in each of the 36 months before the start of the courses, the economic sector corresponding to the longest spell of employment; the labor income and conventional sociodemographic characteristics of the individuals. For ESF courses, they noted a positive impact 12 months after the end of the program on the probability of hiring, 24 months after the end of the program the impact was 6%. The increase was greater for women and older people, but it was not statistically significant for foreign women. Furthermore, the study stated that ESF training courses were more effective than regional ones: The probability of employment was 27% higher for the target group 24 months after the end of the program.
Ghirelli et al. (2019) assessed the impact of on-the-job training for graduates in the Umbria region, financed by ESF 2007-2013 on the probability of employment in 2015, 2 years after the start of the program. They used the PSM approach and sociodemographic characteristics of the individuals, including the field of study of their university degree and not only the level and the number of unemployment spells as control variables. The results showed a positive impact of 12%–14% on the probability of employment of the treated subject.
Duranti and Sciclone (2018) studied the impact of the training programs on the job implemented in Tuscany in the years 2007–2013, applying matching on covariates for continuous covariates (age, years of education, length of unemployment spell, days worked in the previous years, and previous occupation), after an exact matching on categorical ones (sex, nationality, area, period, and sector of activity in the previous 2 years). They found an impact on the probability of employment 18 months after the start of the program of 8.2%. The impact was reduced to 3.7% when employment in permanent contracts was considered. The program was particularly effective in the case of older recipients with the lowest level of education and long-term unemployment. The greatest impact was found in the case of combined on-the-job and off-the-job training programs.
In a later study, Cappellini et al. (2019) used a combined strategy of exact matching and propensity score. The first is on the basis of sex, nationality, “unemployment status,” and quarter of hiring/registration, and the second is on age and local labor system. They found positive short- and long-term effects of vocational training on the probability of employment of the participants in the program both in the short and medium term (2 years).
In other recent studies, Donato et al. (2018), relying on PSM and conventional control variables, found a positive effect from training courses financed in Piedmont in 2014–2015.
Abagnale et al. (2017) provided an aggregate assessment of the impact of the Youth Guarantee. The authors looked at the variation in the probability of finding employment and the evolution of the channels of entry into the world of work for young people aged between 15 and 29 following the introduction of the Youth Guarantee in Piedmont and Sardinia. Their analysis was based on administrative data from the Employment Information Systems, to which they applied a difference-in-differences approach, using people of the 30- to 40-year-old employment centers as a control group, as they were not eligible for the program. The results highlighted a limited effect of the measure on an increase in the probability of being sent to work in Piedmont, but zero effect in Sardinia.
Their approach avoided the criticism of a partial equilibrium approach typical of microeconomic studies. Regardless of the impact of the program on the participants, the authors wondered, more ambitiously, whether, overall, the program had increased the employment opportunities for young people. However, the difference-in-difference approach, if it controls for fixed effects, cannot exclude the presence of concomitant factors (differentiated growth, other legislative, or policy interventions) that act in the same or opposite direction to that of the interventions.
The study also highlights that the program has led to a reduction in the use of short-term and fixed-term contracts and an increase in traineeships for young people, compared to the preprogram period.
Almost all the aforementioned studies found positive net effects of the interventions on the probability of occupation of the participants. The differences between the studies depend on a number of factors (sample size, specificity of the treated and control groups, method of identification of the control group, and so on) and institutional factors (type and modality of intervention, characteristics of the recipients, and so on).
Method
Our econometric analysis follows a counterfactual approach with a comparison between a target and a control group, both chosen for being very similar under several important observed characteristics. The main challenge and advantage of this approach consists in preventing sample selection bias, at least under the observable characteristics, in the identification of the target and control groups. Such bias would result in under/overestimation of the impact of treatment, according to whether the control group presents omitted heterogeneity that is not fully accounted for by our econometric approach. In fact, if for some reasons, the target group were to have better employability characteristics than the control group, the impact of the program itself could be overestimated. In this case, the better values of the outcome variables found for the target group may be attributed to the treatment while, instead, they could be associated with unobserved heterogeneity between the two compared groups. In other words, it is necessary to ensure that the target and control groups only differ in terms of treatment, with all other conditions that affect the employability of an individual (e.g., personal features, previous training, previous employment experience) being equal (see, among others, Angrist, 1998; Angrist & Pischke, 2009; Cerulli, 2015; Sianesi, 2004).
Nonetheless, it is important to clarify from the onset that while our approach is able to control for bias which is due to observable differences among the target and control group, it provides no guarantee against omitted heterogeneity which cannot be observed or that is not proxied by observed characteristics. Omitted heterogeneity may include motivation in finding a job, talent, or skills that are not measured in our data bank. This requires us to be very cautious regarding the selection of the two groups in order to make sure that they are as similar as possible, also from the point of view of omitted heterogeneity. This is never 100% certain, but we have sought to reduce this possible source of bias to a minimum by selecting the target and control groups in such a way as to minimize differences in omitted heterogeneity as discussed in more detail below.
In order to create a “quasi-experimental” observation environment, that is, to minimize observed differences in characteristics between treated and untreated people, we use PSM. This is a statistical matching technique that identifies the control group in untreated subjects having observable characteristics the most similar to the treated subjects.
Following Angrist and Pischke (2009), we estimate the so-called ATT, that is, the Average Treatment Effect on the Treated. 5 The ATT represents the impact of the program on the treated in the event of participation as compared to the counterfactual case where the treated themselves did not participate in the program. At least this would be the condition in the physical sciences, in which experiments can be reproduced by changing only some conditions while holding all the other conditions constant. However, in the case of social sciences, a full counterfactual situation is impossible to reproduce since the same individual cannot be observed as both having and not having been a beneficiary of intervention. The common practice of evaluation to overcome this, which is sometimes called the “missing data” problem, is to select a control group that mimics the treated group.
Once a control group is available with the same characteristics as the target group but which did not participate in the program, it will be possible to define something very similar to an ATT. More analytically:
where D is a variable equal to 1 if the treatment occurs and 0 if it does not occur; Y 1 is the value of the outcome variable given the treatment and Y 0 is the outcome variable in the absence of treatment. In abstract, Y 1 and Y 0 refer to the case when the target group has and has not undergone treatment. But, again, since the same target group cannot be observed after receiving and without receiving the treatment, the ATT is estimated by comparing the values of Y 1 and Y 0 relative to a target group and a control group, that is to say individuals who have exactly the same characteristics as the target group but did not participate in the program. 6
We distinguished three treatment types: (a) on-the-job training, (b) off-the-job training, and (c) a combination of both. In fact, on-the-job training may have different logic and aims and, hence, the effects may be different (Sianesi, 2008). In particular, classroom professional training mainly has an “educational-training” content, while internships are short-term professional experiences in which beneficiaries may also develop work-related competences (Pastore, 2015, 2018).
In order for the PSM to work effectively, it is extremely important to identify a valid control group. In our case, as Angrist (1998) suggested, the control group was drawn from the pool of those who were eligible to join PIPOL between 2014 and December 2016, but who did not join for various reasons. The data bank does not contain specific information on the reasons why they did not participate in the program, but we do know from the organizers that the reasons were essentially of an administrative nature, such as the impossibility to find enterprises which were available for specific on-the-job training or the fact that, for some personal unknown reasons, some people gave up participating in the activities proposed by PESs. Most important is to note that the agencies implementing PIPOL did not impose any process of exclusion. The registered individuals who were eligible, but did not participate, were particularly suitable subjects to be taken as control group as they had signed a declaration of immediate availability to participate in PIPOL and, exactly like those who did eventually attend, had expressed their willingness to participate in a training or internship. Moreover, the number of this group was quite large, almost 20,000 individuals. The target group of those who attended the program represented about 26.5% of the applicants.
Furthermore, we know that those individuals who applied to the program, but were not treated until the end of 2016, were also not treated in the following period (2017–January 2018).
The key assumption for PSM is selection on observables, that is, there are no omitted variables that substantially shift both the selection into the program and the outcome. This is a strong and untestable assumption. In support of that assumption here, we note that we have an unusually rich set of covariates for the type of data used in this type of estimates. The covariates on which PSM is implemented in our case include the following: – age, as measured at the time of the beginning or of enrollment in the program; – gender; – citizenship; – level of education achieved: lower secondary, professional, upper secondary school, university degree, and postgraduate education (master or doctorate); – the presence of work-related skills acquired before the start of the program and the duration of the employment spell; and – province of residence.
7
Several studies (e.g., Caliendo et al., 2017) showed that including previous employment experience in the variables used for matching increases the likelihood of capturing unobservable elements in the differences between the treated individuals and the control group.
In addition, the comparison group is chosen from applicants. Angrist (1998) explains that this is a strong comparison group because both control and target groups share the same level of skills and, more importantly, motivation. To sum up, we do not know in our data how exactly applicants were sorted between the different types of program and treatment. They were all eligible, but there was no prior knowledge in them about the success rate of each treatment and, therefore, there is no reason to believe that the most skilled and motivated chose the most effective program (on-the-job training) and the least skilled and motivated chose the least effective program (off-the-job training). By the way, we may anticipate that the results of the off-the-job training show that that training leading to a qualification is still effective. In other words, even if there was self-selection, this was not driven by skill or motivation: Probably, participants could choose, but due to the lack of information regarding the possible outcomes of the program, it is hard to say how that choice was made. That is why we assume that it was random. This was also the answer of the local organizers of the program.
Moreover, in Italy, there is no prior knowledge of the effectiveness of different active labor market programs. They are relatively new. The country traditionally spends a very little amount of money on it. The European Youth Guarantee, within which PIPOL was organized, is contributing to develop active labor policy, which was almost absent in the country earlier. This explains also the lack of evaluation studies until very recently.
Moreover, youth unemployment is quite widespread and high, and there is little difference between young people attending the program in terms of motivation and skill. The most skilled would not even apply to these types of programs. The segmentation of the labor market is strong and the best way to find a job is by means of informal networks, namely networks of parents and relatives. Other methods are considered appropriate for least skilled individuals. Finally, as already noted, discussions with program staff did not suggest strong reasons for the difference.
As a robustness check, different types of matching were implemented, namely PSM with different individuals and Mahalanobis. We used nearest neighborhood matching in our PSM analysis which matches individuals in the treatment group to an individual in the control group with the closest propensity score. In some cases, as robustness check, we repeated matching by choosing either five, 10, or 15 individuals from the control group with the most similar scores. 8 Mahlanobis distance matching is perhaps more accurate than PSM insofar as it only matches individuals with exactly the same characteristics. We calculated standard errors in different ways, including a bootstrapping method with 50 replications, but results seem to be unaffected by the way standard errors are computed. 9
Data Used
We used three data sources for our study. First, we identified the group of the treated subjects, namely those who had completed PIPOL by December 2016 from the ESF program monitoring data. There were two main sources of this data bank: (a) the “OPOC” system, containing monitoring data of the internships funded by the NOP YEI (National Operation Program for the Youth European Initiative); and (b) NETFORMA, which included all other types of PIPOL programs. These two sources were rather rich in information regarding the sociodemographic characteristics of participants to the program (gender, age, education, province of residence, citizenship), but also with regard to the characteristics of the interventions, such as the start and end date, the type, length, and industrial sector of training provided. 10
Administrative data coming from ERGONET provided the second statistical source of information. This was the database obtained from the labor exchange offices and included information on all people who signed up as unemployed at the labor exchange offices (PESs) but who, for whatever reason, did not participate in the program. This statistical source allowed us identifying the control group and their main sociodemographic characteristics.
Note that, following Angrist (1998), both the treated and the control groups were selected from the same statistical source, confirming, as already mentioned above, that the characteristics of the two groups were similar in terms of motivation and skills. We had no particular reason to believe that there was any type of self-selection into participation of the target group. As noted in the Method section, the divide between participants and nonparticipants was, by and large, random. In a nutshell, there is no information in the data set on the reasons why some registered individuals attended and others did not. However, personal communications with the staff suggest that some of them did not find enterprises providing training places or eventually gave up for some personal reasons unrelated to the program.
The third data source came from the so-called mandatory communications (COBs). These data sources contain information on any type of labor contract which has started, ended, or changed legal form (say move from fixed term to permanent and vice versa) of both samples of treated and untreated. In Italy, whenever an employer hires, fires, or changes the type of labor contract of any wage employee, a formal communication must be made to the PES. A limitation of COBs data is that it excludes three categories of workers: the self-employed; those who find a job in another region; and, obviously, informal workers for whom no formal labor contract is signed by the employer and employee. 11 It means that a higher probability of the treated to find a job than the untreated implies a formal job as a wage employee in the same region as the region of residence. In principle, we cannot exclude that the untreated found an informal job and/or was self-employed and/or was a wage employee in another region. However, as in other papers in the relevant literature (see, among others, Ghirelli et al., 2019), we hypothesized that these phenomena (self-employed, people working in another region and informal workers) would affect both the treated and the control group in the same proportion. In fact, most, if not all of the existing studies of program evaluation in Italy, gather information regarding the employment status of participants in training programs from the same COBs data and consequently make the exact same assumption.
Another limitation of the COBs data source is that these data do not provide information on other possibly interesting variables, such as the earnings of those who find a job.
Our analysis focused on 7,175 participants of training courses and internships completed by the end of 2016. Our sample is constituted overall of about 27,064 job seekers of which 3,911 trainees, 2,945 interns, and 319 recipients of both training and internship within PIPOL. Target were young people, NEETs, and over 30s. Overall, our treatment group entered the program between May 2014 and December 2016 and left PIPOL between September 2014 and December 2016 (Figure 1).
Period in which the treatment group entered (above) and left (below) PIPOL. Source: Our elaborations.
Most of the on-the-job training programs were insertion ones; they are full time (38 hr per week) and last about 6 months. In 20% of the cases, on-the-job training was in the manufacturing sector, followed by trade, liberal professions, scientific (17%), and technical (15%) jobs with the remaining programs representing less than 10% of the total.
Different kinds of off-the-job training were carried out: About 40% were lifelong learning courses, 30% training aimed at acquiring qualifications, and 20% other types, such as foreign language learning courses.
Descriptive Analysis
Table 1 presents results of t tests on the mean differences of the main variables used for the calculation of the propensity score indicator on which matching of the treated and the control groups takes place.
Mean Differences Between Treated and Control Groups Before and After Matching.
Source. Own elaboration.
Note. Matching is based on propensity score matching and 10 observations in the control group for every observation in the target group. AVVPREPIPOL = Employment contract between 2012-2014, before starting PIPOL.
The table shows that, before matching, the two groups are different from several points of view as shown also in Figure 2. In particular, the treatment group includes a younger sample of individuals (age differences are apparent: 3.5 years), a larger share of women (57% vs. 54%), and, on average, a higher level of education. The treated group includes a larger proportion of individuals with university education attainment or higher (+7%), while the control group includes a larger proportion of individuals with a secondary school diploma and compulsory education (+ 12%). Finally, the target group includes a smaller proportion of foreigners (−8%) which, on average, are harder to employ. In terms of work experience before entering PIPOL, there are no statistically significant differences between treated individuals and the control group in having at least one active labor contract in the period 2012–2014 (around 40% of the total); the treated, in any case, tend to have contracts with a shorter duration. Overall, the table confirms that there is notable observed heterogeneity between the two groups which would certainly affect the final impact of the program on the chosen outcomes if estimated simply by probit on the two unmatched samples.
Common support in propensity score matching estimates, with five observations (sx) and 15 observations (dx). Variable employed on January 2018. Source: Our elaborations.
Table 2 shows the t-test results of the mean difference between treated and untreated in terms of employment status before matching and after matching. It is divided into four panels: – Panel (a): all participants, – Panel (b): participants who have completed the internship, – Panel (c): participants who have only done professional training, and – Panel (d): participants who have received both types of vocational training.
Mean Differences in Employment Rates, Treated and Untreated, Before Matching.
Source. Our elaborations.
Note. As already noted in the Data section, the employment status is observed in January 2018. Diff. = difference; Prob. = probability.
The differences highlighted in the table cannot in any way be interpreted as causal effect of the program on the probability of finding employment by the participants as the differences in productivity characteristics between target and control groups might explain at least in part the different probability of employment of the two groups.
The overall sample of the treated shows a higher probability than the untreated of being employed by about 10%. On the other hand, there is no statistically significant higher employment rate if only permanent employment is considered: As we can see, the treated group has an open-ended employment rate of 16%, compared with 14% of the control group.
Looking at the different subgroups of recipients, the results are fairly high and statistically significant for those receiving an internship on the probability of finding a job (+20%) and for being employed with an open-ended contract (+4%).
For the recipients of vocational training, the gross results on the employment variables are lower. However, in general, the impact is more on employability than on employment per se. In contrast to the observations of this study, professional training may still influence the probability of finding employment in a later period, namely after 2018. In fact, at least in principle, training should increase employability and, therefore, also increase employment chances in the long run. Future research should test this hypothesis empirically after collecting suitable data in the following years from the COBs on both the target and control groups.
Matching is quite effective. In fact, differences between treated and untreated after matching are much reduced and negligible in most cases. Nevertheless, PSM seems to be less able than matching based on Mahalanobis distance, as it is apparent after comparison of Table 2 with Figure 2.
Like for PSM, as well as for Mahalnobis distance matching, there may still be some observations in the treatment group which do not match those in the control group. To make sure that the two groups were similar after matching, a specific test was run. Figure 2 shows the results of such statistical test of the differences between the target and control groups without matching and after matching. The figure clearly shows that once matching is achieved, there is no longer any statistically significant difference between the two groups in the observed variables, which instead existed before matching. In the unmatched case, the differences between target and control groups are in some cases noticeable, sometimes in favor of the target group and other times in favor of the control group. The target group has a higher share of highly educated subjects and women, while the control group includes more foreigners, a higher share of people living in Trieste (the capital city of the region) and holding lower education attainment levels. The prematching samples correspond closely to the descriptive analysis depicted in Table 1.
Figure 3 contains a graphical representation of the common support in the estimates of the PSM for different types of observations (to five and 15) used for matching. 12 The graphs show that there was a wide common support that allowed us to adequately control for the different characteristics observed for the control and target groups. The common support was present in all estimates, but the shape and structure of our data are such that we have a small share of the sample of the treated in the far right of the figures which have a small common support. There is indeed some very small common support also for those observations to the far right, which, however, are a small share of the total. In fact, this does not prevent us from getting the results of PSM estimates. In any case, we have implemented as a robustness check the option “common.” The results do not change whenever we use this option. We also run regression based on Mahalanobis matching which is based by definition on exact matching of observations one by one and results do not change. 13
Results of Mahalanobis matching for different groups and for the variable employed on January 2018. Note. The figure measures the difference between the characteristics of the treated and that of the control group/
In order to test for the actual difference in coefficients relative to different types of treatment, we apply the test of homogeneity which implies comparing the differences between observed and expected values. Expected values have been computed taking the total and assuming that the coefficient is the same for each target group. The resulting statistics is a c 2 with (r − 1) (c − 1) degrees of freedom, whereas r is the number of rows and c of columns of the table. The statistics is computed as follows:
where
Econometric Analysis of the Impact of PIPOL
Before presenting our findings, it is important to specify that, as highlighted in the methodology section, our methods assume that selection was random, within the sample of applicants and the observed covariates. This is a strong assumption. In as much as it is violated, the estimates are not causal.
Table 3 shows the main results of the analysis. We report only the results of the probit models on the unmatched target and control groups as well as the results of probit estimates based on the matched samples. We have left only matching based on PSM with 10 matched observations and Mahalanobis distance. Other estimates based on one, five, and 15 matched observations were very similar and were therefore dropped for shortness sake. Findings are organized in different panels one for each type of treatment and on different columns, one for each outcome variable.
Impact of PIPOL on the Probability to be Employed.
Source. Our elaborations.
Note. The table shows the results of the regression model (probit) on the sample of treated and untreated matched with different matching criteria: propensity score matching and Mahalanobis. The first probit estimate in every panel is based on before matching comparison. The employment status is observed in January 2018 on the basis of the administrative data (Comunicazioni obbligatorie). Matching is implemented with replacement. Unreported estimates based on one, five, and 15 matched observations were very similar. They are available on request from the authors. Standard errors are in parentheses. ATT = Average Treatment Effect on the Treated; PIPOL = Piano d’azione per il sostegno all’accesso, rientro o permanenza nel mercato del lavoro.
Significance level: *.5 < p < 1. **.1 < p < .5. ***p < .1.
The results in terms of job placement tout court are equal to +5 pp for the entire target group (Panel [a]), about half of the unconditional mean difference, and statistically significant. There are no statistically significant effects on the probability to be employed with a permanent contract. These estimates are in line with those found in the literature review of the third section regarding Italy.
Interestingly, most part, if not all of the effect of the program on outcome variables, stems from the effect of those who undertook internships (Panel [b]). They are more than twice (+13.4 pp) more likely to be employed than the total sample average. The effect is about half of the unconditional difference. The ATT is generally slightly lower than the effect estimated by probit regression, confirming that part of the impact captured by the latter is attributable to heterogeneity which is not fully accounted for in the probit regression.
The small difference between probit and ATT results suggests that the value added of PSM, in this case, is not particularly high. In other words, we do control for sample selection bias, but, in our case, it is not a very big issue. This indirectly confirms our claim in the Method section that applicants are very similar to each other in terms of observed and also unobserved characteristics between target and control groups. This is typical of countries where youth unemployment is a mass problem, rather than a problem of a few individuals. In Italy, so many young people are unemployed that individual factors are not so important. More important is the lack of jobs in the economy for young people and the common problem of all: a huge gap of work experience, while possessing increasing educational levels (Pastore, 2019; Pastore et al., 2020).
The estimate based on the Mahalanobis method is slightly higher than the PSM one and suggests that part of the effect is lost with PSM because of the less accurate selection of target and control group. Having carried out an internship also significantly affects the probability of finding a permanent job, but with smaller impact (+3.3 pp based on PSM and +5 pp based on Mahalanobis distance). Instead, other types of intervention do not seem to affect in a statistically significant manner on the probability to find permanent employment. Vocational training (Panel [c]), on the other hand, has no statistically significant effect on the probability of being employed.
Slightly different is the case of those who receive both professional training in the classroom and internships in the company (Panel [d]). In this case, the net effect equals +9.6 pp, less than internship only, probably because the internship part of training is shorter and therefore both the professional content of the work experience and the ability to develop a network of knowledge in the company are lower. The statistical insignificance of the effect on open-ended contracts is to be attributed also to the small number of observations.
These first general results are interesting for two main reasons: First, they confirm the analysis of those observers who highlight the difficulty of young people in developing work-related skills, rather than the theoretical ones linked to general education, which is due to the sequential nature of the school-to-work transition typical of Italy (first education and then work experience; Pastore, 2016, 2019), and second, under certain conditions, temporary employment can be a good alternative to active employment policies, if it really helps to develop work-related skills and to broaden the social and informal network of young people.
The Heterogeneity of the Effects
In the previous paragraphs, we have estimated the average impact of different types of intervention (training, internship) on several outcome variables. However, as seen in the literature review, the impact of the programs may differ for heterogeneous group. For example, gender, age, and educational qualification differences could be not fully captured simply by using these as control variables. We focused only on the two types of program—traineeships and internship—for which there are also more observations. Moreover, among the many outcome variables available, we used only employment in January 2018 for the sake of brevity.
Impact by gender
As for Table 4, the positive impact of PIPOL is stronger for women (+5.8–6 pp) than for men (+4.8 pp), differently from what was found for the case of Umbria (Ghirelli et al., 2019) and Tuscany (Irpet, 2011) but in line with Bazzoli et al. (2018) for Trentino. Again, the main positive effect of the program stems from internships for both women (+15.3 pp) and men (+13.3 pp). For training, coefficients are not statistically significant.
Impact of PIPOL by Gender.
Source. Our elaborations.
Note. See notes under Table 3.
In this case, like in all others below cases, the homogeneity test easily rejects to null of equality of coefficients across treatments, suggesting that coefficients are statistically different from each other. 14
Impact by age
Table 5 shows that the effect of PIPOL is slightly higher for young people under 30: +6 pp against +4 pp. Again, the homogeneity test confirms that such differences are highly significant from a statistical point of view. Training does not affect the probability of being employed. It is even associated with a lower probability to find a job for the under 30, probably because of what Pastore (2016, p. 29) calls the “training trap,” namely the tendency to prefer training itself rather than actively seeking a job.
Impact of PIPOL by Age.
Source. Our elaborations.
Note. See notes under Table 3.
Internships seem to favor more the over 30 (+18.1 pp) rather than the under 30 (+12.6 pp), which is confirmed as a statistically significant difference by a homogeneity test of coefficients. This is probably because the over 30s already have some general work experience and are therefore more able to benefit from on-the-job training since they are able to develop the more valuable job-specific rather than general skills they already possess.
Impacts by citizenship
Here, we aim to understand if there are different effects for recipients with foreign citizenship, which are 10% of the total recipients of PIPOL and who are more present in training activities (13%) than in internships (7%).
Table 6 shows that, overall (Panel [a]), PIPOL is much more effective on foreigners (+16.1 pp) than on Italians (+4 pp) in terms of employment probability by a factor of 4. The effect of on-the-job-training is very high (+30 pp). A homogeneity test suggests these statistical differences in coefficients are statistically significant. Moreover, interestingly, for foreigners also vocational training has a positive effect on the probability of being employed. This finding is hardly surprising and suggests that the program is more effective for those who have fewer or no alternatives.
Impact of PIPOL by Citizenship.
Source. Our elaborations.
Note. See notes under Table 3.
Impacts by educational qualification
Similar to some previous studies (IRPET, 2011), Table 7 shows that PIPOL has been effective above all for those with a lower educational qualification, but there are important differences by type of policy.
Impact of PIPOL by Level of Education.
Source. Our elaborations.
Note. See notes under Table 3.
Overall, there are few differences between those with secondary school first and second grade (6.7 pp and 5.7 pp, respectively), while the impact on those who have a university degree is null on average. Overall, such differences are confirmed by the homogeneity test.
Even internships (Panel [b]), which have a positive impact on all levels of education, show double impact in the case of those who are up to high secondary school (+15–17 pp) as compared to those who have a university degree or higher (+7 pp). Among others, this finding is in line with Cerulli-Harms (2017), who even finds negative effects of traineeships for graduates in the short term, which then fade away over time. However, Ghirelli et al. (2019) find a positive effect of the same size as ours for university graduates.
Vocational training shows no effect on low education levels and even a negative impact on the probability of being employed in 2018 for those with the highest education level. This result may be due to the fact that in terms of training and competences which are developed in the classroom, graduates are already well-equipped for the labor market with the result that the added value of training is not as high in comparison to the control group, which is also made up of graduates.
Furthermore, the analysis should be completed by including additional elements to fully assess the impact, for example, wages and quality of work: Training might not have an effect on the employment probability but on the “quality” of the employment found.
The effects of different types of intervention
Here, we aim to test whether the impact of the program is different for different types of proactive schemes provided. We focus our attention only on the impact on employment condition in January 2018. This part of the analysis allows us to examine the demand side.
The effects of different types of training
An interesting question that many evaluation studies ask is: Which kind of intervention is more successful and which is not? The average result is the algebraic sum of not always positive effects and if it turns out that some types of training are less effective than others or that they actually have negative effects on employment, it may be appropriate to concentrate efforts on the most effective interventions, thereby reducing waste of public resources. In some countries, evaluation studies of the effectiveness of individual programs and subprograms have helped to maximize the effectiveness of public spending in the sector over the years.
Our data set allows us to analyze the relative effectiveness of different types of professional training (Table 8). We find that some types of classroom training have an effect on the probability of employment in the short term, although overall training is ineffective. In particular, training courses to gain a qualification have positive and statistically significant effects, similar to those found for PIPOL as a whole (+5 pp). It should be noted that internships are a compulsory part of the pathway to achieve the basic qualification of skilled training and this may therefore be the reason for this positive result. Even for permanent training, which alone accounts for more than 40% of all financed courses, there is some weak indication of a positive effect, though the analyses should be repeated over time.
Impact of PIPOL by Type of Training (Only People in Training Measures).
Source. Our elaborations.
Note. See notes under Table 3.
Training courses to form competences consistent with the repertoire of regional qualifications and linguistic training have a negative effect on the employment status. On the other hand, it should be noted that the main purpose of linguistic training is not employment placement per se but a skills improvement. About 70% of the participants themselves suggest that their expectation is above all to improve their language skills when filling in the entry form with the different information required by the Friuli Venezia Giulia region.
The effect by industrial sector of the internships
Here, we try to draw some indication regarding the industrial sector where internships were carried out. We focus only on three macro-partitions due to data limitations on other industrial sectors: manufacturing, construction, and services.
Table 9 shows that the sector where training is most effective is manufacturing (+20 pp). This probably derives from the fact that manufacturing is the sector in which work-related skills can be learned more easily through an internship and that there are no alternative ways to achieve the same skills, thus making internship more valuable. This result indicates a high added value of the program as the demand side factor could only partially explain this finding: In fact, it is true that the labor force of manufacturing (excluding construction) has gone up from about 123,000 employed in 2014 to 125,000 in 2016, but the trend is lower than that of the tertiary sector over the same period.
Impact of PIPOL by Sector of Internship.
Source. Our elaborations.
Note. See notes under Table 3.
In any case, internships in construction and services are also effective. The impact in the construction field is only a few percentage points lower than that in manufacturing. In the service sector, the employment impact is around 10 pp.
Robustness Checks
In this section, we carried out two other robustness checks which confirm our main findings. First, we aimed to address whether the choice of a cutoff date for observing the employment status of program participant makes a big change in the average impact of the program itself as found in the previous estimates. We divided the whole sample into three groups on the basis of the date the group members enrolled or signed up at the labor office declaring themselves as unemployed and available to work: those who had entered within 24 months before January 2018, between 24 and 36 months, and more than 36 months. This test aimed at verifying whether these groups of individuals had a remarkable different impact and, if it did, how big the difference was.
Table 10 presents the results for the three different periods and different types of treatment. For off-job-training, the first result is that those who enter the program earlier still exhibit positive effect, though lower than average (about +3 pp), while in the short term, the results are slightly negative and statistically significant, confirming the existence of a lock-in effect (Pastore, 2016; Van Ours, 2004). In the case of on-the-job training, differences in the effects across individuals entering the program in the three periods considered are negligible. Overall, the results confirm the findings presented above, especially for on-the-job training.
Impact of PIPOL by Enrollment Period (Employed).
Source. Our elaborations.
Note. See notes under Table 3.
Last but not least, we aimed to study the differences in the impacts of the different types of intervention, namely off-the-job and on-the-job training more in depth. As seen above, on-the-job training programs have a positive impact whereas off-the-job training programs do not show any statistically significant effect. In principle, this result could depend on self-selection, namely the fact the participants in the two programs would have different employability characteristics, even in the absence of any formal selection procedure to assign people to the two programs. Personal communications with the program organizers confirm that there was no particular factor affecting the selection in the two programs and that the two groups were randomly distributed in the two programs.
It is possible to observe that people who participated in classroom training are older and with a lower human capital than people participating at on-the-job training. In order to reduce these differences, we applied our matching approach using the recipients of on-the-job training as the treated group and the people receiving off-the-job training as a control group. Estimates confirm that the impact of on-the-job training is higher than that of off-the-job training by about +16.8 pp for both types of matching methods and are statistically significant. Overall, these differences are similar to those obtained when comparing interns with nonparticipants to the program and seem to confirm the robustness of our main findings. 15
Concluding Remarks
This article studied the effect of PIPOL, an integrated program of active labor policies, launched by the Italian Region Friuli Venezia Giulia in 2014, aimed at supporting people seeking employment. Different population targets, divided by need band, were the object of intervention and different mixes of off- and on-the-job training were provided to participants.
The evaluation exercise aimed to understand the impact of PIPOL on the employment integration of benefit recipients. To address the issue, we resorted to a counterfactual approach: A control group was extracted by means of propensity score or Mahalanobis distance matching among those who had registered in the program over the years 2014–2016, without having benefited from the program. This allowed us to control for observed heterogeneity through a battery of control variables (age, gender, citizenship, education, province of residence, and also, interestingly, preprogram work experience). We drew the control group from the same pool of individuals registered in the program but who did not attend for a number of different reasons. Overall, participating in the program can be seen as a random process.
We used different data banks coming from the administration of the program regarding the observed preprogram characteristics of the target and control groups. Moreover, information on outcome variables was obtained from compulsory communications that employers have to make to PESs whenever a labor contract is signed, terminated, or modified.
We found that the net impact of PIPOL was equal to +5.6 pp on average, but no impact was found for classroom training. The greatest impact was found for on-the-job training or internships (+14.1 pp). The latter also affected the probability of finding permanent employment (+3 pp). The mix of classroom training and internship had also a strongly positive effect (+9.6), mainly due to the training needed to gain a qualification, which has a strong content of internship (+6.4 pp). This was consistent with the view of a youth labor market where young people have excellent theoretical competences, but very little work experience and work-related competences, both general and job-specific (Pastore, 2015, 2017).
The off-the-job training programs showed no statistically significant impact on employment. These results are partly due to a lock-in effect, namely the tendency of those who attend training programs to delay their job search activities.
Interestingly, we found that the program had a different impact for different typologies of recipients and different types of intervention. The scheme seems to have a greater net impact in the case of women, foreigners, and lower educated young people. Some forms of off-the-job training nevertheless had a positive net impact on employment chances (training to gain a qualification). Internships in manufacturing and construction showed a greater impact than in the service sector.
To sum up, our findings suggest that an active labor market policy is more effective in Italy when it generates work-related competences.
Appendix
Methodology
The PSM approach is quite well-known by now, but here we recall some of its assumptions and logics. The correct identification of the ATT requires that at least three conditions being met: conditional independence hypothesis or analytically: Stable Unit Treatment Value Assumption hypothesis, and common support assumption:
where X represents a set of covariates that can complicate the analysis as they are both related to the selection in the probability of receiving the treatment and the possible outcome.
Hypothesis 1 implies that the outcome is independent of participation in the program conditioned on the observation of the characteristics X. In other words, checking for all observable characteristics, the decision to participate in the program should not be related to the possible outcome. The extent to which this occurs depends on data availability. This hypothesis actually suggests that the proposed approach is not exempt from the problems of endogeneity due to unobserved variables exactly as ordinary least squares.
Hypothesis 2 excludes the possibility of spillover effects or general economic effects that are determined by the program and indirectly influence the result of the program. This could happen, for example, if active employment policies (treatment) entailed an increase in public spending and, consequently, in the aggregate demand to increase the probability of finding employment for everyone and especially for those taking part in the program. This hypothesis is certainly verified in the case under consideration, given the relatively small size of spending involved and the relatively small number of participants in the program.
Hypothesis 3 implies that for any given value of the observed characteristics of individuals in the sample X, the probability of participating in the program is not certain. In other words, for any given observed characteristic of individuals, there should be no particular reason why an individual who possesses it is more likely to be in the target group or control group. This hypothesis is verified if we find a common support that is statistically not different between target and control group, that is, the two groups must have similar characteristics (see the discussion about this point contained in the descriptive analysis).
The treatment effect model is best described as a two-step procedure. In analytical terms, this procedure consists of the estimate of the determinants of an outcome variable (O), as a function of an indicator variable (T), a number of individual characteristics (X), and an error term (ε). We could write, then:
We used as the main outcome variable a dummy of the employment status in January 2018 and being employed with a permanent labor contract.
Instead, T is a dummy variable, which takes the value of 1 for program participants and 0 for nonparticipants.
Estimating directly Equation A1 by probit could be affected by self-selection bias if the individuals included in the target group are more motivated or skilled than those in the control group. The treatment effect model suggests to estimate Step 1 first, by estimating the determinants of the probability to experience our treatment variable:
where
Equation A2 is typically estimated by a standard probit model. It represents the probability of being in the target group and, therefore, of receiving the treatment as based on the actually observed variables (ϕ) with given coefficients (γ). Equation A2 allows the investigator controlling for the bias due to observed heterogeneity, not necessarily to also unobserved heterogeneity. Under this respect, the difference from OLS is the greater accuracy of PSM to control for observed heterogeneity, although at the cost of reducing the sample size.
On the basis of the estimated Equation A2, a propensity score is then predicted for each individual in the control group as based on their individual characteristics (Bryson et al., 2002). Then, we selected a sample with the most similar score to that observed for individuals in the target group from the pool of those who did not attend the program.
At Step 2, we estimated Equation A1, namely the probability of experimenting the outcome variable for program participants (the target group) and a sample of individuals enrolled in the program, but not attending in it, with the same productivity characteristics (control group) as based on our propensity score and different types of distance from the target group. If the third hypothesis mentioned above is respected, the PSM procedure ensures that the target group and control group should have the same result in terms of counterfactual potential.
The coefficient of the treatment variable (training program) in being employed (δ in Equation A1) gives the estimated ATT.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
