Advancing Methods in Entrepreneurship Education Impact Research: The Benefits and Limitations of Register Data

Objectives in Entrepreneurship Education Evaluation

Previous literature on entrepreneurship education has focused on start-up and job creation, stimulating entrepreneurial skills, and increasing entrepreneurial spirit, culture and attitude as main categories of objectives (Samwel Mwasalwiba, 2010). In terms of policy, the first would be a clear and wanted outcome to create economic growth (Gilbert et al., 2004), while the second and third might, at least for full programmes in entrepreneurship, need a more nuanced and detailed view on what broader impact they create. In such cases, the usage and value of obtained skills need to be explored and analysed in how they are applied in the alumni’s careers. The same applies to the entrepreneurial spirit, culture, and attitude – how are these affective factors influencing and benefitting the individual alum’s career or entrepreneurial efforts? These questions and problems relate to the connection between some educational outputs (e.g., subjective changes in intentions, skills, or mind-set) and the actual behaviour and impact created by the individual alum. Several calls for such research have appeared in the literature (e.g., Rauch & Hulsink, 2015), and while there are results showing the usage of alumni’s skills in their careers (Alsos et al., 2023) and how these skills influence the selection of jobs (Kucel et al., 2016), the long-term connection between entrepreneurship education’s subjective outcomes and post-graduation activities is still under-researched.

In addition, as more objective outcomes are investigated in long-term evaluations, these are few in number and the methodological rigorousness have been questioned by several (Duval-Couetil, 2013; Longva & Foss, 2018; Nabi et al., 2017; Rideout & Gray, 2013). For instance, in terms of the entrepreneurship education’s objective of creating jobs and new firms, a known issue is that entrepreneurship often occurs later in individuals’ careers (Burton et al., 2016; Marshall & Gigliotti, 2020). Some students might not start their own venture when graduating (or ever) (Dahlstrand & Berggren, 2010; Sørheim et al., 2021), but might apply for a job in an established firm or other start-ups, or pursue alternative careers (Dahlstrand & Berggren, 2010). Thus, those who become entrepreneurs and create jobs might do so later in their careers. This is not unexpected, as many entrepreneurs fail in their efforts, and starting a new venture requires investment and risk handling from the individual that might be undesirable if not necessary (Burton et al., 2016). Hence, expecting that entrepreneurship education meets a very high rate of contemporaneous start-ups among its alumni is perhaps unrealistic, and short-term evaluation might miss the correct career characteristics of the alumni. For other educational disciplines, like nursing, engineering, or business education, the outcome and objective might be investigated through employability; entrepreneurship education, on the other hand, needs to consider multiple outcome measures and objectives.

Sampling Issues in Entrepreneurship Education Evaluation

Another methodological issue, especially for education being elective, is the problem with control groups (Rauch & Hulsink, 2015). Students in elective courses or programs might be entrepreneurs before enrolling in their education; insights into how they behave after their education, without controlling for their entrepreneurial attraction, give a limited understanding of the education’s impact. Experiment designs have been applied and advocated to solve this challenge (Camuffo et al., 2020; Rauch & Hulsink, 2015), but practical as well as ethical issues may arise in such studies. For the studies that have applied an experimental design, the focus has often been on subjective outcomes and less on behavioural measures (Longva & Foss, 2018).

Related to the previous is the problem with self-selection (Rauch & Hulsink, 2015). Students that apply for a program or select elective courses or modules might have an interest in entrepreneurship or a goal of becoming an entrepreneur, which often means they have a positive attitude toward entrepreneurship (Karimi et al., 2016). As such, the difference in self-reported outcomes might differ pre and post education between students in elective and mandatory entrepreneurship education due to the students’ initial interest (see e.g., Karimi et al., 2016), and even in some instances, decrease the students’ outcome measure (see Oosterbeek et al., 2010). In the latter case, students who have some insights into entrepreneurship might think they have a good skillset but could experience the opposite when engaging in entrepreneurship education. This would, nevertheless, be positive as learning would have occurred (Karimi et al., 2016), and with good designs, the actual competence or skill development could be mapped (see e.g., DeTienne & Chandler, 2004). However, whether the development is creating an impact later in the students’ careers is still an under-researched and important question.

Long-Term Impact Evaluation of Entrepreneurship Education

In recent literature reviews on entrepreneurship education evaluations (Loi & Fayolle, 2021; Longva & Foss, 2018; Nabi et al., 2017; Rideout & Gray, 2013), the result show that few studies have investigated the long-term impact of entrepreneurship education on the individual’s behaviour. Fayolle and Gailly (2015) investigate the impact of entrepreneurship education on entrepreneurial intentions and its antecedents in the short term and after six months, finding evidence for a positive influence on attitude and perceived behavioural control after six months. Rauch and Hulsink (2015) also use the theory of planned behaviour in their study but also investigate entrepreneurial behaviour 18 months after the program. They find that entrepreneurship education positively influences students’ attitudes, perceived behavioural control, and intentions to become entrepreneurs, the latter of which also is found to influence entrepreneurial behaviour in the longer term. The individual’s entrepreneurial behaviour is also investigated by Gielnik et al. (2017). They study the impact of a training program on undergraduate students over 32 months, with measurements before and one, 12, and 28 months after the training program. In their study, they focus on the short- and long-term effects the educational efforts on passion, entrepreneurial self-efficacy, and business creation. The results show the complexity of the effects of entrepreneurship education, and how these interrelates and changes over time. However, while their study is longitudinal, they still call for more research using “[p]anel studies with an even longer scope”, as this could give more insights into the development of the individual’s entrepreneurial activities (Gielnik et al., 2017, p. 349). Such calls are also made by Rauch and Hulsink (2015), pointing to the fact that many start their entrepreneurial careers later in life. Thus, being able to look at the individual over a longer period would help in investigating the impact of education.

Case Presentation

The case is a two-year master’s degree in entrepreneurship, where the students are starting their own ventures as a part of the educational design. The programme is open to all individuals that hold a three-year bachelor’s degree or equivalent, and the students are coming from all backgrounds, for instance, nursing, engineering, economics, and so forth. The experiences and outcomes from their venturing activities are used in the education as important sources for learning, and the program can be labelled a venture creation programme (Lackéus & Williams Middleton, 2015). As the programme is located at a technical university in a tech-oriented region, most of the start-ups founded during the programme are technology-based, and several of the courses in the programme have a tech focus. In addition, the students sit in an incubator to which only the programme’s students have access, and where the students’ experiences consequently are shared in a community of peer professionals. The evaluations described here are therefore of a programme that is student-directed (action-based) and where its design is using authentic technology-based contexts in giving students relevant learning.

The main objective of the programme is to educate business developers, where this includes business developers in established, young and self-started firms. Hence, students need skills to be capable to create value in various employment positions, affective capabilities making them interested in working with entrepreneurial endeavours (meaning business development in an entrepreneurial manner), but also the ability to and behaviour in terms of creating their own start-up. Therefore, evaluating the programme’s impact needs to include investigation of affective, skill-based, and start-up outcomes, but also to match this with the educational approach and contextual influences.

Rationale and Data Collection

The program has an admission process that consists of applying for enrolment through the university’s admissions office. The application includes a motivational letter, CV, and transcript of grades, and based on these documents, it is decided which students are invited for an interview with the programme’s faculty. After the academic staff has conducted the interview, the candidates are thoroughly assessed and discussed among the faculty, before a group of applicants is nominated for admission to the study program. In recent years, for the programme’s 35 places, there have been approximately 350 students who have applied, of which around 130 have been invited for an interview, and where most of the remaining have not submitted a letter of motivation.

In order to control for the admission process and motivation of the students, as well as compare against students from higher education in general, the evaluations have been designed to examine four groups of former students: (a) alumni who have completed entrepreneurship education (the program in focus), (b) alumni who applied for the program and participated in interviews, but were not offered a place, (c) alumni who applied but were not invited for an interview, and (d) alumni who have not applied for the program but have a master’s degree from the same university. The evaluations, therefore, include multiple groups that are related and have similar characteristics.

Because there are few students in the individual cohorts (on average 23 students in each cohort in the programme; from the class of 2013, the number of enrolled students increased by 50 percent), it will also be important to gain insight into a larger number of alumni from the program to do good analyses, in accordance with Kirkpatrick’s (1996) implementation guidelines. The evaluations therefore include analyses on alumni in the four groups over the years 2005 (the first graduating class from the programme) until the graduating cohort in 2019. The graduation year 2019 is chosen because 2020 is the last year with the most up-to-date data from the empirical sources and it is desirable to study cohorts that have at least one year’s work experience registered in the data. The sample sizes among the different groups are: (a) 321 alumni, (b) 310 alumni, (c) 1293 alumni, and (d) 38,406 alumni.

To get reliable information about the individuals and their activities, register data is collected from the university, Statistics Norway (national statistical institute responsible for official statistics), and the Norwegian Research Council. This includes data on income, employment, firm ownership, personal status, firm data, address, and demographic variables, as a few examples. More than one thousand different variables are available through Statistics Norway, giving opportunities to investigate multiple relations and developments. However, it should be noted that obtaining such data might require a lot of resources as the data might not be easily accessible. In this case, an immense application process is required, both at the university’s ethics committee and the official bureaus, and the process could potentially last years, depending on the project. This is due to the complexity and privacy regulations, which could hinder access entirely in other situations and contexts.

Meeting Affective Objectives

The first general objective of entrepreneurship education is to increase entrepreneurial spirit, culture, and attitude (Samwel Mwasalwiba, 2010). Through participation in education, the individual gets a sense of belonging in a network of like-minded people, and the individual builds relationships that can be active long after graduation and in relation to the educational institution (Hall, 2011). Having a degree from a given education can also help influence the attractiveness of the individual in the labour market (Hall, 2011). Although entrepreneurship education may result in the individual starting his or her own business in or after the education, the competence of the individual is also attractive to both newly established and established companies (Alsos et al., 2023). As such, alumni from a specific education could benefit from their insights and social network or the attractiveness of their educational background in their careers. Thus, contributing to the community through investments, being on the board of directors, or working in entrepreneurial firms might be a proxy for entrepreneurial spirit, culture, and attitude.

To investigate this objective both behaviourally and in terms of results, multiple viewpoints should be included in an evaluation. The register data can, for instance, be used to explore whether graduating from entrepreneurship education can affect the individual’s choice (opportunities) of jobs, as well as investigate whether the network established in such education affects where the individual works. Working in start-ups (not as a founder) or taking a board role in other alumni’s start-up companies would be clear behavioural activities relevant to the affective objective, which register data could give insights into.

Furthermore, by studying whether alumni from entrepreneurship education work in the same start-up (meaning alumni recruit alumni), contribute to other alumni through investments in the latter’s start-up companies, and whether the companies that are founded to a greater extent are co-located (by postal code), more result-oriented insights can be extracted from the register data. This data could show regional development over time if firms stemming from the education cluster in a specific region.

Meeting Skill-Based Objectives

For the skill-based objective, it has already been shown that students from entrepreneurship education can have various positions and use their skills in new and established firms (Alsos et al., 2023). In the work by Kucel et al. (2016), they also conclude that entrepreneurship education contributes to skills that make the students find a good job match. Their conclusion builds on the assumption that these students are more prepared for working in a changing environment, and further, that these skills also contribute to higher productivity in such environments. As such, the students with entrepreneurial education obtain better jobs (Kucel et al., 2016). However, how the former students that utilise these skills develop their careers is to a lesser extent explored, although the calls have been many (e.g., Pittaway & Cope, 2007). Moreover, having experience as an entrepreneur from the program could give, for instance, special management skills that could also be of value in an established firm (Alsos et al., 2023).

Investigating the behavioural effects on the skill objective is done by investigating the position of the individual and the employing firm, whether this is a newly founded firm or an established company, and if the position is at the management level or similar. The register data give insights into these details and show how the individuals are positioned in the working life, but also show how this evolves, giving indications of the skill base and -usage. This evaluation of the programme therefore shows insights into education’s influence on future entrepreneurial efforts in start-ups, indicating whether a more student-directed approach gives the students the skills and experience which could make them pursue other start-up initiatives later in their careers.

When investigating the results of the education on the skill-based level, the former students’ positions and firm characteristics, but also the salary level the former student, can provide indications of how alumni’s skills or profile are utilised among firms. As found in the work by Kucel et al. (2016), a better job-match could give better performance through the efficiency of the alum, which could be reflected through a higher salary or position. By investigating the register data, the value of the competence obtained from the education could therefore be illustrated. This insight could give an indication of the quality of the students from the programme, but also where the alumni work, showing the variation of the students’ potential jobs and where their competence is valued.

Meeting Start-up Objectives

The last objective of entrepreneurship education is start-up initiatives. As already mentioned, not all students start working in their own ventures upon graduation (Dahlstrand & Berggren, 2010), and the entrepreneurial career might start later in the individuals’ lives (Burton et al., 2016; Marshall & Gigliotti, 2020). While research has investigated the long-term behaviour of students in terms of entrepreneurial activities and business creation (Gielnik et al., 2017; Rauch & Hulsink, 2015), the time frame of these is rather short.

To investigate the behavioural impact of the programme on start-up activities, the starting of new ventures is of interest, and especially over time. For instance, if the students are serial entrepreneurs, meaning they start multiple sequential ventures over time, would be of interest. Also, where the ventures are located or the industry where they are initiated would be of interest, as this could give insights into how students’ educational backgrounds and contexts influence the venturing activity. Hence, by using register data, it is possible to gain insights into these numbers and show the behavioural impact of the education.

While the numbers of new ventures are easy to investigate, also the quality of the firms is of interest, but this could require other analytical approaches. When investigating the result level of the start-up objective, performance indicators on the individual’s firm could give valuable insight into the impact of the education, especially for firms founded by former students. Specifically, performance measured through the start-up’s growth could give result insights, for instance by using register data to explore capital invested in the individual’s start-ups, financial value creation from the firm’s statements (for growth investigation), and information about government grants and assets as a measure for how well these firms acquire such funds. As such, it is possible to use register data to explore whether the education gives the students a learning experience that has an impact on their entrepreneurial performance as alumni, this illustrating a result-level evaluation of the start-up objective.

Register Data and Methodological Issues of Entrepreneurship Education Evaluation

Of the three methodological issues identified in the previous sections, register data is a good source to handle all of these, given entrepreneurship education programmes like the one described above. The longitudinal data is key in solving the issue regarding the time-to-entrepreneurship problem and that entrepreneurship in start-ups could occur in phases of an individual’s career (Burton et al., 2016). By using the data collected from official records on the four groups in the case, the results are rigorous and will give insights into the individuals’ actual activities over time. Researchers can therefore investigate different activities and outcomes connected to the individual, giving rich insights into former students’ career paths.

Moreover, by also collecting information about relevant control groups, the issue regarding self-selection is solved, although this design is not a true experiment in terms of complete randomisation of treatment or not. The different groups included help in diminishing several methodological problems. First, students’ interest in entrepreneurship is controlled through two groups; the ones that have the most motivation and understand they need to submit a letter of motivation, and the ones that have some interest in entrepreneurship education and only apply. Second, those that submit their letter of motivation, but are not enrolled in the program, still have a high quality and are difficult to separate from those who are enrolled in the program. These two groups can therefore be regarded as equal for most of the groups’ members and can furthermore be used in an experimental design although there is no complete randomness in the admission process. However, creating full randomness in the application process might cause unintended and unwanted outcomes, in addition, to having the potential of being somewhat unethical. Third, the last group is also included as a control group, investigating whether having a master’s degree from a technical university is influencing the alumni more than the education in focus. As such, it is controlled for treatment and no treatment, and high interest, interest, and higher education, handling the self-selection and control group problem found in current research (Rauch & Hulsink, 2015).

Thus, in terms of methodology, the studies described here align well with Kirkpatrick’s (1996) evaluation guidelines regarding time to behaviour and results, in addition to the inclusion of control group(s).

Register Data and Objectives in Entrepreneurship Education Evaluation

In terms of objectives, the previous show that register data have the possibility to contribute to evaluating entrepreneurship education, however, strictly investigating one outcome measure might not give insights into the full impact of the education. Students could be influenced differently by the same offerings (Longva & Foss, 2018), and as such, evaluations should investigate several measures to understand entrepreneurship education’s impact. As an example, the objective of improving students’ affective characteristics could also influence the creation of jobs and start-ups, although indirectly through investments, board participation or other support activities, for instance through supporting their former education (Antal et al., 2014). Hence, looking at job creation alone might not give a thorough understanding of the impact of the education. At the same time, investigating affective changes might not give insights into start-up creation, and skill changes do not say anything about the quality of potential start-ups nor if these skills are utilised in entrepreneurial activities post-graduation.

Therefore, by designing evaluation studies with different objectives in mind, in addition to considering the context and educational design, the programme’s impact can be evaluated more rigorously. The programme used as an example in this paper aims at creating business developers with special experience within technology-based business development. Alumni from this programme might therefore work in established firms, or start-ups created by themselves or others, but also contribute to business development through participating in boards, with investment or as advisors for other entrepreneurs. Hence, solely looking at one of these outcomes will not give a full impression of the programme’s impact, and I, therefore, recommend investigating multiple objectives with multiple outcome measures. As such, the different objectives of entrepreneurship education need different evaluation forms in Kirkpatrick’s (1996) model’s levels, and to capture the potential outcomes at various levels, different ideas, approaches, and designs are needed.

Limitations of Register Data in Entrepreneurship Education Evaluation

Regarding limitations, when looking at Kirkpatrick’s (1996) implementation guidelines, a pre-evaluation of results is difficult, if not impossible, for some of the objectives in entrepreneurship education, as most individuals in this context have never been working or been entrepreneurial through starting new ventures. As such, expecting students to have high results prior to the education might be too ambitious, however, not out of the question. Especially the affective objectives could have a potential change in results, for instance in investments in start-ups (equity crowdfunding, as a possible example). Thus, there might be situations where students change their result indicators before and after an education, but for the skill-based and start-up objectives, assuming students are early to mid-twenties, this might be less likely. However, for situations where the students are more likely to have entrepreneurial activities prior to the education in focus, the register data could also be gathered and analysed prior to the educational efforts. The complexity of the analysis would increase in such situations, depending on the age of the students and their prior careers.

For the behavioural evaluation level in Kirkpatrick’s (1996) model, a study using register data will lack information from the individual in focus, its colleagues, partners, and similar. While the previous issue with pre-post result analysis could be handled and in some instances being irrelevant, the second issue is more problematic, at least when evaluating the individual, as is the case in this paper’s examples and in Kirkpatrick’s (1996) model. When the use of register data gives rigorous results and insights into the individual over time, a critique of the use of register data alone is that it might be limited in explaining beyond the empirical level. Without interviews or surveys with the individual, their colleagues, partners, and similar, it is difficult to understand the individual’s actions and what is causing the actions. To understand the outcomes’ connection to the education, the use of mixed-methods and follow-up studies is recommended. Building theoretical contributions from the empirical data might be difficult, for instance, on an alum’s employment in other alumni’s start-ups, but combining these data with qualitative investigations might help us in understanding the phenomenon. Using register data alone shows us the magnitude and difference between groups, indicating an impact on various outcome measures; understanding the magnitude and difference requires us to re-engage with our former students. Hence, if we would like to build theory when investigating entrepreneurship education at the individual level, register data might give us insights into the what and how; the why might need additional data (Whetten, 1989).