Enhancing law enforcement training: A gamified approach to detecting terrorism financing

Abstract

Tools for fighting cyber-criminal activities using new technologies are promoted and deployed every day. However, too often, they are unnecessarily complex and hard to use, requiring deep domain and technical knowledge. These characteristics often limit the engagement of law enforcement and end-users in these technologies, which despite their potential, remain misunderstood. For this reason, in this study, we describe our experience in combining learning and training methods and the potential benefits of gamification to enhance technology transfer and increase adult learning. In this case, participants are experienced practitioners in professions/industries that are exposed to terrorism financing (such as law enforcement officers, financial investigation officers, private investigators). We define training activities on different levels for increasing the exchange of information about new trends and criminal modus operandi among and within law enforcement agencies, intensifying cross-border cooperation and supporting efforts to combat and prevent terrorism-funding activities. A game (hackathon) is designed to address realistic challenges related to the darknet, crypto-assets, new payment systems and dark web marketplaces that could be used for terrorist activities. The entire methodology was evaluated using quizzes, contest results and engagement metrics. In particular, training events show that ∼60% of participants complete the 11-week training course, whereas the hackathon results, gathered in two pilot studies (Madrid and The Hague), show increasing expertise among participants (progression in the points achieved on average). At the same time, more than 70% of participants positively evaluate use of the gamification approach, and more than 85% consider the implemented use cases suitable for their investigations. These outcomes are further discussed to detect the introduced approach's benefits and limitations and improve future events.

Keywords

Gamifying training hackathon counter-terrorism training law enforcement agencies police financial intelligence units

Introduction

Tackling terrorist financing through investigation, prosecution and prevention is a worldwide issue that extends beyond Europe. Every day, terrorists find new channels through which to communicate, campaign and finance their activities. For example, as reported by Europol (2021) in the Internet Organised Crime Threat Assessment (IOCTA), two main trends are related to crowdfunding campaigns and generating market revenue. In the first case, the schema is pretty easy: terrorists start a crowdfunding campaign to gather funds for their activities. In the second case, they try to generate revenue by selling extremist versions of common products or merchandise (such as Nazi-related items, ISIS promotional materials), as well as other legal and illegal goods (counterfeit products, firearms, explosives, everyday items, etc.) to the general public or other extremists/terrorists. In both cases, to maintain anonymity, they often employ a combination of cryptocurrencies and markets in darknet technologies (Europol, 2022).

To tackle these needs and combat cybercrime, new paradigms, such as artificial intelligence (AI) and big data, are being used alongside conventional software to create novel investigation tools (Maher, 2017); for example, the use of emerging technologies such as ChatGPT in good practices and strategies to tackle cyberwar (Mijwil et al., 2023. However, these tools typically include multiple steps for collecting, processing, analysing and visualising information related to financial data (e.g. transactions, electronic invoices) and correlating the data with context data extracted from social media analysis, forums and phishing among others (Kilger and Choo, 2022). This pipeline allows for improvements in the quality and potential of the tools. However, it also enhances the tools’ complexity, requiring specialised domain expertise and technical profiles for their use. As a result, law enforcement officers (LEOs), financial intelligence officers and even private financial investigators may be deterred from using these technologies (Klingberg, 2022).

In that sense, in several European Union (EU) projects, partners have tried to address these problems and increase engagement and technology transfer to law enforcement agencies (LEAs) through different events such as training sessions. For example, in the GRACE (2020) project, LEAs were trained to use federated-learning AI tools to fight the propagation of child sexual exploitation and abuse (CSEA) materials. In the DANTE (2018) project, similar structured training activities were organised with the aim of enhancing analysis of terrorist-related content. In the i-LEAD (2023) project, specific training sessions in using project technologies for tackling cybercrime and performing forensics investigations were planned, whereas in CYCLOPES (2023), events for training and testing participants on specific digital forensics tools were organised. Joint live exercises were explored in the CTC (2023) project for providing both theoretical and practical knowledge about counter-terrorism financing. Field labs events were organised in the TITANIUM (2019) project with the aim of training LEAs on studying criminal trade flow through cryptocurrency analysis. Yet, these events were based on capture the flag (CtF) exercises, as for hackathon events deployed in the ASGARD (2016) project.

In this article, our research objective is to demonstrate that technical knowledge transference and student engagement can be achieved and evaluated in with the context of high stress and dynamic prioritisation typical in crime investigations. In particular, we propose the hypothesis that the research objective can be met by combining learning, training and gamification techniques in a unique methodology, and we present the deployment and results obtained in the Anti-FinTer (2023) project (AFT). The methodology combines traditional teaching techniques, like lectures, with a moderated virtual learning environment (VLE), workshops and exercises with gamification techniques to facilitate interaction and engagement between participants. The aim of the AFT project is to train LEAs and financial intelligence units (FIUs) to enhance their ability to use emergent technologies and complex pipelines to reveal the financing of terrorist activities. In this way, it will be possible to increase Europe's ability to use novel tools for investigating terrorist financing and also promote EU technical and strategical sovereignty. For this reason, the AFT project exploits four tools that have been developed in previous EU projects; namely, Graphsense for virtual assets analytics (Haslhofer et al., 2021), the Visual Analytics tool for forensic image processing, Ordainsare as a transaction anomaly detector based on the model presented in Zola et al. (2019) and and Dark Web Monitor (https://cflw.com/dwm/) for analysing the darknet content.

The methodology followed in the AFT project is based on two main pillars: learning and training, and gamification. Learning and training events are organised on four levels: asynchronous courses, knowledge hub meetings (KHM), train-the-trainers (TTT) and face-to-face (FTF) training, with different aims and target groups. By contrast, the gamification task is based on hackathon events, which are designed as CtF exercises that allow participants to learn effectively how to use AFT tools (alone and linked) in their day-to-day work to reveal terrorist financing activities.

All these activities and events have been designed as experiments following an iterative approach that allows temporal comparison and improvement validation, and defining the generation of evidence during the whole learning and training process (completion measures for the objective indicators and satisfaction for the subjective ones).

Restricted access for practitioners, determined by the number of events and participants in each, has led to application of the entire methodology to all. The availability of practitioners to participate has been the primary constraint we need to manage, resulting in participation disparities in each learning activity. These restrictions prevent a deeper ablation study that would allow us to compare the impact of each learning/training activity over the overall goal.

Although we present the whole methodology, the main analysis is performed on the outcomes achieved during the asynchronous courses, as well as the general results obtained in two AFT FTF training and hackathon events, the first held in Madrid in 2022, and the second held in The Hague in 2023. At each of these events, AFT stakeholders and external users were involved in two days of activities. The results indicate a satisfactory level of engagement among participants for both sessions, and attendees positively valued the general AFT contents and tools. However, they had doubts about their use in day-to-day investigations. This analysis also helped identify limitations and organisational weaknesses that will be addressed in preparation for the future (and last) AFT FTF training and hackathon event scheduled to be held in Vienna at the end of the project. The obtained results and findings can be easily transferred to learning and training related to other cybercrime investigation domains that share the same tools and similar methods, like CSEA, weapon/drug trafficking, money laundering, ransomware or political corruption.

Background

Andragogy and pedagogy training

In general, training participants are experienced practitioners in professions/industries that are exposed to terrorism financing, and all arrive with a preconceived idea of the topic. In line with the expected learning outcomes of the project, we must incorporate a central objective of understanding foundational and prevailing knowledge associated with terrorism financing and adjacent issues, while providing participants with the autonomy to engage with learning materials that they feel best suit their needs. This approach is critical with the target of educating adults (andragogy). The andragogical response to the training structure must leverage the learning opportunities provided by these activities while also appreciating the project participants’ diverse skill sets. In this sense, the main objectives of the training structure can be summarised as follows:

appreciate the requirements of the adult participants;

provide alternative routes of learning within the materials;

generate a learning environment rooted in experiential learning;

provide knowledge that can be perceived as immediately applicable or useful;

create a learning environment that can leverage the insights of training participants.

Applicable theories of learning

Given the limited time available to disseminate the learning materials, an instructor-led approach may not, on its own, be conducive to achieving satisfactory training outcomes. Terrorism financing and its associated issues are a complex and ever-evolving topic requiring intricate insight and knowledge across various subjects. An instructor-led approach for this environment would necessitate the use of behaviouristic learning materials, in which instructors condition learners to reach a preconceived standard of competency through reinforcement, repetition and variation (Carlile and Jordan, 2005). Achieving behaviouristic learning outcomes while adhering to the training structure’s objectives may be impractical.

It is important to leverage the prior knowledge and subject area of interest expressed by the participants and use this as the central focus of the teaching environment. This approach allows the production of deeper insights over the short engagement period. In this sense, it is important to diminish the instructor's role as the exclusive provider of knowledge and reallocate a degree of control and accountability to the learners. A number of learning theories are compatible with this type of learning environment although still adhering to the objectives. Nevertheless, certain elements of the humanist approach need to be incorporated in the training structure. Autonomy is provided to participants to learn independently and without the burden of stress typically associated with assessment structures. The intention behind this approach is to instil a deep sense of personal appreciation for the subject matter, such that participants desire further learning following the cessation of the training (Governors Western University, 2020).

There is a need to account for the limited contact hours that the instructors have with the participants. A lack of consistent contact hours can lead to a disjointed learning base for participants, limited engagement with the learning materials and suboptimal learning outcomes. Therefore, creating an empathetic and shared social environment is paramount to achieving the expected learning outcomes. The social constructivist learning environment leverages the diverse array of knowledge and experiences offered by each learner to develop social discussions within the learning environment. In addition to constructing their own view on the knowledge provided by the instruction materials, participants can reach a higher plane of understanding by simultaneously sharing with their peers their experiences, perspectives and concerns on the subject matter being addressed (Topping, 1998). Developing an interaction-based environment has been heavily linked with student learning and satisfaction. Furthermore, instructor and peer dialogue has been shown to develop trust and social interactions in learning settings (McLean, 2018).

Achieving practical learning

Teaching philosophies identified as suitable for the AFT training structure (humanism, social constructivism) can inform an appropriate learning structure to implement within the learning environment. A key aspect of this is identifying that training will take place both online and in a face-to-face environment. The andragogical aim of the face-to-face learning environment is to establish a community of inquiry, an andragogic framework in which learning occurs at the intersection of social, cognitive and teaching presence (Garrison and Arbaugh, 2007), and a community of practice in which knowledge is embedded in the activities, social relations and expertise of specific communities (O'Neill and McMahon, 2005). Pre-prepared learning materials (cognitive) are paired with synchronous discussions on the topics covered (social). The structure is designed to foster individual perspectives on issues related to terrorism financing (cognitive) and encourage interactivity and feedback among peers as a means of constructing new insights and perspectives (social). These are underpinned by specific prompts that are introduced into the learning environment to spark discussion among the participants. Moreover, identification of a sufficiently motivating problem that serves as a platform for investigation is important in inquiry-based learning environments (Finkel, 2000). In addition, practice-based learning is incorporated into the learning environment because all participants are stakeholders in investigating terrorism financing. Although inquiries provide valuable insights, introducing case study examples of terrorism-financing issues can lead to practical, ready-to-apply knowledge. Introducing case studies to explore terrorism financing is particularly effective in enhancing learning outcomes when real-world problems remain unresolved and ill-structured (Barrows, 2002). Once this problem is introduced, the subsequent investigation and discussion are where the learning takes place. Participants are highly intrinsically motivated to learn what is necessary to solve this problem (Auman, 2011), given that counteracting terrorism financing and discussing the issues of the prevention of terrorism financing is a common interest shared by all participants.

Gamification

CtF is a type of information security contest in which participants are challenged to solve a range of tasks to obtain a designated item called a flag. At a high level, there are many flavours of cybersecurity competition, as well as platforms for managing them. In this sense, it is also difficult to define a finite set of CtF strategies. Several studies ENISA (2021) and Švábenskyỳ et al. (2021) define two main CtF strategies: attack/defend and jeopardy-style. In particular, the lack of attack/defend team specialisation, the presence of multiple users and especially the structured learning goals of the AFT project led us to choose the jeopardy-style format to create a competitive environment and engage multiple users simultaneously.

The CtF strategy has resulted in wide success in terms of introducing and learning cybersecurity-related concepts (Švábenskỳ et al., 2021), but also motivating continued learning after the exercise (McDaniel et al., 2016). For example, in Huang (2011), CtF is used for solving as a differential game, whereas in Eagle and Clark (2004), this strategy is used to educate students to act as crackers and find new vulnerabilities in existing systems (data, files, devices, etc.). Another interesting work is presented in Chicone and Ferebee (2020), where authors compare two CtF platforms (Facebook CtF and CtFd) for cybersecurity learning. In Hanafi et al. (2021), the authors study how gamification can be applied to introduce students in cybersecurity education, who are, in fact, non-technical backgrounded targets. As already mentioned, this gamification approach is also used in many EU projects for improving knowledge and technology transfer.

Inspired by these previous studies, here we combine learning and gamification strategies for facilitating knowledge sharing and best practices exchange among AFT stakeholders and end-users. This approach has shown to be helpful in improving law enforcement capacity and developing expertise in using emerging AI tools for terrorist financing investigations.

Methodology

Designing curricula and training programmes for crypto-asset analysis technologies is challenging because of their intricate and rapidly evolving nature. In this sense, in the ATF project, we combine both asynchronous and synchronous learning to maximise the impact of the activities and reach a broader audience (Figure 1). More specifically, we started designing an asynchronous course deployed in a learning management system (LMS) with the aim of standardizing attendees’ knowledge related to the general topics treated in the project, such as crypto-finance, financial regulation, dark web structure, crypto-ecosystem and machine learning for investigation. By contrast, three different activities are taken into account for synchronous learning, such as KHM, TTT and FTF training, as shown in Figure 1. Finally, the AFT training methodology also includes the gamification (a hackathon) of domain tasks for practising the acquired knowledge. Although we describe all the activities in the following sections, this study mainly focuses on presenting the results obtained from three of them: asynchronous course, TFT and hackathon.

Figure 1.

Learning scheme followed in the project.

Asynchronous course

Given the profile of the AFT participants, embedding flexibility was at the forefront of the programme design and consideration was taken in leveraging key dimensions of learning and teaching. Several principles and established pedagogical approaches are applied, appropriate both to the discipline and the level of the award. An asynchronous, active online learning approach underlies the conception of teaching and provides a strategic filter through which the course is shaped and supported. AFT end-users are typically experienced individuals in full-time employment and are viewed as not just participants, but also active learners and critical thinkers with their own emerging theories about the world. This view also supports the principal tenets of adult learning in that they desire and enact a tendency towards self-directedness, practical learning (tools usage), experiential learning and a flexible learning environment. This relative informality is not without structure, and clear expectations are set regarding performance and rules of engagement as this learning journey is one embarked on together.

In fact, participants’ own experiences are an essential resource for learning. In particular, given their rich and diverse backgrounds, participants bring unique perspectives and insights to the course, creating fertile ground. Thus, considering their experiences as valuable input, the AFT course design and delivery promote a process of interaction between what is known and what is to be learned, because a reflection on experiences and understanding is the foundation upon which something new is built. By seeking participants’ points of view and using this to establish their current conceptions, the objective is knowledge construction (and sometimes reconstruction) rather than knowledge reproduction.

The asynchronous AFT course, hosted on the LMS, comprises 12 lectures organised into two parts over a 6-week period. Part one (lectures 1 to 6) focuses on understanding cryptocurrencies, their operations and their impact on regulatory, law enforcement and tax authorities. Conversely, part two (lectures 7 to 12) delves into the technical concepts utilised by AFT tools like machine learning, image analysis, follow-the-money approach and the darknet, alongside tool presentations. Lecture 12 is a reflective session, gathering feedback and discussing tailored learning paths for different organisational needs.

Prerequisites for effective teaching in this regard are interaction, dialogue and reflection. For this reason, a considered level of meaningful interactivity is designed into the AFT course to encourage participants to critically engage and enhance the process of cognitive development that results in more significant evidence of the achievement of learning outcomes, because this allows for deeper engagement with concepts as multiple perspectives challenge existing assumptions. This interactivity has been provided using the discussion forums added to each subject and with programmed live sessions to directly ask the subject coach. The research method used to evaluate knowledge acquisition is the results of short quizzes related to each subject of the course and the final completion of all of them as the main indicator.

Knowledge hub meeting

KHM is a pan-European and multidisciplinary knowledge hub in which end-users, LEOs, tax authorities, financial services institutions, and also academia, the private sector and any interested stakeholder, can participate to discuss best practices and policy recommendations about new crypto-threats, investigation trends and novel detected modus operandi. This knowledge-sharing approach is implemented through a series of at least 20 workshops (physical and online) over a period of 2 years, which increases the attendees’ expertise and helps the technical partners understand the needs of users and stakeholders, allowing them to improve their tools. Therefore, the KHM aims to create a community (long-term goal) supported by the European Anti-Cybercrime Technology Development Association (EACTDA) to keep up with developments beyond the lifetime of the project. This activity has been designed as an open discussion and network creation among practitioners, and the engagement has been demonstrated with the attendance at each event.

Train-the-trainers

A TTT event is online training that involves developing the skills and knowledge of individuals who are responsible for training others. This activity initially focused on technical trainers, but it is now open to end-user trainers. More specifically, TTT is carried out with two main goals. On the one hand, it presents how the AFT tools can enrich the traditional investigation methods in a complementary way. On the other hand, it shows participants how to use appropriate teaching and good practice for coaching colleagues first and then other organisation members. Participants’ existing knowledge and their interest in the subject area become crucial, serving as the pivotal point around which the teaching environment revolves. This approach aims to enhance the quality of the insights within the limited duration of engagement. In that sense, we removed the instructor as the sole purveyor of knowledge and transferred a degree of control and responsibility into the hands of the learners. The instructor's primary focus in this approach is to facilitate learning and promote participant engagement rather than to act as the sole source of knowledge. This shift from instructor-centred to learner-centred teaching requires an environment that encourages autonomy in constructing knowledge, active participation and knowledge sharing among participants. The research method used to evaluate the improvement in knowledge transference is the results of the survey related to trainer evaluation completed after each FTF training and hackathon event.

Face-to-face training

FTF training events are structured to provide a comprehensive explanation and practical demonstration of AFT tool functionalities while ensuring participants are not overwhelmed by an excessive influx of new concepts. In this way, AFT end-users and external stakeholders who take part in the hackathon event are trained for the next day when they effectively use these tools (alone) to address realistic challenges.

Both the Madrid and The Hague FTF events adhered to an identical schedule: six 30-min sessions, totalling 3 h. The sessions covered introductory domain concepts on crypto governance and regulations, followed by presentation of the four AFT project tools. Each tool was showcased through a 20-min demonstration and a 10-min question and answer session. The final segment introduced an integrated platform, complying with AFT project specifications, enabling swift access/switching between tools.

The choice of these temporal slots allows us to ensure enough time to present each tool and demonstrate it, but also avoid overloading the participants with too many new concepts. The research method used to evaluate knowledge acquisition was included in the survey provided after each FTF training and hackathon event.

Hackathon

AFT hackathon events are based on the gamification of investigations related to terrorism-financing activities, and are oriented towards FTF-trainer users. This is performed using a CtF structure and deploying different domain challenges following the jeopardy-style format in which participants should provide a response in the shape of a ‘tag’ for each question presented and where the difficulty increases step by step. In particular, to accomplish the AFT project goal, the challenges are aligned with possible cybercrimes such as financing terrorism, money laundering and fraud detection that involve dark web, crypto-assets, new payment systems and darknet marketplaces. In this sense, AFT technical partners are responsible for designing realistic challenges based on the modus operandi analysed during the project. However, they also need to consider that hackathon participants are not familiar with these new AFT tools, so challenges need to be created accordingly. For this reason, throughout the AFT project, different strategies are drawn and deployed to create the challenges and to guide the users through the tools, gradually diminishing the constraints and tool-related hints for specific tasks. More specifically, we started the project with a more structured and guided approach (Madrid hackathon), then proceeded to lessen the constraints and limitations of the tools (The Hague hackathon) to finally enable participants to attain full independence in choosing the appropriate tools for the right tasks (final event).

Regarding the structure of the hackathon, each event is separated into sessions, during each of which participants are required to tackle progressively more challenging tasks, beginning with an exploration of basic functionalities and operations, and culminating in demonstrating the true capabilities of AFT tools and their practical relevance in real investigations and complex scenarios. This approach thereby promotes the involvement of AFT users and external stakeholders, creating a more competitive playground, and allowing AFT technical partners to speed up development/validation cycles with respect to traditional innovation processes.

To improve the learning process, each hackathon event follows a different strategy in the challenge definition. In particular, the first hackathon adopted a tool-centric strategy, whereas the second implemented a challenge-oriented strategy. Consequently, these strategies influence the duration of each session and of the event as a whole.

The tool-centric approach (Madrid hackathon) included separate sessions for testing each AFT tool with easy challenges to aid user familiarity, guide actions and assess usability. Tasks were self-explanatory, ensuring clarity and minimising errors. Over 5 h, distributed in 60-min blocks, participants progressed from the basics to more complex challenges requiring deep expertise.

By contrast, the challenge-oriented strategy (The Hague hackathon) began with a tool-familiarisation session for both new and returning participants, as planned in the Madrid event. However, this time, instead of being split by tool, challenges were consolidated to accelerate learning. Subsequent sessions targeted a real-world scenario: addressing terrorism financing via the Luckp47 darknet market, a niche darknet market that allows the purchase of various illegal items using Bitcoin and is purportedly affiliated with a paramilitary group (Jiang et al., 2021). The challenge-oriented event ended with a session in which participants were asked to use their acquired knowledge to address more complex scenarios, which could also involve the use of more than one tool at a time (similar to the last session in the Madrid event). In The Hague event, each session required considerably more time to complete, and as a consequence of this approach, the event extended beyond 5 h.

Both events ended with a timeslot to award the participants with the highest score and collect feedback.

The research method used to evaluate the knowledge acquisition has been gathered in the Facebook Capture the Flag (FBCtF) tool using the challenge-solving metrics to analyse the completion, the fails and the speed of challenge-solving. The number of completed challenges is also a measure of engagement with the content and tools provided. The survey provided after each FTF training and hackathon event allows us to evaluate the supporting activities and improve them in the next hackathon.

Study methodology

Each learning and training activity has some type of evaluation metric to gather, enabling the generation of more robust engagement and validation of knowledge acquisition. These metrics are used as indicators and reflected formally in AFT deliverables. The results are shared between the technical partners of the project, allowing them to improve the learning and training material available for the next synchronous event (Madrid, The Hague or even the final one in Vienna). Although the number of participants in the synchronous learning events is variable, and their workload changed over time in the asynchronous events, normalisation of the gathered results and trend analysis allow us to validate improvements in knowledge acquisition and engagement.

Furthermore, it should be noted that for the guiding principle for both the design of tools for law enforcement and the protection of research subjects, the project was aligned with the ALLEA framework (https://allea.org/horizon-europe/). This European research integrity framework allows the creation of a strategic response to professional, legal and ethical responsibilities, and acknowledges the importance of the institutional settings in which research is organised. For example, for gathering feedback at each event, participants were informed about the anonymity of the survey and the voluntary involvement.

Asynchronous course validation

Validation context

All lectures are fully online, asynchronous and moderated by the course leaders. Discussion forums, with prompting questions, are embedded in the design to encourage collaboration with peers and course leaders, and to create a community of practice, drawing on the different backgrounds and experiences of the participants. Each week, course leaders hosted an online synchronous session to address specific questions and to further engage with participants. In this sense, participant engagement with assessment activities is a requirement of the course to ensure that learning outcomes are met. A reflective space is provided, allowing participants to evaluate their own learning experience and critically reflect on the content each week. Learning can be complex, with each individual adopting their own strategy to navigate and engage with the content. For this reason, it is important to provide various motivational aspects and a multimodal approach to deliver support to each participant in their learning journey.

Each lesson, excluding the last, requires the completion of components (e.g. quizzes) to demonstrate a participant’s achievements and engagement level, but also to track lesson features and evaluate the learning outcomes. On achievement of proficiency levels, the next course becomes available to the participant. Completion reporting is also used to track activity for each course requirement in the programme and award a certificate to successful participants.

Results and lessons learned

Figure 2 reports the results in terms of the academic performance of the group and their progress through the various elements of the VLE. The figure shows strong and consistent engagement throughout the course, although there was some attrition over the 6-week duration. This was gradual, and no one element of the course caused a precipitative fall in participant numbers. Of the 81 participants at the start of the course, 52 from across the EU completed all 11 units. Furthermore, excluding lecture 1, the number of participants who passed the quizzes is very similar to the number of course participants, i.e. almost all the participants were able to complete the lessons successfully.

Figure 2.

Overview of completion of required components (quizzes).

A completion rate of ∼64% (52 of 81) for an online offering aimed at busy professionals can be viewed as a success. Feedback was positive, and engagement on the discussion boards was reasonably strong. In fact, the discussion boards allowed the creation of a community of practice among participants and helped participants to have a direct chat with the teachers/moderators. Furthermore, positive feedback was also obtained regarding the structure of the course, its context and technical content.

FTF training validation

Validation context

FTF events are targeted towards hackathon participants, who are mainly AFT end-users and external stakeholders from LEAs and FIUs. For this reason, to evaluate the events, a unique survey with ten questions is prepared. The first five questions are related to the satisfaction grade of the training materials, lessons learned and the understandability of guides and platforms. These questions have responses on a five-point Likert scale ranging from ‘strongly disagree’ (1) to ‘strongly agree’ (5). The final five are open questions related to the most engaged and distanced training moments, and which training actions participants consider most helpful or puzzling.

For the sake of simplicity and the aim of this study, only the first five questions are reported and discussed. Indeed, the results from the open-ended questions hold significance only within the context of the project, rather than being pertinent to the presentation of the methodology's advantages and limitations.

Results and lessons learned

In the Madrid event, 15 of the 17 participants filled out the survey, whereas in The Hague, the number was 13 of 15 participants. This number of participants can be considered limited to provide an extensive validation of the methodology. However, our idea is to follow and analyse the participants’ trends during the different events to check their learning progress.

Participants showed very broad opinions about how the knowledge gained in the training event is helpful for their immediate (short-term) investigations (Figure 3A). In fact, ∼50% of participants in both events ‘strongly’ or ‘simply’ agree on that, whereas ∼26% (Madrid) and ∼15% (The Hague) have a different vision and disagree. The remaining participants do not have a clear opinion. Different results are instead obtained by looking at Figure 3(B), i.e. analysing how the training information would be beneficial to their day-to-day work in the long term. In this case, in the Madrid event, just 66.7% of the participants ‘strongly’ or ‘simply’ agreed with this claim, whereas in The Hague, all participants expressed certainty in that regard. A similar trend is also found in relation to how participants perceive the engagement and obtained benefit of other players (Figure 3C). Finally, further interesting feedback is obtained with respect to the quality of the learning materials (guides, platforms, leaflets, etc.). Figure 3(D) shows that in the Madrid event, although a significant portion of participants (66.7%) endorsed the thoroughness and comprehensibility of the materials, 20% expressed dissent. However, in the second event, only 7.7% of participants assessed the material negatively, whereas more than 80% considered it valuable. The results presented clearly indicate that, on the whole, the AFT FTF training events receive a positive assessment from end-users and stakeholders. These users also express a sense of engagement in acquiring domain-specific and technical knowledge relevant to their daily responsibilities. Their recommendations were fundamental to detecting limitations and situations to be refined. In this sense, the iterative approach followed in this project contributes to a higher level of satisfaction, as evidenced by the enhanced outcomes witnessed in the second event.

Figure 3.

Training results gathered in Madrid and The Hague events.

Hackathon validation

Validation context

Feedback is gathered just after the awards ceremony, using surveys, and has its own timeslot, looking for a high engagement. First, participants are collectively informed about the anonymity of the survey and their voluntary involvement. The survey is then delivered covering questions about the events, the organisation and the tools. More specifically, opinions from AFT end-users and external stakeholders as well as from AFT technical partners were gathered.

AFT technical partners

A specific questionnaire is prepared for AFT technical partners, consisting of two main sections: exercise preparation and performance evaluation. The former includes ten questions related to the materials, location, facilities, complaints and problems in the hackathon organisation. The latter contains 15 questions separated into four different topics: tool installation (eight questions), configuration/default parameters (two questions), integration with other project tools (three questions) and data evolution (two questions).

AFT users and external stakeholders

Two evaluation frameworks are used to collect this feedback: one based on the common state-of-the-art learning schemes and the other based on objective metrics. In the learning schemes case, a survey with 35 questions is designed. This questionnaire is created following two distinct learning schemes: the System Usability Scale (SUS) (Peres et al., 2013) and the Kirkpatrick Model (Smidt et al., 2009). The SUS is composed of ten standardised questions used to assess the usability of a wide range of systems. These questions are rated on a five-point scale from ‘strongly disagree’ (1) to ‘strongly agree’ (5). The survey is improved with 25 (five-point Likert scale and open) questions based on the Kirkpatrick Model. This model comprises four criteria levels: reaction, learning, behaviour and results (Bates, 2004). The first level aims to gather information on whether the participants perceive the task as relevant to their daily job. The second assesses the knowledge acquired by participants during the hackathon. The third focuses on evaluating the change in participants’ attitudes and behaviours resulting from the newly acquired knowledge, which is a critical step because it can be challenging to validate effectively. Finally, the fourth level directly evaluates the performance results obtained during development of the hackathons. In the objective metrics case, regarding objective metrics, data and logs are directly extracted from the FBCtF platform and analysed to evaluate the trends and statistics of each participant as well as their engagement level. This analysis gives us an overview of the difficulties and problems encountered by each participant and, at the same time, helps us to determine the winners of the hackathon event. Yet, the results allow us to fill the gap between the trainees’ perceptions of their learning process and their actual performance.

It is important to note that although objective metrics have relevance within the scope of the project, they do not directly influence presentation of the methodology's strengths and weaknesses. Therefore, in this work, we report only the number of participants who have correctly addressed each challenge and their final score to show their engagement level in the events. By contrast, considering the extensive number of questions (35) in the survey only those related to evaluation of the general methodology are further analysed and discussed.

Results and lessons learned

During the Madrid event, the survey was completed by 14 of 16 participants, whereas in The Hague event, it was filled out by 11 of 15 attendees. As pointed out previously, because of the limited number of participants, the main idea of this work is to monitor participants’ progress across various events to track their learning trajectory and the benefits and limitations of the proposed methodology.

Figure 4 reports the distribution of participants’ expertise in both events. The figure shows that, in both events, the number of officers who work directly on real investigations (operational level) is far greater than the number of strategic officers. In fact, in both cases, operational officers represented more than 75% of the attendees, although, in the Madrid event, there were more FIU agents than LEAs. These outcomes align with expectations, because the objective of the AFT is to showcase the effectiveness of innovative tools for detecting terrorism financing primarily intended for use by operational officers.

Figure 4.

Anti-FinTer users and external stakeholders’ distribution in Madrid (left) and The Hague (right) event.

According to information gathered in The Hague event (second hackathon), 7 of the 11 participants who completed the survey (63.6%) were also present at the Madrid event (first hackathon). As a result, three comparative questions were directed exclusively towards this subgroup. In particular, all of them positively evaluate the assessment and the improvements made in this second event, especially the new format of the hackathon based on a challenge-oriented strategy, the new functionalities introduced in the tools, and the design of the challenges related to a real investigation (Figure 5).

Figure 5.

Comparative questions asked to users who participated in Madrid and The Hague events.

Figure 6(A,B) shows that all the participants felt confident with the technical and domain information acquired during the second event. In the Madrid hackathon, two participants considered the acquired domain knowledge insufficient, whereas one participant expressed the same opinion regarding the technical aspects. At the same time, four (domain) and five (technical) participants did not express a clear opinion (neutral). By contrast, in the second event, just one participant was neutral about the tool knowledge, and all the others appreciated the gained information in both aspects. Regarding the usefulness of the concepts learned during the hackathon, participants still had issues with understanding how they can be integrated into their daily duties (concrete use cases). In that sense, Figure 6(C) shows that ∼47% and 64% of the participants – in the Madrid and The Hague events, respectively – did not consider that the lessons learned were relevant or were neutral. By contrast, if participants were asked about their perception of using the knowledge in real investigations – from a more general point of view and not only related to specific day-to-day use cases – they had a different opinion (Figure 6D). In fact, especially in the second event, more than 80% of participants perceived that the lessons learned could be applied effectively in practical investigations.

Figure 6.

Results gathered during the Madrid and The Hague hackathons.

In terms of the participants’ engagement level, evaluated using objective metrics, the Madrid event achieved relatively homogeneous results. Figure 7 (left) shows that four participants (25%) achieved extraordinary scores of >2,000 points (of the 2850 available; red dotted line), and the other eight participants (of 12 in total, 75%) achieved >1,425 points (black dotted line). Two further participants reached at least 1,000 points, whereas only two were below this threshold. For almost all the participants, session 2 was the one in which they earned more points; this was expected because session 2 had a longer duration and involved two AFT tools. A different trend is observed in The Hague event (Figure 7 right), where despite good feedback and comments gathered in the survey, only one player completed all the tasks and nearly attained the highest possible score. Six additional participants (<50% of attendees) were able to reach at least half of the maximum score (black dotted line).

Figure 7.

Participant score in Madrid (left) and The Hague (right) hackathon events.

Conclusion

This article describes a framework for training LEAs and FIUs to enhance their ability to use emergent technologies and complex pipelines to reveal terrorism-financing activities. The methodology combines learning and training with gamification activities. Learning events are organised to facilitate the exchange of information related to new criminal payment systems and modus operandi among different stakeholders, whereas training events are designed as CtF exercises, allowing participants to learn effectively how to use AFT tools in their day-to-day work. The goal is to enhance the number of engaged participants coming from different fields and with different expertise.

The methodology was evaluated at two events held in Madrid (2022) and The Hague (2023). The asynchronous course, FTF training and hackathons validate knowledge transfer through quizzes, contest results and engagement metrics like quiz completion (>60%, gradually declining in the 11-week course). There was an observed progression between events, with an average of 1,500 points in Madrid and 2,000 points in The Hague. The presented overview of these results shows a satisfactory level of engagement among the participants in both training and the hackathon, as can be seen in Figure 5, where 71.4% of the participants agreed that the hackathon format was interesting, and 85.7% thought that the raised Use Cases were useful. Attendees expressed a favourable view of the general AFT content and tools, and recognised the potential of the acquired knowledge (domain and technical) in their investigations. The metrics in Figure 6(A,B) are proof of that, where data in the histogram show that participants felt confident. However, participants highlighted hesitation in using AFT tools in their day-to-day duties, as shown in Figure 6(C). For this reason, as a lesson learned, AFT technical partners need to spend more time understanding the day-to-day needs of the agents to enhance the challenges for the following events. At the same time, AFT partners should keep working on trying to provide new ideas to foster cooperation and share knowledge and experiences on hot AFT topics like new crypto-threats, financing trends and terrorism modus operandi. In this way, it is possible to create a more homogeneous community from which LEAs and FIUs can take inspiration for detecting and defining new investigation policies and share experiences about new market products that are useful for their investigations. The final future target would be to improve the number of LEA and FIU participants. In that sense, online participation can represent an interesting and valid solution by exploiting the pilot test in The Hague events.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work was partially funded by the European Union's Internal Security Fund—Police as a part of the Anti—FinTer project (grant agreement No. 101036262).

ORCID iD

Francesco Zola

Author biographies

Francesco Zola has a master's degree in computer engineering and a PhD in industrial science and technologies. His work is mainly on applying machine learning algorithms for anomaly detection and classification in graphs in cybersecurity environments. Francesco joined Vicomtech as a researcher in the Digital Security department. In 2021, he was first a visiting researcher at University College London and then at Netherlands Forensic Institute.

Lander Segurola studied mathematics at the University of the Basque Country (UPV) and subsequently completed a master's degree in mathematics and applications at the Autonomous University of Madrid (UAM). Since 2019, he has been a research assistant at Vicomtech in the Digital Security and Cybersecurity department, developing projects focused on anomaly detection using machine learning technologies. At the end of 2021, Lander started his PhD in collaboration with the UPV, focused on anomaly detection and machine learning applied to cybersecurity.

Erin King works at the Kemmy Business School, University of Limerick, in the Department of Accounting and Finance. She is an experienced Accounting and Finance department instructor with a demonstrated history of working in the higher education and digital education industry. Erin is skilled in negotiation, analytical skills, management and teamwork. She is a strong finance and education professional with an MSc in financial services focused on finance, risk, compliance regulation and portfolio management.

Martin Mullins is a professor in risk and insurance at the Kemmy Business School, University of Limerick. He has worked mainly on insurance-related research projects, including four EU Commission-funded projects around emerging technologies and risk transfer. Professor Mullins maintains strong links with the international insurance industry and works closely with Lloyd's of London and XL Catlin on emerging risks. He has been appointed to a European expert group to advise European Insurance and Occupational Pensions Authority on the development of digital responsibility principles in insurance.

Raul Orduna graduated in computer engineering and obtained a PhD in computer science and artificial intelligence at the Public University of Navarra. He is currently the Director of Digital Security at Vicomtech and leads projects related to ethical hacking, forensic analysis, access control and cryptography. He has participated in many successful funded projects, and he is the author of more than ten published original articles in international journals. He is also a reviewer of fuzzy sets and systems.

References

Anti-FinTer (2023) Versatile artificial intelligence investigative technologies for revealing online cross-border financing activities of terrorism. Available at: https://anti-finter.eu/ (accessed 4 March 2024).

ASGARD (2016) Analysis System for Gathered Raw Data. Available at: https://www.asgard-project.eu/ (accessed 4 March 2024).

Auman

(2011) Using simulation games to increase student and instructor engagement. College Teaching 59: 154–161.

Barrows

(2002) Is it truly possible to have such a thing as dPBL? Distance Education 23: 119–122.

Bates

(2004) A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning 27: 341–347.

Carlile

Jordan

(2005) It works in practice but will it work in theory? The theoretical underpinnings of pedagogy. Emerging Issues in the Practice of University Learning and Teaching 1: 11–26.

Chicone

Ferebee

(2020) A comparison study of two cybersecurity learning systems: Facebook’s open-source capture the flag and CTFd. Issues in Information Systems 21: 202–212.

CTC (2023) Cut The Cord. s.l., s.n. Available at: https://ctc-project.eu/ (accessed 4 March 2024).

CYCLOPES (2023) Cybercrime Law Enforcement Practitioners Networks. Available at: https://www.cyclopes-project.eu/ (accessed 4 March 2024).

10.

DANTE (2018) Detecting and Analysing Terrorist-Related Online Contents and Financing Activities. Available at: https://www.h2020-dante.eu/ (accessed 4 March 2024).

11.

Eagle

Clark

(2004) Capture-the-flag: Learning computer security under fire . Proceedings from the Sixth Workshop on Education in Computer Security (WECS6). Calhoun.

12.

ENISA (2021) All you need to know about Capture the Flag competitions. Available at: https://www.enisa.europa.eu/news/enisa-news/capture-the-flag-competitions-all-you-ever-wanted-to-know (accessed 4 March 2024).

13.

Europol (2021) Internet Organised Crime Threat Assessment (IOCTA). Available at: https://www.europol.europa.eu/publications-events/main-reports/internet-organised-crime-threat-assessment-iocta-2021 (accessed 4 March 2024).

14.

Europol (2022) European Union Terrorism Situation and Trend Report, Publications Office of the European Union . Available at: https://www.europol.europa.eu/publication-events/main-reports/european-union-terrorism-situation-and-trend-report-2022-te-sat (accessed 4 March 2024).

15.

Finkel

(2000) Teaching with your mouth shut. Heinemann.

16.

Garrison

Arbaugh

(2007) Researching the community of inquiry framework: Review, issues, and future directions. The Internet and Higher Education 10: 157–172.

17.

Governors Western University (2020) What is humanistic learning theory in education? . Available at: https://www.wgu.edu/blog/what-humanistic-learning-theory-education2007.html (accessed 4 March 2024).

18.

GRACE (2020) Global Response Against Child Exploitation. Available at: https://www.grace-fct.eu/ (accessed 4 March 2024).

19.

Hanafi

AHA

Rokman

Ibrahim

, et al. (2021) A CTF-based approach in cyber security education for secondary school students. Electronic Journal of Computer Science and Information Technology 7(1).

20.

Haslhofer

Stütz

Romiti

, et al. (2021) GraphSense: A General-Purpose Cryptoasset Analytics Platform. arXiv preprint arXiv:2102.13613:2021 Feb 26 .

21.

Huang

Ding

Zhang , et al. (2011) A differential game approach to planning in adversarial scenarios: A case study on capture-the-flag. 2011 IEEE International Conference on Robotics and Automation, pp.1451–1456. IEEE.

22.

i-LEAD (2023) innovation – Law Enforcement Agency’s Dialogue . Available at: https://cordis.europa.eu/project/id/740685 (accessed 4 March 2024).

23.

Jiang

Foye

Broadhurst

, et al. (2021) Illicit firearms and other weapons on darknet markets. Trends and Issues in Crime and Criminal Justice 622: 1–20.

24.

Kilger

Choo

K-KR

(2022) Do Dark Web and Cryptocurrencies Empower Cybercriminals? s.l., s.n. International Conference on Digital Forensics and Cyber Crime, pp.277. Springer International Publishing.

25.

Klingberg

(2022) Countering terrorism: digital policing of open source intelligence and social media using artificial intelligence. In: Artificial Intelligence and National Security. Springer International Publishing, pp.101–111.

26.

Maher

(2017) Can artificial intelligence help in the war on cybercrime? Computer Fraud & Security 2017: 7–9.

27.

McDaniel

Talvi

Hay

(2016) Capture the flag as cyber security introduction. 2016 49th hawaii international conference on system sciences (hicss), pp.5479–5486. IEEE.

28.

McLean

(2018) This is the way to teach: insights from academics and students about assessment that supports learning. Assessment & Evaluation in Higher Education 43: 1228–1240.

29.

Mijwil

Aljanabi

and ChatGPT (2023) Towards artificial intelligence-based cybersecurity: the practices and ChatGPT generated ways to combat cybercrime. Iraqi Journal For Computer Science and Mathematics 4: 65–70.

30.

O’Neill

McMahon

(2005) Student-centred learning: What does it mean for students and lecturers . Available at: https://eprints.teachingandlearning.ie/id/eprint/3345/1/O'Neill%20and%20McMahon%202005.pdf (accessed 4 March 2024).

31.

Peres

Pham

Phillips

(2013) Validation of the system usability scale (SUS) SUS in the wild. Proceedings of the human factors and ergonomics society annual meeting, pp.192–196. Sage Publications.

32.

Smidt

Balandin

Sigafoos

, et al. (2009) The Kirkpatrick model: A useful tool for evaluating training outcomes. Journal of Intellectual and Developmental Disability 34: 266–274.

33.

Švábenskỳ

Čeleda

Vykopal

, et al. (2021) Cybersecurity knowledge and skills taught in capture the flag challenges. Computers & Security 102: 102154.

34.

TITANIUM (2019) Tools for the Investigation of Transactions in Underground Markets. https://titanium-project.eu/ [Accessed: 04/03/2024].

35.

Topping

(1998) Peer assessment between students in colleges and universities. Review of Educational Research 68: 249–276.

36.

Zola

Eguimendia

Bruse

, et al. (2019) Cascading machine learning to attack bitcoin anonymity. 2019 IEEE International Conference on Blockchain (Blockchain), pp.10–17. IEEE.