Fuzzy linguistic descriptions for execution trace comprehension and their application in an introductory course in artificial intelligence

1.1 Computer games for motivating students

Keeping students motivated is an important goal for undergraduate students [1]. In particular, a challenge in teaching introductory artificial intelligence (AI) is to provide students with significant experiences and frequent feedback [2]. In this sense, computer games-based learning methodologies are being employed to provide students with motivating experiences.

Computer games have a long history of motivating learning computer science. In fact, integrating games into the computer science curriculum has been gaining acceptance in recent years, particularly when they are used to improve student engagement in introductory courses. It is important to note that computer games are not used for teaching in the classic way (serious games), but students design and implement their own computer games and artificial intelligence agents.

Feedback and motivation are also indispensable components of an effective teaching and learning environment in education [3–6]. Additionally, personalisation offers possibilities to deliver feedback that is the most appropriate for the user’s expertise, cognitive abilities and that addresses their current moods and attentiveness [7]. However, providing students with personalised, immediate and motivating feedback is a complex task, and it is usually a standardised process (every student receives the same feedback, e.g., knowledge of the correct response) due to the large number of students [7]. In writing skill, for example, truly immediate feedback is impractical [8].

Therefore, our idea is to provide students with two benefits: a learning methodology for artificial intelligence based on computer games and personalised feedback.

1.2 Challenges for incorporating games into an introductory AI subject

In 2011, a project-based learning methodology based on computer games was designed and implemented into the introductory artificial intelligence subject at the University of Bío-Bío.

The AI projects consist of developing a 2D computer game (see Fig. 1) based on the classic Pac Man computer game with some important differences. The player navigates through a maze containing four rewards and three opponents. The goal of the game is to have energy as long as possible by collecting the rewards and escaping from opponents. The three opponents roam the scene and chase the player. The player begins with 20 energy points. If the player comes into contact with an opponent, he loses 5 energy points. If the players earn a reward, he gets 10 extra energy points. The game ends when all the energy has been collected. The player loses 1 point of energy every five seconds.

OPEN IN VIEWER

Fig. 1

Example of a computer game developed by a student at the University of Bio-Bio. In the figure is shown a complete scenario (down) and a 2D scenario with the entities explained in Section 3 (top).

The player and opponents are computer game bots designed and implemented by the students using the heuristic algorithms studied during the course. The student must design and implement adequate heuristics metrics to design believable agents. An important project requirement is that the bots must be implemented by using heuristic algorithms, while opponent bots must be programmed by using the breadth-first search algorithm.

The traditional assessment of this kind of project has been performed by visually checking if the bots developed by the students have been correctly designed (revising the code developed for each student) and implemented (revising the codes functionality). This process has important flaws.

It is a time-consuming task, mainly due to the excessive time required by the teacher to check the project’s functionality. This becomes a serious problem when the number of students is high, and there is only one teacher. It produces difficulty for using individual project-based learning.

A possible solution is to analyse execution traces or employ code debugging so that the programs can be analysed, but as it has been mentioned and explained in several works [9], even this alternative could result in a very complex task due to the large amount of generated data. For example, for an actual play session, each line of a log file is formed by almost 21 values which are generated each second, so the total size for this file is approximately 800 lines (see Fig. 2 to understand the difficulty of interpreting this kind of file).

OPEN IN VIEWER

Fig. 2

An example of an execution trace. For each line is shown the values for each variables. Description of the variables is also provided in the figure. Comprehending an execution trace can be a very difficult task.

In this scenario, fuzzy linguistic descriptions technology could play a key role in the computer game-based methodology since it would allow us to:

Interpret large amounts of numerical data quickly and accurately.

In the literature, some works have been proposed to provide learners and/or teachers with resources based on fuzzy linguistic descriptions from data generated in the teaching-learning process. In the non-fuzzy area, works can be grouped into five main categories: statistical, natural language processing (NLP), information extraction (IE), clustering and integrated-approaches [10]. Several examples of successful applications can be found in the literature: automatic creation of summary assessments for intelligent tutoring systems [11], automatic generation of formative feedback in the university classroom for specific concept maps scaffold students’ writing [12], a framework to provide students with feedback on algebra homework in middle-school classrooms [13], automatic test-based assessment of programming [14], automatic assessment of free text answers using a modified BLEU algorithm [10], and feedback for serious computer games to provide learners with useful and immediate information about the player’s performance [15].

In the fuzzy logic area [16], the following works should be mentioned: linguistic summaries of graph datasets using ontologies [17], automatic textual reporting in learning analytics dashboards [18], feedback reports for students based on several performance factors [19], and reports describing the learner’s rating in a specific learning activity [20]. In [21] linguistic descriptions were used for improving the player experience in a computer game called YADY (your actions define you). There are remarkable differences with respect to the present work. In [21], the feedback aimed to improve the player experience; the current work focuses on providing users (students) with written, immediate and interpretable feedback that aims to support the teaching-learning process.

In this paper, a methodology and a data-driven architecture software are proposed to automatically generate personalised and technical feedback from the data generated during the algorithm execution implemented by the students. A combination of three computational techniques is proposed: bot’s behaviour analysis, computational perception networks and natural language generation based on templates. The idea is that each student receives immediate, technical and personalised feedback about their mistakes made during the development of the project, and they learn about how heuristic algorithms can be employed for programming computer game bots. Additionally, another important challenge for us is to assess the ”believability” of the AI agents implemented by the students, hence a similarity measure between linguistic descriptions for evaluating and comparing the behaviour between agents and human expert players is also proposed (see Restrictive Equivalent Functions in Section 2).

Additionally, this approach is very beneficial to the teachers since it allows them to:

Save time for evaluating other aspects of the projects, which implies a better understanding of the projects.

It is important to note that our approach aims to support the traditional process of project evaluation and not to replace it. Additionally, our approach can be seen as a useful tool for improving the decison-making process into the classroom. As is mentioned in [22] “decision-making performs a vital role in our daily life” and teachers/students are countinuosly confronted with challenging situations that demand decision making [23]. As a consequence fuzzy data-driven decision making approaches should be taken into account [24–26]

To show and explore the possibilities of this new technology, a web platform has been designed and implemented by one of the authors following the phases and steps established in the methodology detailed in Section 1. Finally, our framework is evaluated by using a survey directed to students and comparing the results of the application of the proposed approach with the human expert assessment.

The structure of the paper is as follows. Section 2 introduces several general concepts regarding project-based learning in artificial intelligence and provides a very brief review of the state-of-the-art of the different involved disciplines. Then, in Section 3, a methodology for incorporating linguistic descriptions of data into the AI projects is proposed. Section 4 details the software architecture for providing teachers and students with personalised and technical feedback. Afterwards, Section 6 explains the experimentation and evaluation carried out on the projects of the student by employing an adaptation of the Turing test. Finally, Section 7 provides future work and some concluding remarks.

2.1 Automatic generation of reports in natural language and fuzzy linguistic descriptions technology

The automatic generation of reports in natural language is a sub-field of AI, which allows us to produce natural language (and/or graphs) as output on the basis of data input. There are two main methods (which are compatible with each other) for generating reports in natural language from datasets: natural language generation (NLG) and linguistic description discipline (LD).

The NLG field is focused on converting any kind of data into informative texts. NLG models and techniques have been applied for textual reporting in various domains, such as meteorological data [27, 28], care data [29], project management [30], and air quality [31]. An important and extensive survey of the state-of-the-art in this discipline can be found at [32]

The LD discipline [33] combines several sub-areas and as is mentioned in [34]: “it is a young field, to achieve a general approach capable of building different types of linguistic descriptions for any kind of application domain is still an open challenge, although some steps have been made in this direction”.

In this sense, we can mention here the linguistic description of the data (LLD) approach, which has been applied in several practical cases where data is the main input 1 . LDD is a research area with similar objectives as NLG. However, as it is mentioned in [35], ”in the fuzzy logic and soft computing field, this task is performed by employing fuzzy logic machinery (mainly linguistic variables [36], inference fuzzy rules and fuzzy quantifiers)”. More formally, following [35], LDD is defined as the task of extracting knowledge in natural language sentences and combining it with useful and explainable graphs from some input data by producing an abstraction composed of linguistic terms and inference fuzzy rules [35]. The LDD models and techniques have been applied for textual reporting in various domains such as air quality index textual forecasts [37], and weather forecasts [38].

Additionally, the linguistic description of complex phenomena (LDCP) [39] paradigm aims to extract and represent knowledge by using natural language sentences as if they were produced by a human expert, describing the most relevant aspects of a phenomenon for certain users in specific contexts. The LDCP technique has been used in domains such as deforestation analysis [40], big data [41], advice for saving energy at home [42], self-tracking physical activity [43], cosmology [44, 45], and driving simulation environments [46]. The construction of sentences in LDCP is a process influenced by the computational perception concept. The algorithms employed in LDCP approaches generate all possible sentences combinations to create candidate descriptions from data, which have been previously transformed in variable and linguistic terms. Then, linguistic terms are summarised, and template descriptions are defined to automatically generate linguistic descriptions which are put together with explainable graphs to provide users with more complete textual and visual information.

LDCP is based on the Computational Theory of Perceptions [47] and it grounds on the fact that human cognition is based on the role of perceptions, and their remarkable capability to granulate information in order to perform physical and mental tasks without any traditional measurements and computation. LDCP is based on the concept of Computational Perception (CP). A CP is a pair (A,W) described as follows:

A = (u₁, …, u _n ) is a vector of linguistic expressions (words or sentences in Natural Language) that represents the whole linguistic domain of CP.

W = (w₁, …, w _n ) is a vector of the validity degrees w _i  ∈ [0, 1] of each u _i . w _i represents the suitability of a _i to describe the current perception of a specific aspect of the monitored phenomenon.

For example, suppose the following values:

u₁= “The current situation is dangerous”, w₁ = 0.8

u₂= “The current situation is risky”, w₂ = 0.2

u₃= “The current situation is easy”, w₃ = 0.0

u₄= “The current situation is safe”, w₄ = 0.0

We use Perception Mappings (PM) to aggregate CPs. We distinguish two kind of PMs, namely, First Order PMs (1PMs) and Second Order PMs (2PMs). We define a 1PM as a triple (Z,y,g); where Z is a special type of CP with a numerical value z, y is a 1CP, g is a function W = g (z), the function g (z) can be implemented by using membership functions w _i  = μ _i  (Z) associated with each component a _i of A and therefore: W=(μ₁ (Z) , μ₂ (Z) , …, μ _n  (Z)) where n es the elements of elements in A. A 2PM is a tuple (U,y,g); where U is a vector of input CPs, y is the output CP and g is an aggregation function implemented by using a set of fuzzy rules (see several examples of PM later on).

2.2 Similarity measures and restricted equivalence functions

Measuring and evaluating the believability of the agents acting in a computer game is an important challenge in AI [48–50]. A method based on the similarity between fuzzy linguistic descriptions is proposed here. The main idea consists in establishing a similarity measure between linguistic reports by computing a similarity degree between its components.

An essential component of the fuzzy linguistic descriptions are the well-known linguistic vectors which employs linguistic terms for its implementation. At this level, similarity measures can be employed in order to measure the similarity between the corresponding components. We have selected the called restricted equivalence functions (REF) because we proved in [51] that they work very well in practice. Of course, others similarity measures could be used to compare fuzzy linguistic descriptions (for example [52]).

Finally, bots and human players can be compared from their automatically generated respective reports, which finally provides us with a measure of believability. It will be studied in detail in the Section 1.

A REF [53] is a function that establishes similarity between the elements of a domain. A REF can be formally defined as follows:

Definition 2.1. A REF, f, is a mapping [0, 1] ² ⟶ [0, 1] that satisfies the following conditions:

f (x, y) = f (y, x) for all x, y ∈ [0, 1]

For example, g (x, y) =1 - |x - y| satisfies conditions (1)-(5) with c (x) =1 - x for all x ∈ [0, 1].

3.1 Phase 1. Bots behaviour analysis

In this phase, a set of actions performed by the agents is analysed to establish a set of behaviour patterns. The first step is the selection of entities, attributes and their interactions to determine which variables must be considered for generating some kind of behaviour. In our case, four entities have been identified: agent, opponents, rewards and obstacles. An agent is defined by using four attributes: position X and Y in the scenario, energy E and time T employed to capture each reward. The rest of the entities (opponents, rewards and obstacles) are defined by using two attributes (position x and y).

The second step is the definition of the metrics from the entities, attributes and interactions selected in the previous step. The metrics provide the designer with useful information on the nature of the problem and the kind of behaviour that could be inferred. Two kinds of metrics are considered here: quantitative and qualitative. Quantitative provides us with information about the performance of the algorithms (memory occupied and iterations performed), and qualitative provides us with information about the behaviour of the heuristic algorithms in terms of “believability” (protection, distance, energy, time, reward). In particular, a set of metrics for analysing the behaviour of the heuristic algorithms are defined as follows:

Protection. Number of obstacles between the agent and the opponent _i , a rectangular area is created from the position of the agent and the opponent _i , respectively.

The third step is the definition of a computational procedure to capture numeric data, in our case, traces of execution have been employed as a computational procedure for capturing and storing data. Trace recording or tracing is a commonly used technique useful in debugging and performance analysis. Specifically, trace recording implies detection and storage of relevant events during run-time, for later off-line analysis. We use a trace recording that stores the metrics defined in the previous item. The result is stored in a text file containing values for each metric defined in Fig. 2.

The fourth step is the determination of behaviour patterns where a set of basic behaviour patterns can be established on the input data captured by using knowledge representation techniques. Patterns are determined by an expert. A behaviour pattern is associated with a set of actions; that is, when a set of actions tupleA₁, A₂, …, A _n is produced then a set of effects tupleE₁, E₂, …, E _m is performed.

For example, movements performed by the player could be conditioned by the movements performed by the opponents, e.g., if an opponent is close to the player, then the player will go far away from the opponent, so the player and opponent are related, and it provides us with an interesting behaviour pattern. Here, the action is “to move close to the opponent” and the effect would be “to move far away from the opponent”. Note that several actions and effects could be given at the same time in a particular instant of time, for example an action could be given by “to move close to the reward” and the effect could be “to move closer to the reward”. The idea of behavior pattern will be formalized by using computational perceptions.

Note finally that patterns are related to the metrics defined in the previous step and the metrics should be define from them. For example, attitude provide us with information about how the agent acts with respect to the reward, hence the distance between the opponent and the reward must be evaluated. On the hand, situation provide us with information about how the agent acts with respect to the opponent, in this case the energy and the protection must be evaluated. Additionally, Kind of move provide us with information about which one is the result of a movement, here distance between the agent and the reward and the opponents must be studied. Finally, Performance is about which is the performance of each movement, time and memory must be measured, here memory and iterations must be taken into account.

The result of this phase is a set of behaviour patterns derived from actions performed by entities in the virtual world. These behaviour patterns are designed and implemented by using a computational perception network (see Fig. 4).

3.2 Phase 2. Linguistic descriptions of data in a 2D virtual world for automatically generating behaviour profiles

The aim of this phase is to establish a cognitive computational model from previously identified behaviour patterns. Knowledge representation techniques can be employed here to generate linguistic descriptions using the following steps:

Selection of behaviour patterns to be studied. The behaviour patterns defined in the previous module are analysed and selected according to our particular interest in them. The idea is to determine which behaviour patterns are truly important to create the “behaviour profile” from the data selected in the previous steps. For example, a particular sequence of movements is not relevant for us, but the reason for which these movements were performed is truly important because it provides us with interesting cause-effect information, which allows us to employ if-then rules for modelling.

This phase provides us with a complete specification of the behaviour profile, which is a graphical and textual report describing the most relevant information about the behaviour of the computer game bots acting in the computer game.

3.3 Evaluation: A Turing test for computer game bots based on LLD and REFs

There is an important difference between the algorithms employed for programming bots and classic algorithms, for example, to get the optimal way to go from one place to another one. While classic algorithms aim to simulate near-optimal intelligent behaviour, the gaming bot algorithms aim to provide us with interesting and fun opponents for human players, not optimal opponents [55].

A variation of the Turing test was proposed and designed in [56] to test the abilities of the computer game bots to impersonate human players. The idea is as follows: “Suppose we are playing an interactive video game with some entity. Could you tell, solely from the conduct of the game, whether the other entity was a human player or a bot? If not, then the bot is deemed to have passed the test”.

This kind of Turing test is redefined and adapted for our methodology by using LDCP and REFs. Our idea is to establish a formal and effective method of measuring the believability of the agents acting in the computer game. A comparison between the automatically generated profiles (bots and players) is performed by establishing a similarity measure between them.

For us, a heuristic algorithm is near-behavioural (for us it is an “optimal” algorithm) when bots profiles are similar to human profiles (defined by the teacher as an evaluation pattern, i.e., requirements that must met by the bots). This novel process of assessment for heuristic algorithms in computer games will be detailed in the Section 5.

The tracing module aims to implement the functionality explained in phase 1, and it is incorporated into an event handler layer of the computer game architecture. This starts when a movement is performed by the player or the opponents, then an event is launched, and a function is called, which creates a new execution trace row. Each row is formed by the values of the metrics described in Section 1, namely: player position; opponent position (for each opponent); player energy; protection with respect to the opponent (for each opponent); distance between the player and the closest reward; distance between the opponent and the closest reward (for each opponent), time elapsed, if the reward was captured in this movement (true or false), number of iterations, memory used and which entity performed the movement (character ’J’ for the player, character ’A’ for the opponent).

At the end of each play session, these values are written in the output trace file. Note that, the output of this module is a file containing the needed data captured during the execution of the algorithms implemented in the project (see Fig. 2).

4.2 Computational perception network module

A computational perception network is used to implement the functionality explained in phase 2. An extension of the computational perception network presented in [21] is performed. In this case, additional variables must be considered, and the computational perceptions network must be enhanced. Additionally, rules and templates must also be updated for these new requirements.

Please note that the design of computational perceptions depends on the designer criterion. Here, after intensive experimentation we decided to make the linguistic model simple and functional but to keep it easy tounderstand. A computational network is formed by two kind: first-order computational perceptions (called 1CP) are the metrics perceived by the agents (distance,protection, energy, time, iterations, memory) and second-order computational perceptions (called 2CP) are the mental objects built by using metrics in the world [39].

4.2.1 First-order computational perceptions

The process of construction of a computational perception network starts with the selection of the metrics and the creation of a set of linguistic variables from them. An important requirement here is that the problem domain must be known by the designer. We use 1CP to process the input data. Here the inputs are numerical data obtained from varaibles and metrics defined in previous sections.

CP Distance. The CP _Distance is the perception by which an agent perceives if him/her is close or far away from something. The distance depends on the number of cells among the different actors (see Fig. 4). Given two points A (x, y) and B (x′, y′), the distance between both points is calculated by means of the following equation: ( x - x ′ ) 2 + ( y - y ′ ) 2 . We define three different CPs of Distance: Distance between the player and the opponent (Dpa*); Distance between the player and the closest reward with respect to him (Dpr*) and Distance between the opponent and the closest reward with respect to the player (Da* r *). Since these parameters are real numbers, we define this 1CP as follows:

OPEN IN VIEWER

Fig. 4

Computational perception network: CP means Instant CP; ∑CP means play session CP.

Z = [0, N], being N the maximum distance in the game world.

A = (close, normal, far)

g: this function is built using three linguistic labels that are represented with trapezoidal membership functions close (0, 0, 4, 7) , normal (6, 9, 11, 14), far (13, 16, N, N)

CP Protection. The protection degree is the number of walls between the player and the opponent (see Fig. 4). This is also a metric whose values are represented by a real number. Given two points A (x, y) and B (x′, y′), being the positions of the player and the opponent, respectively. The number of obstacles between both points can be calculated by using the following steps: i) calculating the rectangle formed by both points; ii) counting the number of obstacles in such a rectangle. Since this parameter is a real number, we define this 1CP as follows:

Z = [0, N], being N the total number of walls in the game world.

A = (low, intermediate, high)

g: this function is built using three linguistic labels that are represented with trapezoidal membership functions (0, 0, 0, 2), (1, 3, 3, 5) and (4, 6, 380, 380)

CP Energy. The energy indicates the current energy of the player. This is a metric whose value is represented by a real number.

Z = [0, N], being N the total of energy of the player (100 in our case)

A = (low, intermediate, high)

g: this function is built using three linguistic labels that are represented with trapezoidal membership functions (0, 0, 3, 6), (4, 7, 9, 12) and (10, 13, 100, 100)

CP Iterations. This metric indicates the current number of iterations performed by the algorithm used for implementing the artificial intelligence of the agents. This is a metric whose value is represented by a real number.

Z = [0, N], being N the total number of iterations of the agent

A = (little, normal, large)

g: this function is built using three linguistic labels that are represented with trapezoidal membership functions (0, 0, 18, 30), (18, 30, 42, 54) and (42, 54, 104857600, 104857600)

CP Memory. The memory indicates the current memory required by the algorithm employed to implement the artificial intelligence of the agent. This is a metric whose value is represented by a real number.

Z = [0, N], being N the total of memory for the agent (104857600 in our case)

A = (low, normal, high)

g: this function is built using three linguistic labels that are represented with trapezoidal membership functions (0, 0, 768, 1280), (768, 1280, 1792, 2304) and (42, 54, 104857600, 104857600)

CP of Time. This CP measures the spent time to capture the rewards distributed in the game world (see Fig. 4). Given two points A (x, y) and B (x′, y′), the time is calculated by measuring the needed time to go from A to B. For example, suppose that the player is at the position (0, 0) and the rewards are at the positions (3, 4) and (8, 8). CP-Time measures the time that the user has needed to go from (0, 0) to (3, 4) and then from (3, 4) to (8, 8). Since the parameter is a real number, we define this 1CP as follows:

Z = [0, N], being N a maximum fixed in seconds for capturing a reward.

A = (short, large)

g: this function is built using two linguistic labels that are represented with triangular membership functions short (0, 0, 7) , large (5, N, N)

Finally, a set of if-then rules is defined by aggregating these linguistic terms, which have an associated fuzzy set (see Example 1).

4.3 Behaviour profile report generation module

A special kind of CP is employed here which is called CP play session, and it is denoted by ΣCP. Each ΣCP can be formally defined by using a vector of linguistic expressions ((a₁, w₁) , …, (a _n , w _n )). Each ΣCPs represents the whole linguistic domain. These kinds of CPs allow us to obtain the total number of times in which a value (a₁, …, a _n ) occurred during the execution.

These kinds of CPs provide us with a set of variables, their associated value and a degree α, which indicates the average for a particular value. For example, a value for CP situation could be “Safe” with 0.8 at an instant i and “Safe” with 0.7 at instant i + 1, and so on. Therefore, at the end of the execution, we have that a _i (in the example “Safe”) has been given N times with N different degrees β₁, …, β _n (of course, some of these degrees could be equal). Thus, the final degree is calculated as follows: α _i  = ((β₁ + … + β _n )/N). For example, the following summaries can be obtained from different ΣCP (see Fig. 4).

The generation of the report is performed by using a set of ΣCP. For each CP a linguistic description is created in the function of the pair (a _i , w _i ) ∈ ΣCP. Percentages are calculated for each ΣCP. The percentage p _i is then transformed into a linguistic term of quantity as follows: few is when p _i  ∈ [0, 1/3]; several is when p _i  ∈ [1/3, 2/3] or many is when p _i  ∈ [2/3, 1]. Next, we consider the following four cases:

There exists a pair (a _i , p _i ) ∈ ΣCP whose p _i is greater than 66 percent.

The system selects, the most suitable linguistic expressions from among the available possibilities to describe the input data. Linguistic descriptions for each CP are stored in a list 1, 2, 3, 4 as follows:

Descriptions for CP situation={① “Definitely, [degree] situations were [value];”, ② “[degree] situations were [value]”; ③ “[degree] situations were [value₁], although the [degree] situations were also [value₂]”; ④ “Diverse situations were detected during most of the play session” }

Descriptions for CP attitude={① “During most of the play session, the bot showed [degree] attitudes [value]”; ② “The bot showed [degree] attitudes [value]”; ③ “The bot showed [degree] attitudes [value₁] but also showed [degree] attitudes [value₂]”; ④ “The bot does not show a particular attitude during the play session” }

Descriptions for CP movement={① “Certainly, the [degree] of the movements performed by the bot was [value]”; ② “The bot proved to be capable of performing [degree] movements [value]”; ③ “The bot proved to be capable of performing [degree] movements [value₁] but also performed [degree] movements [value₂]”; ④ “The bot performed several movements indistinctly during the play session ” }

Descriptions for CP ability={① “Clearly, the bot displayed a/an [value] player [degree] times”; ② “The bot displayed a/an [value] player [degree] times”; ③ “The displayed a/an [value] player [degree₁] times, however [degree₂] times it acted as a/an [value₂]”; ④ “No kind of player has been identified ” }

Descriptions for CP Skill={① “Certainly, the bot proved to be [value] [degree] times”; ② “The agent proved to be [value] [degree] times”; ③ “The agent proved to be [value₁] [degree₁] times, nevertheless degree₂ times proved to be [value₂]”; ④ “No kind of skill can be proved during the current play session” }

Several cases may happen simultaneously, in this case (④, ③, ②,①), it would be the priority.

A complete example of the generation of behaviour profiles from an execution trace file is detailed in Example 1.

Example 1. Assume the unprocessed execution trace is described in the Fig. 7. A random real line of an execution trace file is as follows: 1, 13, 4, 12, 2, 12, 3.60, 3.16, 1.41, 17, 5.0, 2.0, 1.0, 15.26, 17.08, 13.0, 13.89, 15995,false. 42, 924, J First, the data are processed as follows: the values (1,13) correspond to the player position 2 and it is stored in the variable P; (3, 16), (4, 12) and (2, 12) are the positions of the opponent 1, 2, 3, respectively and they are stored in the variables (O₁, O₂, O₃). The values 3.60, 3.16, 1.41 are the distance between the player and the opponents and they are stored in the variables (Dist (P, O₁),Dist (P, O₂),Dist (P, O₃)). The value 17 is the energy of the player at this instant of time and it is stored in the variable E. The same process is followed for computing the protection between the player and the opponents (Protect (P, O₁) =5.0, Protect (P, O₂) =2.0, Protect (P, O₃) =1.0), and distances between player and opponents with respect to to the closest reward (Dist (P, R *) =15.26; Dist (O₁, R *) =17.08; Dist (O₂, R *) =13.0; Dist (O₃, R *) =13.89). Finally, Time = 15995; Iterations = 42 and Memory = 924. Second, linguistic terms are created for each variable as it was explained in Section 1. Therefore, we have: Dist (P, O₁) =3.60 is a close distance with 1.0; D (P, O₂) =3.16 is close distance with 1.0; Dist (P, O₃) =1.41 which is a close distance with 1.0; Protect (P, O₁) is high protection with 0.5; Protect (P, O2) is a normal protection with 0.5; Protect (P, O₃) is a low protection with 0.5; Dist (P, R *) =15.26 is a far distance with 0.75; Dist (O₁, R *) =17.08 is a far distance with 1.0; Dist (O₂, R *) is a normal distance with 0.33; Dist (O₃, R *) =13.89 is a far distance with 0.29. Time = 15995 is a small time with 1.0; Iterations = 42 is a normal number of iterations with 1 and Memory = 924 needed is a low required memory with 0.69. Then, each CP is instantiated with the values of the linguistic terms and if-then fuzzy rules are computed by using the average as the aggregation operator for computing the computational perceptions as follows (the closest opponent is represented by O*):

Attitude=(Cautious,0.54) ← Dist(O*,R*)=(High,0.29), Dist(P,R*)=(Normal, 0.33), Energy=(High,1) Situation=(Dangerous,0.83) ← Protection=(Low,0.5),Distance(P,O*)=(Close,1),Energy = (High,1) Movement=(Bad,0.77) ← Distance(P,R*)=(Normal,0.33),Distance(P,O*)=(Close,1), Energy=(High,1) Ability=(Dummy,0.76) ← Attitude=(Cautious,0.54), Situation=(Dangerous,0.83), Movement=(Bad,0.91) Skill=(Improvable,0.81) ← Attitude= (Cautious,0.54), Movement=(Bad,0.91), Time=(Small,1) Resources=(Efficient,0.84) ← Memory=(Low,0.69), Iteration=(Normal,1)

Subsequently, the ΣCP are computed as it was explained in Section 1:

ΣCP _Attitude  = {(wise,17.53), (brave,101.55), (cautious,14.05), (passive,10.78) }

Finally, the instantiation, for each template shown in the Section 4.3, is performed by using the generated ΣCP. An example of instantiation is shown in the Fig. 5.

OPEN IN VIEWER

Fig. 5

Instantiation template for the execution trace of the Example 1 and the similarity between behaviour profile reports: human player versus bots.

The aim of this section is to show how fuzzy linguistic descriptions can be used to get more information about the algorithms implemented by the students. The classic human expert assessment for the computer game-based project at the University of Bío-Bío is based on the following guidelines. There are minimal requirements to pass the subject: 2D scenario, 4 rewards, 3 opponents, 1 player, opponents implemented with breadth-first algorithms all functionality; crash with stop (+1); crash without stop (-1); images (+3); life (+3); lose life (+3); Show life (+3); Stop play session correctly (+3); a star algorithm implementation (+3); additional tasks about the project performed in class (+2 per task); additional tasks about the project not performed in class (-2 per task); exceptional IA (+8)

In order to perform an empirical comparation, we are going to use the scores obtained in a classical assessment evaluation by the teacher and those automatically generated by the software tool (see Fig. 6). A comparative table was created in order to show both resulting scores. The empirical study is simple, illustrative. We must take into account the time employed in a classical assessment of the algorithm functionality.

Supposing that teacher assesses each algorithm in five minutes, the teacher would need 50 minutes for evaluating 10 projects.

OPEN IN VIEWER

Fig. 6

Scores obtained by students in the project and the scores obtained automatically.

OPEN IN VIEWER

Fig. 7

Bar Graphs obtained from the answers for the survey directed to the students.

The above examples support our initial hyphotesis. It is hard for a teacher to assess all the features involved in an implementation. However, although the teacher could do it, this is a task which needs too much time to be completed. The use of linguistic descriptions allows us to automatise this step, making the task easier for the expert and reducing time. The teacher could spend more time reviewing other features of the project that he might not be able to do because of lack of time.

Additionally, there are considerable differences between both scores. While in the first one, the difference between the respective scores is large, the scores obtained by using our methodology are more equidistant. In a second and unofficial round of revision, the teacher could check that the scores obtained are more adjusted to reality. However, a more deep analysis should be performed in the future work, for example by implementing intelligent artificial tutors based on fuzzy linguistic descriptions. It will be a great challenge for us which starts with the present work.

To measure the impact of teaching innovation, a survey directed to the students of the artificial intelligence class was carried out. Very interesting opinions from the students have been observed. A number of opinions were positive, which appears to indicate that a project-based methodology is very useful in allowing the students to achieve the competencies pertaining to the profile of a graduate. Other opinions were negative but constructive, which indicates that certain aspects of the class, namely, those related to the organisation and presentation of materials and assessments, should be improved. Others were negative but constructive with regards to the context of the class and pre-existing gaps in knowledge. Finally, a smaller number of negative, non-constructive opinions were registered, which criticised the teaching abilities of the instructor and the class contents. Measures have been taken in light of these findings, and a final methodology has been designed as a result. The results show that 38% of students very strongly agreed, 27% strongly agreed, and 10% agreed with the statement that “The development of the artificial intelligence project was motivating and exciting” whereas 13% slightly agreed, and 12% disagreed.

Positive results were also obtained regarding the statement. The project-based method greatly helped in understanding the theoretical concepts of the classž with 81% of students agreeing, strongly agreeing or very strongly agreeing, while 10% remained neutral, and 9% disagreed. Additionally, 75% of the students agreed, strongly agreed or very strongly agreed; 21% remained neutral, and 4% disagreed with the statement. The project-based method greatly helped in understanding how to implement the research techniques reviewed in class. For the next question, 79% of students agreed, strongly agreed or very strongly agreed; 15% remained neutral, and 6% disagreed with the statement “Carrying out this project autonomously helped me improve my professional skills and further prepared me for future employment”. The results of this survey indicate that the methodology made it possible for the project-workshop to be motivating and exciting. However, it failed to establish a connection between theoretical and practical content. It has been observed that autonomous work allows students to acquire working and programming skills. The majority of the students see this as a challenge. In general, the students stated that this type of methodology helped them achieve the desired abilities to learn a better understanding of the studied algorithms.