Operational profile development methodology for normative multi-agent systems

Abstract

Software reliability refers to the ability of a system to perform its intended function under specified conditions for a specified period of time. The first critical step in the software reliability testing process is to create a Software Operational Profile (SOP). Several methodologies for creating SOP have been proposed. Nonetheless, nearly all the proposed studies have neglected the uniqueness of the new software paradigms, despite the fact that these are generally distinguished by their own concepts and methodologies. One of these software paradigms is multi-agent systems. Rather than using a generic one, it would be more useful to propose a specific methodology for creating SOP. In this paper, we propose a methodology for developing Operational Profile for specific kinds of multi-agent systems (so-called normative multi-agent systems). A detailed case study is used to demonstrate this methodology.

Keywords

Software operational profile reliability testing process normative multi-agent systems

1. Introduction

Technological advances over the years have made software ubiquitous and more complex. Ensuring the quality of such software is becoming more and more important. Software quality is generally modeled by a group of characteristics such as reliability, maintainability, availability, modifiability, performance, security, usability, and testability. Among these characteristics, reliability is one of the most important quality characteristics. Software reliability refers to the probability that a system will perform its intended function under specified conditions for a specified period of time [1]. Several studies have also been conducted on the development, management, and measurement of software reliability [2, 3]. Determining the Software Operational Profile (SOP) is important for ensuring software reliability because it’s the first step in measuring software reliability. A SOP corresponds to a quantitative description of how the system is used from the user’s perspective, presenting a set of system operational scenarios and their associated probabilities [4, 5]. The SOP improves test productivity, reliability, and speed by efficiently allocating resources in terms of utilization and importance [6, 7]. On the other hand, multi-agent systems are relatively recent software paradigms that try to incorporate artificial intelligence to ensure the autonomy of systems. Like all software paradigms, multi-agent systems are characterized by inherent specificities that require specific development methods and methodologies. Several works, in particular, have been proposed to investigate testing aspects of multi-agent systems [8, 9, 10]. However, testing the reliability of such systems remains unaddressed. In order to develop a basic support for testing the reliability of normative multi-agent systems (NorMAS), we propose, in this paper, Operational Profile for Normative Multi-Agent Systems (OP4NorMAS), a new methodology to develop operational profile for such systems. This work is an extension of our previous work [11].

The rest of this paper is organized as follows: Section 2 presents a review of the literature relevant to developing software operational profiles. Section 3 describes the detailed development process for the OP4NorMAS. Section 4 illustrates the proposed methodology using a detailed case study. Section 5 draws some conclusions and points out future directions.

2. Literature review

Several studies on SOP development can be found in the literature. Musa introduced SOP in [4, 5]. The SOP development process includes five steps: customer profile, user profile, system mode profile, functional profile, and operational profile. He demonstrated his approach with a case study of a telecommunications switching system. [12] shows how to create a behavioral profile by decomposing operational profile into its two components: a configuration profile and a usage profile. This paper includes an example of a project (AT&T Private Branch Exchange) that first used this method. This approach works best when the customers, applications, and configurations of the product are diverse and complex.

[13] describes a structured way to develop an operational profile for interactive users and gives examples. It also demonstrates the use of a test case selection tool that facilitates the tester’s task by using the information obtained during the development of the operational profile as input. Another study proposed an improved methodology to generate more accurate operational profiles that truly represent the different usage patterns of customers [14]. Clustering analysis supports improved methods for identifying customer groups with similar characteristics. The operational profile model has been extended in [15]. In addition to the operational profile, the extended version consists of a process profile, a structure profile, and a data profile. A process profile describes the processes and frequency that occur in a typical user application. A structural profile is the system structure on which an application runs. The application input values from one or more users represent a data profile.

Development of systematic operational profiles for software components is presented in [16]. This method uses both intended usage assumptions and usage data to discover a usage structure, usage distribution, and parameter characteristics. In [17], software previously exposed to the company’s normal planned test cycle is tested by the operational profile. A SOP is developed and test cases are allocated.

A multi-agent framework for automatically generating an operational profile for distributed systems is presented in [7]. The work also proposes a restructured metric for determining operational criticality. [18] proposes a framework for developing SOP-based test case allocation using fuzzy logic. This work presents two case studies that demonstrate the applicability of the proposed framework. Another method based on uniform design was proposed two years later to develop SOP [19]. A review of a SOP, its extension, implementation, and shortcomings is presented in [6], while [20] is devoted to collecting, analyzing, and classifying operational profiles that characterize the type and frequency of software inputs. [21] proposes a method for effective SOP development with some modifications to Musa’s approach. To make an efficient operational profile, usage and configuration profiles have been added to the existing model. If the configuration profile defines how the customer sets up the system, then the usage profile indicates that the users have unequal usage shares.

Recently, a novel study was published in [22] and considered using SOP as a resource for assuring and improving software quality from the user’s perspective. The study suggests a strategy for users to use the SOP to evaluate and adapt an existing test suite to rank the impact of the defects on software behavior. This helps with pricing defects. A behavior-driven development approach [23] is used as an information source for semi-automatic generation of the operational profile. The proposed approach was evaluated using the social networking software Diaspora, an open-source project.

From the above, we can conclude two main observations. First, despite the fact that the first works appeared thirty years ago, the field is still an active one, with new contributions being made [21, 22, 23]. Second, the evolution of software development paradigms was excluded from the proposed studies. In fact, each new software paradigm introduces its own concepts, methods, and methodologies. Obviously, these concepts will affect the entire software development process. In this paper, we will develop an operational profile for normative multi-agent systems (OP4NorMAS).

3. Proposed operational profile development methodology for normative multi-agent systems

The proposed methodology is shown in Fig. 1. It generates an operational profile (OP4NorMAS) from an initial operational profile (MAS-OP) and norms specified using the NorMAS-ML modeling language [24] by applying a set of proposed rules. The resulting OP4NorMAS will be used to generate NorMAS test cases and to add norm action probability as a new norm constraint to the NorMAS-ML specification.

We devote the following sub-sections to detailing this methodology.

Figure 1.

Proposed operational profile development methodology.

3.1 Norms specification

The norms specification is the first element in our methodology. It’s a normative specification that contains the basic goals of the system, grouped by missions and assigned to various agents. These missions are performed according to norms based on deontic concepts that allow us to describe the behavioral constraints of system agents, such as: obligation, permission, recommendation, and prohibition. Application or non-application of norms by system agents may result in rewards or penalties. Agents will be penalized for performing prohibited acts or ignoring their obligations. In recommended acts, agents are rewarded for applying the norm, but violating the norm has no consequences. With permissions, there are no consequences. Table 1 gives an example of norm specification.

Table 1
An example of norms specification

Norm	Rolei	Type	Mission	Consequence
N1	Role1	Obligation	Mission-Role1 $=$ {goal1, goal2, goal3}	Punishment
N2	Role2	Prohibition	Mission-Role2 $=$ {goal4}	Punishment
N3	Role2	Permission	Mission-Role3 $=$ {goal5, goal6}	None
…	…	…	…	…
Ni	Rolei	Recommendation	Mission-Rolei $=$ {goal7, …}	Reward

Thanks to NorMAS-ML [24], normative multi-agent systems and their properties can be modelled using a structural diagram called a “norm diagram”. In fact, the norm diagram includes the following new graphic elements:

3.1.1 Norm

The norm is represented by a rectangle with an angle in the upper right and lower left corners (see Fig. 2). The norm superior compartment includes the norm class name and the deontic concept associated with that norm, using one of the following stereotypes: $<<$ permission $>>$ , $<<$ obligation $>>$ , or $<<$ prohibition $>>$ . We can register resources regulated by the specification in the intermediate compartment. The resource can be an entity (agent, agent role, organization, or environment) or a property (goal, belief, attribute, method, action, plan, protocol, association, or message). The bottom compartment describes the standard activation limits.

Figure 2.

Norm representation [24].

3.1.2 Organization and agent role classes

Figure 3 shows their graphical representation. The organization representation is a combination of a rectangle and an ellipse with three compartments: the top compartment contains the instance name; the middle contains goals and beliefs; and the bottom compartment contains plans and actions. The agent role representation is a rectangle with downward curved edges and three compartments: the top contains instance name; the middle contains goals and beliefs; and the bottom contains actions and communication protocol details.

Figure 3.

Representation of organization and agent role classes [24].

3.1.3 Restrict relationship

Is represented as simple line with a filled square in its end point (see Fig. 4). It indicates the entity restricted by a norm (environment class, organization class, agent role class, or agent class).

Figure 4.

Restrict relationship representation [24].

3.1.4 Sanction relationship

It indicates the norm sanction and is used between norm classes. A sanction relationship is represented as a simple line with a filled (punishment) or empty (reward) pentagon at its end point. The relationship is shown in Fig. 5.

Figure 5.

Sanction relationship representation [24].

For more details, you can refer to the work [24] to learn more about the other concepts and relationships such as Context, Ownership, Inhabit, and Play.

3.2 Developing initial operational profile for multi-agent systems

A proposed methodology [11] (see Fig. 6) for developing an initial operational profile of a multiagent system is summarized in the following five steps: Identify the MAS customer role profile; define the user role profile for the MAS; define the MAS mode profile; determine the MAS goal profile; and determine the initial MAS operational profile.

Figure 6.

Initial operational profile for multi-agent system [11].

3.2.1 Developing customer role profile

A customer is an actor or a group of actors who acquires the system. A customer role group is defined by a set of customer roles that use the system in the same way. A set of customer role groups and their associated probabilities define a MAS customer role profile.

3.2.2 Identifying user role profile

A user can be an agent or a group of agents who use the system. A user role group is a set of user roles that use the system in the same way. The MAS user role profile is the user role group that uses the system differently associated with their occurrence probabilities.

3.2.3 Defining mode profile

It is defined as a set of goals grouped together to determine the execution environment. An operational profile must be determined for each system mode.

3.2.4 Determining goal profile

A MAS goal profile is a set of goals with their occurrence probabilities. To create a MAS goal profile, you need to find an initial goal list that describes the system’s main goal as well as environment variables that may affect system goals during execution. It then combines the initial goal list with environmental variables to determine the final goal list.

3.2.5 Determining initial operational profile

The initial MAS-OP is a set of operations with their respective occurrence probabilities. Each goal of each MAS mode is decomposed into a list of sub-goals, which in turn is decomposed into a list of actions (operations). The overall probability of each action is calculated by multiplying the initial action probability by the probability of its goal.

3.3 Generating operational profile for normative multi-agent systems

As we explained previously in Sub-section 3.2, MAS-OP specifies the occurrence probability of each action. Hence, it is important to recalculate these probabilities according to the norm’s specification presented in Sub-section 3.1. Compared to traditional systems, norms enhance the flexibility of the system by providing the possibility of executing or ignoring some actions. Of course, applying norms or not implies the execution of consequences. Consequently, the occurrence probabilities of normative actions should be recalculated by introducing their consequences because, in each case, we can execute an action or its consequence. For this goal, we propose the following rules to calculate the occurrence probabilities of normative actions. This step result is the OP4NorMAS.

3.3.1 Rule 1

If a norm is an obligation or prohibition, the overall occurrence probability (OOP) of each normative action (NA) must be multiplied by the rate of agents applying the norm (AAN Rate) to obtain the new occurrence probability (NOP) of this action (Eq. (1)). In the rest of the cases (i.e., when the agent ignores the norm), the system will execute the consequence of the norm (NC). Consequently, we calculate the occurrence probability of the consequence as the result of the overall occurrence probability of the normative action multiplied by the rate of agents ignoring the norm (AIN Rate) (Eq. 2). For example, if we consider that an obligated action will occur with an overall probability equal to OOP (OOP $=$ 0.20). In addition, 90% of system agents will apply this action while the remaining 10% ignore it and execute its consequences. Consequently, the new occurrence probability of this action will become 0.18. On the other hand, the consequence will occur with a probability of 0.02.

(1)NOP(NA) = OOP * AAN_Rate (2)NOP(NC) = OOP * AIN_Rate
3.3.2 Rule 2

If a norm is a recommendation, the overall occurrence probability (OOP) of each normative action (NA) must be multiplied by the rate of agents applying the norm (AAN Rate) to obtain the new probability (NOP) of this action (Eq. (1)). Knowing that a recommended action must be rewarded, the consequence will be executed with the same occurrence probability as the action (Eq. (3)). For example, if we consider that a recommended action will be executed with an occurrence probability equal to OOP (OOP $=$ 0.20) and only 30% of system agents will apply this recommendation under normal conditions, then the new occurrence probability of this action will be (NOP $=$ 0.30 * 0.20 $=$ 0.06). This new occurrence probability of the recommended action is equal to the occurrence probability of the reward consequence.

(3)NOP(NC) = NOP(NA)
3.3.3 Rule 3

The permission norms specify actions that have no consequence. In this case, the agent can choose to execute the normative action or not. Consequently, the overall occurrence probability of this action will not be changed.

After applying Rules 1, 2, and 3, the probabilities of the resulting OP4NorMAS actions are modified, which modifies the global probability of each system mode. Therefore, the generated OP4NorMAS needs to be normalized.

4. Case study

To illustrate our approach, let us consider the same example of a NorMAS presented in [11] and illustrated in Fig. 7.

4.1 Norms specification of university system

The university system consists of two major sub-organizations: Course and Research. System agents are distributed between these two sub-organizations and are supervised by the university authorities (authority agents). All system agents are obliged, permitted, or recommended to satisfy their goals. For example, students are obliged to satisfy their goals (guided_autonomous_practice, pass_evaluation, and respect_regulations) grouped by mission (Mission-S). The same agents are permitted to attend courses. Table 2 describes the different system norms.

Table 2
Norms specification of university system

Norm	Agent role	Type	Mission	Consequence
N1	Teacher	Obligation	Mission-T $=$ {goal1 (Teach), goal2 (Evaluate), goal3 (Research), goal4 (Respect_regulations)}	C1 $=$ Decrease_salary
N2	Teacher	Recommendation	Mission-T $=$ {goal5 (Occupy_senior_position)}	C2 $=$ C3 $=$ Increase_salary
N3	Project leader	Recommendation	Mission-Pl $=$ {goal5 (Occupy_senior_position)}
N4	Student	Obligation	Mission-S $=$ {goal6 (Guided_autonomous_practice), goal7 (Pass_evaluation), goal4 (Respect_regulations)}	C4 $=$ Decrease_evaluation
N5	Student	Permission	Mission-S $=$ {goal8 (Attend_courses)}	None
N6	Researcher	Obligation	Mission-R $=$ {goal9 (Collaborate_with_researchers), goal3 (Research), goal10 (Attend_seminars), goal4 (Respect_regulations)}	C6 $=$ Decrease_reputation
N7	Researcher	Permission	Mission-R $=$ {goal1 (Teach)}	None
N8	Project leader	Obligation	Mission-Pl $=$ {goal3 (Research), goal11 (Make_industrial_contacts), goal12 (Sign_contracts), goal13 (Propose_approach), goal14 (Supervising_researcher), goal4 (Respect_regulations)}	C8 $=$ Decrease_reputation

Figure 7.

University system [11].

Figure 8.

Modeling of the norm and consequence N1 and C1.

Figure 9.

Modeling of the norm and consequence N4 and C4.

Figure 10.

Modeling of the norm and consequence N2 and C2.

Figure 11.

Modeling of the norm and consequence N3, N5 and C3.

Figure 12.

Modeling of the norm and consequence N6, N7 and C6.

Figure 13.

Modeling of the norm and consequence N8 and C8.

Table 3

User role profile of university system

University user role	Overall user role group probability
Teacher agent	0.23
Student agent	0.7
Researcher agent	0.06
Project leader agent	0.01

Table 4

User role profile of university system

University mode	Occurrence probability
Course mode	0.884
Research mode	0.116

Table 5

Initial goal list of course mode

Goals	Agent role	Course mode (0.884)
		Occurrence probability	Overall occurrence probability
Goal 1 (Teach)	Teacher, Researcher	0.3	0.2652
Goal 2 (Evaluate)	Teacher	0.1	0.0884
Goal 4 (Respect_regulations)	Teacher, Student	0.1	0.0884
Goal 5 (Occupy_senior_position)	Teacher	0.1	0.0884
Goal 6 (Guided_autonomous_practice)	Student	0.2	0.1768
Goal 7 (Pass_evaluation)	Student	0.1	0.0884
Goal 8 (Attend_courses)	Student	0.1	0.0884

Table 6

Initial goal list of research mode

Goals	Agent role	Research mode (0.116)
		Occurrence probability	Overall occurrence probability
Goal 3 (Research)	Teacher, Researcher, Project leader	0.3	0.0348
Goal 4 (Respect_regulations)	Researcher, Project leader	0.1	0.0116
Goal 9 (Collaborate_with_researchers)	Researcher	0.075	0.0087
Goal 10 (Attend_seminars)	Researcher	0.075	0.0087
Goal 5 (Occupy_senior_position)	Project leader	0.2	0.0232
Goal 11 (Make_industrial_contacts)	Project leader	0.025	0.0029
Goal 12 (Sign_contracts)	Project leader	0.025	0.0029
Goal 13 (Propose_approach)	Project leader	0.1	0.0116
Goal 14 (Supervising_researcher)	Project leader	0.1	0.0116

Table 7

Environment variables of university system

NorMAS goals	Environmental variables
	NorMAS agent and actor variables	NorMAS variables
Goal 1 (Teach)	Absence of student agent role (Abs-SR)	NorMAS Resources Problems
Goal 2 (Evaluate)		(NorMAS-RP)
Goal 3 (Research)	Absence of project leader agent role (Abs-LR)
Goal 4 (Respect_regulations)	–
Goal 5 (Occupy_senior_position)
Goal 6 (Guided_autonomous_practice)	Absence of teacher agent role (Abs-TR)
Goal 7 (Pass_evaluation)
Goal 8 (Attend_courses)
Goal 9 (Collaborate_with_researchers)	–
Goal 10 (Attend_seminars)	–
Goal 11 (Make_industrial_contacts)
Goal 12 (Sign_contracts)
Goal 13 (Propose_approach)
Goal 14 (Supervising_researcher)	Absence of researcher agent role (Abs-RR)

Table 8

Goal profile of course mode

Goals	Agent role	Course mode (0.884)
		Environmental variables	Occurrence probability
Goal 1 (Teach)	Teacher, Researcher	NorMAS-RP (0.01)	0.002652
		Abs-SR (0.01)	0.002652
		Normal conditions (0.98)	0.259896
Goal 2 (Evaluate)	Teacher	NorMAS-RP (0.01)	0.000884
		Abs-SR (0.01)	0.000884
		Normal conditions (0.98)	0.086632
Goal 4 (Respect_regulations)	Teacher, Student	NorMAS-RP (0.01)	0.000884
		Normal conditions (0.99)	0.087516
Goal 5 (Occupy_senior_position)	Teacher	NorMAS-RP (0.01)	0.000884
		Normal conditions (0.99)	0.087516
Goal 6 (Guided_autonomous_practice)	Student	NorMAS-RP (0.01)	0.001768
		Abs-TR (0.01)	0.001768
		Normal conditions (0.98)	0.173264
Goal 7 (Pass_evaluation)	Student	NorMAS-RP (0.01)	0.000884
		Abs-TR (0.01)	0.000884
		Normal conditions (0.98)	0.086632
Goal 8 (Attend_courses)		NorMAS-RP (0.01)	0.000884
		Abs-TR (0.01)	0.000884
		Normal conditions (0.98)	0.086632

Table 9

Goal profile of research mode

Goals	Agent role	Research mode (0.116)
		Environmental variables	Occurrence probability
Goal 3 (Research)	Teacher, Researcher, Project leader	NorMAS-RP (0.01)	0.000348
		Abs-LR (0.01)	0.000348
		Normal conditions (0.98)	0.034104
Goal 4 (Respect_regulations)	Researcher, Project leader	NorMAS-RP (0.01)	0.000116
		Normal conditions (0.99)	0.011484
Goal 9 (Collaborate_with_researchers)	Researcher	NorMAS-RP (0.01)	0.000087
		Normal conditions (0.99)	0.008613
Goal 10 (Attend_seminars)		NorMAS-RP (0.01)	0.000087
		Normal conditions (0.99)	0.008613
Goal 5 (Occupy_senior_position)	Project leader	NorMAS-RP (0.01)	0.000232
		Normal conditions (0.99)	0.022968
Goal 11 (Make_industrial_contacts)		NorMAS-RP (0.01)	0.000029
		Normal conditions (0.99)	0.002871
Goal 12 (Sign_contracts)		NorMAS-RP (0.01)	0.000029
		Normal conditions (0.99)	0.002871
Goal 13 (Propose_approach)		NorMAS-RP (0.01)	0.000116
		Normal conditions (0.99)	0.011484
Goal 14 (Supervising_researcher)		NorMAS-RP (0.01)	0.000116
		Abs-RR (0.01)	0.000116
		Normal conditions (0.98)	0.011368

Table 10

Initial operational profile of course mode: Goal 1, Goal 2, Goal 3, Goal 4 and Goal 5

Agent role	Actions (operations)	Course mode (0.884)
		Occurrence probability	Environmental variables	Overall occurrence probability
Teacher, Researcher	Goal 1 (Teach)
	Prepare_course	0.2	NorMAS-RP (0.01)	0.0005304
			Abs-SR (0.01)	0.0005304
			Normal conditions (0.98)	0.0519792
	Present_course	0.3	NorMAS-RP (0.01)	0.0007956
			Abs-SR (0.01)	0.0007956
			Normal conditions (0.98)	0.0779688
	Guided_practice	0.3	NorMAS-RP (0.01)	0.0007956
			Abs-SR (0.01)	0.0007956
			Normal conditions (0.98)	0.0779688
	Prepare_autonomous_practice	0.2	NorMAS-RP (0.01)	0.0005304
			Abs-SR (0.01)	0.0005304
			Normal conditions (0.98)	0.0519792
Teacher	Goal 2 (Evaluate)
	Prepare_evaluation_work	0.2	NorMAS-RP (0.01)	0.0001768
			Abs-SR (0.01)	0.0001768
			Normal conditions (0.98)	0.0173264
	Prepare_typical_answer	0.2	NorMAS-RP (0.01)	0.0001768
			Abs-SR (0.01)	0.0001768
			Normal conditions (0.98)	0.0173264
	Correct_evaluation_work	0.5	NorMAS-RP (0.01)	0.000442
			Abs-SR (0.01)	0.000442
			Normal conditions (0.98)	0.043316
	Assign_marks	0.1	NorMAS-RP (0.01)	0.0000884
			Abs-SR (0.01)	0.0000884
			Normal conditions (0.98)	0.0086632
Teacher, Student	Goal 4 (Respect_regulations)
	Know_regulations	0.2	NorMAS-RP (0.01)	0.0001768
			Normal conditions (0.99)	0.0175032
	Practice_rights	0.4	NorMAS-RP (0.01)	0.0003536
			Normal conditions (0.99)	0.0350064
	Apply_duties	0.4	NorMAS-RP (0.01)	0.0003536
			Normal conditions (0.99)	0.0350064
Teacher	Goal 5 (Occupy_senior_position)
	Occupy_senior_position	1	NorMAS-RP (0.01)	0.000884
			Normal conditions (0.99)	0.087516

Table 11

Initial operational profile of course mode: Goal 6, Goal 7 and Goal 8

Agent role	Actions (operations)	Course mode (0.884)
		Occurrence probability	Environmental variables	Overall occurrence probability
Student	Goal 6 (Guided_autonomous_practice)
	Achieve_guided_practice	0.6	NorMAS-RP (0.01)	0.0010608
			Abs-TR (0.01)	0.0010608
			Normal conditions (0.98)	0.1039584
	Achieve_autonomous_practice	0.4	NorMAS-RP (0.01)	0.0007072
			Abs-TR (0.01)	0.0007072
			Normal conditions (0.98)	0.0693056
	Goal 7 (Pass_evaluation)
	Responds_evaluation_work	1	NorMAS-RP (0.01)	0.000884
			Abs-TR (0.01)	0.000884
			Normal conditions (0.98)	0.086632
	Goal 8 (Attend_course)
	Attend_course_presentation	1	NorMAS-RP (0.01)	0.000884
			Abs-TR (0.01)	0.000884
			Normal conditions (0.98)	0.086632

Table 12

Final operational profile of course mode: Goal 1, Goal 2, Goal 3, Goal 4 and Goal 5

Agent role	Actions (operations)	Norm type	Course mode (0.849) $\downarrow$
			Environmental variables	AAN and AIN rates	Overall occurrence probability
Teacher	Goal 1 (Teach)
	Prepare_course	Obligation	NorMAS-RP (0.01)	/	0.0005304
			Abs-SR (0.01)	/	0.0005304
			Normal conditions (0.98)	AAN (0.9)	0.04678128 (NA)
				AIN (0.1)	0.00519792 (NC)
	Present_course		NorMAS-RP (0.01)	/	0.0007956
			Abs-SR (0.01)	/	0.0007956
			Normal conditions (0.98)	AAN (0.9)	0.07017192 (NC)
				AIN (0.1)	0.00779688 (NC)
	Guided_practice		NorMAS-RP (0.01)	/	0.0007956
			Abs-SR (0.01)	/	0.0007956
			Normal conditions (0.98)	AAN (0.9)	0.07017192 (NC)
				AIN (0.1)	0.00779688 (NC)
	Prepare_autonom_practice		NorMAS-RP (0.01)	/	0.0005304
			Abs-SR (0.01)	/	0.0005304
			Normal conditions (0.98)	AAN (0.9)	0.04678128 (NA)
				AIN (0.1)	0.00519792 (NC)
	Goal 2 (Evaluate)
	Prepare_evaluation_work	Obligation	NorMAS-RP (0.01)	/	0.0001768
			Abs-SR (0.01)	/	0.0001768
			Normal conditions (0.98)	AAN (0.98)	0.016979872 (NA)
				AIN (0.02)	0.000346528 (NC)
	Prepare_typical_answer		NorMAS-RP (0.01)	/	0.0001768
			Abs-SR (0.01)	/	0.0001768
			Normal conditions (0.98)	AAN (0.98)	0.016979872 (NA)
				AIN (0.02)	0.000346528 (NC)
	Correct_evaluation_work		NorMAS-RP (0.01)	/	0.000442
			Abs-SR (0.01)	/	0.000442
			Normal conditions (0.98)	AAN (0.98)	0.04244968 (NA)
				AIN (0.02)	0.00086632 (NC)
	Assign_marks		NorMAS-RP (0.01)	/	0.0000884
			Abs-SR (0.01)	/	0.0000884
			Normal conditions (0.98)	AAN (0.98)	0.008489936 (NA)
				AIN (0.02)	0.000173264 (NC)
	Goal 5 (Occupy_senior_position)
	Occupy_senior_position	Recommendation	NorMAS-RP (0.01)	/	0.000884
			Normal conditions (0.99)	AAN (0.3)	0.0262548 (NA) $\downarrow$
					0.0262548 (NC) $\downarrow$
Teacher,	Goal 4 (Respect_regulations)
Student	Know_regulations	Obligation	NorMAS-RP (0.01)	/	0.0001708
			Normal conditions (0.99)	AAN (0.8)	0.01352736 (NA)
				AIN (0.2)	0.00338184 (NC)
	Practice_rights		NorMAS-RP (0.01)	/	0.0003416
			Normal conditions (0.99)	AAN (0.8)	0.02705472 (NA)
				AIN (0.2)	0.00676368 (NC)
	Apply_duties		NorMAS-RP (0.01)	/	0.0003416
			Normal conditions (0.99)	AAN (0.8)	0.02705472 (NA)
				AIN (0.2)	0.00676368 (NC)

Table 13

Final operational profile of course mode: Goal 6, Goal 7 and Goal 8

Agent role	Actions (operations)	Norm type	Course mode (0.849) $\downarrow$
			Environmental variables	AAN and AIN rates	Overall occurrence probability
Student	Goal 6 (Guided_autonomous_practice)
	Achieve_guided_practice	Obligation	NorMAS-RP (0.01)	/	0.0010608
			Abs-TR (0.01)	/	0.0010608
			Normal conditions (0.98)	AAN (0.8)	0.08316672 (NA)
				AIN (0.2)	0.02079168 (NC)
	Achieve_autonom_practice		NorMAS-RP (0.01)	/	0.0007072
			Abs-TR (0.01)	/	0.0007072
			Normal conditions (0.98)	AAN (0.8)	0.05544448 (NA)
				AIN (0.2)	0.01386112 (NC)
	Goal 7 (Pass_evaluation)
	Responds_evaluation_work	Obligation	NorMAS-RP (0.01)	/	0.000884
			Abs-TR (0.01)	/	0.000884
			Normal conditions (0.98)	AAN (0.95)	0.0823004 (NA)
				AIN (0.05)	0.0043316 (NC)
	Goal 8 (Attend_course)
	Attend_course_presentation	Permission	NorMAS-RP (0.01)	/	0.000884
			Abs-TR (0.01)	/	0.000884
			Normal conditions (0.98)	AAN (0.95)	0.0823004 (NA)
				AIN (0.05)	0.0043316 (NC)
	$\sum$				0.8489936

Table 14

Normalized operational profile of course mode: Goal 1 and Goal 2

Agent role	Actions (operations)	Norm type	Course mode (1)
			Environmental variables	AAN and AIN rates	Overall occurrence probability
Teacher	Goal 1 (Teach)
	Prepare_course	Obligation	NorMAS-RP (0.01)	/	0.000624734
			Abs-SR (0.01)	/	0.000624734
			Normal conditions (0.98)	AAN (0.9)	0.055101625 (NA)
				AIN (0.1)	0.006122402 (NC)
	Present_course		NorMAS-RP (0.01)	/	0.000937102
			Abs-SR (0.01)	/	0.000937102
			Normal conditions (0.98)	AAN (0.9)	0.082652438 (NC)
				AIN (0.1)	0.009183604 (NC)
	Guided_practice		NorMAS-RP (0.01)	/	0.000937102
			Abs-SR (0.01)	/	0.000937102
			Normal conditions (0.98)	AAN (0.9)	0.082652438 (NC)
				AIN (0.1)	0.009183604 (NC)
	Prepare_autonom_practice		NorMAS-RP (0.01)	/	0.000624734
			Abs-SR (0.01)	/	0.000624734
			Normal conditions (0.98)	AAN (0.9)	0.055101625 (NA)
				AIN (0.1)	0.006122402 (NC)
	Goal 2 (Evaluate)
	Prepare_evaluation_work	Obligation	NorMAS-RP (0.01)	/	0.000208245
			Abs-SR (0.01)	/	0.000208245
			Normal conditions (0.98)	AAN (0.98)	0.019999849 (NA)
				AIN (0.02)	0.000408160 (NC)
	Prepare_typical_answer		NorMAS-RP (0.01)	/	0.000208245
			Abs-SR (0.01)	/	0.000208245
			Normal conditions (0.98)	AAN (0.98)	0.019999849 (NA)
				AIN (0.02)	0.000408160 (NC)
	Correct_evaluation_work		NorMAS-RP (0.01)	/	0.000520612
			Abs-SR (0.01)	/	0.000520612
			Normal conditions (0.98)	AAN (0.98)	0.049999623 (NA)
				AIN (0.02)	0.001020400 (NC)
	Assign_marks		NorMAS-RP (0.01)	/	0.000104122
			Abs-SR (0.01)	/	0.000104122
			Normal conditions (0.98)	AAN (0.98)	0.009999924 (NA)
				AIN (0.02)	0.000204080 (NC)

Table 15

Normalized operational profile of course mode: Goal 4, Goal 5, Goal 6, Goal 7 and Goal 8

Agent role	Actions (operations)	Norm type	Course mode (1)
			Environmental variables	AAN and AIN rates	Overall occurrence probability
Teacher	Goal 5 (Occupy_senior_position)
	Occupy_senior_position	Recommendation	NorMAS-RP (0.01)	/	0.001041224
			Normal conditions (0.99)	AAN (0.3)	0.030924381 (NA)
					0.030924381 (NC)
Teacher,	Goal 4 (Respect_regulations)
Student	Know_regulations	Obligation	NorMAS-RP (0.01)	/	0.000201177
			Normal conditions (0.99)	AAN (0.8)	0.015933286 (NA)
				AIN (0.2)	0.003983321 (NC)
	Practice_rights		NorMAS-RP (0.01)	/	0.000402355
			Normal conditions (0.99)	AAN (0.8)	0.031866572 (NA)
				AIN (0.2)	0.007966643 (NC)
	Apply_duties		NorMAS-RP (0.01)	/	0.000402355
			Normal conditions (0.99)	AAN (0.8)	0.031866572 (NA)
				AIN (0.2)	0.007966643 (NC)
Student	Goal 6 (Guided_autonomous_practice)
	Achieve_guided_practice	Obligation	NorMAS-RP (0.01)	/	0.001249469
			Abs-TR (0.01)	/	0.001249469
			Normal conditions (0.98)	AAN (0.8)	0.097958445 (NA)
				AIN (0.2)	0.024489611 (NC)
	Achieve_autonom_practice		NorMAS-RP (0.01)	/	0.000832979
			Abs-TR (0.01)	/	0.000832979
			Normal conditions (0.98)	AAN (0.8)	0.065305630 (NA)
				AIN (0.2)	0.016326407 (NC)
	Goal 7 (Pass_evaluation)
	Responds_evaluation_work	Obligation	NorMAS-RP (0.01)	/	0.001041224
			Abs-TR (0.01)	/	0.001041224
			Normal conditions (0.98)	AAN (0.95)	0.096938044 (NA)
				AIN (0.05)	0.005102002 (NC)
	Goal 8 (Attend_course)
	Attend_course_presentation	Permission	NorMAS-RP (0.01)	/	0.001041224
			Abs-TR (0.01)	/	0.001041224
			Normal conditions (0.98)	AAN (0.95)	0.096938044 (NA)
				AIN (0.05)	0.005102002 (NC)
	$\sum$				1

We define a set of eight norms for the university system. Among these norms we find: four norms are obligations (N1, N4, N6, and N8), two are recommendations (N2 and N3), and two are permissions (N5 and N7). Violation of these norms has six consequences (C1, C2, C3, C4, C6, and C8). The scenario for each norm is presented below.

N1 defines that the teacher agent role of the course sub-organization is obligated to satisfy the teach, evaluate, research, and respect_regulations goals, and assumes the consequence C1 $=$ decrease_salary (sanction: punishment) when it is violated. N2 and N3 describe that the teacher and project leader agent roles of the university organization are recommended to occupy_senior_position and apply the consequence C2 $=$ increase_salary (sanction: reward) when they are fulfilled. N4 describes that the student agent role of the course sub-organization is obligated to achieve guided_autonomous_practices, pass_evaluation, and respect_regulations, and must have the consequence C4 $=$ decrease_evaluation (sanction: punishment) when it is violated. N5 defines that a student agent role of the course sub-organization is permitted to attend courses. N6 describes that the researcher agent role of the research sub-organization is obligated to collaborate_with_researchers, research, attend_seminars, and respect_regulations, and assumes the consequence C6 $=$ decrease_reputation (sanction: punishment) when it is violated. N7 defines that Researcher agent role of the research sub-organization is permitted to teach. N8 describes that Project leader agent role of the research sub-organization is obligated to satisfy the research, make_industrial_contacts, sign_contracts, propose_approach, supervising_researcher and respect_regulations goals; and the consequence C8 $=$ decrease_reputation (sanction: punishment) when it is violated.

The university organization and its two sub-organizations (course and research) are represented as organization classes, and the university environment is represented as an environment. The university organization belongs to the university environment through an inhabit relationship. The authority agent role (represented as an agent role class) belongs to the university organization and is an extension of the teacher and project leader agent roles. The course sub-organization consists of the agent roles of teacher and student, while the research sub-organization is made up of the agent roles of researcher and project leader. All agent roles have lists of beliefs, goals, and actions. Goals and beliefs are represented as attributes of the goal and belief types, respectively.

All norms are modeled using the norm diagram defined in NorMAS-ML [24]. They are defined in the context (context relationship) of the university organization and its two sub-organizations (course and research). N1 and N4 restrict, respectively, the teacher and student agent roles’ behaviors (restriction relationship) by stating an obligation to execute the actions (prepare_course, …, achieve_ guided_pract, …) associated with the teacher and student agent roles’ goals (teach, …, attend_courses, …). For example, the action prepare_course is represented as an attribute with the stereotype $<<$ AtomicExecute $>>$ , the entity (teacher), and the action in the intermediate compartment of norm. C1 and C4 are sanctions (punishments) of N1 and N4 that restrict the authority agent role behavior by stating an obligation to decrease salary and decrease evaluation if one action or more of teacher or student agent role are not executed. Figures 8 and 9 show the modeling of the norms N1, N4, C1 and C4.

Figures 10 and 11 show the modeling of the norms N2, N3, N5, C2 and C3. N2 and N3 restrict, respectively, the teacher and project leader agent roles’ behaviors by stating a recommendation for the execution of the action occupy_senior_position. N5 restricts the student agent role behavior by stating permission to execute the attend_course_presentation action. The authority agent role behavior is restricted by C2 and C3 which are sanctions (rewards) of N2 and N3 that indicates an obligation to increase salary if N2 and N3 are fulfilled.

N6, N7, N8, C6 and C8 are shown in Figs 12 and 13. N6 and N8 restrict, respectively, the researcher and project leader agent roles’ behaviors by stating an obligation to achieve their goals (research, …, propose_approach, …). For example, the goal of research is represented as an attribute with the stereotype $<<$ AtomicAchive $>>$ , the entity (researcher) and this goal. N7 restricts the researcher agent role behavior by stating permission to achieve the teaching goal. C6 and C8 are sanctions (punishments) of N6 and N8 that limits the authority agent role behavior by stating an obligation to reduce reputation if one or more goals of the researcher or project leader agent role are not met.

4.2 Initial university operational profile

4.2.1 University system customer role profile

The university system does not use customer role profiles because there is no customer role type. Therefore, the probability that should be assigned to the customer role profile is 1.

4.2.2 University system user role profile

The university user role profile is shown in Table 3. There are four types of user agent roles: teacher agent, student agent, researcher agent, and project leader agent. We also assume that their occurrence probabilities are 0.23, 0.7, 0.06, and 0.01 respectively (see Table 3). In this case, the customer role group probability is 1, so we know that the user role group overall probability is the same as the user role group probability.

4.2.3 University system mode profile

We identify two modes in the university system: course and research. We assume that course mode uses 80% of the teacher agent role and 100% of the student agent role. The remaining percentage of teacher agent role (20%), along with 100% of researcher and project leader agent roles, are used in research mode. The overall probability of course mode is calculated by multiplying the occurrence probabilities of user role groups (Teacher agent, 0.23; and Student agent, 0.7) by the proportion of use (0.8; 1). Use the same formula to calculate the overall probability of research mode. The derived system mode profile is shown in Table 4.

4.2.4 University system goal profile

Initial goal list

We need to identify the initial goal list for each system mode identified in the previous step. The initial goal lists for both course and research modes are shown in Tables 5 and 6, respectively.

Environmental variables

Table 7 shows the environmental variables considered in this case study. The resource problem (NorMAS-RP) is considered as a variable related to the university system. For different agent roles in the system, we consider missing role variables (missing of student agent role, missing of project leader agent role, …).

Final goal list

It’s calculated by multiplying initial goal lists and environmental variables’ probabilities. The final result is the goal profiles shown in Tables 8 and 9 for both course and research system modes, respectively.

4.2.5 Initial operational profile of university system

In this final step, all university system goals are divided into sub-goals, which in turn are divided into actions (operations). For example, a teaching scenario (Goal 1) contains four actions: Prepare_course, in which the teacher agent works on the presentation; Present_course demonstrating what a teacher agent should learn; Guided_practice occurs when a teacher agent does something similar to what is done during course presentations with students; and prepare_autonomous_practice occurs when the teacher agent prepares a personal work and allows the student to reinvest what he has learned during the two previous steps.

Due to space limitations, the rest of this paper will only detail the course mode of the university system. The initial operational profile of course mode is shown in Tables 10 and 11.

4.3 Final operational profile for university system

From the initial operational profile of the university system generated in Table 9, the occurrence probability of each action must be recalculated according to the norm type by applying the rules defined in Section 3.3 by Eqs (1)–(3), respectively.

In the case of obligation or prohibition, Rule 1 is applied. For example, if the action present_course is an obligation, then if we consider that 90% (AAN Rate) of teacher agent roles will present their courses under normal conditions while the remaining 10% (AIN Rate) are absent, the OOP, which is 0.0779688, will be multiplied by 90% to obtain the probability of applying this obligation (Eq. (1)) and will be multiplied by 10% to obtain the probability in case this obligation is ignored (Eq. (2)). The remaining 10% of teacher agent roles should assume the consequence of ignoring the action.

In the case of recommendation, Rule 2 is applied. For example, if we consider that 30% of teacher agent roles will apply the recommended action Occupy_senior_position under normal conditions, the OOP, which is 0.087516, will be multiplied by 30% (AAN Rate) to obtain the normative action overall probability (Eq. (1)). The same 30% of teacher agent roles must be rewarded by the authority agent role with the same probability (Eq. (3)).

When the normative action is a permission, Rule 3 is applied. The overall occurrence probability of this action remains unchanged. For example, the OOP of the permitted action attend_cours_presentation applied by student agent roles, which is 0.086632, will not be changed.

The resulting operational profile (OP4NorMAS) for course mode in the university system is shown in Tables 12 and 13. Course mode probabilities will change according to the changes we make by applying Rules 1, 2 and 3. For example, the probability of applying the Occupy_senior_position action is reduced, reducing the overall probability of the mode from 0.884 to 0.849. Therefore, we need to normalize OP4NorMAS. The normalized results are shown in Tables 14 and 15.

5. Conclusion

In order to test the reliability of normative multi-agent systems (NorMAS), this paper introduces a methodology for developing an operational profile (OP4NorMAS) for this type of system. The proposed methodology is based on a multi-agent systems operational profile (MAS-OP) [11], and the specification of norms using the NorMAS-ML language [24]. In addition, this methodology is illustrated through a concrete case study of a NorMAS called the University System. This methodology takes into account the specificities of NorMAS, like using role and goal concepts. Furthermore, it distinguishes between the agent and the system levels. In addition, the proposed methodology helps generate the operational profile using the norms identified in the specification phase of the software development.

As future work, we plan to extend the proposed methodology to NorMAS reliability testing. In fact, the generated operational profile of NorMAS can be considered as a cornerstone for generating test cases.

Footnotes

Acknowledgments

This research work was supported by the General Direction of Scientific Research and Technological Development (DGRSDT) of the Algerian Higher Education and Scientific Research Ministry. We would like to thank the DGRSDT for its support in the achievement of this work.

Part of this work is done during an internship at GISMA University of Applied Sciences in Potsdam, Germany. We would thank this university for its support.

Author’s Bios

	Yahia Menassel is an Assistant professor at the Department of Mathematics and Computer Science of Laarbi Tbessi University (Tebessa) in Algeria. He is a member of DISE (Distributed Intelligent Systems Engineering) team at ReLa(CS)2 (Research laboratory on computer science complex systems) Laboratory at Larbi ben M’hidi University (Oum El Bouaghi) in Algeria. His areas of interest include agent-oriented software engineering and multi-agent systems testing.
	Toufik Marir is a senior lecturer at the department of Mathematics and Computer Science of the University of Oum El Bouaghi in Algeria. He holds a university accreditation in Computer Science (Distributed Artificial Intelligence) awarded by Larbi ben M’hidi University (Oum El Bouaghi) in Algeria. Currently, he is the head of ReLa(CS)2 (Research laboratory on computer science complex systems) Laboratory. His main areas of interest include software engineering, multi-agent systems and artificial intelligence applications.
	Farid Mokhati is a full professor of Computer Science at the Department of Mathematics and Computer Science of the University of Oum El-Bouaghi in Algeria. He holds a university accreditation in Computer Science (Distributed Artificial Intelligence) awarded by Badji Mokhtar University (Annaba) in Algeria. Currently, he is the head of DISE (Distributed-Intelligent Systems Engineering) team at ReLa(CS)2 (Research laboratory on computer science complex systems) Laboratory as well as the president of the Scientific Council of the Faculty of Exact Sciences. His main areas of interest include object and agent-oriented software engineering, software quality assurance, software maintenance and embedded software engineering.

References

Koziolek

, Operational Profiles for Software Reliability, in: Seminar “Dependability Engineering”, Oldenburg, Germany, 2005, pp. 1–17.

Neufelder

A.M.

, Ensuring Software Reliability, 1st ed., CRC Press, 1993.

Din

Shen

Han

and Tang

, A Survey on Software Reliability Demonstration, in: IOP Conference Series: Materials Science and Engineering, 1043(3), 2021, p. 032008.

Musa

J.D.

, The operational profile in software reliability engineering: an overview, in: Proceedings Third International Symposium on Software Reliability Engineering, IEEE Computer Society, 1992, pp. 140–154.

Musa

J.D.

, Operational profiles in software-reliability engineering, IEEE Software 10(2) (1993), 14–32.

Muhammad Ali-Shahid

and Sulaiman

, A review of software operational profile in software reliability engineering, in: 8th. Malaysian Software Engineering Conference (MySEC), IEEE, 2014, pp. 1–6.

Yamany

H.E.

and Capretz

M.A.M.

, A Multi-Agent Framework for Building an Automatic Operational Profile, in: Advances and Innovations in Systems, Computing Sciences and Software Engineering, Springer, Dordrecht, 2007, pp. 161–166.

Earle

C.B.

and Fredlund

L.Å.

, A Property-based Testing Framework for Multi-Agent Systems, in: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS ’19), International Foundation for Autonomous Agents and Multiagent Systems, Richland, 2019, pp. 1823–1825.

Machado

R.A.

and Gonçalves

E.M.

, Testing Multiagent Systems Under Organizational Model Moise Using a Test Adequacy Criterion Based on State Transition Path. in: Brazilian Conference on Intelligent Systems (BRACIS 2020), Springer, Cham, 2020, pp. 154–168.

10.

Nascimento

Alencar

Lucena

and Cowan

, A metadata-driven approach for testing self-organizing multiagent systems, IEEE Access 8 (2020), 204256–204267.

11.

Menassel

Marir

and Mokhati

, An Operational Profile for Normative Multi-agent Systems, in: International Conference on Sustainability in Software Engineering & Business Information Management: Innovation & Applications (SSEBIM 2022), University of Applied Sciences and Arts Northwestern Switzerland, Olten, Switzerland, September 23–24, 2022.

12.

Juhlin

B.D.

, Implementing operational profiles to measure system reliability, in: Proceedings of the Third International Symposium on Software Reliability Engineering (ISSRE’02), IEEE Computer Society, 1992, pp. 286–295.

13.

Pant

Franklin

and Everett

, A structured approach to improving software-reliability using operational profiles, in: Proceedings of Annual Reliability and Maintainability Symposium (RAMS), IEEE, 1994, pp. 142–146.

14.

Elbaum

and Narla

, A methodology for operational profile refinement, in: Annual Reliability and Maintainability Symposium, 2001 Proceedings, International Symposium on Product Quality and Integrity (Cat. No.01CH37179), IEEE, 2001, pp. 142–149.

15.

Gittens

Lutfiyya

and Bauer

, An extended operational profile model, in: 15th. International Symposium on Software Reliability Engineering, IEEE, 2004, pp. 314–325.

16.

Shukla

Carrington

and Strooper

P.A.

, Systematic operational profile development for software components, in: 11th. Asia-Pacific Software Engineering Conference, IEEE, 2004, pp. 528–537.

17.

Arora

and Misra

R.B.

, Software reliability improvement through operational profile testing, in: Annual Reliability and Maintainability Symposium, 2005 Proceedings, IEEE, 2005, pp. 621–627.

18.

Kumar

K.S.

and Misra

R.B.

, Software operational profile-based test case allocation using fuzzy logic, International Journal of Automation and Computing 4(4) (2007), 388–395.

19.

J.P.

and Lu

M.Y.

, Develop software operational profile with uniform design, in: 2009 IEEE International Conference on Industrial Engineering and Engineering Management, IEEE, 2009, pp. 842–846.

20.

Smidts

Mutha

Rodriguez

and Gerber

M.J.

, Software testing with an operational profile: OP definition, ACM Computing Surveys (CSUR) 46(3) (2014), 1–39.

21.

Amrita and Yadav

D.K.

, Development of software operational profile, International Journal of Applied Engineering Research 12(22) (2017), 11865–11873.

22.

Cavamura

, Operational Profile and Software Testing: Aligning User Interest and Test Strategy, in: 2019 12th. IEEE Conference on Software Testing, Validation and Verification (ICST), 2019, pp. 492–494.

23.

Fazzolino

and Rodrigues

G.N.

, Feature-Trace: an approach to generate operational profile and to support regression testing from BDD features, in: Proceedings of the XXXIII Brazilian Symposium on Software Engineering (SBES 2019), 2019, pp. 332–336.

24.

Freire

E.S.S.

Cortés

M.I.

Da Rocha Júnior

R.M.

Gonçalves Ênyo

J.T.

De Lima

G.A.C.

, NorMAS-ML: Supporting the modeling of normative multi-agent systems, ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal 8(4) (2019), 49–81.