Agile and Value Creation in Agent-Based Social Simulation

Abstract

Agile software development is having a profound impact on the software industry. Agent-Based Social Simulation (ABSS) has led a paradigm shift in the way social scientists understand and manipulate complex systems. However, little is known about the relationship between the two and how they can create meaningful synergies. A review of the evidence available on the connection agile-ABSS was conducted. The review investigates what is currently known about agile adoption by ABSS and the value and limitations that can be derived from the connection. The main implication for researchers is the need to implement agile methods in a greater number of ABSS projects, on which to support further research and experimentation. For practitioners, the review raises awareness on the topic and reveals that there is a long road ahead of challenges and opportunities for those willing to unlock value from ABSS. For both, it would be highly beneficial if they joined forces and agreed on a common research agenda.

Keywords

Agile agent-based modelling value creation social science software development

Introduction

The use of Agent-Based Social Simulation (ABSS) has grown to the point of being recognized as a third way of doing science and expressing theories, in contrast with the two ‘standard’ scientific inquiry methods: deduction and induction (Axelrod, 2005; Gilbert, 1999; Hummon & Fararo, 1995). ABSS’s freedom of design, high-level of flexibility, and intuitiveness provide a compelling approach to tackle real-life problems and operate on highly complex systems (Omicini et al., 2000; Schweitzer, 2002). Many social challenges, seemingly intractable until now, have started to sketch an answer through ABSS, favoured by a substantial increase in data availability and data processing capacity.

Notwithstanding the above, there are several threats hanging over ABSS’s adoption, partly due to a concomitant lack of robust software development processes and standardization. Hence, for ABSS to deliver on its promises, it needs to be engineered with quality and rigour. It is in this context that agile software development tools can create value for ABSS. However, further investigation is needed on the adoption of agile methods within ABSS, as well as on the benefits and limitations that their joint use can provide.

This review seeks to collect evidence on the current state of adoption of agile within ABSS, and how agile can contribute to achieve greater recognition of agent-based modelling and simulation in the social sciences. The results of the investigation will be relevant to people in the research community and the software industry who want to identify shortfalls and existing challenges for improving ABSS applications and increase their adoption.

The paper is organized as follows. Section ‘Background’ gives an overview of the background and context of ABSS and agile methods. The research questions to be addressed are stated in this section too. Section ‘Method’ outlines the steps followed in the research. Section ‘Results’ details the main results achieved. Section ‘Discussion’ examines the current level of adoption of agile, its benefits and limitations, and key implications for researchers and practitioners. Section ‘Conclusions’ presents the final conclusions.

Background

Both agile and ABSS are transformative frameworks that have led to a paradigm shift in their respective fields of practice. As such, they have shaken the way of thinking and working of the software development and social science research communities, respectively. However, little is known about the relationship between the two and how they can create meaningful synergies.

This section describes the context and core ideas behind ABSS, particularly within the broader field of computational social science (CSS). Then, an introduction to agile software development methods is made that lays the groundwork for further discussion on improving ABSS development processes and adoption.

ABSS and Computational Social Science

ABSS is the flagship toolkit of CSS, a fast-growing field that seeks to nurture our understanding of real-life social systems and processes (Gilbert, 2007) by building artificial computational worlds that we can feed with data, engineer, scrutinize, and test against empirical data (Conte et al., 2012). CSS involves a new kind of scientific inquiry process that starts from a representation (model) of social systems as information-processing organizations, from which a new type of data, made available through information and communications technology (ICT) applications, is used to simulate social phenomena (Cioffi-Revilla, 2010).

CSS specifically focuses on the entities that form a real socio-economic system (e.g., individuals, people, institutions, societies), as well as their links within the system. These entities, called agents, are then modelled as computational objects that encompass autonomous dynamics and traits that allow them to interact with a preset degree of freedom, and so to adapt to a changing environment (Borrill & Tesfatsion, 2010).

The resulting ‘agentification’ of social systems is thereby devised through agent-based models, aiming to both explain emergent and dynamical phenomena and predict their evolution through data-driven simulations (Cioffi-Revilla, 2010; Gilbert & Troitzsch, 2005). By abstracting the agents’ behaviour into a virtual environment, making it operational, and speeding the way in which we test our theories and ideas (Conte & Paolucci, 2014; Jackson, 2014), ABSS essentially comes to fulfil the main goals pursued by CSS.

Nonetheless, the outcomes from ABSS remain modest to date and have contributed little outside the academy (Lucas, 2009), despite the growing number of stakeholders and policymakers who are calling for more effective and practical guidance from social simulation. This further highlights several software development challenges that constrain ABSS’s ability to deliver on its promises (Siebers & Klügl, 2017).

Agile Software Development

The 12 principles of the agile manifesto (Beck et al., 2001) articulate a set of practices that prioritize the implication of the customer in the software development process, promote simplicity, and proactively embrace change. These principles essentially differ from ‘traditionalist’ (aka, heavy-weight) software development approaches, which emphasize extensive planning, codified processes, extensive documentation, and rigorous reuse (Dybå & Dingsøyr, 2009; Meso & Jain, 2006). By contrast, agile methods believe in the power of iterative development cycles aimed at continuously releasing software products that are submitted to the evaluation, feedback, and approval of the customer (Dybå & Dingsøyr, 2008).

Since the introduction of Extreme Programming (XP) as the first agile software development approach in 1999, agile has grown to become mainstream within software developers. Many other methods have followed, including Scrum, the Crystal family of methodologies, Feature-Driven Development, the Rational Unified Process, Dynamic Systems Development, Adaptive Software Development, and open source development (Abrahamsson et al., 2017).

Agile principles and methods seem to be particularly well suited to address the challenges of our unpredictable world, to tackle complexity, and to fit the needs of (social) computational models (Jackson, 2014). Furthermore, the introduction of agile practices involves a value creation process for software development (Neto et al., 2019; Racheva et al., 2009). It is thus evident that for the ABSS adoption to grow, there will be a need for improved software development methodologies that are more adaptable and can provide a quick response to changing uncertainties and requirements.

Research Questions

Despite the abundant extant literature on the topics of agile methods and of agent-based modelling (ABM), and the large number of studies showing how they have respectively impacted the software and social research communities to the point of causing a paradigm shift in their respective domains, no review combining the two approaches has been previously published.

This paper is intended to be useful to both researchers and practitioners, as to clarify some assumptions on the adoption of agile within ABSS and support them with scientific evidence. With this scope in mind, the purpose of this review is to answer the following two research questions:

To what extent have agile methods been adopted within ABSS development?

What benefits and limitations do agile methods have for ABSS?

In addition to gathering evidence on agile within ABSS, this review also aims to provide guidance to researchers and practitioners on the ways to improve the quality and rigour of ABSS software development, therefore helping to meet the growing expectations it has generated among stakeholders and policymakers (Lucas, 2009).

Method

Based on the methodological framework provided by systematic reviews in the social sciences and the recommendations and procedures set forth by Petticrew and Roberts (2006), the author carried out a literature review distributed in the following phases: (a) review protocol, (b) inclusion and exclusion criteria, (c) sources and search strategy, (d) critical appraisal, (e) data extraction, and (f) synthesis of results.

The review protocol set out the process and methods to be applied throughout the review, the description of and the rationale for the review questions, and the specifics on how different types of evidence were to be found, managed, assessed, and synthesized.

The inclusion and exclusion criteria determined the types of studies found in the literature search that were eligible for an in-depth review, and those that were excluded. Studies were eligible if they provided evidence of the adoption of an agile software development method within ABSS and had a minimum quality threshold. Inclusion of studies was not circumscribed to any specific type of publication, journal, conference, or publisher. Both qualitative and quantitative studies were considered in the review, published from the year 2000 onwards and written in English.

The search strategy covered electronic databases and hand searches on the main peer-reviewed specialized journals. The following electronic databases were searched:

IEEE Xplore

ACM Digital Library

Web of Science

ScienceDirect

SpringerLink

Google Scholar

The selection of studies was conducted in four consecutive stages (see Figure 1). In stage 1, candidate studies were searched through the electronic databases and specialized journals using the following search terms:

Agent-based AND agile

Agent-based AND (extreme programming OR XP)

Agent-based AND scrum

Agent-based AND (features driven development OR fdd)

Agent-based AND (UML OR unified modelling language)

It should be noted that UML, though not a declared agile method, is a de-facto standard in agile modelling, and a technique closely related to efficient agile modelling design, visualization, and testing (Braude, 2017; Rumpe, 2017).

A second search was carried out for studies containing in their title the terms ‘agent’, ‘modelling’ or ‘simulation’, plus the name of an ABM platform, plus a reference to an agile method in its content. The search string was as follows:

TITLE = ((agent OR modeling OR simulation) AND (Netlogo OR Repast OR Gama OR SODA OR Gaia OR Mason OR Swarm OR Starlogo OR Anylogic OR Obeus OR Simagent OR Agentsheet)) AND ALL=(“agile” OR “extreme programming” OR “xp” OR “scrum” OR “features driven development” OR “fdd” OR “unified modeling language” OR “uml”)

This ‘search by term’ strategy provided 649 results, which resulted in 553 unduplicated and error-free citations. Excluded from the search were results obtained from reviews, news, workshops, panels and editorials.

Figure 1.

Source: Own elaboration.

Table 1.

Quality Assessment Criteria

C1. How reliable/credible are the findings?

C2. How defensible is the research design?

C3. How well defended is the sample design/target selection of cases/documents?

C4. How well was the data collection carried out?

C5. How well has the approach to and formulation of the analysis been conveyed?

C6. How well have detail, depth, and complexity (i.e., richness) of the data been conveyed?

C7. How clear are the links between data, interpretation and conclusions?

Source: Own elaboration.

After stage 4 of the search process, there were left 12 studies that provided evidence on agile software development, agent-based modelling and simulation, and the social sciences. These references went into the next stage of the review which consisted of a quality assessment.

The objective of the quality assessment was to determine if any of the individual references was affected by bias as to render it useless to answer the research questions or to make any valuable contribution to the review. Each study was assessed according to seven quality criteria (Table 1), chosen among the criteria contained in the framework for assessing qualitative evaluations of the UK’s Government Chief Social Researcher’s Office (Spencer et al., 2004). After this evaluation, one study was excluded, leaving 11 studies qualified for data extraction and synthesis.

It was then that the author proceeded to extract data from the 11 studies meeting the inclusion and quality criteria. A narrative method was used to synthesize the data extracted from the studies, as they were too heterogeneous to enable a quantitative exploration (meta-analysis) of the findings. Finally, given the heterogeneity of the studies involved, a conceptual categorization was made that enabled to group the studies into affinity clusters for ease of comparison.

Results

An in-depth review was conducted of 11 studies focused on the adoption of agile software development methods within ABSS. The studies were categorized into three main groups: (a) conceptual applications, (b) survey /literature reviews, and (c) case studies. An overview of the extracted data is provided below, along with the quality assessment. Next, a description of the evidence on the adoption of agile development methods within ABSS is presented.

Overview of Studies

The agile conceptual applications found in the studies were highly diverse, including economics and finance, alcohol consumption, environmental and urban planning, and politics. The remaining studies were survey/literature reviews (3), and one case study.

An overview of the studies reviewed according to their source of publication shows that the largest number of studies (64%) were publications in scientific journals, followed by conferences (18%) and books (18%), with two studies each. With regards to the year of publication, no study was reviewed before 2004, but an increase from this date is noted with one study in 2004, two in 2007, one in 2009, one in 2011, two in 2012, three in 2015, and one in 2017.

Quality Assessment

The quality criteria evaluated the outcomes of each study, as well as the main stages of the research process performed. A summary of the criteria used to assess the quality of the studies is presented in Table 1.

The seven criteria were assessed on a dichotomous scale, whereby ‘1’ meant ‘yes’ (the study met the criteria) and ‘0’ meant ‘no’ (the study did not meet the criteria). Note that only criterion 1 was used for including or excluding a study. The results of the critical appraisal are shown in Table 2.

Only four studies [R2, R4, R6 and R9] got a full score (7/7) on the quality assessment. Four studies scored 6/7, and three of them got three negative answers to a final score of 4/7.

Adoption of Agile Methods

Seven studies (64%) mentioned UML or some of its variants (e.g., AgentUML), as either one of or the only agile-related technique used in ABSS projects. The reported use of generic agile methods was next (27%), followed by XP (18%). Only one study reported the implementation of Scrum.

Table 2.

Quality Assessment

Study	C1	C2	C3	C4	C5	C6	C7	Total Score
R1	1	1	1	1	1	0	1	6
R2	1	1	1	1	1	1	1	7
R3	1	1	0	0	1	0	1	4
R4	1	1	1	1	1	1	1	7
R5	1	1	1	1	1	0	1	6
R6	1	1	1	1	1	1	1	7
R7	1	1	0	0	1	0	1	4
R8	1	1	1	1	1	0	1	6
R9	1	1	1	1	1	1	1	7
R10	1	1	1	1	1	0	1	6
R11	1	1	0	0	1	0	1	4

Source: Own elaboration.

Few studies informed the adoption of more than one agile method. This is the case of Moyo et al.’s alcohol consumption study [R4], which reported the use of Scrum to organize the workflow and guide the development process, in conjunction with UML as a modelling tool to describe the function of the model and conceptualize abstract representations of the environment.

Other more heterogeneous approaches, as in Nachamai et al. [R7], show the potential offered by XP and agent-oriented software engineering (AOSE) methodologies like Tropos. Vermeir and Bersini [R6] and Bersini [R7] address the integration of UML and object-oriented modelling (OOM). Klügl [R11] presents a more hybrid approach where UML integrates with the ODD (Overview, Design and Details) protocol and the iterative development method provided by Pattern-Oriented Modelling.

Discussion

In this section the research questions are addressed, beginning first with the current status of agile adoption within ABSS. Next, an analysis of the benefits and limitations that agile software development may have for ABSS is provided. Finally, a discussion is drawn on the implications for scholars and practitioners, and on the limitations of this review.

Adoption of Agile by ABSS

To draw an informed conclusion on the current status of adoption of agile methods within ABSS, a combination of the following key metrics was analysed on the basis of the evidence reviewed: (a) the number of agile implementations, (b) quality, (c) consistency, and (d) directedness. The number of reported implementations shows that only 7 of the 11 studies reviewed were real-world agile implementations within ABSS. The remaining studies were indirect references to the potential benefits of agile for ABSS, or conceptual approaches to the use of these techniques. Consequently, it can be argued that, based on the total evidence found, the number of agile implementations within ABSS is low.

Likewise, the analysis of the quality, consistency, and directedness metrics reinforced the thesis about the low level of implementation of agile methods within ABSS. Therefore, no decisive conclusions can be drawn, let alone relevant practical guidance, on the agile adoption process in an academic or professional setting.

Benefits and Limitations of Agile for ABSS

The reviewed studies addressing the adoption of agile provide a succinct overview of very few, but key methods used in ABSS.

UML is widely reported as the most agile-related adopted method within ABSS, to the point of having become the standard for graphical visualization of ABSS modelling development. UML encourages researchers to spend more time on modelling rather than on coding, which means that more effective progress towards algorithmic problem solving can be achieved within ABSS.

Among the more general agile development practices reported, iterative development cycles are gaining momentum within ABSS teams. Iterative cycles provide with the capacity to evolve ABSS from simple models to highly complex (more realistic) ones, as well as to extend the scale of analysis from local to global as more properties from the real-world are added (Gonzalez-Palacios & Luck, 2007).

There are studies that reported potential benefits from the implementation of Extreme Programming (XP), like short development cycles, pair programming, and test-driven development; as well as from Scrum and its project management-oriented strategies (Moyo et al., 2015; Nachamai et al., 2011; Siebers & Davidsson, 2015).

With regards to human factors, few of the studies provided a detailed glimpse of the way agile had contributed to structure ABSS teams. Given that most ABSS were developed in academic settings with very few team members with clearly defined competencies (e.g., domain and computational experts), it seems that otherwise software development critical tasks such as how teams are formed, who should compose them, or how tasks should be distributed, were often overlooked. In this line, Moyo et al. (2015) identify developers’ low experience and little knowledge of software development techniques as a major drawback for modelling and simulation in academia.

It seems appropriate that the ABSS community becomes fully aware that software engineering (SE) techniques are a key prerequisite for running successful (ABSS) software projects (Siebers & Davidsson, 2015). An important step towards a sound methodology and more professional SE practices might come from embracing existing standards that are relevant, before pushing to develop new and more specific ones (Bersini, 2012). The adoption of more standard practices would render more readable, scalable and usable software, thus contributing to overcome some of the limiting factors affecting ABSS growth today. The adoption of open source and sharing practices through community platforms are some other tools that could accelerate standardization and boost ABSS.

Agile development methods seem to be particularly well suited for pushing ABSS in the right direction, even more so if they find the right way to integrate with techniques such as agent-oriented software engineering (AOSE), multi-agent systems, and advanced object-oriented programming, to create long-lasting synergies. Finding effective ways for users to play an active role in defining meaningful requirements through, for instance, user stories (Lucassen et al., 2017; Wautelet et al., 2017), can also become a relevant success factor of agile adoption by ABSS. However, it should be noted that the path for ABSS improvement is not without difficulties, as drawbacks lurk along the way, mainly those concerned with the applicability and comprehensiveness of agent-based development methods.

Implications for Research and Practice

This review contains several implications for researchers and practitioners. For researchers, further empirical studies on the agile-ABSS connections are needed, as agile is having a profound impact on the software industry, which in turn has not been accompanied by the same interest in the ABSS development community. Furthermore, as few but highly efficient results-oriented steps can usually foster big mindset changes, ABSS development teams might start experimenting with some of the most well-established agile practices (e.g., user stories, test-driven development, short cycles, early demos deliveries) to learn and improve fast.

A special point of interest would also be to study how agile methods can join and integrate with AOSE practices, particularly in highly complex and distributed systems. Furthermore, if we want to leverage on the potential benefits that agile methods may have for ABSS, we need to devote more attention to the human factors behind ABSS projects.

For practitioners, this review shows that we are dealing with an emerging area of knowledge and practice that will significantly influence the next generation of research methods in the social sciences. Notwithstanding the foregoing, in order to unlock greater value from ABSS, it shall be desirable that more ABSS projects adopt agile methods. Only in this way will it be possible to unravel the usefulness of the ABSS-agile relationship and provide evidence to address further research questions. Hence, a common research agenda that bridges the gap between those who benefit/commission ABSS and those who develop them seems to be a priority. Hopefully, this review will have contributed to raising awareness of the opportunity ahead.

Limitations of This Review

Among the main limitations of this review are the potential bias introduced by the author in the selection of studies and the errors or omissions that could be made when extracting the data. There may also be inaccuracy in the synthesis of the information and bias in weighting some conclusions over others.

To help minimize bias in the studies selection process, a review protocol was established, and the research questions were posed beforehand. The research questions were the basis for determining the keywords used in the search strategy for relevant literature. However, there is a risk that the selection of keywords left out other terms that could have been valuable for the investigation. Inclusion and exclusion criteria were also provided before starting the search process and assessing the references. Last but not least, a critical appraisal of the selected studies was carried out based on seven quality criteria. In this way, the importance given to the interpretation of studies’ outcomes was weighted according to a quality score.

Conclusions

There are a number of reviews in the literature dealing with agile methods from different perspectives. Most of them feature practitioners’ success stories, lessons learned, and case studies, in which the issue of bias receives little attention. In contrast, this review is the first known to the author that investigates the connections between agile methods and ABSS development, and provides scientific evidence drawn from the literature.

It became apparent that the adoption of agile methods by ABSS development teams can provide a more robust, systematic, and productive way of undertaking software development processes. Some limitations were also identified that could hinder the adoption of agile within ABSS, more notably in academic development environments. However, the amount and strength of evidence found was strikingly low, making it difficult to offer specific guidance for ABSS developers in the industry and academia. Therefore, the true value of this article lies in highlighting some important gaps that exist in the current development practices of ABSS, in contributing to the debate on the importance of implementing proven and professional software development techniques, and in stimulating the comparison between current ABSS developments and the shortcomings detected in this research.

A conclusive finding from this review is that before proceeding with further and deeper research into the role of agile within ABSS, it would be highly beneficial to conduct an awareness-raising process, mainly in academic circles, to strengthen the advocacy of the benefits of adopting software development methods and, more specifically, agile methods. In this way, and after having implemented a reasonable number of ABSS demonstration projects where agile had been tested, then further viable research could be carried out. In this context of insufficient evidence, but also of great opportunities for the growth of ABSS, there is a clear need to establish an agenda for new agile deployments as more and better field studies will be developed using ABSS.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author received no financial support for the research, authorship and/or publication of this article.

Appendix A: Studies Included in the Review

ORCID iD

Francisco J. Navarro-Meneses

References

Abrahamsson

Salo

Ronkainen

Warsta

(2017). Agile software development methods: Review and analysis. ArXiv Preprint ArXiv:1709.08439 .

Axelrod

(1997). Advancing the art of simulation in the social sciences. In Simulating social phenomena (pp. 21–40). Springer.

Beck

Beedle

Van Bennekum

Cockburn

Cunningham

Fowler

Grenning

Highsmith

Hunt

Jeffries

(2001). Manifesto for agile software development . https://agilemanifesto.org/

Bersini

(2012). UML for ABM. Journal of Artificial Societies and Social Simulation , 15(1), 9.

Borrill

P. L.

Tesfatsion

(2010). Agent-based modeling: The right mathematics for the social sciences? Working Paper No. 10023. https://doi.org/10.4337/9780857938077.00018

Braude

(2017). Incremental UML for agile development: Embedding UML class models in source code. 2017 IEEE/ACM 3rd International Workshop on Rapid Continuous Software Engineering (RCoSE) , pp. 27–31.

Cioffi-Revilla

(2010). Computational social science. Wiley Interdisciplinary Reviews: Computational Statistics , 2(3), 259–271. https://doi.org/10.1002/wics.95

Conte

Gilbert

Bonelli

Cioffi-Revilla

Deffuant

Kertesz

Loreto

Moat

Nadal

J.-P.

Sanchez

(2012). Manifesto of computational social science. The European Physical Journal Special Topics , 214(1), 325–346.

Conte

Paolucci

(2014). On agent-based modeling and computational social science. Frontiers in Psychology , 5, 668.

10.

Dybå

Dingsøyr

(2008). Empirical studies of agile software development: A systematic review. Information and Software Technology , 50(9–10), 833–859.

11.

Dybå

Dingsøyr

(2009). What do we know about agile software development? IEEE Software , 26(5), 6–9.

12.

Gilbert

(1999). Simulation: A new way of doing social science. American Behavioral Scientist , 42(10), 1485–1487.

13.

Gilbert

(2007). Computational social science: Agent-based social simulation . Centre for Research on Social Simulation.

14.

Gilbert

Troitzsch

K. G.

(2005). Simulation for the social scientist . McGraw-Hill Education.

15.

Gonzalez-Palacios

Luck

(2007). Extending gaia with agent design and iterative development. International Workshop on Agent-Oriented Software Engineering , pp. 16–30.

16.

Hummon

N. P.

Fararo

T. J.

(1995). The emergence of computational sociology. The Journal of Mathematical Sociology , 20(2–3), 79–87. https://doi.org/10.1080/0022250X.1995.9990155

17.

Jackson

P. J.

(2014). Software solutions for computational modelling in the social sciences. In Theories and simulations of complex social systems (pp. 5–17). Springer.

18.

Lucas

(2009). Usefulness of simulating social phenomena. In Adaptive and emergent behaviour and complex systems. Proceedings of the 23rd Convention of the Society for the Study of Artificial Intelligence and Simulation of Behaviour (pp. 7–12). AISB 2009.

19.

Lucassen

Robeer

Dalpiaz

van der Werf

M. J. M. E.

Brinkkemper

(2017). Extracting conceptual models from user stories with Visual Narrator. Requirements Engineering , 22(3), 339–358.

20.

Meso

Jain

(2006). Agile software development: Adaptive systems principles and best practices. Information Systems Management , 23(3), 19–30.

21.

Moyo

Ally

A. K.

Brennan

Norman

Purshouse

R. C.

Strong

(2015). Agile development of an attitude-behaviour driven simulation of alcohol consumption dynamics. Journal of Artificial Societies and Social Simulation , 18(3), 10.

22.

Nachamai

Vadivu

M. S.

Tapaskar

(2011). Enacted software development process based on agile and agent methodologies. International Journal of Engineering Science and Technology , 3(11).

23.

Neto

G. T. G.

Santos

W. B.

Fagundes

R. A. A.

Margaria

(2019). Towards an understanding of value creation in agile software development. Proceedings of the XV Brazilian Symposium on Information Systems , pp. 1–8.

24.

Omicini

Tolksdorf

Zambonelli

(2000). Engineering societies in the agents world. First International Workshop, ESAW 2000 . https://books.google.es/books?id=c6RqCQAAQBAJ

25.

Petticrew

Roberts

(2006). Systematic reviews in the social sciences: A practical guide . John Wiley & Sons.

26.

Racheva

Daneva

Sikkel

(2009). Value creation by agile projects: Methodology or mystery? International Conference on Product-Focused Software Process Improvement , pp. 141–155.

27.

Rumpe

(2017). Agile modeling with UML: Code generation, testing, refactoring . Springer.

28.

Schweitzer

(2002). Modeling complexity in economic and social systems . World Scientific Publishing Company.

29.

Siebers

P.-O.

Davidsson

(2015). Engineering agent-based social simulations: An introduction. Journal of Artificial Societies and Social Simulation , 18(3), 13.

30.

Siebers

P.-O.

Klügl

(2017). What software engineering has to offer to agent-based social simulation. In Simulating social complexity (pp. 81–117). Springer.

31.

Spencer

Ritchie

Lewis

Dillon

(2003). Quality in qualitative evaluation: A framework for assessing research evidence . UK Government Social Research Profession and Cabinet Office.

32.

Wautelet

Heng

Kiv

Kolp

(2017). User-story driven development of multi-agent systems: A process fragment for agile methods. Computer Languages, Systems & Structures , 50, 159–176.