The verification and validation (V&V) of agent-based models (ABMs) is challenging. The underlying structure of the model and the agents can change over time. Furthermore, the theoretical context of the model is often very different from established models of the same phenomenon. In an effort to overcome these issues, trace validation is becoming a common V&V mechanism within the agent-based modeling community. In trace validation, characteristics of agents and the model are tracked over time and then analyzed by subject matter experts (SMEs) to gain insight into unexpected and potentially invalid output. Here, we present our tool, the V&V Calculator, which applies predicates employed in the field of software engineering. The result is a structured trace validation approach with quantifiable measures that facilitates SME exploration and insight into the causes of unexpected output within ABMs.
BonabeauE.Agent-based modeling: methods and techniques for simulating human systems. PNAS2002; 99: 7280–7287.
3.
MacalCMNorthMJ.Tutorial on agent-based modelling and simulation. J Simul2010; 4: 151–162.
4.
TakadamaKKawaiTKoyamaY.Micro- and macro-level validation in agent-based simulation: reproduction of human-like behaviors and thinking in a sequential bargaining game. J Artif Soc Soc Simul2008; 11(2): 1–17.
5.
BalciO.Verification, validation and testing. In: BanksJ (ed) Handbook of simulation: principles, methodology, advances, applications, and practice. Hoboken, NJ: John Wiley & Sons, Inc., 1998, pp.335–393.
6.
BharathyGKSilvermanBG.Validating agent based social systems models. In: Proceedings of the winter simulation conference (ed JohanssonBJainSMontoya-TorresJ), Baltimore, MD, 5–8 December 2010, pp.441–453. Piscataway, NJ: IEEE.
7.
SargentRG. Verifying and validating simulation models. In: Proceedings of the winter simulation conference (ed TolkADialloSYRyzhovIO.), Savannah, GA, 7–10 December 2014, pp.118–131. Piscataway, NJ: IEEE.
8.
XiangXKennedyRMadeyG. Verification and validation of agent-based scientific simulation models. In: Spring Simulation Multiconference (SpringSim’05), Vista, CA, 3–7 April 2005, pp.47–55. IEEE.
9.
GoreRReynoldsPFJrKamenskyD.Statistical debugging with elastic predicates. In: Proceedings of the 26th IEEE/ACM international conference on automated software engineering, Lawrence, KS, 6–10 November 2011, pp.492–495. Piscataway, NJ: IEEE.
10.
WhitnerRBBalciO. Guidelines for selecting and using simulation model verification techniques. In: Proceedings of the winter simulation conference (ed MacNairEMusselmanKHeidelbergerP), Washington, D.C., 4–6 December 1989, pp.559–568. Piscataway, NJ: IEEE.
11.
SargentRG. An overview of verification and validation of simulation models. In: Proceedings of the winter simulation conference (ed ThesenAGrantHKeltonWD), Atlanta, GA, 14–16 December 1987, pp.33–39. New York: ACM.
12.
SargentRG. Verification, validation and accreditation of simulation models. In: Proceedings of the winter simulation conference (ed FishwickPKangKJoinesJ.), Orlando, FL, 10–13 December 2000, pp.50–59. Piscataway, NJ: SCS.
13.
NorthMJMacalCM.Managing business complexity: discovering strategic solutions with agent-based modeling and simulation. New York: Oxford University Press, 2007.
14.
WindrumPFagioloGMonetaA.Empirical validation of agent-based models: alternatives and prospects. J Artif Soc Soc Simul2007; 10(2): 1–19.
15.
BagniRBerchiRCarielloP.A comparison of simulation models applied to epidemics. J Artif Soc Soc Simul2002; 5(3): 1–18.
16.
XuCHeHSHuY. Latin hypercube sampling and geostatistical modeling of spatial uncertainty in a spatially explicit forest landscape model simulation. Ecol Modell2005; 185: 255–269.
17.
EpsteinJMAxtellRL.Growing artificial societies: social science from the bottom up. Washington, DC: Brookings Institution Press, 1996.
18.
CourdierRGuerrinFAndriamasinoroFH. Agent-based simulation of complex systems: application to collective management of animal wastes. J Artif Soc Soc Simul2002; 5(3): 1–17.
GoreRDialloS. The need for usable formal methods in verification and validation. In: Proceedings of the winter simulation conference (ed PasupathyRKimSTolkA.), Washington, DC, 8–11 December 2013, pp.1257–1268. Piscataway, NJ: IEEE.
21.
GoreRReynoldsPFJrKamenskyD. Statistical debugging for simulations. ACM Trans Model Comput Simul2015; 25(3): 1–26.
22.
NainarPAChenTRosinJ. Statistical debugging using compound Boolean predicates. In: Proceedings of the 2007 international symposium on software testing and analysis, London, UK, 9–12 July, 2007, pp.5–15. New York: ACM.
23.
LiMYMuldowneyJS.Global stability for the SEIR model in epidemiology. Math Biosci1995; 125: 155–164.
24.
DunhamJB.An agent-based spatially explicit epidemiological model in MASON. J Artif Soc Soc Simul2005; 9(1): 231–244.
25.
GoreRReynoldsPF. Applying causal inference to understand emergent behavior. In: Proceedings of the winter simulation conference (ed MasonSHillRMonchL.), Miami, FL, 8–10 December 2008, pp.712–721. Piscataway, NJ: IEEE.
26.
CioppaTMLucasTW.Efficient nearly orthogonal and space-filling Latin hypercubes. Technometrics2007; 49(1): 45–55.
27.
ReynoldsCW.Flocks, herds and schools: a distributed behavioral model. ACM SIGGRAPH Comput Graph1987; 21(4): 25–34.
28.
GilbertN.Agent-based social simulation: dealing with complexity. Complex Syst Netw Excell2004; 9(25): 1–14.
29.
EpsteinJM.Generative social science: studies in agent-based computational modeling. Princeton, NJ: Princeton University Press, 2006.
30.
SalgeCPolaniD.Digested information as an information theoretic motivation for social interaction. J Artif Soc Soc Simul2011; 14(1): 1–18.
31.
QueraVHerrandoSBeltranFS. An index for quantifying flocking behavior. Percept Mot Skills2007; 105: 977–987.
32.
BajecILZimicNMrazM.The computational beauty of flocking: boids revisited. Math Comput Model Dyn Syst2007; 13: 331–347.
33.
SchellingTC.Micromotives and macrobehavior. New York: WW Norton & Company, 2006.
34.
HatnaEBenensonI.The Schelling model of ethnic residential dynamics: beyond the integrated–segregated dichotomy of patterns. J Artif Soc Soc Simul2012; 15(1): 1–13.
35.
HegselmannRThomasC.Schelling and the computer: Some notes on Schelling’s essay “on letting a computer help with the work”. J Artif Soc Soc Simul2012; 15(4): 1–6.
36.
RossiterSNobleJBellKRW. Social simulations: improving interdisciplinary understanding of scientific positioning and validity. J Artif Soc Soc Simul2010; 13(1): 1–29.
37.
HammondRA.Complex systems modeling for obesity research. Prev Chronic Dis2009; 6(3): 1–10.
38.
NishinariKSugawaraKKazamaT. Modelling of self-driven particles: foraging ants and pedestrians. Phys A Stat Mech Appl2006; 372: 132–141.
39.
WiemanCEAdamsWKPerkinsKK.PhET: simulations that enhance learning. Science2008; 322: 682–683.
40.
VicsekTCzirkABen-JacobE. Novel type of phase transition in a system of self-driven particles. Phys Rev Lett1995; 75: 1226–1229.
41.
HuepeCAldanaM.Intermittency and clustering in a system of self-driven particles. Phys Rev Lett2004; 92: 168701.
42.
JonesAHarroldMJ. Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE/ACM international conference on automated software engineering. Long Beach, CA, 7–11 November, 2005, pp.273–282. New York, NY: ACM.
