Validating the simulation metamodel: Some practical approaches

Abstract

This paper stresses the usefulness of developing a metamodel as an auxilliary model in simulation analysis and emphasizes the importance of validating the metamodel in order to determine whether it accurately approximates the simulation-generated data. A simulation model of the M/M/s queuing system was used for demonstration purposes. Two statistical validation procedures appropriate for simulation metamodels were discussed and demonstrated: use of a holdout sample and double cross- validation. The metamodel developed from the M/M/s queuing system simulation data proved to be valid as an approximation not only to the simulation but also to the real-world system itself.

Keywords

metamodel metamodel validation simulation output statistical validation

Get full access to this article

View all access options for this article.

References

Berk, K.N. "Validating Regression Procedures with New Data." Technometrics 26:4 (1984), 331-338.

Blanning, R.W. "The Sources and Uses of Sensitivity information ." Interfaces 4:4 (1974), 32-38.

Blanning, R.W. "Response to Michel, Kleijnen, and Permut" Interfaces 5:3 (1975), 24-25.

Blanning, R.W. "The Construction and Implementation of Metamodels ." SIMULATION 24:6 (1975), 177-184.

Geoffrion, A.M. "The Purpose of Mathematics is Insight, Not Numbers" Interfaces 7:1 (1976), 81-92.

Ignall, E.J. ; Kolesar, P. , and Walker, W. E "Using Simulation to Develop and Validate Analytic Models: Some Case Studies." Operations Research 26-2 (1978 ), 237-253.

Kleijnen, J. P C "A Comment on Blanning's Metamodel for Sensitivity Analysis: The Regression Metamodel in Simulation." Interfaces 5:3 ( 1975), 21-23.

Kleijnen, J.P.C. "Regression Metamodels for Generalizing Simulation Results." IEEE Transactions on Systems, Man, and Cybernetics SMC-IX: 2 (1979), 93-96.

Kleijnen, J.P.C. "Regression Analysis for Simulation Practitioners ." Journal of The Operational Research Society 32:1 (1981), 35-43.

10.

Kleijnen, J.P.C. ; Van Den Burg, A.J. ; and Van Der Ham , R. Th. " Generalization of Simulation Results." European Journal of Operational Research 3:1 ( 1979), 50-64.

11.

Lawless, R.W. ; Williams, L.H. ; and Richie, C.O. "A Sensitivity Analysis Tool for Simulation with Application to Disaster Planning." SIMULATION 17:6 (1971), 217-223.

12.

Montgomery, D.C. and Peck, E.A. Introduction to Linear Regression Analysis Wiley and Sons, New York, N.Y. (1982).

13.

Neter, J. ; Wasserman, W. ; and Kutner, M.H. Applied Linear Regression Models. Irwin, Homewood, III. (1983).