Home

Introduction

Welcome to the SimDesign wiki! This wiki is for front-end users to add their own example Monte Carlo simulations, potentially linking to published articles which have used SimDesign. As well, some additional examples have been included to highlight different types of simulation experiments, and contains a previous presentation file related to the details of simulations and SimDesign. See also the 2020 presentation by Matt Sigal for related presentation information.

If you are interested in a more in-depth description of the package and its use in real-world simulation experiments refer to Chalmers, R. P. & Adkin, M. C. (2020). Writing effective and reliable Monte Carlo simulations with the SimDesign package. The Quantitative Methods for Psychology, 16(4), 248--280 , For a (somewhat dated) discussion of using SimDesign for statistical pedagogy please refer to Sigal, M. J. & Chalmers, R. P. (2016). Play It Again: Teaching Statistics with Monte Carlo Simulation. Journal of Statistics Education, 24, 136-156.

Simulation Examples with SimDesign

• Power Curve Plot - Simple example of how to build empirical power-curve plots.
• Type I error and Power Rates - Example of how to construct a more realistic Type I error and power analysis simulation. Compares the Welch and Independent t-test when model assumptions are violated.
• Parameter Recovery - Realistic example of how to compare parameter estimates using different estimators. Compares the mirt and lavaan approaches to item factor analysis and how effectively item parameters are recovered.
• Recovering Secondary Estimates - Example of how to use SimDesign to compute secondary estimates that are of interest (e.g., factor scores and latent trait values). Uses the mirt package, particularly the fscores() function.
• Internal parallelization - Example of how to use SimDesign for computerized adaptive testing when parallel computations are distributed within each replication rather than between. Uses the mirtCAT package.
• Coverage - Simple example of how to determine the coverage rate of an estimator. Example demonstrates the coverage of the t and z-distributions when recovering a population mean.
• Missing Data - Example with MCAR and MNAR missing data mechanisms when recovering correlation coefficients.
• Parametric Bootstrap - Example showing how to use the SimDesign package as a Monte Carlo generation workhorse for parametric bootstrap functions.
• Calling External Binary Programs - Example demonstrating how to utilize Mplus within the SimDesign framework using both single and multi-core estimation strategies.

Into the wild

Example code for simulations which have been previously been published or are visible elsewhere.

• Brown, M. B. and Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346), 364--367.
  • This file contains the simulation code rendered through an Rmarkdown file, organized by design-rows, where not applicable combinations are omitted. Alternatively, this file contains the main R script file to be executed, while this file contains the extra user-defined functions required to perform the Monte Carlo simulation. The final simulation results are available in this .rds file
• Kang T, Cohen AS. IRT Model Selection Methods for Dichotomous Items. Applied Psychological Measurement. 2007;31(4):331-358. doi:10.1177/0146621606292213
  • This file contains a partial replication of Kang and Cohen's second simulation (partial since the MCMC results were omitted as MCMC methods have evolved better in the more recent years; e.g., computing true-blue Bayes factors via bridge sampling). The results generally replicate well using the mirt software, however selecting a three-parameter logistic model using the likelihood ratio test and AIC criteria was notably better using mirt than when using the older BILOG-MG software. Convenience functions used in this simulation can be found here
• Nordstokke, D. W., & Zumbo B. D. (2007). A Cautionary Tale About Levene's Tests for Equal Variances. Journal of Educational Research and Policy Studies, 7, 1-14.
  • This file reconstructs the Monte Carlo simulation described. Additionally, the Brown-Forsythe approach is included, mostly because R >= SPSS. The final simulation results are available in this .rds file, while the executable R script file can be found here.
• Flora, D. B. & Curran, P. J. (2004). An Empirical Evaluation of Alternative Methods of Estimation for Confirmatory Factor Analysis With Ordinal Data. Psychological Methods, 9, 466-491
  • This simulation demonstrates an important result for item factor analysis using a limited information estimation method commonly available in SEM software when the model is slightly mis-specified. Specifically, this study investigated the effects of generating non-normal latent traits and how robust these limited information estimators are to this violation. For the executable R script files see here for the main simulation file using SimDesign, while see this file containing the helper functions for generating syntax with lavaan. The final simulation results are available in this .rds file

For-loop issues

The article written by Hallgren, Kevin A. (2013) “Conducting Simulation Studies in the R Programming Environment.” Tutorials in Quantitative Methods for Psychology, 9 (2). NIH Public Access: 43. explained how to run Monte Carlo simulations in R, but unfortunately introduced the topic using the for loop strategy. So, I decided to compare the code to SimDesign's.

• The result was that, when written in SimDesign's framework, the code was not only cleaner and easier to read, manipulate, and manage, but actually ran about 2-6x faster on a single core. Oddly enough, it also demonstrated some of the pitfalls discussed by Sigal and Chalmers (2016) very nicely, and therefore serves as another good comparison between the coding approaches.