Home

Introduction

Welcome to the SimDesign wiki! This wiki is for front-end users to add their own example Monte Carlo simulations, potentially linking to published articles which have used SimDesign. As well, some additional examples have been included to highlight different types of simulation experiments, and contains a previous presentation file related to the details of simulations and SimDesign.

If you are interested in a more in-depth description of the package and its use in simulations and teaching, see Sigal, M. J. & Chalmers, R. P. (2016). Play It Again: Teaching Statistics with Monte Carlo Simulation. Journal of Statistics Education, 24, 136-156.

Simulation Examples with SimDesign

• Power Curve Plot - Simple example of how to build empirical power-curve plots.
• Type I error and Power Rates - Example of how to construct a more realistic Type I error and power analysis simulation. Compares the Welch and Independent t-test when model assumptions are violated.
• Parameter Recovery - Realistic example of how to compare parameter estimates using different estimators. Compares the mirt and lavaan approaches to item factor analysis and how effectively item parameters are recovered.
• Recovering Secondary Estimates - Example of how to use SimDesign to compute secondary estimates that are of interest (e.g., factor scores and latent trait values). Uses the mirt package, particularly the fscores() function.
• Internal parallelization - Example of how to use SimDesign for computerized adaptive testing when parallel computations are distributed within each replication rather than between. Uses the mirtCAT package.
• Coverage - Simple example of how to determine the coverage rate of an estimator. Example demonstrates the coverage of the t and z-distributions when recovering a population mean.
• Missing Data - Example with MCAR and MNAR missing data mechanisms when recovering correlation coefficients.
• Parametric Bootstrap - Example showing how to use the SimDesign package as a Monte Carlo generation workhorse for parametric bootstrap functions.
• Calling External Binary Programs - Example demonstrating how to utilize Mplus within the SimDesign framework using both single and multi-core estimation strategies.

Into the wild

Example code for simulations which have been previously been published or are visible elsewhere.

• Brown, M. B. and Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346), 364--367.
  • This file contains the simulation code rendered through an Rmarkdown file, organized by design-rows, where not applicable combinations are omitted. Alternatively, this file contains the main R script file to be executed, while this file contains the extra user-defined functions required to perform the Monte Carlo simulation. The final simulation results are available in this .rds file
• Nordstokke, D. W., & Zumbo B. D. (2007). A Cautionary Tale About Levene's Tests for Equal Variances. Journal of Educational Research and Policy Studies, 7, 1-14.
  • This file reconstructs the Monte Carlo simulation described. Additionally, the Brown-Forsythe approach is included, mostly because R >= SPSS. The final simulation results are available in this .rds file, while the executable R script file can be found here.
• Flora, D. B. & Curran, P. J. (2004). An Empirical Evaluation of Alternative Methods of Estimation for Confirmatory Factor Analysis With Ordinal Data. Psychological Methods, 9, 466-491
  • This simulation demonstrates an important result for item factor analysis using a limited information estimation method commonly available in SEM software when the model is slightly mis-specified. Specifically, this study investigated the effects of generating non-normal latent traits and how robust these limited information estimators are to this violation. For the executable R script files see here for the main simulation file using SimDesign, while see this file containing the helper functions for generating syntax with lavaan. The final simulation results are available in this .rds file

For-loop issues

The article written by Hallgren, Kevin A. (2013) “Conducting Simulation Studies in the R Programming Environment.” Tutorials in Quantitative Methods for Psychology, 9 (2). NIH Public Access: 43. explained how to run Monte Carlo simulations in R, but unfortunately introduced the topic using the for loop strategy. So, I decided to compare the code to SimDesign's.

• The result was that, when written in SimDesign's framework, the code was not only cleaner and easier to read, manipulate, and manage, but actually ran about 2-6x faster on a single core (2x faster on a computer with 24GB of RAM, 6x faster on a Linux computer with only 8GB of RAM). Oddly enough, it also demonstrated some of the pitfalls discussed by Sigal and Chalmers (2016) very nicely, and therefore serves as another good comparison between the coding approaches.

Contacting information

If you have further examples you would like to share, please feel free to contact me at [email protected] and I'll add the files to the appropriate locations on Github.