Simulations reading group

Meeting 2: Maurer and Lock (2016)

Selling points

  1. Realistic and thoughtful design of a randomized controlled experiment
  2. Two dimensional response variable: ARTIST-scaled topic scores for confidence intervals and hypothesis testing
  3. Estimate the treatment effect of simulation-based inference curriculum on responses after adjusting for pre-treatment measurements

Main point 1

Is it worth implementing the simulation-based inference curriculum?

  1. Small to modest gains on a 10 point scale: 95% CI (0.04,1.39) for CI questions, 95% CI (-0.40, 1.01) for hypothesis testing questions
  2. Most of the variance in responses attributable to Midterm and specific Lab/HW.

Main point 2

Can we “get” the gains elsewhere?

  1. For CI questions, a 15 percentage point increase in midterm performance would “compensate” for the treatment effect.
  2. For hypothesis testing questions, an 8 percentage point increase in midterm performance would “compensate” for the treatment effect.

Main point 3

Based on the variance decompositions, “despite the increased complexity of the simulation-based material and the shortened exposure to theory-based inference concepts, there was no significant detriment to students’ performance in conducting inference using theory-based methods.”

Things to think about

  • Ethical challenges in conducting an experiment like this
  • A more granular analysis by looking at the responses to the questions + add “random effects” which the authors were worried about
  • Smaller-scale interventions which can give rise to paired comparisons?
  • Framing hypothesis tests as confidence intervals instead? Emphasize hypothesis tests for situations with no confidence interval counterpart?

Strange findings for Figure 3

  1. Refer to Figure 3.
  2. Look at the text, “The simulation-based inference group had lower variability than the traditional inference group on the ARTIST question set for confidence intervals, but higher variability on all other scores.”
  3. The description and the figure are in conflict.

Categorizing contributed questions

  • Design and detection
  • Access and accountability