Simulations reading group

Meeting 3: Case and Jacobbe (2018)

Context

  1. AP Statistics course
  2. 14 in-class experiences with simulation-based inference
  3. For each data type: Statistic, Simulate, Strength (3S strategy from ISI), then theory-based
  4. Seems to be in conflict with “… simulation-based inference activities were incorporated throughout the year beginning on the first day of class, with theory-based inference introduced in the final third of the course.”
  5. Collected qualitative data

Things of note

  1. A definition of simulation-based inference: “Significance tests that use simulations to model the null hypothesis will be called simulation-based inference methods. (page 11)”
  2. Simulation here vs simulation elsewhere: “In other disciplines, simulation models are often constructed as a best-guess representation of reality and used to predict what would happen in the real-world. (page 11)”

Tidbits

  1. Learning sampling distributions vs learning derivatives

    • “Students can understand the process of drawing a single random sample and computing a summary number like a mean. But the transition from there to the sampling distirbution as the probability distirbution each of whose outcomes corresponds to taking-a-sample-and-computing-a-summary-number is … a hard transition.”
    • What is the analog for derivatives?
    • Are there similar instances elsewhere?

Tidbits

  1. Capacity to use modus tollens

  2. How do changes in representation or the exposure to multiple representations affect learning?

Struggles

  1. To distinguish samples and sampling distributions
  2. To transition from the sample level to the sampling distribution level
  3. To distinguish between simulation and replication

Contributed questions

  1. How much of the observed challenges can be attenuated with changes in the curriculum and simulation user interface/experience?

  2. Why are students not able to envision the multi-level nature of inference without having an applet open in front of them?

  3. How can we get students to a place where they must face their own contradictions in reasoning/recognize their reasoning contains contradictions?

2 x 3

Real-world Hypothetical
Population Actual population or true relationship Hypothetical population or relationship
Sample Empirical distribution Distribution of one simulated sample
Sampling distribution Distribution of statistics produced through replication Distribution of statistics produced through simulation

In the mind vs Physical/computational

Real-world Hypothetical
Population Actual population or true relationship Hypothetical population or relationship
Sample Empirical distribution Distribution of one simulated sample
Sampling distribution Distribution of statistics produced through replication Distribution of statistics produced through simulation

Static vs dynamic

Real-world Hypothetical
Population Actual population or true relationship Hypothetical population or relationship
Sample Empirical distribution Distribution of one simulated sample
Sampling distribution Distribution of statistics produced through replication Distribution of statistics produced through simulation

Real versus fictional

Real-world Hypothetical Fictional
Population Actual population or true relationship Hypothetical Fictional population or relationship
Sample Empirical distribution Distribution of one simulated sample
Sampling distribution Distribution of statistics produced through replication Distribution of statistics produced through simulation

Coin tossing

Real-world Hypothetical
Population A very long sequence of coin tosses of length \(N\) A very long sequence of fair coin tosses of length \(N\)
Sample One actual sequence of coin tosses with length \(n\) One artificially generated sequence of fair coin tosses with length \(n\)
Sampling distribution Literally tossing a coin \(n\) times repeatedly Use the computer to toss a fair coin \(n\) times repeatedly

Brainstorming possible solutions

  1. Role-playing as a chance skeptic

    • How do you address your own skepticism?
    • What are the sources of your skepticism?

Brainstorming possible solutions

  1. Introduction to probability in an intro-stats course

    • Introduce simulation as a device
    • Separate inferential reasoning from the implementation of inferential reasoning?

Brainstorming possible solutions

  1. Change the language used

    • samples, resamples, replicates, simulations
    • reducing inference as a verbal argument first
    • research hypotheses v. null models

Brainstorming possible solutions

  1. Is the software layer the problem?

    • Tactile vs applet
    • Emphasizing simulation as mimicry?

Brainstorming possible solutions

  1. Will it help to show that null hypothesis testing is a procedure with guarantees provided you follow a specific argument?