Skip to content

Survey Design Checklist Interactive

Run the Survey Design Checklist Fullscreen

Edit in the p5.js Editor

About This MicroSim

"Time to squirrel away this knowledge!" Sylvia cheers. "Everything we've learned about sampling comes together here. Can you spot the problems that make a survey unreliable?"

This interactive tool presents real-world survey scenarios and challenges students to identify the methodological problems. It's a practical application of all the bias and design concepts covered in Chapter 11.

Problem Categories

Category Issues to Identify
Sampling Method Convenience sample, Voluntary response
Coverage Issues Undercoverage, Nonresponse bias, Bad timing
Question Wording Loaded language, Leading questions, Double-barreled

How to Use

  1. Read the scenario describing a survey situation
  2. Check all problems you can identify from the checklist
  3. Click "Check Answers" to see if you found them all
  4. Review the explanation to understand why each issue matters
  5. Click "Next →" to try another scenario

The Four Scenarios

  1. Magazine Reader Survey — Classic voluntary response
  2. Teen Social Media Study — Convenience sample and undercoverage
  3. Political Opinion Poll — Loaded and leading question wording
  4. Community Health Survey — Multiple coverage issues

Key Concepts

Evaluating Any Survey

When you encounter a survey in the news or research, ask:

About Sampling: - Who was actually asked? (sampling frame) - How were they selected? (random or convenience?) - Could anyone choose to participate? (voluntary response?)

About Coverage: - Who might be missing? (undercoverage) - Who might not respond? (nonresponse) - When and how was contact made? (accessibility)

About Questions: - Is the wording neutral? (no loaded terms) - Does it suggest an answer? (not leading) - Does it ask one thing at a time? (not double-barreled)

"Good surveys don't just happen," Sylvia reminds us. "Every choice—from who you sample to how you phrase questions—affects the quality of your data."

Embedding This MicroSim

1
<iframe src="https://dmccreary.github.io/statistics-course/sims/survey-design-checklist/main.html" height="552px" width="100%" scrolling="no"></iframe>

Lesson Plan

Learning Objectives

By the end of this activity, students will be able to:

  1. Apply survey design principles to evaluate real-world scenarios
  2. Identify multiple types of bias in a single survey
  3. Explain why each identified problem affects validity
  4. Suggest improvements to flawed survey designs

Target Audience

  • AP Statistics students (high school)
  • Introductory statistics college students
  • Research methods courses
  • Media literacy education

Prerequisites

  • Understanding of sampling methods (SRS, convenience, voluntary response)
  • Knowledge of bias types (undercoverage, nonresponse, response)
  • Familiarity with question wording issues

Classroom Activities

Activity 1: Solo Practice (15 minutes)

Students work through all four scenarios independently, recording their answers and scores.

Activity 2: Pair Discussion (10 minutes)

Partners compare which issues they identified and which they missed. Discuss disagreements.

Activity 3: Design Improvement (15 minutes)

For each scenario, students propose specific fixes:

  1. Magazine survey → Random sample from subscriber list with follow-ups
  2. Teen study → Stratified sample across multiple school types
  3. Political poll → Rewrite with neutral language
  4. Health survey → Multiple languages, cell phones, evening calls

Extension Activities

Create Your Own Scenario: Students write a survey scenario with intentional flaws for classmates to identify.

Real-World Analysis: Bring in actual survey results from news articles. Students evaluate the methodology.

Fix It Workshop: Given a flawed survey, students redesign it to eliminate all identified problems.

Assessment Questions

  1. A website asks visitors to click a button to share their opinion on a new product. What type of sampling is this? What bias is most likely?

  2. List three specific questions you would ask to evaluate whether a survey result reported in the news is trustworthy.

  3. A survey about transportation preferences is conducted at a train station during rush hour. Identify at least two problems with this approach.

  4. Explain why a survey can have multiple bias issues simultaneously, and why each issue makes the results less reliable.

References