Skip to content

Evaluation Metrics Calculator

Run the Evaluation Metrics Calculator MicroSim Fullscreen

You can include this MicroSim on your website using the following iframe:

<iframe src="https://dmccreary.github.io/robot-arm/sims/evaluation-metrics-calculator/main.html"
        height="612px" width="100%" scrolling="no"></iframe>

Description

Change the number of trials, successes and violations in a test of an agent and watch the success rate and the violation rate. Then solve six problems about rates, a mean and a range. Used in Chapter 17.

Explore mode starts with the result of the chapter's lab: 26 of 40 trials finish, which is 65.0 percent, with 0 violations. The quiz gives six problems in a fixed order. You type a number in the unit shown and press Check, and an answer within 0.1 of the correct value counts as correct.

Lesson Plan

Grade level: Middle school and up (age 12+). Time: 10 to 15 minutes.

  1. Predict first. Before you move a slider, say what will happen to each rate. What happens to the success rate when the trials go up and the successes stay the same?
  2. Explore. Set the violations to 1 with a high success rate, and read the note. A high success rate does not make an agent safe.
  3. Problems. Press Start. Six problems: type each answer as a number and press Check. You get one try for each problem, and mastery is 5 of 6.
  4. Think about it. The success rate is allowed to fall when the model gets worse. Why must the violation rate stay at zero?

A rate is a count divided by the number of trials, as a percentage. The range is the largest value minus the smallest, and it is not an average.

Specification

The learning objective, evidence of mastery and content for this MicroSim are specified in Chapter 17: Safety Layers and Evaluation for Agent-Controlled Arms.