Lab 35: Capstone — Design, Benchmark and Report

Time: one to three weeks | Prerequisites: Lab 34 | Hardware: Pico 2 + kit

Your turn

Thirty-four labs ago you'd never opened Thonny. Now you can capture audio, transform it, hand-write the assembly that does the transforming, and — the part most engineers never learn — measure it honestly. Time to use all of it on a question nobody handed you.

What You'll Deliver

An experiment of your own: a question, a hypothesis written down before measuring, an implementation, measurements taken under the discipline of Lab 26, and a written report that states what your benchmark excludes.

A negative result, honestly reported and explained, earns full marks. An unexplained positive one does not.

Learning Objectives

• Formulate a research question with a measurable answer
• Design an experiment with controls and a stated hypothesis
• Implement a variant or application using the techniques from Modules 4–8
• Measure with warm-up, best-of-N, and stated exclusions
• Report results including limitations and negative findings

Concepts Introduced

ID	Concept
478	Experimental Design
479	Research Question
480	Independent Variable
481	Dependent Variable
482	Methodology Section
483	Results Presentation
484	Limitations Statement
485	Conclusion Drawing
486	Project Scoping
487	Peer Review

Choosing a Project

Your instructor will select which of these tracks are available. They differ in difficulty, not in how much you'll learn.

Track A — Optimize (hardest)

Invent a new FFT variant and measure it against the six from Lab 34.

Idea	Why it's interesting
Radix-4	~25% fewer multiplies in theory. But it needs four complex values plus three twiddles live at once, and the baseline already spills registers. Does the saving survive the spill traffic? Genuinely open — nobody in this course has measured it.
Dual-core	The RP2350 has two M33 cores and _thread is available. The transform is under a millisecond, so synchronization may cost more than it saves. Expected to disappoint, which makes it instructive.
Interleaved layout	Store [re,im] pairs to halve address arithmetic. Plan 02 measured the kernel at 1.28× — the fastest of any variant — but conversion cost destroyed it. Can you avoid the conversion?
Fixed-point Q15	Would need a C toolchain, since MicroPython's assembler exposes no DSP instructions (Lab 28 proves it). A scoping study is a legitimate deliverable.

Track B — Apply

Build something that uses the FFT for a real purpose.

• A spectrogram that scrolls, showing frequency over time
• A DTMF decoder that recognises phone keypad tones
• A vibration monitor that learns a machine's normal spectrum and flags changes
• A musical instrument identifier using harmonic ratios from Lab 12
• A voice-activity detector distinguishing speech from silence and noise

Deliverable: a working device plus measurements showing it meets its real-time deadline.

Track C — Investigate

Answer a question with an experiment. No new FFT required.

• How does FFT size trade resolution against frame rate? Map the curve.
• Does clock speed scale performance linearly? Try 100–200 MHz.
• How much does window choice cost in cycles, and is Hanning always right?
• Does the Lab 24 stage profile change with N? At what size does drawing overtake the FFT?
• How accurate is the Lab 23 tuner against a reference source, across its range?

Track D — Replicate and Challenge

Take a result from these labs and try to break it.

• Lab 33 found VFMA worth ~0%. Under what conditions would it pay?
• Lab 32 found specialization worth 1.11×. Does that hold at N=128? N=2048?
• Lab 26 claims best-of-N beats mean. Construct a case where it misleads.

Replication is real science and is treated as such here.

Required Structure

1. Question and hypothesis

State both before measuring. Include your reasoning — a hypothesis without a "because" is a guess.

"Processing two frames per call should save 5–10%, because Lab 24 showed loop control is roughly 30% of the cost and this halves the per-frame setup."

2. Method

• What varies (independent variable), what you measure (dependent variable)
• What you hold constant
• Test signals and why you chose them (Lab 15)
• Trials, warm-up, and the statistic you report (Lab 26)

3. Correctness

Speed numbers are meaningless until correctness is established. Show your variant agrees with a trusted reference on every test signal, with the tolerance you chose and why (Lab 15).

4. Results

Tables and, where useful, a chart. Report best, mean and spread — never a bare number.

5. Discussion

• Did it match your hypothesis? By how much?
• If not, what does the gap reveal about the machine?
• What does your benchmark exclude?
• What would you measure next?

6. Limitations

At minimum: one board, one firmware, one temperature, one set of signals. Say so.

The Starter Template

# Lab 35: Capstone Template
#
# A starting skeleton for your capstone experiment. It already does the
# things that are easy to get wrong:
#
#   * builds every variant before measuring any of them   (Lab 32)
#   * discards a warm-up run                              (Lab 26)
#   * reports best, mean AND spread                       (Lab 26)
#   * checks correctness before reporting speed           (Lab 15)
#   * states its exclusions                               (Lab 26)
#
# Replace MyVariant with your own idea and fill in the write-up prompts at
# the bottom.

import gc
import machine
import math

import v0_baseline          # the reference every claim is measured against

machine.mem32[0xE000EDFC] = machine.mem32[0xE000EDFC] | (1 << 24)
machine.mem32[0xE0001000] = machine.mem32[0xE0001000] | 1
FREQ = machine.freq()

N = 512
TRIALS = 20
SAMPLE_RATE = 12800


def rd():
    return machine.mem32[0xE0001004]


# =========================================================================
# 1. Your research question
# =========================================================================
RESEARCH_QUESTION = "..."          # e.g. "Does processing two frames at
                                   # once amortise the loop overhead?"
HYPOTHESIS = "..."                 # e.g. "Expect 5-10%, because Lab 24
                                   # showed loop control is ~30% of cost."
PREDICTION_WRITTEN_BEFORE_MEASURING = True    # be honest about this


# =========================================================================
# 2. Test signals -- design them, do not just grab noise
# =========================================================================
# Each signal should test something specific (see Lab 15).
signals = {
    "single tone (bin-exact)": [math.sin(2 * math.pi * 40 * i / N)
                                for i in range(N)],
    "two tones": [0.7 * math.sin(2 * math.pi * 40 * i / N)
                  + 0.3 * math.sin(2 * math.pi * 111 * i / N)
                  for i in range(N)],
    "impulse (flat spectrum)": [1.0 if i == 0 else 0.0 for i in range(N)],
    "silence": [0.0] * N,
}


# =========================================================================
# 3. Build everything BEFORE measuring anything  (Lab 32)
# =========================================================================
baseline = v0_baseline.Variant(N)
# candidate = my_variant.Variant(N)
candidate = v0_baseline.Variant(N)      # <-- replace with your variant
gc.collect()


# =========================================================================
# 4. The harness
# =========================================================================
def bench(variant, signal, trials=TRIALS):
    re, im = variant.make_buffers()

    def once():
        for i in range(N):
            re[i] = signal[i]
            im[i] = 0.0
        s = rd()
        variant.run(re, im)
        return (rd() - s) & 0xFFFFFFFF

    once()                                       # warm-up, discarded
    runs = [once() for _ in range(trials)]
    lo = min(runs)
    mean = sum(runs) / len(runs)
    sd = (sum((r - mean) ** 2 for r in runs) / len(runs)) ** 0.5
    return lo, mean, sd, re, im


def check(variant, signal):
    """Return relative error against the baseline on this signal."""
    bre, bim = baseline.make_buffers()
    for i in range(N):
        bre[i] = signal[i]
        bim[i] = 0.0
    baseline.run(bre, bim)
    peak = max(max(abs(x) for x in bre), max(abs(x) for x in bim)) or 1.0

    cre, cim = variant.make_buffers()
    for i in range(N):
        cre[i] = signal[i]
        cim[i] = 0.0
    variant.run(cre, cim)

    worst = 0.0
    for i in range(N):
        worst = max(worst, abs(cre[i] - bre[i]), abs(cim[i] - bim[i]))
    return worst / peak


# =========================================================================
# 5. Correctness FIRST
# =========================================================================
print("Research question: %s" % RESEARCH_QUESTION)
print("Hypothesis       : %s" % HYPOTHESIS)
print()
print("=== Correctness ===")
all_ok = True
for name, sig in signals.items():
    err = check(candidate, sig)
    ok = err < 1e-3
    all_ok = all_ok and ok
    print("  %-26s rel err %.2e  %s" % (name, err, "PASS" if ok else "FAIL"))

if not all_ok:
    print()
    print("Correctness failed. Speed numbers below are meaningless until")
    print("this is fixed. Stop here and debug (Lab 15: bisection).")


# =========================================================================
# 6. Speed
# =========================================================================
print()
print("=== Speed ===")
print("%-26s %10s %10s %8s %10s" % ("signal", "baseline", "candidate",
                                    "speedup", "stddev"))
speedups = []
for name, sig in signals.items():
    b_lo, b_mean, b_sd, _, _ = bench(baseline, sig)
    c_lo, c_mean, c_sd, _, _ = bench(candidate, sig)
    speedups.append(b_lo / c_lo)
    print("%-26s %10d %10d %7.3fx %9.0f"
          % (name, b_lo, c_lo, b_lo / c_lo, c_sd))

avg = sum(speedups) / len(speedups)
print()
print("mean speedup across signals: %.3fx" % avg)

budget_us = N / SAMPLE_RATE * 1e6
c_lo, c_mean, c_sd, _, _ = bench(candidate, signals["two tones"])
print("candidate: %.1f us = %.2f%% of the %.0f us frame budget"
      % (c_lo * 1e6 / FREQ, c_lo * 1e6 / FREQ / budget_us * 100, budget_us))


# =========================================================================
# 7. Report honestly
# =========================================================================
print()
print("=== Report ===")
print("Trials per signal        : %d, after one discarded warm-up" % TRIALS)
print("Statistic reported       : best-of-N, with mean and stddev")
print("Excluded from the timing : table construction, buffer loading")
print("Prediction written first : %s" % PREDICTION_WRITTEN_BEFORE_MEASURING)
print()
print("Now answer these in your write-up:")
print("  1. Did the result match your hypothesis? By how much?")
print("  2. If it did not, what does the gap tell you about the machine?")
print("  3. What does this benchmark EXCLUDE that a user would care about?")
print("  4. What would you measure next, and why?")
print()
print("A negative result, honestly reported and explained, is worth more")
print("than a positive one you cannot account for.")

It already handles the things that are easy to get wrong: allocating before measuring (Lab 32), discarding a warm-up (Lab 26), reporting spread, checking correctness first (Lab 15), and stating exclusions.

Establish your noise floor first

Run the template unmodified — comparing the baseline against itself. It reports about 0.996×, which tells you your harness has roughly ±0.4% of noise. Any effect smaller than that is not a result, it's weather. Measure this before you trust a small number.

Assessment

Criterion	What good looks like
Question	Specific, measurable, with a stated reason for expecting the answer
Method	Controlled, reproducible, exclusions stated
Correctness	Verified before speed, tolerance justified
Measurement	Warm-up, best-of-N, spread reported
Honesty	Negative results reported plainly; predictions not retro-fitted
Insight	Explains why, not just what

The one unforgivable error

Changing your hypothesis after seeing the data and presenting it as a prediction. Every engineer is tempted; it destroys the value of the measurement. If you were wrong, say you were wrong — that page is usually the most interesting one in the report.

What You've Actually Learned

Look back at the numbers you've measured yourself:

Implementation	Time per 512-point FFT	vs. real-time budget
Brute-force DFT (Lab 16)	~21,000 ms	530× over
Python FFT (Lab 20)	140 ms	3.5× over
Assembly FFT (Lab 31)	0.85 ms	2.1% of budget
Best variant (Lab 34)	0.59 ms	1.5% of budget

A 35,000× improvement from your first working transform to your best one — on a chip that costs less than a sandwich.

But the durable skill isn't the assembly. It's that you can now look at a performance claim and ask the right questions: what was excluded, was there a warm-up, is that best or mean, and does the fast version still compute the right answer?

That transfers to every system you'll ever work on.

You made it, signal hunter

Thirty-five labs from "what's a Thonny?" to hand-encoded ARM instructions and a benchmark you can defend. You taught a $5 chip to listen to the world and understand it in real time. Now that's a superpower. Go build something with it.

---

Previous: Lab 34 | Back to all labs