The AI HAT+ and Neural Network Fundamentals¶
Summary¶
This chapter introduces the AI HAT+ accelerator -- the neural processing unit and its TOPS performance rating, available in 13 TOPS (Hailo-8L) and 26 TOPS (Hailo-8) variants -- and the machine learning vocabulary needed to use it: pretrained models, model inference versus model training, neural networks, and convolutional neural networks in particular. It covers image classification and object detection concepts -- bounding boxes, confidence scores, class labels -- and the datasets and accuracy metrics (training/validation datasets, model accuracy, false positives and negatives, overfitting) used to evaluate a model. Students finishing this chapter will be able to explain what the AI HAT+ accelerates and interpret a model's confidence score and accuracy metrics.
Concepts Covered¶
This chapter covers the following 22 concepts from the learning graph:
- AI HAT Plus
- Neural Processing Unit
- TOPS Performance Metric
- Machine Learning Model
- Pretrained Model
- Model Inference
- Model Training
- Training Versus Inference
- Neural Network
- Convolutional Neural Network
- Image Classification
- Object Detection
- Bounding Box
- Confidence Score
- Class Label
- Labeled Dataset
- Training Dataset
- Validation Dataset
- Model Accuracy
- False Positive
- False Negative
- Overfitting
Prerequisites¶
This chapter builds on concepts from:
- Chapter 1: Computational Thinking and Debugging for Physical Computing
- Chapter 14: Raspberry Pi 5 Hardware: Storage, Camera, and Cooling
A Different Kind of Program
Every program you've written so far in this book follows rules you wrote yourself: if the distance is under 10 centimeters, stop the motors. This chapter introduces something genuinely different — a program that writes its own rules by studying examples. Take this one slowly; it's dense, but every idea here builds on the last one. Let's build something!
Chapter 1 defined an algorithm as a finite, unambiguous sequence of instructions that a person designs, step by step, to produce a specific output. Everything you've built in this book so far — a blinking NeoPixel pattern, a robot's obstacle-avoidance loop — followed that model: you decided the rules, and the hardware executed them exactly as written. This chapter introduces a different approach to solving a problem, one where the rules themselves are learned from examples rather than hand-written, and it introduces the hardware add-on built specifically to run the result of that approach quickly on a Raspberry Pi 5: the AI HAT+.
What Machine Learning Actually Means¶
A machine learning model is a program that learns to make predictions or decisions by finding patterns in example data, rather than following rules that a person explicitly wrote out in advance. Instead of a programmer deciding "if the light level is below X, turn on the LED," a machine learning model is shown thousands of examples — pictures labeled "cat" and pictures labeled "not cat," for instance — and it gradually adjusts itself until it can correctly guess the label for pictures it has never seen before. The rules it ends up using aren't written anywhere as readable code; they're encoded as thousands or millions of numbers inside the model itself.
Berry's Key Insight
This is still computational thinking — just aimed at a different target. Instead of decomposing a problem into steps you write yourself, you're decomposing it into examples good enough that a model can find the pattern on its own. Abstraction still matters: you decide which details (pixels, in this case) the model gets to see.
The specific kind of machine learning model behind almost every camera-based AI project in this book is a neural network: a machine learning model loosely inspired by how biological neurons connect and pass signals, built from layers of simple computing units that each combine their inputs and pass a result forward to the next layer. Each of those simple computing units is called a neuron, and understanding what a single neuron actually computes is worth walking through with real numbers before looking at a whole network.
A neuron receives one or more input values, and it combines them using a weighted sum: each input is multiplied by its own adjustable number called a weight, all those products are added together, and one more adjustable number called a bias is added on top. If a neuron has inputs \( x_1, x_2, \ldots, x_n \), weights \( w_1, w_2, \ldots, w_n \) (one weight per input, learned during training), and a bias \( b \), the weighted sum \( z \) is:
Concretely, suppose a tiny neuron has just two inputs, with learned weights and a bias already assigned. Here is that exact arithmetic laid out one term at a time.
| Symbol | Meaning | Value |
|---|---|---|
| \( x_1 \) | First input | 0.5 |
| \( w_1 \) | Weight on the first input | 2 |
| \( x_2 \) | Second input | 0.8 |
| \( w_2 \) | Weight on the second input | -1 |
| \( b \) | Bias | 0.1 |
| \( z \) | Weighted sum: \( (w_1 x_1) + (w_2 x_2) + b \) | \( 1.0 - 0.8 + 0.1 = 0.3 \) |
That single number, \( z = 0.3 \), then passes through one more small step called an activation function, which decides how strongly the neuron "fires" based on that value — the details of specific activation functions aren't needed here, but the key idea is that this weighted-sum-plus-activation pattern, repeated across many neurons and many layers, is the entire computational engine behind a neural network.
Now that a single neuron's math is concrete, it's worth seeing how many of them connect into a full network before moving on.
Diagram: Neural Network Layer Explorer¶
Run the Neural Network Layer Explorer MicroSim fullscreen
Neural Network Layer Explorer (interactive diagram)
Type: interactive-diagram
sim-id: neural-network-layer-explorer
Library: p5.js
Template: https://github.com/dmccreary/linear-algebra/tree/main/docs/sims/neural-network-architecture
Status: Specified
Learning objective: Students will explain (Bloom L2: Understand) how a neuron computes a weighted sum of its inputs and how neurons connect across layers to form a neural network.
Canvas: 700x460px, responsive — recompute neuron positions as fractions of width/height inside windowResized(), keeping three layers readable down to 480px wide by shrinking neuron circle radius before reducing spacing.
Layout: three vertical columns of circles representing an "Input Layer" (2 neurons), a "Hidden Layer" (3 neurons), and an "Output Layer" (1 neuron), connected by lines representing weighted connections. Line thickness varies slightly to hint that each connection has its own weight value, without requiring students to read exact numbers from every line.
Interaction: clicking any hidden-layer or output-layer neuron opens an infobox beneath the diagram showing that specific neuron's weighted-sum calculation using the exact worked example from the surrounding chapter text (inputs, weights, bias, and the resulting \( z \) value), reinforcing the same numbers the reader already saw in prose. A createSlider() labeled "Input 1 Value" (range 0-1, step 0.1, default 0.5) lets students change one input and watch the connected neurons' displayed weighted-sum values update live. A createButton() labeled "Reset" restores the default input values.
Implementation: p5.js. Store neurons as objects with {x, y, layer, inputs, weights, bias} and connections as line segments between neuron pairs. Recompute each neuron's weighted sum from its inputs and fixed weights whenever the slider changes, and redraw the displayed value next to each neuron. Hit-testing via dist() for neuron clicks.
Convolutional Neural Networks: Built to Look at Images¶
A plain neural network like the one just described treats every input as an independent number in a flat list, with no built-in sense that some inputs are spatially near each other. That's a poor fit for images, where a pixel's meaning depends heavily on the pixels right next to it — a fixed-layout network would need an enormous number of weights to even begin recognizing that an edge or a curve looks similar no matter where in the image it appears. A convolutional neural network — often abbreviated CNN — solves this by using small, reusable grids of weights called filters that slide across the entire image, checking for the same pattern (an edge, a curve, a patch of color) at every position rather than learning a separate detector for every single pixel location.
You've Got This!
If "a small grid of weights sliding across an image" doesn't fully click yet, that's completely normal — this is one of the genuinely hard ideas in this book. Picture a cookie cutter moving across a sheet of dough, checking the same shape at every position, instead of a person memorizing where every single cookie will end up. The diagram below lets you drive that cutter yourself.
That sliding-filter process is why convolutional neural networks became the dominant approach for image classification and object detection, both covered later in this chapter — they detect the same visual pattern regardless of where it appears in the frame, which is exactly the kind of flexibility a real camera pointed at a moving world needs.
Diagram: Convolution Filter Explorer¶
Run the Convolution Filter Explorer MicroSim fullscreen
Convolution Filter Explorer (MicroSim)
Type: microsim
sim-id: convolution-filter-explorer
Library: p5.js
Template: https://github.com/dmccreary/linear-algebra/tree/main/docs/sims/convolution-operation
Status: Specified
Learning objective: Students will apply (Bloom L3: Apply) a small convolution filter to a grid of pixel values and predict the resulting feature map value at each position.
Canvas: 700x460px, responsive — two grids (a small 6x6 source image grid and a resulting smaller output feature-map grid) recomputed as fractions of width inside windowResized(), stacked vertically below 560px wide instead of side by side.
Layout: left grid shows a 6x6 grid of simple pixel values (0-9, grayscale-shaded cells); a 3x3 highlighted "filter window" outline sits on top of the grid at its current position. Right grid shows the resulting smaller output feature map, filled in only for positions the filter has already visited.
Controls: a createButton() labeled "Step" that slides the filter window one position to the right (wrapping to the next row when it reaches the edge) and computes and fills in the corresponding output cell; a createButton() labeled "Run All" that animates the filter across every remaining position automatically; a createButton() labeled "Reset" that clears the output grid and returns the filter to the top-left position.
Interaction: while the filter window sits at a given position, a small readout beneath the grids shows the exact multiplication-and-sum calculation being performed (each of the 9 filter weights times its corresponding pixel value, summed), so students can verify the output cell's value before or after clicking "Step." Hovering any filled output cell re-highlights the source region that produced it.
Implementation: p5.js. Store the source grid and a fixed 3x3 filter (e.g., a simple edge-detection kernel) as 2D arrays. On each "Step," compute the dot product of the filter and the currently covered 3x3 region, write it to the output array, and advance a stored (row, col) position for the filter window. Redraw both grids from their arrays each frame.
Training a Model vs Running It: Two Very Different Jobs¶
Machine learning has two distinct phases, and mixing them up is one of the most common sources of confusion for anyone new to the field. Model training is the process of showing a neural network many examples and repeatedly adjusting its weights and biases so that its predictions get closer and closer to the correct answer for those examples. Training is slow, computationally expensive, and typically done once, ahead of time, on powerful computers — not on the small Pi 5 sitting on your desk. Model inference is the process of running an already-trained model on a new input it has never seen before, to produce a prediction, using the fixed weights and biases training already worked out. Inference is comparatively fast and lightweight, which is exactly why it's realistic to run on a Raspberry Pi 5.
Training versus inference captures that division of labor as a single comparison: training happens rarely, requires enormous datasets and computing power, and produces a finished model; inference happens constantly, requires comparatively little computing power per prediction, and consumes that finished model without changing it. Every project in this book that uses the AI HAT+ performs inference only — you will never train a neural network from scratch on a Pi 5 in this book.
That's possible because of a pretrained model: a neural network that has already completed the training process, using someone else's dataset and computing resources, and is distributed ready to use for inference. Rather than collecting a million labeled images and training for days, you download a pretrained model file and immediately start running inference with it.
Now that both phases are defined clearly, here's how they compare side by side.
| Model Training | Model Inference | |
|---|---|---|
| What happens | Weights and biases are repeatedly adjusted using example data | A fixed, already-trained model produces a prediction for one new input |
| Typical hardware | Powerful cloud or datacenter computers | A Raspberry Pi 5 with the AI HAT+ |
| Frequency | Rare — done once (or occasionally re-done) ahead of time | Constant — every camera frame, every prediction |
| What this book uses | Not covered — pretrained models are used instead | Every AI HAT+ project in this book |
Diagram: Training vs Inference Workflow¶
Run the Training vs Inference Workflow MicroSim fullscreen
Training vs Inference Workflow (interactive infographic)
Type: interactive-infographic
sim-id: training-vs-inference-workflow
Library: p5.js
Template: https://github.com/dmccreary/linear-algebra/tree/main/docs/sims/ml-pipeline
Status: Specified
Learning objective: Students will compare (Bloom L4: Analyze) the training and inference phases of a neural network pipeline in terms of hardware, frequency, and output.
Canvas: 700x420px, responsive — two horizontal swim lanes ("Training" and "Inference") recomputed as fractions of width/height inside windowResized(), stacked with reduced label text below 480px wide.
Layout: the "Training" lane shows a sequence of three boxes — "Huge Labeled Dataset" → "Powerful Training Computer (days of work)" → "Trained Model File" — connected by arrows. The "Inference" lane shows "Trained Model File" (shared visually with the training lane's output) → "Raspberry Pi 5 + AI HAT+" → "One Prediction, in Milliseconds."
Interaction: clicking any box in either lane opens an infobox beneath the diagram with a one-sentence description drawn from the surrounding chapter comparison table. A createButton() labeled "Highlight Shared Model" pulses a highlight around the "Trained Model File" box to emphasize that it is the single link between the two otherwise very different processes.
Implementation: p5.js. Store both lanes as arrays of box objects with fixed relative x positions per lane. Draw connecting arrows between sequential boxes within a lane. Hit-testing via rectangular bounds. Use a frameCount-based sine pulse for the "Highlight Shared Model" animation, toggled on by the button and off by clicking elsewhere on the canvas.
The Data Behind the Model: Labeled, Training, and Validation Datasets¶
Every pretrained model you'll use started life as a large collection of examples, and the vocabulary around that collection matters for understanding how trustworthy a model's predictions are. A labeled dataset is a collection of example data where each item has been tagged with the correct answer by a human or a trusted process — a photo tagged "dog," a photo tagged "bicycle." Without labels, a model has no correct answer to compare its guesses against during training, so labeling is the foundation the entire training process depends on.
A labeled dataset is typically split into two separate parts that serve different purposes. The training dataset is the portion of the labeled data actually shown to the model during training, used to adjust its weights and biases. The validation dataset is a separate portion of labeled data, deliberately withheld from training, used afterward to check how well the model performs on examples it has never adjusted itself to fit. Keeping these separate matters because a model can look excellent on data it was trained on while actually being poor at generalizing to new, unseen examples — which is precisely the failure mode covered next.
How Good Is the Model? Accuracy, Errors, and Overfitting¶
Once a model is trained, you need a way to judge whether it's actually good at its job. Model accuracy is the proportion of predictions a model gets correct out of all the predictions it makes on a given dataset, usually expressed as a percentage. A model with 95% accuracy on its validation dataset got the right answer 95 times out of 100 on data it had never specifically trained on.
Accuracy alone doesn't tell the whole story, though, because not all mistakes look the same. A false positive is a case where a model predicts something is present when it actually is not — for example, a security camera model reporting "person detected" when the frame is empty. A false negative is the opposite case, where a model fails to detect something that actually is present — the same camera missing a person who is clearly in frame. Depending on the project, one of these errors can matter far more than the other: a false negative in a safety application (missing a real hazard) is often much more costly than a false positive (a harmless false alarm).
Berry's Gentle Warning
Watch out for overfitting — when a model performs extremely well on its training dataset but poorly on new data it hasn't seen, because it has essentially memorized the specific training examples instead of learning the general pattern behind them. A model reporting suspiciously perfect training accuracy but disappointing validation accuracy is very likely overfit, not brilliant.
Now that all four evaluation ideas are defined, it helps to see how they relate to each other as a single explorable model rather than four separate definitions.
Diagram: False Positive and False Negative Explorer¶
Run the False Positive and False Negative Explorer MicroSim fullscreen
False Positive and False Negative Explorer (interactive diagram)
Type: interactive-diagram
sim-id: false-positive-negative-explorer
Library: p5.js
Status: Specified
Learning objective: Students will classify (Bloom L4: Analyze) a model's prediction outcomes as correct detections, false positives, or false negatives, and connect the pattern of those outcomes to overall model accuracy.
Canvas: 700x460px, responsive — a 2x2 grid of quadrant boxes recomputed as fractions of width/height inside windowResized(), stacking into a single column below 480px wide.
Layout: a 2x2 grid labeled by "Actual: Present" / "Actual: Absent" (rows) and "Predicted: Present" / "Predicted: Absent" (columns), producing four quadrants: correct detection (top-left, circuit green #2E7D32), false positive (top-right, raspberry #C2185B), false negative (bottom-left, copper gold #D4AF37), correct rejection (bottom-right, gray). Each quadrant shows a running count of example items sorted into it.
Controls: a createButton() labeled "Add Example" that generates one new randomly-outcome example (weighted toward mostly-correct results with occasional errors) and animates it dropping into the correct quadrant based on its actual/predicted values; a createSlider() labeled "Model Accuracy" (range 50-99, default 90) controlling how frequently new examples land in the two error quadrants versus the two correct quadrants.
Interaction: clicking any quadrant opens an infobox defining that quadrant's term (using the chapter's definitions for false positive and false negative, plus one-sentence definitions for the two correct-outcome quadrants) and shows that quadrant's current count as a fraction of the total examples added so far, connecting back to the model accuracy definition. A running "Overall Accuracy" readout beneath the grid recalculates after every added example.
Implementation: p5.js. Track four quadrant counts in a simple object. "Add Example" uses the slider's accuracy value as a probability to decide whether the new example is correct or an error, and a second coin flip to decide which of the two error types (if an error) or which of the two correct types (if correct). Animate the new example as a small circle moving from a spawn point into its target quadrant using lerp(), then increment that quadrant's stored count and redraw.
Two Vision Tasks: Image Classification and Object Detection¶
With training, inference, datasets, and accuracy all defined, you're ready for the two specific tasks nearly every camera-based AI HAT+ project in this book performs. Image classification is the task of assigning a single label to an entire image, answering the question "what is the main thing in this picture?" without saying where in the picture it is. Object detection is a more demanding task: finding every instance of one or more objects within an image and reporting both what each object is and where it is located.
That "where it is located" answer takes a specific visual form. A bounding box is a rectangle, typically defined by its corner coordinates, drawn tightly around a detected object to mark its location and extent within the image. Alongside every bounding box, a detection model reports a class label — the specific category name the model assigns to that detected object, such as "person," "car," or "dog," drawn from the fixed list of categories the model was trained to recognize — and a confidence score: a number, usually between 0 and 1 (or expressed as a percentage), representing how certain the model is that a given detection is correct. A confidence score of 0.95 means the model is highly certain; a confidence score of 0.31 means the detection is a weak guess that a project might reasonably choose to ignore.
Here's how the two vision tasks compare directly, now that both are fully defined.
| Image Classification | Object Detection | |
|---|---|---|
| Answers the question | "What is the main subject of this image?" | "What objects are present, and where?" |
| Output per image | One class label (plus a confidence score) | Multiple bounding boxes, each with a class label and confidence score |
| Handles multiple objects in one frame | No — one label for the whole image | Yes — one detection per object found |
Berry's Key Insight
Object detection is really image classification's more ambitious cousin: instead of one label for the whole picture, it's running that same "what is this?" judgment on many candidate regions of the picture at once, then reporting a box, a label, and a confidence score for every region it's confident enough about.
Diagram: Bounding Box and Confidence Threshold Explorer¶
Run the Bounding Box and Confidence Threshold Explorer MicroSim fullscreen
Bounding Box and Confidence Threshold Explorer (MicroSim)
Type: microsim
sim-id: bounding-box-confidence-explorer
Library: p5.js
Status: Specified
Learning objective: Students will apply (Bloom L3: Apply) a confidence threshold to decide which of a model's candidate detections should be kept or discarded.
Canvas: 700x460px, responsive — a single image-panel canvas recomputed as a fraction of width/height inside windowResized(), scaling all bounding boxes proportionally with the panel.
Layout: a simple illustrated scene (drawn with basic p5.js shapes representing a few objects — a rectangle "car," an oval "dog," a rounded rectangle "person") with six candidate bounding boxes overlaid, several deliberately overlapping and low-confidence, each labeled with its class label and confidence score (e.g., "dog: 0.87," "dog: 0.22").
Controls: a createSlider() labeled "Confidence Threshold" (range 0.0-1.0, step 0.05, default 0.5).
Interaction: as the threshold slider moves, any bounding box with a confidence score below the current threshold visibly fades out and its border switches to a dashed gray outline labeled "below threshold — discarded," while boxes at or above the threshold remain solid and fully colored. A live count beneath the canvas reads "Detections kept: X of 6" and updates as the slider moves. Hovering any bounding box (kept or discarded) shows its exact class label and confidence score in a tooltip.
Implementation: p5.js. Store the six candidate detections as objects {label, confidence, x, y, w, h}. On every frame, compare each detection's confidence to the current slider value to decide its rendered opacity and border style. Draw class label and confidence score as text above each box using textAlign(LEFT, BOTTOM).
The AI HAT+: Hardware Built to Run Neural Networks Fast¶
Everything covered so far in this chapter is hardware-independent — the same neural network math runs, in principle, on any processor. But running a neural network's weighted sums and convolutions for every pixel of every video frame, dozens of times per second, is a huge number of repeated multiply-and-add operations, and a general-purpose processor like the Pi 5's CPU is not specifically optimized for that particular kind of math. That's the gap the AI HAT+ closes.
The AI HAT Plus is an official add-on board for the Raspberry Pi 5, connecting through the PCIe interface and GPIO header you met in an earlier chapter, built around a chip specifically designed to run neural network inference far faster and more efficiently than the Pi's own CPU can. At the heart of the AI HAT+ sits a neural processing unit — often abbreviated NPU — a specialized processor whose circuitry is built specifically to perform the weighted-sum and convolution operations neural networks depend on, rather than being a general-purpose processor pressed into that role.
A neural processing unit's speed is usually described using a single headline number: the TOPS performance metric — Trillion Operations Per Second — measures how many basic arithmetic operations a neural processing unit can perform each second, giving a rough, comparable sense of how much inference workload it can handle. The AI HAT+ ships in two variants built around different Hailo-brand neural processing unit chips: a 13 TOPS version built on the Hailo-8L chip, and a 26 TOPS version built on the more powerful Hailo-8 chip, at a higher price, for projects that need to run larger models or process more frames per second.
Berry's Key Insight
A higher TOPS number isn't automatically "better" for every project, the same way a bigger engine isn't automatically the right choice for every car. A simple object-detection demo may run comfortably within the 13 TOPS Hailo-8L's budget; a more demanding project running a larger model at a higher frame rate is where the 26 TOPS Hailo-8 variant starts to earn its higher price.
That TOPS rating is the last new hardware term this chapter needs, and it closes the loop back to Chapter 14: the AI HAT+ plugs into the same PCIe interface and GPIO header you already met, and the case-and-ventilation habits from Chapter 15 apply here too, since a neural processing unit running flat-out generates real heat right alongside the Pi 5's own CPU.
You've Got This!
That's a lot of new vocabulary in one chapter — neurons, weighted sums, convolutions, training, inference, datasets, accuracy, TOPS. You don't need all of it memorized cold. What matters most going forward is the shape of the whole picture: a pretrained model, running inference, on hardware built specifically for the job. The next two chapters give you plenty of hands-on practice to make these terms stick.
Bringing It Together¶
You now have the full vocabulary behind every AI HAT+ project in this book: what a neural network actually computes, how a convolutional neural network adapts that computation to images, why training and inference are entirely separate phases, how datasets and accuracy metrics reveal whether a model is trustworthy, and what makes image classification different from object detection. You also know exactly what hardware sits on the AI HAT+ and why its TOPS rating matters. The next chapter takes all of it and puts it to work: preparing a real pretrained model, running it through the AI HAT+ at real-time speed, and measuring exactly how fast and how accurate the resulting pipeline actually is.
You Unlocked a Superpower!
That's berry impressive — you just learned how a neural network actually thinks, one weighted sum at a time, and what hardware makes that thinking fast enough for a live camera feed. STEM is our superpower, and this one's now yours. Let's build something — see you in Chapter 17!