Building a Real-Time Object Detection Pipeline

Summary

This chapter covers how a pretrained model actually runs in real time on the AI HAT+. It explains model quantization and compression, and common model formats (TFLite, ONNX) used for edge deployment, along with the general concept of edge AI and the inference latency and frames-per-second metrics used to evaluate a running pipeline. It walks through the full camera-to-model pipeline -- image preprocessing, region of interest, non-maximum suppression, and detection threshold -- and covers transfer learning as a way to add a custom object class to a pretrained model. Students finishing this chapter will be able to set up a Pi 5 and AI HAT+ to run a pretrained real-time object detection demo.

Concepts Covered

This chapter covers the following 18 concepts from the learning graph:

1. Model Quantization
2. Model Compression
3. TFLite Model Format
4. ONNX Model Format
5. Edge AI
6. Inference Latency
7. Frames Per Second
8. Hardware Accelerator
9. Hailo Accelerator
10. Model Zoo
11. Transfer Learning
12. Custom Object Class
13. Region Of Interest
14. Non Maximum Suppression
15. Image Preprocessing
16. Real Time Video Pipeline
17. Camera To Model Pipeline
18. Detection Threshold

Prerequisites

This chapter builds on concepts from:

• Chapter 2: Choosing the Right Raspberry Pi Product
• Chapter 14: Raspberry Pi 5 Hardware: Storage, Camera, and Cooling
• Chapter 16: The AI HAT+ and Neural Network Fundamentals

---

From Vocabulary to a Working Pipeline

Last chapter gave you the words: neural network, pretrained model, inference, bounding box. This chapter connects those words into one continuous, running pipeline — camera in, detections out, fast enough to feel instant. By the end, you'll know exactly what happens between a photon hitting the camera's image sensor and a labeled box appearing on your screen. Let's build something!

Every idea in this chapter exists to answer one practical question: how does a model small enough to fit in memory and fast enough to keep up with a live camera actually get onto a Raspberry Pi 5, and what happens to each frame of video between capture and result? This chapter follows that path stage by stage, starting with why the Pi 5 doesn't just send video to a distant server in the first place.

Edge AI: Keeping Inference on the Device

Edge AI describes running a machine learning model's inference directly on a local device — a phone, a camera, a Raspberry Pi 5 — at the location where data is captured, instead of sending that data over the internet to a distant server for processing. The alternative, sending every camera frame to a cloud server and waiting for a response, introduces network delay, depends on a reliable internet connection, and raises real privacy questions about where captured images end up. Edge AI avoids all three problems by keeping the entire inference process local: no round trip, no dependency on connectivity, and image data that never has to leave the device unless a project deliberately chooses to send it somewhere.

Making that local option fast enough requires specialized hardware. A hardware accelerator is a chip designed to perform one specific category of computation — in this case, the weighted sums and convolutions behind neural network inference — dramatically faster and more efficiently than a general-purpose CPU running the same math in software. The AI HAT+'s neural processing unit, introduced in the previous chapter, is one example of a hardware accelerator. More specifically, it's a Hailo accelerator — a neural processing unit chip made by Hailo, the company whose Hailo-8L and Hailo-8 chips power the AI HAT+'s 13 TOPS and 26 TOPS variants respectively. Every real-time object detection pipeline in this chapter runs its inference stage on this Hailo accelerator rather than on the Pi 5's own CPU.

Berry's Key Insight

Edge AI is the sense-think-act cycle from Chapter 1, kept entirely local. "Sense" is the camera capturing a frame; "think" is the Hailo accelerator running inference right there on the board; "act" is whatever your program does with the result — no network hop in between any of those steps.

Making a Model Small Enough for the Edge: Quantization and Compression

A model straight out of training is usually too large and too computationally heavy to run efficiently on edge hardware like the AI HAT+, so it typically goes through a shrinking process before deployment. Model compression is the general practice of reducing a trained model's size and computational cost — through several possible techniques — while trying to preserve as much of its original accuracy as possible. The most common compression technique used for edge deployment is model quantization: converting a model's internal numbers (its weights and the calculations it performs) from a high-precision format, such as 32-bit decimal numbers, down to a lower-precision format, such as 8-bit whole numbers, which takes less memory to store and less computation to process.

Quantization isn't free — representing a number with fewer bits necessarily loses some precision — but for most vision models the accuracy cost is small compared to the speed and size benefit, especially on hardware accelerators specifically built to work efficiently with those lower-precision numbers.

Berry's Gentle Warning

Don't assume quantization is completely free just because the accuracy drop is usually small. A model that was already borderline accurate before quantization can cross from "good enough" to "not reliable enough" after compression. Always re-check a model's accuracy after quantization rather than trusting the pre-compression numbers.

Now that compression and quantization are defined, the chart below shows the kind of tradeoff a real project has to weigh.

Diagram: Model Compression Tradeoff Explorer

Run the Model Compression Tradeoff Explorer MicroSim fullscreen

Model Compression Tradeoff Explorer (interactive chart)

Type: chart sim-id: model-compression-tradeoff-explorer
Library: Chart.js
Status: Specified

Learning objective: Students will analyze (Bloom L4: Analyze) the tradeoff between model size, inference speed, and accuracy as a model is progressively quantized and compressed.

Chart type: dual-axis line chart, responsive Chart.js canvas.

Purpose: show how model size (shrinking) and inference speed (increasing) move in opposite directions from accuracy (slightly decreasing) as a slider moves a model from full precision toward heavier quantization.

X-axis: a single interactive slider (HTML <input type="range">, not a chart axis) labeled "Quantization Level," from "None (32-bit)" to "Heavy (8-bit)" in four steps.

Y-axis: left axis shows model size in megabytes and relative inference speed (normalized 0-100); right axis shows model accuracy as a percentage, using Chart.js's dual y-axis support.

Data series: three lines — "Model Size (MB)" (indigo #3F51B5, decreasing), "Inference Speed" (circuit green #2E7D32, increasing), "Accuracy (%)" (raspberry #C2185B, very slightly decreasing) — re-plotted as the slider moves between the four quantization levels.

Interaction (required): moving the slider re-renders all three lines' current-position markers and updates a text readout beneath the chart summarizing the tradeoff in one sentence (e.g., "Heavy quantization: 4x smaller, 3x faster, accuracy down 2 points"). Hovering any line's marker shows its exact value in a tooltip.

Implementation: Chart.js line chart with two y-axes (yAxisID set per dataset) and responsive: true. Store four preset data points per series keyed by quantization level, and update chart.data.datasets[i].data plus call chart.update() on the slider's input event.

Model File Formats: TFLite and ONNX

A trained and compressed model needs to be saved in a specific file format before a program can load and run it, and two formats show up constantly in edge AI work. TFLite model format — TensorFlow Lite — is a compact model file format designed specifically for running inference on mobile devices and edge hardware with limited memory and processing power. ONNX model format — Open Neural Network Exchange — is a model file format designed to be shared across different machine learning tools and frameworks, so a model trained in one tool can be exported to ONNX and then loaded and run by a completely different tool without retraining.

The AI HAT+'s software toolchain accepts models converted from formats like these into its own optimized format for the Hailo accelerator, so in practice a maker downloads or exports a model as TFLite or ONNX and then runs it through a conversion step before deploying it to the AI HAT+ — a detail handled by setup scripts rather than something you need to do by hand for the projects in this book.

Finding a Model to Start From: The Model Zoo

Given everything covered so far — training, quantization, file formats — it would be a lot of work to build a usable model completely from scratch for every project. A model zoo is a curated online collection of pretrained models, often organized by task (image classification, object detection, pose estimation) and already available in common formats like TFLite or ONNX, ready to download and deploy without training anything yourself. Hailo and Raspberry Pi both maintain model zoo resources specifically tuned for the AI HAT+, including common object-detection models already quantized and converted for immediate use.

Berry's Tip

Always check a model zoo before assuming you need to train or convert something yourself. For the vast majority of "detect common objects" projects in this book, a model zoo already has a pretrained, pre-quantized model that does exactly what you need.

The Camera-to-Model Pipeline

With a model in hand, it's time to trace exactly what happens to a single video frame from the moment light hits the camera's image sensor to the moment a labeled detection appears. The camera-to-model pipeline is the complete sequence of stages a captured frame passes through — capture, preparation, inference, and interpretation of the result — before a final detection is available to the rest of your program. Running that pipeline continuously, frame after frame, fast enough to feel live rather than like a slideshow, is what makes it a real-time video pipeline: a processing pipeline structured specifically to keep up with an incoming video stream at a usable frame rate rather than processing frames after a long delay.

Two stages inside that pipeline deserve their own definitions before you see the whole thing laid out. Image preprocessing is the set of adjustments made to a raw captured frame — resizing it to the exact pixel dimensions the model expects, adjusting color values, normalizing brightness — before that frame is handed to the model for inference, since a model trained on images of one specific size and format generally can't accept a differently-shaped input. A region of interest — often abbreviated ROI — is a specific sub-area of a full frame that a pipeline chooses to focus processing on, rather than the entire frame, which can speed up detection when a project already knows roughly where in the frame useful content is likely to appear (for example, cropping out a fixed background region that's never going to contain a moving object).

Now that preprocessing and region of interest are both defined, here's the full pipeline as one connected sequence.

Diagram: Camera-to-Model Pipeline

Run the Camera-to-Model Pipeline MicroSim fullscreen

Camera-to-Model Pipeline (interactive workflow diagram)

Type: workflow sim-id: camera-to-model-pipeline
Library: vis-network
Status: Specified

Learning objective: Students will decompose (Bloom L4: Analyze) a real-time object detection pipeline into its component stages and identify which stage is responsible for a given delay or error.

Canvas: 700x420px, responsive vis-network canvas with autoResize: true, laid out left-to-right with layout: { hierarchical: { direction: "LR" } } so the pipeline reads as a clear left-to-right sequence, switching to top-to-bottom below 560px wide.

Layout: five connected nodes in sequence — "1. Capture" (camera reads a frame from the image sensor) → "2. Preprocess" (resize, normalize, optionally crop to a region of interest) → "3. Inference" (Hailo accelerator runs the neural network) → "4. Postprocess" (apply detection threshold and non-maximum suppression, covered next in this chapter) → "5. Action" (draw boxes, trigger an event, or log a result) — colored in a left-to-right gradient from raspberry #C2185B through copper gold #D4AF37 to circuit green #2E7D32.

Interaction: clicking any node opens an infobox beneath the diagram with a one-sentence description of that stage (matching the chapter's definitions) and a representative timing contribution (e.g., "Capture: ~2 ms · Preprocess: ~3 ms · Inference: ~15 ms · Postprocess: ~4 ms · Action: ~1 ms"), so students can see which stage dominates total pipeline time. A createButton()-style HTML button labeled "Simulate a Slow Frame" temporarily colors the "Inference" node red and updates its infobox timing to a much larger value, illustrating how a single slow stage becomes the bottleneck for the whole pipeline regardless of how fast the other four stages run.

Implementation: vis-network with a nodes DataSet (five nodes with id, label, color) and an edges DataSet connecting them in sequence with arrows (arrows: "to"). Attach a network.on("click", ...) handler to open the infobox based on the clicked node's id. Implement the "Simulate a Slow Frame" behavior by updating the Inference node's color and a separate timing-data object, then calling nodes.update().

Cleaning Up Detections: Detection Threshold and Non-Maximum Suppression

A model's raw output for a single frame is rarely clean. It's common for a detector to propose dozens of candidate bounding boxes per frame, many overlapping the same real object, and many representing weak, low-confidence guesses. Two postprocessing steps clean that raw output into something usable, and both belong to the "Postprocess" stage of the pipeline diagram above.

The detection threshold is a minimum confidence score, chosen by the developer, below which a candidate detection is discarded entirely rather than reported — the same idea explored in the previous chapter's bounding box explorer, now placed in its exact spot within the full pipeline. Raising the detection threshold reduces false positives (weak, likely-wrong guesses get thrown out) but risks increasing false negatives (a real but visually ambiguous object might also fall below the raised bar).

Berry's Key Insight

Notice that "detection threshold" is just the false positive / false negative tradeoff from Chapter 16, now turned into a single number you can directly control. There's no universally correct threshold — a security application might accept more false positives to avoid ever missing a real detection, while a different project might prefer the opposite.

Even after weak detections are discarded, a model frequently reports several separate, overlapping boxes around the very same real object — the model essentially "seeing" the same dog three times with slightly different box positions and confidence scores. Non-maximum suppression — often abbreviated NMS — is a postprocessing algorithm that resolves this by keeping only the highest-confidence box in each cluster of significantly overlapping boxes for the same class, and discarding the rest as duplicates of that same detection.

Before the demo, here is non-maximum suppression's logic written as pseudocode, in the same style used throughout this book:

detections = all boxes with confidence >= detection threshold
SORT detections by confidence, highest first
kept = empty list
FOR each detection in detections (highest confidence first):
    IF detection does not significantly overlap any box already in kept:
        ADD detection to kept
RETURN kept

Reading that pseudocode alongside the earlier flowchart vocabulary from Chapter 1: this is a loop with a decision inside it, keeping a box only if it clears both the confidence check (already applied by the detection threshold) and the overlap check against boxes already accepted.

Diagram: Non-Maximum Suppression Demo

Run the Non-Maximum Suppression Demo MicroSim fullscreen

Non-Maximum Suppression Demo (MicroSim)

Type: microsim sim-id: non-maximum-suppression-demo
Library: p5.js
Status: Specified

Learning objective: Students will apply (Bloom L3: Apply) a non-maximum suppression overlap threshold to eliminate duplicate overlapping bounding boxes while keeping the highest-confidence detection in each cluster.

Canvas: 700x460px, responsive — single image-panel canvas recomputed as a fraction of width/height inside windowResized().

Layout: an illustrated scene with one real object (a simple oval "dog" shape) surrounded by five overlapping candidate bounding boxes with varying confidence scores (e.g., 0.91, 0.85, 0.78, 0.64, 0.55), all clustered around the same object, drawn with semi-transparent fills so overlaps are visible.

Controls: a createSlider() labeled "Overlap Threshold" (range 0.1-0.9, step 0.05, default 0.5, describing how much two boxes must overlap to be considered duplicates); a createButton() labeled "Run NMS" that executes the suppression algorithm at the current threshold; a createButton() labeled "Reset" that restores all five original boxes.

Interaction: clicking "Run NMS" applies the pseudocode logic from the surrounding chapter text: the highest-confidence box (0.91) is always kept and highlighted solid green; any other box overlapping it by more than the current threshold fades to a dashed gray outline labeled "suppressed (duplicate)"; boxes that do not sufficiently overlap the kept box remain visible as separate candidates for a second pass. A live readout beneath the canvas states "Kept: X · Suppressed: Y" after each run. Moving the "Overlap Threshold" slider after a run and clicking "Run NMS" again lets students see stricter or looser thresholds change the outcome.

Implementation: p5.js. Store the five candidate boxes as objects {confidence, x, y, w, h}. Implement standard rectangle intersection-over-union (IoU) as the overlap measure. On "Run NMS," sort boxes by confidence descending and apply the greedy suppression loop from the pseudocode, storing each box's resulting kept/suppressed state for rendering.

Measuring Pipeline Performance: Latency and Frames Per Second

Once a pipeline runs end to end, two numbers tell you whether it's actually fast enough to feel real-time. Inference latency is the time elapsed between handing a single preprocessed frame to the model and receiving its output — a direct measure of how long just the inference stage of the pipeline takes for one frame. Frames per second — FPS — is the number of complete frames the entire pipeline can process from capture through action in one second, a measure of the whole pipeline's throughput rather than just the inference stage alone.

These two metrics are related but not identical: a pipeline could have low inference latency for a single frame yet still achieve a mediocre overall FPS if the capture, preprocessing, or postprocessing stages are slow, since FPS depends on every stage in the pipeline diagram, not inference alone. Measuring inference latency directly in Python is a matter of recording a timestamp immediately before and after the inference call and taking the difference:

start_time = time.time()
result = model.run_inference(frame)
end_time = time.time()
latency_ms = (end_time - start_time) * 1000

This records start_time right before the inference call, end_time right after it returns, and converts the difference from seconds to milliseconds — a unit that reads more naturally for something this fast, since a well-optimized AI HAT+ pipeline often completes inference in well under 50 milliseconds per frame.

Berry's Key Insight

A common mistake is chasing a lower inference latency number and assuming FPS automatically follows. It doesn't — FPS is a property of the whole pipeline diagram from earlier in this chapter, capture through action, so a slow camera read or a heavy postprocessing step can cap your frame rate even with a lightning-fast inference stage.

Diagram: Inference Latency and Frames Per Second Benchmark Chart

Run the Inference Latency and Frames Per Second Benchmark Chart MicroSim fullscreen

Inference Latency and Frames Per Second Benchmark Chart (interactive chart)

Type: chart sim-id: fps-latency-benchmark-chart
Library: Chart.js
Status: Specified

Learning objective: Students will interpret (Bloom L4: Analyze) a benchmark comparing inference latency and frame rate across CPU-only inference and two AI HAT+ hardware accelerator variants.

Chart type: grouped bar chart, responsive Chart.js canvas.

Purpose: compare representative inference latency (lower is better) and resulting frames per second (higher is better) across three hardware configurations, showing why a hardware accelerator matters for a real-time pipeline.

X-axis: three hardware configurations — "CPU Only," "AI HAT+ (13 TOPS Hailo-8L)," "AI HAT+ (26 TOPS Hailo-8)."

Y-axis: dual-axis — left axis shows inference latency in milliseconds, right axis shows frames per second, using Chart.js's dual y-axis support.

Data series: two bars per configuration — "Inference Latency (ms)" (raspberry #C2185B, showing CPU Only far higher than either accelerator) and "Frames Per Second" (circuit green #2E7D32, showing the inverse pattern, CPU Only far lower than either accelerator).

Interaction (required): a <select> control lets students choose a model size ("Small," "Medium," "Large") which re-renders all six bars with representative values showing that the gap between CPU-only and accelerated inference widens as model size increases. Hovering any bar shows its exact value in a tooltip.

Implementation: Chart.js grouped bar chart with two y-axes (yAxisID set per dataset) and responsive: true. Store three preset datasets keyed by model size and swap chart.data.datasets on the <select>'s change event, then call chart.update().

You've Got This!

Latency, FPS, thresholds, suppression — this section packs in a lot of measurement vocabulary at once. The good news: you don't calculate any of this by hand in a real project. Once your pipeline is running, these numbers just appear in your terminal, and your job becomes reading them, not deriving them.

Teaching the Model Something New: Transfer Learning

Every model discussed so far in this chapter recognizes only the object categories it was originally trained on — common classes like "person," "car," or "dog." Real projects often need to recognize something more specific: a particular product on a shelf, a specific piece of lab equipment, a school mascot. Retraining a whole neural network from scratch for one new category would require the huge labeled dataset and heavy computing resources described in the previous chapter — resources well beyond a classroom project's reach.

Transfer learning avoids that cost by starting from a pretrained model that already knows how to recognize general visual features — edges, textures, shapes — and retraining only its final layers on a much smaller, new labeled dataset built around the specific category you want to add. Because the early layers already understand general vision, transfer learning needs far fewer example images and far less computing time than training from scratch, while still producing a model that recognizes a custom object class: a new category, not present in the original pretrained model, added specifically for a particular project's needs.

Here is the same idea as a short connected sequence, matching the earlier camera-to-model pipeline diagram's style.

Diagram: Transfer Learning Workflow

Run the Transfer Learning Workflow MicroSim fullscreen

Transfer Learning Workflow (interactive workflow diagram)

Type: workflow sim-id: transfer-learning-workflow
Library: p5.js
Status: Specified

Learning objective: Students will sequence (Bloom L3: Apply) the steps of transfer learning used to add a custom object class to a pretrained model.

Canvas: 700x420px, responsive — four connected step boxes recomputed as fractions of width inside windowResized(), switching from a horizontal row to a vertical stack below 560px wide.

Layout: four connected step boxes: "1. Start with a Pretrained Model" (general vision features already learned) → "2. Collect a Small Custom Labeled Dataset" (a few hundred images of the new object class, not millions) → "3. Retrain Only the Final Layers" (early layers stay frozen and unchanged) → "4. Fine-Tuned Model" (recognizes the original classes plus the new custom object class).

Controls: a createButton() labeled "Compare to Full Training" that temporarily overlays a second, grayed-out path above the main sequence reading "Full Training: Millions of Images -> Weeks of Compute -> New Model," letting students directly compare the size of both efforts. A createButton() labeled "Hide Comparison" removes that overlay.

Interaction: clicking any of the four main step boxes opens an infobox with a one-sentence explanation matching the chapter's transfer learning definition, plus, for step 3 specifically, a note that "frozen" means those layers' weights are not updated during this retraining pass. Only one infobox is shown at a time.

Implementation: p5.js. Step data as an array of four objects {title, description} with fixed relative x positions recomputed on resize. Hit-testing via rectangular bounding boxes. Toggle the comparison overlay's visibility with a boolean state variable set by the two buttons.

Berry's Tip

A few hundred well-chosen, varied photos of your custom object class — different angles, different lighting, different backgrounds — will usually train a far better transfer-learned model than a thousand nearly-identical photos taken in one spot. Variety in a small dataset beats sheer volume.

Bringing It Together

You've now followed a single video frame through the entire real-time object detection pipeline: captured by the camera, preprocessed and possibly cropped to a region of interest, quantized and compressed to run efficiently, run through inference on a Hailo accelerator, cleaned up with a detection threshold and non-maximum suppression, and finally measured in milliseconds of latency and frames per second. You also know how to extend that pipeline with transfer learning when a project needs to recognize something the original pretrained model never saw. That's the complete path from a bare Raspberry Pi 5 and AI HAT+ to a working real-time vision project — and every hardware and software concept from this book's Pi 5 chapters fed directly into building it.

You Unlocked a Superpower!

That's berry real-time work! You can now trace a single frame's entire journey from the camera's image sensor to a labeled, confidence-scored detection on screen — and explain exactly why each stage exists. STEM is our superpower, and this one's now yours. Let's build something!

See Annotated References