Skip to content

A Convolutional Layer Is a Learned FIR Filter

Copy this iframe to your website:

1
<iframe src="https://dmccreary.github.io/signal-processing/sims/conv-layer-as-filter/main.html" width="100%" height="702px" scrolling="no"></iframe>

Run MicroSim in Fullscreen

Description

This MicroSim trains a tiny one-dimensional convolutional neural network (CNN) in the browser. The gradient descent is written by hand in the sketch, with no machine-learning library. Each input is a 64-sample segment that holds either a low tone (\(f = 0.08\) cycles/sample) or a high tone (\(f = 0.30\)) with a random phase, plus white Gaussian noise of standard deviation \(\sigma\). The network has \(F\) kernels of \(K\) taps each:

\[z_f[n] = \sum_{k=0}^{K-1} w_f[k]\,x[n+k] + b_f, \qquad a_f = \max_n \mathrm{ReLU}\big(z_f[n]\big), \qquad p(\text{high}) = \frac{1}{1 + e^{-\left(\sum_f v_f a_f + c\right)}}\]

Training minimizes the cross-entropy on 256 training segments, 16 per step, and accuracy is also measured on 256 separate test segments. The stem plots show each kernel \(w_f[k]\). The next plot shows its frequency response \(|H_f(f)| = \left|\sum_k w_f[k]\,e^{-j2\pi f k}\right|\), with gold markers at the two tone frequencies. The table lists each kernel's peak frequency, its gain ratio \(|H(0.30)|/|H(0.08)|\) in dB and its output weight \(v_f\). The loss and accuracy curves show training (solid) and test (dashed) results. Four test segments are drawn in blue over their clean tone (gold), with the network's prediction. Colored bars mark the \(K\) samples where each kernel responded most.

The MicroSim opens paused showing the untrained network. Run trains for up to 400 steps (about 7 s), Step applies one gradient step, Reset restores the starting weights, and New data draws a new training and test set. The sliders set the noise \(\sigma\) (0 to 2), the learning rate \(\eta\) (0.001 to 3), the kernel size \(K\) (5 to 16) and the number of filters \(F\) (1 to 3). A slider change restarts training from the same seeded weights. The output weights start at \(+0.5, -0.5, +0.5\).

This MicroSim is used in Chapter 15: Signal Processing Applications and AI Integration.

Lesson Plan

Learning Objectives

After using this MicroSim, students will be able to:

  1. Explain how a convolutional layer computes \(z_f[n]\) as an FIR filter (written as a cross-correlation), followed by ReLU and global max-pooling
  2. Analyze a trained kernel's frequency response to decide which tone it detects, using the peak of \(|H(f)|\), the ratio \(|H(0.30)|/|H(0.08)|\) and the sign of its output weight \(v\)
  3. Relate the kernel length \(K\) to frequency resolution and predict when a kernel can form a peak at the low tone
  4. Evaluate how the learning rate and the noise level affect convergence, stability and test accuracy
  5. Design a hand-made kernel that detects the high tone and compare its frequency response with the learned kernel's

Target Audience

Undergraduate juniors and seniors in a digital signal processing or machine-learning-for-signals course. The four activities take about 40 minutes.

Prerequisites

  • FIR filters, convolution and correlation (Chapter 4)
  • The frequency response of an FIR filter (Chapters 6 and 9)
  • The idea of a classifier trained by gradient descent on a loss function

Activities

The defaults are \(\sigma = 1.00\), \(\eta = 0.1\), \(K = 12\), \(F = 2\) and data set 1. Reload the page to restore them. Every number below is reproducible because the data and the starting weights are seeded.

Activity 1: Watch a kernel become a filter (10 min).

  1. Press Reset. Before training, \(w_1\) peaks at \(f = 0.000\) and \(w_2\) at \(0.176\) cycles/sample, and the test accuracy is 69.5%.
  2. Press Step five times. \(w_1\) now peaks at 0.305 and \(w_2\) at 0.089, and the test accuracy is 85.9%. Each step moved a kernel toward one of the tones.
  3. Keep pressing Step: the test accuracy first reaches 95% at step 11 (95.7%). Then press Run. At step 400, \(w_1\) peaks at 0.304 cycles/sample with \(|H(0.30)|/|H(0.08)| = +30.4\) dB and \(v = +2.04\). \(w_2\) peaks at 0.081 with \(-26.7\) dB and \(v = -2.00\). The test accuracy is 96.5% (training 99.2%) and the losses are 0.027 (training) and 0.081 (test).
  4. Look at the \(w_1\) stems. They form a short burst of a sinusoid whose period is close to \(1/0.30 \approx 3.3\) samples.

Activity 2: How many filters does the task need? (8 min).

  1. Set \(F = 1\). The single kernel peaks at 0.306 with \(v = +2.42\), and the test accuracy falls to 90.2%. The network can only detect the high tone, so "low" means "no high tone found". The third test segment (a low tone) gets \(p(\text{high}) = 0.55\) and is misclassified.
  2. Set \(F = 3\). The test accuracy rises to 98.4%. \(w_1\) and \(w_3\) peak at 0.306 and 0.310 (high-tone detectors, \(v > 0\)) and \(w_2\) peaks at 0.073 (low-tone detector, \(v < 0\)). Each kernel's role follows the sign of its starting output weight.

Activity 3: Kernel length sets the frequency resolution (10 min).

  1. Set \(F = 2\) and \(K = 5\). \(w_1\) peaks at 0.269, but \(w_2\) peaks at \(f = 0.000\). Five taps hold only 0.4 of a period of the low tone, so that kernel becomes a smoothing (low-pass) filter. The test accuracy drops to 86.7%.
  2. Move the \(K\) slider to 6, 7 and 8, letting each run finish. The low-tone kernel peaks at 0.120, 0.086 and 0.084 cycles/sample.
  3. Set \(K = 16\). The peaks are 0.302 and 0.093 and the test accuracy is 99.2%. A \(K\)-tap kernel has a main lobe about \(2/K\) wide: 0.40 cycles/sample for \(K = 5\) and 0.125 for \(K = 16\). Compare those widths with the 0.16 gap between the low tone at \(+0.08\) and its mirror image at \(-0.08\).

Activity 4: Learning rate and noise (12 min). Start from \(K = 12\) and \(F = 2\).

  1. Set \(\eta = 0.001\). After 400 steps the test accuracy is only 85.2%, the training loss is still 0.593, and \(w_2\) rejects the high tone by only 5.9 dB. The kernels have barely moved.
  2. Set \(\eta = 1\), press Reset and press Step ten times. The test accuracy is 50% and \(w_2\)'s output weight has flipped from \(-0.50\) to \(+0.22\). Press Run: at step 400 both kernels are high-tone detectors (peaks 0.301 and 0.303) and the test accuracy is 88.3%.
  3. Set \(\eta = 3\). The weights blow up at step 11 and the sim reports that training diverged.
  4. Set \(\eta = 0.1\) and \(\sigma = 2.00\). The curves jump from step to step and the test accuracy ends at 63.3% (training 76.6%). With \(\eta = 0.01\) it ends at 71.1% (training 80.9%), and the kernels still peak at 0.313 and 0.087. With \(\sigma = 0.50\) and \(\eta = 0.1\), Reset followed by two Step presses already gives at least 95%, and the run ends at 100%.

Assessment

  1. With \(K = 12\), how many positions \(n\) does the global max-pool compare for each kernel, and which input samples does the winning position use? Answer: \(64 - 12 + 1 = 53\) positions. The winner \(n^*\) uses \(x[n^*], \dots, x[n^* + 11]\), the span marked by the colored bar.
  2. A trained kernel has \(|H(0.30)|/|H(0.08)| = +30\) dB and output weight \(v = +2.0\). Which class does a strong response support, and by what factor does the kernel pass the high tone more than the low tone? Answer: the high tone, because \(v > 0\) raises the logit. +30 dB is an amplitude ratio of \(10^{30/20} \approx 32\).
  3. For a kernel with \(a_f > 0\), the gradient is \(\partial L/\partial w_f[k] = (p - y)\,v_f\,x[n^* + k]\). Explain why gradient descent makes \(w_1\) (with \(v_1 > 0\)) look like the high tone. Answer: each step subtracts \(\eta (p - y) v_1\) times the winning input window. For a high-tone segment \((p - y) < 0\), so a copy of the 0.30 tone is added; for a low-tone segment \((p - y) > 0\), so a copy of the 0.08 tone is subtracted. The noise in these copies averages out, so the kernel becomes a band-pass filter matched to the high tone.
  4. Why does a 5-tap kernel trained on the low tone peak at \(f = 0\) instead of 0.08, and what is the smallest \(K\) in the sim whose low-tone kernel peaks near 0.08? Answer: five taps cover only 0.4 of a period, and the main lobe (about \(2/K = 0.4\) cycles/sample wide) merges the responses at \(+0.08\) and \(-0.08\) into one lobe at 0. In the sim \(K = 6\) gives a peak at 0.120 and \(K = 7\) gives 0.086.
  5. Design a 12-tap kernel that detects the high tone without training, and predict its frequency response. Answer: \(w[k] = \cos(2\pi \cdot 0.30\,k)\) for \(k = 0, \dots, 11\). Its \(|H(f)|\) peaks at 0.302 cycles/sample with a main lobe about \(2/12 \approx 0.17\) wide, close to the learned \(w_1\) (peak 0.304).

Common Misconceptions

"The weights of a neural network cannot be interpreted." The frequency-response plot shows that each trained kernel is an FIR band-pass filter. After pressing Run with the default settings the kernels peak at 0.304 and 0.081 cycles/sample, within 0.004 of the tones they vote for, and the sign of the output weight says which tone that is.

"A CNN convolution is the convolution of Chapter 4." Deep-learning layers compute \(\sum_k w[k]\,x[n+k]\), a cross-correlation with no kernel flip. Flipping a kernel in time changes only the phase of \(H(f)\), not \(|H(f)|\). So the learned kernel is the time-reversed impulse response of an ordinary FIR filter with the same magnitude response.

"A larger learning rate always trains faster." In Activity 4, \(\eta = 1\) flips the role of the second kernel and ends at 88.3% test accuracy instead of 96.5%, and \(\eta = 3\) diverges at step 11.

References

  1. Convolutional neural network - Wikipedia - convolutional layers, shared weights, ReLU and pooling
  2. Finite impulse response - Wikipedia - FIR filters and their frequency response, the view used for every learned kernel
  3. Cross-correlation - Wikipedia - the operation a CNN layer actually computes, and how it differs from convolution
  4. Matched filter - Wikipedia - why a kernel shaped like the target signal maximizes the detection signal-to-noise ratio
  5. Gradient descent - Wikipedia - the update rule used to train the network and the role of the learning rate
  6. Cross-entropy - Wikipedia - the loss function plotted in the loss panel
  7. The Scientist and Engineer's Guide to Digital Signal Processing - Steven W. Smith - free textbook with chapters on FIR filters and on neural networks