22. Entropy Loss in Linear Filters
Filtering Reduces Information
A linear filter is a deterministic operation. Intuitively, it cannot create new information, but can it preserve all existing information? Shannon shows: unless the filter is perfectly invertible, entropy is lost.
The Filter Equation
Input ensemble \(\{x(t)\}\) with spectral density \(S_x(f)\) passes through a linear filter with transfer function \(Y(f)\):
Output spectral density = input spectral density × squared magnitude response.
Entropy Change
The entropy rate changes by:
For an ideal bandpass filter that passes frequencies in \([f_1, f_2]\) and rejects others:
The integral becomes \(\log 1 = 0\) in the passband, but the output is now band-limited, so its maximum entropy is reduced.
The Ideal Low-Pass Filter
Passing white noise through an ideal low-pass filter with cutoff \(W\):
But the output has finite bandwidth, so its entropy per degree of freedom is higher than the input's. The total entropy is preserved, but concentrated into fewer degrees of freedom.
For a non-ideal filter with roll-off, some information is irretrievably lost in the transition band.
Recoverability
If \(|Y(f)| > 0\) for all \(f\) in the signal band, the filter is invertible (at least in principle). The original signal can be recovered by inverse filtering:
(with regularization \(\epsilon\) for numerical stability).
If \(|Y(f)| = 0\) for some frequencies, those frequency components are permanently lost.