Chapter 12: Deep Learning and AI for CT

CT image quality is ultimately a trade against dose. Lower the incident photon count $I_0$ to spare the patient and, as Chapter 4 showed, Poisson noise grows in the projections and the ramp filter amplifies it into a grainy FBP image. The iterative methods of Chapter 8 fight back by modeling the noise statistics and adding prior knowledge (regularization), at the price of computation and tuning. In the late 2010s a third option arrived: learn the mapping from low-dose to high-dose images directly from data. Deep-learning reconstruction (DLR) is now available on several clinical CT platforms.

Image-domain deep learning: residual learning

The most basic setup is supervised learning in the image domain. Take a low-dose reconstruction $y$ as input and a high-dose reconstruction $x$ as the target, and train a neural network $f_\theta$ to map one to the other. The standard trick is not to predict $x$ directly but to predict the noise $y - x$ and subtract it from the input (residual learning):

$$\hat{x} = y - f_\theta(y), \qquad L(\theta) = \mathbb{E}\,\bigl[\,\lVert f_\theta(y) - (y - x) \rVert_2^2\,\bigr]$$

The reason is simple: the network starts from something close to "do nothing" ($f_\theta \approx 0$). It never has to learn the identity mapping, only the zero-mean noise component, so even a small network trains fast and stably. The landmark methods of this family, RED-CNN for low-dose CT and DnCNN for image denoising, all take this form.

Architecture of the small CNN

The network this chapter trains is a genuine convolutional neural network, just a miniature one: three 3×3 convolution layers (channels 1 → 8 → 8 → 1) with ReLU between them. Layer $l$, channel $c$ computes

$$z^{(l)}_{c} = \mathrm{ReLU}\!\left(b_c + \sum_{c'} w_{c c'} * z^{(l-1)}_{c'}\right)$$

It has 737 parameters in total and a 7×7-pixel receptive field. RED-CNN has about 1.8 million parameters, so ours is over 2000× smaller, but convolution, residual learning, and stochastic gradient descent work exactly the same way, and this scale is enough to remove pixel-scale FBP noise.

The training data are synthesized on the fly: generate random ellipse phantoms, forward-project, add Poisson noise, and reconstruct with FBP to get noisy/clean image pairs, then sample 32×32 patches into mini-batches. The optimizer is Adam, which normalizes each parameter's step size by running averages $\hat m_t, \hat v_t$ of the gradient's first and second moments, so one learning rate works across noise levels with essentially no tuning:

$$\theta_{t+1} = \theta_t - \eta\, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}$$

U-Net: the workhorse architecture of medical imaging

The 3-layer CNN's receptive field was 7×7. Long-range artifacts such as streaks need far more context, but naively stacking convolution layers makes both compute and parameters balloon. U-Net's answer is downsampling: halve the resolution and the same 3×3 convolution effectively covers twice the distance. Shrink to read context (the encoder), then expand back to full resolution (the decoder). The detail lost in shrinking is restored by skip connections that hand the same-resolution encoder features across to the decoder. The U shape formed by these bridges gives the network its name.

This structure is the de facto standard of medical image processing. Since FBPConvNet (2017) applied it to sparse-view CT, most low-dose denoising and learned-MAR networks have been U-Net variants: the multi-scale receptive field fits CT's demand of removing long-range artifacts while preserving pixel-scale detail.

The simulation below can train a 2-level miniature: conv 1→8 → average pooling → conv 8→16 → nearest-neighbor upsampling → skip concatenation 8+16→24 → conv 24→8 → conv 8→1, with the same residual subtraction at the output. It has 3057 parameters (about 4× the 3-layer CNN) and a receptive field of roughly 12×12. The original U-Net digs 4-5 levels deep with max pooling, but the principle is exactly this miniature's. Switch the architecture in the simulation and compare convergence speed, final RMSE, and edge preservation.

Simulation: train the tiny CNN in your browser

Press "Start training" and the training loop runs in a Web Worker at tens of steps per second. Watch two curves fall, the training loss (MSE) and the RMSE on a held-out validation image, while the noise fades out of the CNN-output panel in real time. Switching the training noise $I_0$ retrains from scratch for that noise level. Switching the architecture likewise retrains from scratch.

Testing generalization

How well does the trained network work on data it has never seen? The next simulation applies the network you just trained to the Shepp-Logan phantom or your own free-draw phantom, side by side with a classical baseline, Gaussian smoothing (σ = 1.2 px).

There are two things to look for. First, when the test noise matches the training level, the CNN should beat Gaussian smoothing both visually and in RMSE, removing noise while keeping edges, having learned to subtract noise-like patterns rather than blur uniformly. Second, move the test noise away from the training level and the behavior changes: noise stronger than trained is left partially unremoved, while a network trained on strong noise oversmooths cleaner images and erases fine detail. Outside its training distribution, a learned method carries no guarantees, which is a fundamental characteristic of deep learning.

Simulation: apply the trained CNN to unseen data

Train for at least 100 steps in the simulation above, then press "Apply trained CNN". Vary the phantom and the test noise to find the conditions where the CNN excels and where it fails. Draw thin structures in the free-draw phantom and you may watch the CNN mistake them for noise and erase them.

Limits and risks: hallucination and evaluation

For this reason, pixel-wise error metrics such as RMSE and PSNR are considered insufficient for evaluating deep-learning methods: they reward oversmoothed images and cannot measure whether diagnostically relevant detail survives. Task-based evaluation, measuring low-contrast lesion detectability with phantoms and observer studies, is what matters, and when reading clinical dose-reduction claims it matters greatly under which evaluation conditions they were obtained.

Beyond image-domain processing

Post-processing reconstructed images is only the starting point. Research and products have been moving upstream, toward where the data are born.

Projection and dual domains: noise and data loss are native to the sinogram, so networks can correct there, or span both domains connected by a differentiable reconstruction layer. For the metal artifacts of Chapter 11, learned MAR methods (DuDoNet and successors) inpaint the sinogram with a network instead of LI or NMAR.

Unrolled networks: truncate the iterative update of Chapter 8 at a finite number of iterations and replace each one with a learnable layer (LEARN, Learned Primal-Dual):

$$x^{k+1} = \Lambda_{\theta_k}\!\left(x^k,\; A^{\mathsf T}\!\left(A x^k - p\right)\right)$$

The physics model $A$ remains inside the formula while the regularization-like part $\Lambda_{\theta_k}$ is learned. This is an advantage over image-only post-processing, but a finite unrolled network does not automatically guarantee data consistency. The final residual $A\hat x-p$ should still be evaluated.

Diffusion models and generative priors: use a diffusion model, which generates images from noise step by step, as a prior for "what CT images look like", and steer the generation to stay consistent with the measured data. Powerful for severely undersampled problems such as sparse-view and ultra-low-dose CT, but being generative, hallucination management becomes all the more important.

Clinical DLR: several vendors have introduced clinical deep-learning reconstruction, including Canon's AiCE, GE's TrueFidelity, and Philips' Precise Image. Training data, losses, and reconstruction pipelines differ by product and are not always public. Improvements in noise and reader preference have been reported, but the effect depends on anatomy, dose, detection task, and comparator.

Practical considerations

FBP uses an analytic inverse, iterative methods combine a physics model with explicit regularization, and deep learning learns some of that regularizing behavior from data. Its performance depends on the training distribution and may not transfer to unseen acquisition conditions or pathology. Evaluation should include data consistency and task-based measures, not only the appearance of the output image.

References

• Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI, 234–241 (2015).
• Chen H et al. Low-dose CT with a residual encoder-decoder convolutional neural network. IEEE Transactions on Medical Imaging 36, 2524–2535 (2017).
• Jin KH et al. Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Processing 26, 4509–4522 (2017).
• Adler J, Öktem O. Learned Primal-Dual Reconstruction. IEEE Transactions on Medical Imaging 37, 1322–1332 (2018).
• Samei E et al. Performance evaluation of computed tomography systems. Medical Physics 46, e735–e756 (2019).
• Mileto A et al. State-of-the-Art Deep Learning CT Reconstruction Algorithms in Abdominal Imaging. RadioGraphics 44, e240095 (2024).