SAX-NeRF: Structure-Aware Sparse-View
X-ray 3D Reconstruction

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Yuanhao Cai, Jiahao Wang, Alan Yuille, Zongwei Zhou*, Angtian Wang
Johns Hopkins University

Reconstructing internal 3D structures from sparse-view X-rays.

CT Reconstruction During Optimization

Chest

Foot

Head

Abdomen

Pancreas

Pelvis

Jaw

Leg

Aneurism

Bonsai

Carp

Backpack

Box

Engine

Teapot

Full optimization to 75,050 or 150,050 iterations using 50 X-ray projections per scene. One frame per iteration through 500, every 10 iterations through 3,000, and every 100 thereafter, including the final iteration.

Interactive CT Volume

Drag to rotate · Scroll or pinch to zoom

Meshes extracted from the released SAX-NeRF reconstructed CT volumes.

Abstract

X-rays reveal internal structures that visible-light reconstruction cannot observe. SAX-NeRF learns a neural radiodensity field from sparse X-ray projections. Its Lineformer backbone models dependencies along ray segments, while masked local-global ray sampling combines foreground context with projection geometry. On the 15-scene X3D benchmark, it improves novel view synthesis and CT reconstruction over earlier NeRF methods by 12.56 dB and 2.49 dB, respectively.

Method Overview

Lineformer predicts attenuation coefficients along X-rays. Volume integration produces projections for supervision.

SAX-NeRF pipeline, Lineformer, line segment attention, and masked local-global ray sampling
SAX-NeRF: masked local-global ray sampling and a structure-aware Lineformer backbone.

Line Segment-based Transformer

Lineformer embeds sampled 3D positions with a hash encoder and partitions their features into short, ordered segments along each X-ray. A sequence of line segment-based attention blocks models how neighboring samples relate to one another, helping the network distinguish the internal structures traversed by the ray. The resulting features are decoded into point-wise radiodensity values for X-ray rendering.

Hij=Vijsoftmax((Kij)TQijαij)

For segment i and head j, Q, K and V denote query, key and value features; α is a learned scale. The heads are concatenated, projected and combined with positional embeddings. Eq. (9)

Masked Local-Global Ray Sampling

MLG combines dense local patches with scattered global rays to capture both nearby detail and the overall shape of the object. A thresholded foreground mask first identifies informative projection regions. Patch-level sampling selects non-overlapping windows inside the foreground, preserving neighborhood context; pixel-level sampling then draws foreground pixels outside the selected patches, increasing coverage without repeating the same rays.

ℛ=ℛl∪ℛg

Here, ℛl contains the rays from local patches and ℛg contains the globally sampled foreground rays. Their union forms the training batch used to supervise the predicted X-ray projections. Eq. (18)

Comparison between scattered ray sampling and masked local-global sampling
Global rays and local patches jointly supervise the radiodensity field.

Results on X3D

X3D covers medicine, biology, security, and industry. Results below use 50 training projections per scene.

MethodNovel View SynthesisCT Reconstruction
PSNR ↑SSIM ↑PSNR ↑SSIM ↑
InTomo31.960.976830.290.9189
NeRF32.490.977032.150.9354
NeAT36.010.963833.410.9447
TensoRF37.670.971233.780.9387
NAF38.810.978534.760.9535
SAX-NeRF51.370.999437.250.9753

Averages reported in the CVPR 2024 paper. PSNR in dB; higher is better.

X-ray Novel View Synthesis

Novel X-ray projection comparison of SAX-NeRF and previous methods on Backpack and Carp
Novel view synthesis on Backpack (top) and Carp (bottom).
Per-scene novel view synthesis results
SceneInTomoNeRFNeATTensoRFNAFSAX-NeRF
Jaw26.910.993734.840.998132.680.991134.060.996439.890.998842.750.9992
Leg42.530.997645.920.998947.710.998141.400.996950.870.998856.860.9996
Box34.650.996335.670.998536.140.995735.430.997735.980.995539.670.9992
Carp24.040.964820.620.946731.260.962037.350.997329.600.959359.880.9999
Foot39.480.997941.050.998938.240.996337.730.992938.350.991346.640.9994
Head34.830.997729.760.999127.740.929534.430.987830.170.953153.060.9995
Pelvis38.720.996140.790.997237.700.986641.570.994843.760.997553.270.9995
Chest28.950.991536.160.998840.770.999023.610.940242.370.999347.420.9994
Bonsai39.260.995337.670.998347.020.998547.800.998949.030.998955.330.9995
Teapot41.510.997834.660.999329.290.966944.180.999334.920.998552.620.9996
Engine23.990.951721.070.933430.360.885439.720.991831.680.919558.800.9998
Pancreas20.030.853719.850.856037.530.901729.240.803136.230.884449.880.9978
Abdomen27.640.964624.620.955926.740.856327.380.873037.590.985554.220.9996
Aneurism20.810.962124.970.979235.410.993647.990.999739.620.999052.910.9998
Backpack36.090.991839.750.996241.600.996943.160.997742.020.998247.170.9989
Average31.960.976832.490.977036.010.963837.670.971238.810.978551.370.9994

PSNR (upper) / SSIM (lower). Source: Table 1 of the paper.

CT Reconstruction

CT reconstruction comparison on Head, Carp, Box, and Teapot including ground truth and SAX-NeRF
CT reconstruction on Head, Carp, Box, and Teapot (top to bottom).
Per-scene CT reconstruction results
SceneFDKASD-POCSSARTInTomoNeRFNeATTensoRFNAFSAX-NeRF
Jaw28.580.781633.250.932533.130.930131.950.916232.170.911432.530.913931.900.897134.140.935835.470.9525
Leg28.480.669035.390.982635.300.980936.410.988239.270.993840.290.990240.700.992341.280.994043.470.9973
Box24.140.561631.270.922631.200.920030.590.914033.580.949431.580.929832.170.931431.780.930935.330.9602
Carp32.320.817737.630.977736.890.968232.470.949332.990.952936.850.957637.520.968737.930.971142.720.9902
Foot24.530.600029.980.920830.290.929631.430.912730.030.907230.860.922130.460.915331.630.936332.250.9403
Head26.170.715535.270.970734.880.959731.070.930334.150.967235.560.967935.530.967236.460.974339.700.9888
Pelvis26.910.636734.260.949334.380.948130.380.904231.720.917033.730.937035.130.952836.010.965440.400.9870
Chest22.890.786131.130.942232.170.959422.040.746028.400.892531.200.949730.130.930833.050.958134.380.9718
Bonsai24.530.727632.700.952933.020.960028.900.881131.770.938233.200.947633.470.952133.850.958536.510.9761
Teapot31.070.805937.350.980037.380.978736.150.978641.670.994540.850.987242.710.994242.560.992644.320.9970
Engine23.020.540530.810.958030.440.944227.490.926436.850.985836.630.980435.210.972837.840.985938.770.9917
Pancreas9.6410.123218.300.770118.360.700816.010.786517.730.861419.060.854119.750.773719.410.812622.980.9531
Abdomen22.630.603031.460.923131.400.917028.050.875429.710.904931.140.906031.510.907334.450.950135.010.9598
Aneurism28.070.729534.730.986434.760.986430.320.965231.970.935335.800.981937.360.988937.730.987141.460.9956
Backpack23.840.535131.340.930931.320.929428.770.875330.280.919231.900.934533.160.936233.260.950135.970.9688
Average25.120.642232.320.940032.330.934230.290.918932.150.935433.410.944733.780.938734.760.953537.250.9753

PSNR (upper) / SSIM (lower). Source: Table 2 of the paper.

Structural Analysis

Ablation comparison with and without line segment attention and masked local-global sampling
Both line segment attention and masked local-global sampling help recover structural details.

BibTeX

@inproceedings{cai2024structure,
  title={Structure-Aware Sparse-View X-ray 3D Reconstruction},
  author={Cai, Yuanhao and Wang, Jiahao and Yuille, Alan and Zhou, Zongwei and Wang, Angtian},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={11174--11183},
  year={2024}
}