(a) 3D Gaussian splatting (3DGS) results trained for 7K iterations. (b) 3DGS results trained for 30K iterations, in whichmore Gaussian primitives are allocated so quality gets higher, speed gets slower, and storage gets larger compared to (a). (c) SUNDAE results by pruning 90% primitives upon (a), being much smaller in storage, more accurate and a little bit slower than (a). Note that the storage usage is not 10% of (a) because an additional neural compensation head is used.
Recently, 3D Gaussian Splatting, as a novel 3D representation, has garnered attention for its fast rendering speed and high rendering quality. However, this comes with high memory consumption, e.g., a well-trained Gaussian field may utilize three million Gaussian primitives and over 700 MB of memory. We credit this high memory footprint to the lack of consideration for the relationship between primitives. In this paper, we propose a memory-efficient Gaussian field named SUNDAE with spectral pruning and neural compensation. On one hand, we construct a graph on the set of Gaussian primitives to model their relationship and design a spectral down-sampling module to prune out primitives while preserving desired signals. On the other hand, to compensate for the quality loss of pruning Gaussians, we exploit a lightweight neural network head to mix splatted features, which effectively compensates for quality losses while capturing the relationship between primitives in its weights. We demonstrate the performance of SUNDAE with extensive results. For example, SUNDAE can achieve 26.80 PSNR at 145 FPS using 104 MB memory while the vanilla Gaussian splatting algorithm achieves 25.60 PSNR at 160 FPS using 523 MB memory, on the Mip-NeRF360 dataset.
Relationship among Gaussian Primitives. The left panel shows vanilla 3D Gaussian splatting, which requires a large amount of storage as it does not capture the relationship between primitives. The middle panel shows our spectral pruning technique that is based upon the relationship between 3D Gaussians. The right panel shows that the neural compensation head exploits the relationship between 2D feature splatting results to improve rendering.
(a) Pipeline: Our proposed method warms up a 3D Gaussian field firstly, followed by a Graph-based pruning strategy to down-sample the Gaussian primitives, and a convolutional neural network to compensate the losses caused by pruning.
(b) Graph-based Pruning: A graph based on the spatial relationship between the Gaussian primitives, is utilized for pruning post warm-up. Employing a band-limited graph filter, this process facilitates the extraction of fine details from high-frequency components, alongside capturing general features from low-frequency parts, thereby enabling a comprehensive and efficient representation of the entire scene.
Our method effectively balances high rendering performance, speed, and efficient memory usage, demonstrating significant advancements. Specifically, at 30% and 50% sampling rates, it manages to deliver top-tier performance metrics while achieving rapid rendering speeds of 88 FPS and maintaining a manageable memory footprint of 393MB. This balance showcases the strength of our spectral pruning and neural compensation techniques, which adeptly handle the relationships among Gaussian primitives, ensuring minimal quality loss even when reducing the abundance of primitives. The efficiency of our approach is further highlighted by the SUNDAE variant, which at very low sampling rates like 10% and even 1%, still offers remarkable rendering efficiency and reduced memory demands, maintaining competitive image quality. Overall, our method stands out for its robustness and capability to optimize the trade-offs between quality, speed, and memory usage in complex rendering environments.
@misc{yang2024spectrally,
title={Spectrally Pruned Gaussian Fields with Neural Compensation},
author={Runyi Yang and Zhenxin Zhu and Zhou Jiang and Baijun Ye and Xiaoxue Chen and Yifei Zhang and Yuantao Chen and Jian Zhao and Hao Zhao},
year={2024},
eprint={2405.00676},
archivePrefix={arXiv},
primaryClass={cs.CV}
}