Differentiable 3D Gaussian Volumetric Pathtracer
This project is a differentiable volumetric path tracer based on the paper Don't Splat your Gaussians. Their model represents participating media as 3D Gaussians in a formulation more faithful to the radiative-transfer-equation (RTE). I implemented their volumetric pathtracer from scratch in a minimal renderer for a directed-readings course and it was originally going to be the topic of my Master's thesis (before I shifted topics).
The motivation was compression: a mixture of a few thousand Gaussians can stand in for a dense VDB volume at a fraction of the memory, and the formulation does not give up physical accuracy.
The main limitation is being able to generate a good Gaussian mixture from a VDB quickly. For this reason, I implemented (weighted) stochastic Expectation-Maximization to fit a VDB into a Gaussian mixture. That fit is then refined against reference renders through the differentiable pipeline to create the most faithful reconstruction.
Renders
All renders at 256 spp. Use the arrows to scroll and compare; render times and memory are in the table below.
The VDB reference uses delta tracking, ratio tracking, and DDA traversal of a per-brick majorant grid, whereas the Gaussians have closed-form optical depth and are sampled analytically, similar to regular tracking.
Showing → Delta-tracking VDB (reference)
Compression
The reference is the Stanford Bunny VDB: 19.2 million active voxels, ~73 MB of float density. Each Gaussian instead needs just 11 floats (mean, covariance, density, albedo = 44 bytes), so the fits are up to thousands of times smaller and render much faster; all while solving the RTE with no simplifying assumptions.
The last bunny is the VDB itself downsampled to the same ~1.7 MB memory footprintas the 40k-Gaussian fit. The quality is comparable but the Gaussian fit is more efficient to render. Timings were recorded on an Intel Core i7-13700H (14 cores / 20 threads).
| Representation | Primitives | Size | vs VDB | Render |
|---|---|---|---|---|
| Delta-tracking VDB (reference) | 19.2M voxels | 73 MB | — | 4 min 29 s |
| 400 Gaussians | 400 | 17 KB | ~4,400× smaller | 39 s |
| 4k Gaussians | 4,000 | 172 KB | ~440× smaller | 1 min 28 s |
| 40k Gaussians | 40,000 | 1.7 MB | ~44× smaller | 3 min 12 s |
| VDB downsampled to 40k size | 452k voxels | 1.7 MB | ~44× smaller | 4 min 34 s |
| 10k random Gaussians | 10,000 | 430 KB | — | 47 s |