I've been playing with 3D Gaussian splatting recently — a photogrammetry technique that turns a bunch of photos into a 3D scene you can view from any angle. Below are a few scans: the chimney cladding project, our front garden and my scout groups' building that we are renovating.
It's amazing it works at all: I just walked around and recorded a video. Then of course there's a bit of software involved. I threw the video through COLMAP and Lichtfeld Studio and after a bit of back and forth between Docker containers running them, I got the results below. They're a bit downsampled to not have gigabyte-sized files on the website.
Next thing to try is Spirula Studio, which integrates the whole video-to-splat pipeline in a single application. The main video I want to try that on is from the renovation of my scout groups' building, after a day of demolishing some old walls that had to go. Not quite done yet, I might need a bigger computer for that. (Update: I for sure need a bigger computer for that. The Wickiup splat was made through Splatware)
The scenes are downsampled a fair bit to make them a reasonable size for a website, but that does diminish the quality though.
The chimney, after the cladding was finished:
(drag to orbit; scroll to zoom; right-drag to pan — loads the ~26 MB scan on demand)
Front garden:
(loads the ~25 MB scan on demand)
The Wickiup:
The building is getting renovated, but first some walls needed to be demolished. I thought that there would be plenty of keypoints to be found in that scene. (loads the ~34 MB scan on demand)
What is Gaussian Splatting?
It's a way to make a 3D model from a set of images. The 3D model consists of spheres (or rather ellipses) with each a position in space, a color, size and some tranparency. Recreating a single image with these kind of spheres (or circles/ellipses in that case) is quite do-able maybe even manually. It gets interesting when you need to have the same spheres match multiple images of the scane scen from different orientations and points over view. The only way to do that is to actually draw the 3D scene with those spheres.
Doing this for a scene with eg a football or rugby ball would be relatively straightforward, but of course not when the scene involves plants, a table, people or anything really.
Getting the spheres to match a whole video worth of images is an optimisation or training process. That process is Gaussian Splatting, and the spheres/spheroids are called splats.
The training process begins with potentially random splats thrown around the scene. Starting with a pointcloud from a Structure from Motion process improves this step already. That won't look like the images too much initially, so the positions, sizes, colors and transparency or the spheroids is tweaked a little bit, to look a bit more like the images in the dataset. For example, if there is eg. a red couch in a house scene, a single red spheroid might be a good step towards looking like the image of that couch. For more detail, add few more spheres. Maybe brown, stretched spheroids at the floor, etc.
That refinement, improvement process is repeated until a good enough similarity to all of the images is achieved.
Of course, there is lots more detail in each and every of those steps. You can have 3D spheres, but to represent the average flat wall, floot, table, flat 2D ellipses might be better than 3D spheres.
This is all better explained at https://radiancefields.com/what-is-gaussian-splatting and https://en.wikipedia.org/wiki/Gaussian_splatting
I also had Claude create a little animation for me, since that is now possible. My prompt was just "Make an animation of how Gaussian Splatting works for a simple scene" and this is what I got. I briefly pondered making a set of drawings, with paper and pen but I think this is much clearer:
How Gaussian splatting works, step by step
The animation below walks through the whole training process on a toy scene (a sphere, a box and a floor): press play, scrub the timeline, or jump straight to a stage with the buttons.