What is Gaussian splatting?

Capture is a cornerstone of the visual effects pipeline. Established techniques such as LiDAR and photogrammetry have become standard tools across high-end production workflows. Gaussian splatting builds on those foundations while introducing new possibilities for capturing complex scenes and materials.

Gaussian splatting takes the same fundamental approach as existing scanning methodologies, based on photogrammetry, but offers significant advantages in terms of application, fidelity, and processing speeds. With a Gaussian splat, the heightened ability to capture fine detail in complex materials like hair, smoke, or reflective surfaces is opening up new avenues for asset development. While Gaussian splatting is not a replacement for traditional methods, it expands the range of production and VFX scenarios and offers exciting potential to capture what was once impossible. 

What makes Gaussian Splatting different? 

Gaussian splatting is a method for creating a digital 3D representation of a real scene. It is a sub-category of ‘radiance fields’, an emerging technology which focuses on capturing light’s behaviour within a physical space.

Instead of building a mesh-based 3D model, the method creates a dense cloud of tiny, semi-transparent shapes. Each Gaussian stores information including position, colour, opacity and scale, like an elliptical drop of translucent paint suspended in 3D space. The shape is densest at its centre and gradually fades out towards its edges, and has a specific position, colour, size, and opacity. To render from a 3D form to a 2D frame, these ellipsoids are flattened or ‘splatted’ by projecting them onto a plane from the perspective of a virtual camera. A volume of these shapes can then be splatted, in real time, from any point of view to create a seamless 3D rendering that is photorealistic from any angle.

One of the greatest strengths of Gaussian splatting is in giving the ability to capture and reproduce subjects that don’t have a solid, defined surface. This sort of element can completely confuse a mesh-based digital system but Gaussian splatting captures these volumetric characteristics directly. Rather than attempting to reconstruct the subject as a hard, physical piece of geometry, Gaussian splatting essentially captures the colour and transparency of light throughout a volume. This means it can represent wisps of smoke, individual strands of hair, the distorted view of water through glass, and all kinds of other details which, until now, would have needed to be simulated from scratch by a VFX artist. If 3D meshes are really what you want they can still be derived from Gaussian splat data, depending on the materials of the subject.

What are the production applications? 

The most immediate impact of Gaussian splatting can be seen across production workflows and on-set efficiency.

An interesting use case is capturing an environment for use in CG. Traditionally, this might require a team to shoot a combination of LiDAR, photogrammetry, texture photography and lighting reference data, which then requires some heavy post work before it’s usable in a CG scene.

With a Gaussian splat, on the other hand, the technician captures a similar dataset but the processed asset retains much of the lighting and material properties that would need time consuming manual simulation in a 3D scan. Translucency, reflection, a little bit of mist on the ground - these kinds of elements are already present in the asset, meaning very little cleanup is required before it can be used. This is a huge gain in terms of production pipeline efficiency and budget. 

Gaussian splats can reproduce moving subjects just as well as static scenes, with each frame rendered as an individually calculated splat. We refer to this as 4D, with the fourth dimension representing ‘time’. From the authentic motion of a real explosion, the subtle way light filters through a forest canopy, or the sweat dripping from a boxer’s brow, Gaussian splatting enables realism, creative flexibility, and the potential for new approaches to storytelling.

Directors can approach the lighting, staging and action as they would any other scene, with the authentic performances captured in detail but with the flexibility of deciding the camera position in post. Gaussian splatting provides particular utility for crowd work, capturing small groups of performers in motion and then bringing them into new digital environments, and the same nuanced performances of actors captured in a studio environment can be transferred to distant environments such as spaceships or more grounded locations based on real-world locations. 

The potential applications of Gaussian splatting go beyond VFX. In the virtual production space, its real-time rendering capabilities mean it can be used to create photorealistic digital backdrops for LED volumes. As a pre-visualisation tool, it can allow directors to walk through virtual sets and plan shots, and it’s playing an emerging role in the video game industry, particularly with VR content.

How is Gaussian Splatting being used today? 

It’s still early days, but Gaussian splatting is already being used in targeted and strategic ways in high-end TV production, especially in its capacity to complement existing tools. The technique excels in capturing non-rigid or volumetric phenomena – elements like fire, water, smoke, and reflections. Studios and providers are exploring how best to utilise this capability using data directly from the scan. 

Others are exploring the post-production gains of Gaussian splatting, the primary benefit of which is the elimination of long processing times for iteration. As a result of gains in rendering times, an artist working with a scene captured via Gaussian splatting can move the virtual camera and see the result instantly. We are also beginning to see experimentation with editing the content of a splat, including re-lighting, and changes to rigging and animation. These aspects represent a massive leap from traditional workflows, where even small changes could require hours of rendering to preview.

Is this a new normal for capture? 

Gaussian splatting is proving its value in cutting-edge projects. Its promise of flexibility, speed, and its ability to reproduce incredibly complex materials makes it a powerful addition to the production toolkit. At Clear Angle Studios, it is already a practical solution for many real-world production challenges, and will support new creative directions in television.

As the underlying radiance field technology continues to evolve, Gaussian splatting is moving beyond research and into practical production use. While challenges around editability, relighting and pipeline integration remain active areas of development, progress is rapid with new workflows continuing to emerge. Gaussian splatting’s greatest impact is likely to come as a technology that expands what productions can capture efficiently and convincingly, further cementing its place in the future of digital content creation.

Alex Dewar is lead capture technician at Clear Angle Studios