Union & Clear Angle develop new crowds pipeline

Union and Clear Angle Studios have developed a new crowds pipeline using relightable Gaussian Splats. 

The workflow has been created for a current project, but the pair believe it can be used more widely in the industry.

Gaussian Splats recreate objects and people using millions of tiny volumetric points that blend together into incredibly photoreal assets instead of building traditional CG characters from scratch. This reduces the need for manual cleanup, texturing, lighting and animation.

For this workflow, Clear Angle’s new Volumetric Capture Rig (VCR) uses 40 cameras, running at up to 60 frames per second,  with controllable LED lighting to capture performances within a 3m by 3m area. 

The scanned data then runs through a custom training pipeline developed jointly by the two studios before being brought into Union’s VFX pipeline as fully usable USD CG assets. The resulting splats can be viewed from any camera angle, dropped into any environment, and rendered through standard CG renderers.

Union image segmentation technology, which allows artists to isolate and adjust costume colours and materials, creating much more visual variety in crowds. In practice, as the VCR can comfortably capture small groups of performers together, a small number of 20-second captures can now become a crowd of tens of thousands by applying time-offset animation, varied placement, and segmentation.

The issue in the past would have been relighting these creations, and David Schneider, VFX & technical supervisor at Union, said: “There are experiments out there that ‘relight’ splats using relighting baked in at training time or render-time relighting, resulting in the introduction of artefacts. 

“Our solution allows the splats to be lit with the same lighting setups used in other CG workflows without specialised render support.  They behave like traditional CG assets, but are far more realistic and come complete with animation, hair and fabric movement.  They can also be lit and rendered within CG environments and alongside traditional CG assets.

“This results in incredibly realistic splat crowds that accurately match your choice of lighting environments while keeping everything looking natural and consistent. Splats hold up way closer to camera than traditional CG crowd, allowing productions to create more convincing crowds with fewer unique elements.”

In addition, the capture area can accommodate devices that enable movement such as a treadmill, bike rollers, a gimbal or wires; action captured can then be placed into a CG environment creating believable movement.

Tim Caplan, CEO and founder, Union VFX, said: “It’s really exciting technology. As far as we know, we’re the only studio currently offering truly relightable Gaussian Splats in production. Being able to relight splats opens up huge creative possibilities for digi-doubles with realistic hair and cloth simulation, giving far greater realism and utility than building them traditionally.

The costume variation tools, alongside the human imperfection of the captured material, make our digital crowds feel far more natural and diverse than a traditional CG approach. 

The earlier productions start thinking about how they can utilise this tech, the more time and money they can save across the production process.”

Dominic Ridley, CEO of Clear Angle Studios, added: “Our entire ethos is about being faithful to reality. The VCR delivers a dataset that fully preserves the talent’s original, nuanced performance in all its glory. By providing VFX teams with robust, accurate data as a foundation, we make life easier downstream and ensure the final result is completely grounded in truth.”