This page records a 4D Gaussian Splatting pipeline built around a synchronized multi-camera capture. It is an experimental post-processing workflow rather than a built-in Camera Server process task. Tool versions, scripts, and model-training settings change quickly, so verify each external project's current instructions before starting.

Capture and organize the source dataset

1. Use Camera Server Camera control to connect, synchronize, and trigger the cameras.
2. Capture a complete image sequence with consistent exposure, focus, and lighting.
3. Keep the original dataset intact and make a working copy for frame extraction and reconstruction.
4. Confirm that camera numbering, timestamps, filenames, and audio or trigger references are preserved.

The quality of the temporal result depends on consistent capture timing and camera registration. Fix missing or misnumbered source files before moving to training.

Example pipeline

The legacy workflow used these stages:

1. Camera control and trigger: Xangle Camera Server.
2. Sound synchronization and frame extraction: PeakAlign.
3. Camera registration: RealityScan or a compatible RealityCapture workflow.
4. Sparse point-cloud generation per frame: RealityScan driven by a custom script.
5. Gaussian Splat training: Postshot using a custom script and an MCMC-based training workflow.
6. Floater removal: PointNuker.
7. Compositing and review: After Effects with a Gaussian Splat importer and a flicker-reduction tool where needed.

This list is a pipeline map, not a claim that each tool is required or supported by Camera Server. Keep the intermediate files for every stage so a failed training run does not require a new capture.

Camera registration and frame preparation

Register the cameras using a clean reference dataset and the current RealityScan or RealityCapture workflow. Review the solved camera positions before generating per-frame point clouds.

Use PeakAlign or an equivalent tool to relate the sound or trigger event to the extracted image frames. Check the offset on a short test sequence before extracting the complete production range. Keep the source audio, frame numbering, and offset values together in the working project.

Training and cleanup

Train the splat from the registered per-frame data using settings appropriate for the available GPU and the intended playback quality. Training time and memory use vary with camera count, frame count, resolution, and scene complexity.

After training:

• Review the motion for temporal drift and flicker.
• Remove floaters conservatively so valid moving details are not lost.
• Check the beginning, middle, and end of the sequence, not only a representative frame.
• Export an intermediate preview before committing to a long final render.

Feed-forward 4DGS evaluation (July 2026)

The pipeline above is per-scene optimization: registration plus training per capture, measured in hours. Feed-forward 4DGS models predict the dynamic Gaussian representation in a single forward pass, which makes them candidates for a built-in Camera Server process task comparable to the Extractor. Findings below come from reviewing the feed-forward section of the awesome-4dgs catalog.

Candidate models

Sparse-view options (4C4D, UFO-4D, HiReFF) become relevant only if rig density drops to 2 to 4 cameras.

Rig context

Xangle rigs are 4 to 250 cameras in fixed positions. Two consequences:

• Calibration is reusable. Fixed rigs mean poses are constant across captures. Even though NoPo4D runs pose-free, known intrinsics and extrinsics can be fed in or used to verify predictions, removing the biggest feed-forward failure mode.
• Camera count is the open risk. NoPo4D benchmarks top out at 18 to 21 cameras (Neural 3D Video dataset). At 100+ views, encoder memory and attention distribution shift are unproven. Mitigation: batch cameras into overlapping spatial groups of 8 to 16 adjacent cameras sharing anchor views, reconstruct per group, and stitch using the fixed rig geometry. Groups also parallelize across the process-task queue.

Validation plan

1. 4-camera spike. Run src/inference.py --num_cameras 4 on an existing small-rig capture as-is. Confirm output quality, VRAM use, and runtime per second of capture.
2. 16-camera mid-size test. Run a bullet-time rig subset near N3DV density. This is the threshold that shows whether quality holds at the density the model was benchmarked at.
3. View-batching experiment. Run a full 100+ camera capture through camera groups. Measure per-group cost and check for seams at shared anchor views.

If step 1 passes, write the process-task wrapper with a --cameras camA,camB,... window parameter from day one, so the same runner serves small rigs directly and large rigs through batching without a redesign.

Integration notes

• Environment: Python 3.10+, CUDA GPU, PyTorch 2.x, xformers, gsplat backend, Depth Anything 3 as a vendored submodule. Expect roughly 16 GB VRAM as a working floor (4DGT's stated minimum; NoPo4D does not publish one).
• The Depth Anything 3 submodule license must be verified before shipping, and every vendored dependency needs a THIRD-PARTY-NOTICES entry.
• A test dataset is being prepared from a real rig capture for the spike.

Current limitations

The original article was marked as a work in progress. There is no universal Camera Server button that completes this pipeline. Custom scripts, RealityScan, Postshot, PointNuker, and compositing tools remain external dependencies with their own licensing, hardware, and version requirements.

Document the Camera Server build, camera layout, frame rate, audio offset, registration project, scripts, training settings, and cleanup decisions with each production. That record is more useful than copying a fixed benchmark from an older rig.