Neural Hamiltonian Deformation Fields for Dynamic Scene Rendering

Abstract

Representing dynamic scenes with realistic motion remains challenging as existing methods often produce physically implausible deformations. We introduce NeHaD, a neural deformation field for dynamic Gaussian splatting governed by Hamiltonian mechanics. Our key innovation replaces MLP-based deformation with Hamiltonian neural networks that model Gaussians evolving along energy-conserving trajectories in phase space, ensuring natural dynamics. We introduce Boltzmann equilibrium decomposition for energy-aware static/dynamic separation, and employ symplectic integration with rigidity constraints to handle real-world dissipation. Additionally, we extend NeHaD to adaptive streaming through scale-aware mipmapping. Extensive experiments demonstrate that NeHaD achieves physically plausible dynamic rendering with good quality-efficiency trade-offs, representing the first application of Hamiltonian mechanics to Gaussian deformation.

Method

Hamiltonian Neural Network Decoder

To address the lack of physics-informed inductive biases in standard MLP decoders, we propose a Hamiltonian Neural Network (HNN) decoder. Explicitly defining position-momentum coordinates $(\boldsymbol{q}, \boldsymbol{p})$ for high-dimensional Gaussians is intractable; consequently, we substitute them with implicit latent representations $\boldsymbol{h}$ extracted from hex-planes as the input to the HNN. Meanwhile, the core of simulating Hamiltonian dynamics lies in learning the vector field $\boldsymbol{v}$ from these implicit features. However, directly learning this high-dimensional field is challenging due to potential mode collapse and the difficulty of enforcing conservation constraints. To address this, we propose decomposing the vector field and instead learning two scalar potentials, $F_1$ and $F_2$, to effectively generate the dynamics via automatic differentiation. Specifically, the conservative field $\boldsymbol{v}_c$ (preserving energy) and solenoidal field $\boldsymbol{v}_s$ (preserving volume) are formulated as:

Here, $\boldsymbol{M}$ is the symplectic permutation matrix. The total vector field $\boldsymbol{v} = \boldsymbol{v}_c + \boldsymbol{v}_s$ is finally projected by lightweight adapters $\mathcal{A}$ to obtain deformations for Gaussian attributes ($\Delta \boldsymbol{\mu}, \Delta \boldsymbol{s}, \Delta \boldsymbol{r}$), e.g., $\Delta \boldsymbol{\mu} = \mathcal{A}_{\boldsymbol{\mu}}(\boldsymbol{v})$.

Boltzmann Equilibrium Decomposition

Indiscriminately deforming all primitives leads to computational inefficiency and potential artifacts. We introduce a statistical mechanics approach to adaptively mask Gaussians based on their deviation from learned equilibrium states. We employ distinct decomposition strategies tailored to the visual characteristics of specific attributes:

• Position $\boldsymbol{\mu}$ (Spatial-Temporal): Position dynamics require dual selectivity. We model the energy deviation $E_{st}$ based on both spatial distance and temporal context to filter out static regions.
• Scaling $\boldsymbol{s}$ (Temporal-Only): Scaling contributes to fine-grained surface details and requires spatial universality. We model the energy $E_{t}$ based solely on temporal deviation to preserve texture authenticity.

The soft activation masks are formulated using the Boltzmann distribution. Take position dynamics as an example:

Here, $\beta$ represents the inverse temperature controlling the sharpness of the distribution, and $\gamma$ ensures a minimum responsiveness floor. The symbol $\odot$ denotes the Hadamard (element-wise) product. Consequently, primitives far from equilibrium (high energy) undergo significant deformation, while stable ones remain static.

Physics-Informed Constraints

Real-world dynamics are inherently dissipative due to friction and other non-conservative forces. This deviation from strict energy conservation can mislead the HNN into generating erroneous deformation fields. To mitigate this and strictly enforce physical plausibility, we impose explicit constraints tailored to the attribute dynamics:

• Position $\boldsymbol{\mu}$ (Symplectic Integration): Standard integrators often suffer from numerical energy drift. We adopt the Position Verlet scheme to govern translational evolution. We interpret the predicted deformation $\Delta \boldsymbol{\mu}$ as instantaneous velocity and the negative potential gradient $-\nabla \mathcal{U}$ as the physical force (acceleration) $\boldsymbol{F}$ driving the primitive:
  $$ \tilde{\boldsymbol{\mu}} = \boldsymbol{\mu} + \Delta t \cdot \Delta \boldsymbol{\mu} + \frac{(\Delta t)^2}{2} \boldsymbol{F} $$
• Rotation $\boldsymbol{r}$ (Rigidity Regularization): Unconstrained rotation predictions often lead to unrealistic twisting artifacts. Inspired by As-Rigid-As-Possible (ARAP) principles, we enforce local rigidity by clamping the rotation magnitude. The rotation angle $\phi$ is constrained via a smooth saturation function:
  $$ \phi^{\prime} = \phi_{max} \cdot \tanh\left(\frac{\phi}{\phi_{max}}\right), \quad \boldsymbol{r}^{\prime} = \mathcal{N}(\boldsymbol{r} \otimes \Delta \boldsymbol{r}^{\prime}) $$

Here, $\phi_{max}$ defines the maximum allowable rotation per timestamp. The symbol $\otimes$ represents the standard quaternion multiplication, and $\mathcal{N}(\cdot)$ denotes the normalization operation to ensure valid unit quaternions.