Anonymous NeurIPS 2026 Dataset and Benchmarks submission

PhysEditWorld

A large-scale dataset toward physics-editable world models.

Matched UE5 replay groups keep the scene, action trace, controller, and camera policy fixed while gravity changes, making physical edits measurable instead of hidden inside visual imitation.

Read paper GitHub repository Code and data release

12: Cinematic UE5 scenes
100h+: Gameplay interactions
60M+: Rendered rollout frames
8: Synchronized camera views

One World, Many Physics

A matched replay protocol for editable physical rules.

Existing game world models can look plausible while ignoring the requested physics. PhysEditWorld turns gravity into an explicit control variable: the same authored scenario is replayed under counterfactual gravity settings, so differences in jump arcs, airtime, fall speed, and landing dynamics can be attributed to the physical edit.

Overview of PhysEditWorld with cinematic scenes, gravity variants, interaction traces, camera trajectories, and multimodal render outputs. — Matched rollouts combine cinematic UE5 scenes, gravity variants, multi-view rendering, and synchronized multimodal annotations.

Research Question

Can a world model respond to a physical rule as an editable parameter, rather than treating physics as a fixed dataset regularity?

Hold behavior fixed

Scene, initial state, action sequence, controller, and camera policy stay matched.

Edit gravity explicitly

Gravity multipliers create counterfactual rollouts from the same interaction trace.

Export aligned evidence

RGB, depth, normals, audio, actions, camera paths, engine state, and labels share one frame timeline.

Evaluate physical faithfulness

Utility studies test whether generated motion follows requested gravity ordering.

DataFactory Pipeline

UE5-native capture, replay, rendering, and annotation.

PhysEditWorld uses an in-editor UE5 DataFactory plug-in instead of a separate simulator. It registers authored levels, binds character controllers and camera policies, injects normalized action traces, edits physical parameters, then exports synchronized observations and engine logs.

A Build scenario libraries from human, agent, or scripted inputs.
B Replay the same scenario while editing only physical configuration.
C Expand matched counterfactual replay groups across gravity levels.
D Export multi-view videos, trajectories, modalities, captions, and labels.

PhysEditWorld data pipeline showing input sources, controlled UE5 simulation, matched replay groups, and exported annotations. — A matched group shares scene, action, and cameras while varying only the physical configuration.

Dataset Contents

Every rollout is multimodal, synchronized, and gravity-labeled.

Samples are rendered at 30 FPS and 1280 x 720 resolution, with metadata exported in UE5 world coordinates and physical scale.

Observation

RGB video, depth maps, surface normals, and spatial audio when available.

Interaction

Normalized action traces, camera trajectory, character state, and object state.

Physical Edit

Explicit gravity labels across matched replay groups and counterfactual variants.

Semantic Layer

Per-clip captions generated from rendered frames without simulator metadata.

0.05g 0.1g 0.5g 1.0g 2.0g 5.0g 20.0g

Canonical sample layout

Each sample is one recording-rendering job under a fixed scene, action sequence, camera setup, and physical configuration.

<dataset_root>/
  Video/<sample_id>/<camera_name>.mp4
  Meta/<sample_id>_meta_frame.csv
  Meta/<sample_id>_meta_time.csv
  PhysicalConfig/<physical_config_name>.json
  Aux/<aux_type>/<sample_id>/<camera_name>/<frame_id>.png
  Details.json

Utility Studies

PhysEditWorld exposes gravity failures that visual quality alone can miss.

Initial experiments cover gravity-conditioned video generation, action-conditioned first-person world modeling, and gravity-aware video-language understanding.

Video generation

Gravity alignment reaches 100%

After PhysEditWorld LoRA tuning, Wan2.2-TI2V-5B preserves gravity ordering in a matched free-fall case, improving alignment from 33.3% to 100%.

0.066 zero-shot R2 0.570 tuned R2

World modeling

Action-conditioned motion becomes gravity-sensitive

A rooftop stress test shows baseline models staying near the platform edge, while PhysEditWorld-tuned LingBotWorld departs the platform and scales downward self-motion with requested gravity.

Video-language models

Gravity class accuracy improves to 95.29%

Qwen3-VL-8B improves from below-random zero-shot classification to fine-grained gravity prediction after LoRA SFT on PhysEditWorld.

24.71% zero-shot 95.29% tuned

Qwen3-VL-8B

24.71%

Qwen3-VL-8B + SFT

95.29%

Class accuracy on held-out gravity-aware prediction rollouts.

Release Plan

Code, data, and evaluation scripts will live in this repository.

The public release is planned to include synchronized RGB videos, depth maps, normal maps, gravity annotations, camera trajectories, action traces, and evaluation scripts. The UE5 replay-and-rendering pipeline will be released as an editor plug-in with generation configs and example replay assets.

Some third-party UE5 scene assets may have redistribution limits. When raw assets cannot be redistributed, the repository will provide replacement assets, asset lists, or reconstruction instructions.

Open repository Download paper

Dataset index

Details.json, sample metadata, physical configuration files, and modality paths.

UE5 DataFactory plug-in

Editor workflow for scene setup, replay capture, physical editing, rendering, and recovery.

Evaluation scripts

Gravity-conditioned generation, gravity-response metrics, and VLM gravity prediction utilities.

Example subset

A small downloadable subset for quick inspection and dataloader verification.

Citation

Cite PhysEditWorld

The current manuscript is anonymous for review. The citation block will be updated with author information after the review period.

@inproceedings{physeditworld2026,
  title = {PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models},
  author = {Anonymous Authors},
  booktitle = {Submitted to NeurIPS Datasets and Benchmarks Track},
  year = {2026}
}