Vistral — Warp-Guided Streaming Exo-to-Ego Video Synthesis

Abstract

A free, non-drifting anchor for long-horizon exo→ego generation.

Exo-to-ego translation promises immersive first-person understanding from third-person footage, but the extreme viewpoint change and minimal overlap between the two views make long-horizon generation hard: errors accumulate, drift sets in, and quality collapses.

Vistral treats the exocentric view as persistent warped visual history instead of a static condition. We 3D-lift the exo video into a point cloud, rasterize it into the target ego trajectory at every step, and inject it through a frozen diffusion model's history pathway — turning the exo camera into a stable, non-drifting anchor for long-horizon, camera-controllable generation with only a lightweight LoRA.

On Ego-Exo4D, Vistral beats four baselines (Exo2Ego-V, TrajectoryCrafter, Wan Fun Control, EgoX) on per-frame fidelity in both Seen and Unseen scenarios, with substantially better long-horizon stability than concatenation-based conditioning.

Method

One frozen backbone, warped history for control.

Vistral extends warp-as-history from same-scene camera control to the cross-view, long-horizon exo-to-ego setting. Each chunk: (i) build a cross-view exo warp by 3D-lifting the exo video, (ii) inject it as visibility-selected, target-aligned pseudo-history, (iii) generate autoregressively with prior ego frames as temporal history. Only a lightweight LoRA is trained; the VAE and Helios backbone stay frozen.

4.1Cross-view warp constructionPi3X predicts per-frame depth, intrinsics, and pose; we lift to a 3D point cloud and render it into each target ego camera with z-buffering.
4.2Warped pseudo-historyWarp tokens share the target latent's RoPE index (not history); tokens with valid support below τ=0.1 are dropped.
4.3Streaming rolloutTemporal history = a persistent sink chunk + recent frames, with its own history position embedding.
4.4TrainingRank-128 LoRA on attn1/attn2 projections, flow-matching loss on noisy ego tokens only; recent-history frames are augmented.

Temporal history H_t

Previously generated ego frames — a persistent sink chunk plus recent frames — give temporal continuity with what's already been generated.

Exo warp H̃^exo_t (ours)

The exo point cloud, re-rasterized into the target view at every chunk, re-anchors generation to a static geometry instead of a shrinking window.

Prior methods like EgoX instead concatenate the exo latent with the noisy ego latent at a fixed position — effective for short clips, but unable to re-anchor frames generated far from that window under autoregressive roll-out. Routing the exo view through the history pathway re-places the anchor at every chunk instead, since it's re-rasterized from the fixed exo view each step.

Vistral pipeline: 3D-lift the exocentric video, rasterize along the target ego trajectory to get warped pseudo-history, then feed history tokens, visible warp tokens, and noisy video tokens into a frozen Helios AR video diffuser with LoRA. — **Figure 1.** Vistral 3D-lifts the exo video and rasterizes it along the target trajectory as warped pseudo-history. A frozen Helios model takes history, warp, and noisy-target tokens with shared position embeddings; only a LoRA is trained.

Evaluation

30 s streaming rollouts, evaluated against real GT.

Full 30 s clips on Ego-Exo4D Seen (training scenes) and Unseen (held-out scenes) splits. The first nine frames of each clip are discarded before scoring to avoid leakage from the GT conditioning anchor.

Method	Image Criteria				Video Criteria
Method	PSNR ↑	SSIM ↑	LPIPS ↓	CLIP-I ↑	FVD ↓	Temp. Flicker ↑	Motion Smooth ↑	Dyn. Degree ↑
Seen Scenes
Exo2Ego-V	14.53	0.384	0.569	0.774	622.47	0.960	0.966	0.985
TrajectoryCrafter	13.05	0.375	0.606	0.780	546.09	0.960	0.980	0.947
Wan Fun Control	12.25	0.463	0.617	0.810	595.07	0.968	0.980	0.901
EgoX	16.05	0.556	0.498	0.896	184.47	0.977	0.990	0.974
Vistral (Ours)	16.30	0.540	0.479	0.853	547.12	0.966	0.985	0.917
Unseen Scenes
Exo2Ego-V	12.70	0.439	0.597	0.679	1283.50	0.971	0.976	0.978
TrajectoryCrafter	12.24	0.297	0.619	0.778	821.71	0.966	0.984	0.944
Wan Fun Control	13.59	0.439	0.604	0.799	968.78	0.971	0.985	0.944
EgoX	14.38	0.457	0.552	0.877	440.64	0.981	0.992	0.989
Vistral (Ours)	14.90	0.440	0.506	0.852	539.26	0.968	0.988	0.860

Table 1. Best per column in bold, second-best underlined. Vistral leads on per-frame fidelity (PSNR, LPIPS) in both scenarios; EgoX — a strong non-streaming, fixed-clip baseline — leads on FVD and Dynamic Degree.

Ablation grid: rows GT, Warp, Helios, Helios-1shot, No sink, Ours; columns t=0s to t=27s across a 30s bike-repair clip. Helios and Helios-1shot collapse into severe color drift and artifacts within seconds; No sink diverges later in the clip; Ours stays close to GT throughout. — **Figure 2.** Ablation over a 30 s clip: GT, the warp prior, the base backbone without fine-tuning (Helios), a one-shot variant (Helios-1shot), our model without sink / recent-history augmentation (No sink), and the full model (Ours). Without streaming components, generation drifts sharply within seconds; Ours stays close to GT throughout — confirming the sink + recent-history augmentation is what suppresses autoregressive drift.

Exo GTthird-person input

Warpcross-view pseudo-history

Ego GTground-truth first-person

GeneratedVistral output

Case study — indiana_bike_01_2 (cam04). The same 30 s window shown from all four sides: what Vistral generates, the real egocentric recording, the exocentric input it was generated from, and the sparse geometric warp that conditions generation. The warp is holey and low-fidelity by construction (Sec. 4.1) — the generative prior fills the rest, closely matching Ego GT.

Design comparison

Vistral vs. EgoX: keeping the fisheye.

Same Ego-Exo4D Aria data, opposite preprocessing. EgoX (Kang, Kim, Kim, Park, Hyung & Choo, CVPR 2026) rectifies the native fisheye feed into a synthetic pinhole camera; Vistral keeps the fisheye end-to-end, trading harder warp geometry for full FOV — mitigated since the LoRA trains directly on fisheye video.

Vistral (ours)

Native fisheye (Kannala–Brandt)
448×448, full FOV (square)
Online cross-view warp, re-rasterized every chunk
Helios (distilled), streaming AR

↔ opposite choice

EgoX

Rectified pinhole (virtual camera)
448×448, cropped FOV (rectified pinhole)
Pre-generated ego_Prior.mp4
Wan2.1-I2V-14B, fixed 49-frame window

Table 1 already scores both on the same splits — this panel is about why they differ. Rectifying to pinhole discards FOV at the edges of Aria's ~110°+ fisheye, exactly where hands and near-field content live.

Video wall

33 streaming rollouts, cooking & cycling.

Every clip is genuinely fisheye, generated end-to-end from a single exocentric video and a target head-camera trajectory — kitchen and bike-repair scenes across ten Ego-Exo4D capture sites. Scroll a card into view to play.

Each card: top-left exo input · top-right warp prior (holes = unobserved) · bottom-left Vistral (ours) · bottom-right ground truth

Cite & acknowledge

BibTeX and acknowledgements.

Vistral builds on Helios, Warp-as-History, Pi3, and the Ego-Exo4D dataset.

@misc{chen2026vistral,
  title  = {Vistral: Warp-Guided Streaming Exo-to-Ego Video Synthesis},
  author = {Chen, Yunyi},
  year   = {2026},
  note   = {VITA Lab, EPFL},
  url    = {https://github.com/C-Yunyi/Vistral}
}

Helios — the distilled autoregressive video-diffusion transformer used as the frozen generative backbone.

Warp-as-History — the warp-as-conditioning mechanism this work adapts to the cross-view, long-horizon exo→ego streaming setting.

@misc{wang2026warpashistory,
  title  = {Warp-as-History: Generalizable Camera-Controlled Video Generation from One Training Video},
  author = {Wang, Yifan and He, Tong},
  journal= {arXiv preprint arXiv:2605.15182},
  year   = {2026}
}

Pi3X / π3 — feed-forward geometry estimation used for the exocentric 3D lift.

@article{wang2025pi3,
  title   = {$\pi^3$: Scalable Permutation-Equivariant Visual Geometry Learning},
  author  = {Wang, Yifan and Zhou, Jianjun and Zhu, Haoyi and Chang, Wenzheng and Zhou, Yang and Li, Zizun and Chen, Junyi and Pang, Jiangmiao and Shen, Chunhua and He, Tong},
  journal = {arXiv e-prints},
  pages   = {arXiv:2507},
  year    = {2025}
}

Ego-Exo4D — the paired egocentric/exocentric video dataset used for training and evaluation.