ByteDance's Lucida Turns a Room Video Into Editable 3D

A phone walked slowly through a living room is a cheap thing to produce. A simulator scene that a robot can actually train inside is not. Between those two things sits a problem that has quietly held up embodied AI for years: turning a messy handheld video of a real room into a collection of separate, movable 3D objects, rather than one frozen shell of geometry that looks right from the camera path and falls apart the moment anything is asked to move.

What Happened

On August 31 a team at ByteDance Seed posted a paper called Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling. Nine authors are listed, with Minghan Qin, Yuang Wang and Xiuyu Yang sharing first-author credit and several contributors working through internships from Peking University and Zhejiang University. The stated goal is narrow and practical: take an ordinary video capture of a cluttered indoor space and return the room as complete, individually editable object assets, arranged the way the camera saw them.

The field calls this composable scene modeling, and the standard recipe has three steps. Parse the footage into object instances, generate a 3D asset for each one, then place each asset back where it belongs. The trouble is that every step in that chain quietly assumes something a real capture almost never provides. Parsing wants accurate per-instance geometry. Generation wants unoccluded views of each object. Placement wants assets that already match the observation closely enough for alignment to be a small correction rather than a guess. A chair half-hidden behind a table, photographed once from an awkward angle, breaks all three assumptions at the same time.

Lucida keeps the order but moves the burden. It parses the video into a scene graph whose nodes carry per-instance multi-view evidence — reference frames, masks, partial point clouds, oriented 3D boxes and referring cues — instead of demanding clean geometry up front. It then synthesizes an occlusion-free image of each object from that evidence bundle before lifting it into a complete mesh. Precision is deferred to the very last stage rather than required at the first.

That last stage is the part most likely to be borrowed by other groups. It is called GizmoAct, and it works by handing a vision-language model the controls of a 3D editor. The model receives a set of rendered observations each turn, issues one executable pose edit in the object's local frame — the same translate and rotate gizmo a human artist would drag — and then decides for itself when the alignment is close enough to stop. The authors cap it at four views and twelve refinement steps. Framing pose estimation as multi-turn GUI interaction rather than a single regression is an unusual move, and the reported numbers suggest it pays. On the R2S-Scene benchmark, class-agnostic detection mAP rises from 0.351 for the Boxer baseline to 0.592, a relative gain of roughly 69 percent. On Apple's CA-1M data, strict surface alignment measured by ADD-SB at a 0.05 threshold climbs from 57.8 percent to 83.4 percent. Whole-scene reconstruction F-Score moves from 0.794 for SAM 3D to 0.924.

Why It Matters

Simulation is where robots get the cheap experience they cannot afford to gather in the physical world. A policy can fail ten thousand times in a simulator overnight and cost nothing but electricity; the same ten thousand failures on real hardware cost broken grippers and months. The bottleneck has never really been compute. It has been the scenes themselves, which until now have mostly been hand-built by 3D artists, one kitchen at a time, at a pace that no amount of GPU capacity can accelerate.

The word doing the heavy lifting in Lucida's framing is editable. Plenty of existing systems can produce a photorealistic reconstruction of a room — Gaussian splats and neural fields have been good at this for a while. But those outputs are a single fused surface. You cannot pick up the mug, because there is no mug; there is only a region of geometry that happens to look like one. For a manipulation policy, that distinction is everything. Training requires objects that can be grasped, moved, dropped and rearranged independently, each with its own collision mesh and its own pose.

Getting there from a phone video changes the economics of the whole pipeline. It means a robotics team can walk through a customer's warehouse, a test apartment or a hospital supply room, and come back with a simulation-ready replica of that specific environment — including its specific clutter, which is usually the part that breaks deployed systems. Domain randomization has been the standard workaround for the gap between synthetic training rooms and real ones. Reconstructing the real room directly is a more honest solution to the same problem.

There is also a quieter industrial signal here. ByteDance is not usually the first company that comes to mind in embodied AI, and Seed publishing openly in this area — with a public project page, benchmarks against named baselines and results on third-party datasets — puts it in a conversation that has recently been dominated by NVIDIA, Google DeepMind and a handful of well-funded robotics startups.

Reaction

The paper landed on Hugging Face's daily papers board on September 1 and settled into the No. 2 slot for that day, accumulating upvotes in the high thirties within its first hours. For a real-to-sim paper — a subfield that rarely produces viral results — that is a strong showing, and it came without a model release or a demo anyone could run.

What practitioners seized on was less the reconstruction quality than the GizmoAct framing. There is an ongoing argument in the field about whether vision-language models are useful for precise geometric work, and the usual skeptical position is that they are excellent at describing scenes and poor at metric reasoning. Casting alignment as a sequence of small, reversible editor actions with visual feedback after each one sidesteps that objection rather than arguing with it. The model never has to compute a pose; it only has to judge whether the current render looks closer than the last one, which is exactly the kind of comparison these models are good at. The paper's own figures include a trajectory where the policy makes a bad rotation edit and then recovers from it, which is arguably a better advertisement than any of the headline metrics.

What's Next

The obvious open question is generalization beyond indoor scenes. Every benchmark in the paper — R2S, CA-1M, Aria Digital Twin — is a room. Warehouses, workshops, outdoor sites and vehicles all have geometry and lighting that indoor priors handle badly, and the asset-generation stage in particular leans on the kind of furniture-shaped objects that fill apartments.

The second question is physics. A simulation-ready scene needs more than correctly placed meshes; it needs mass, friction, articulation, and the knowledge that a drawer slides while a table does not. The acknowledgements mention help with physical-plausibility post-processing, which suggests the team is aware this is a separate problem and has not fully solved it. Until an asset knows how it behaves as well as how it looks, a reconstructed room is a very good stage set rather than a working environment. Whether ByteDance releases code or weights will also matter a great deal; the project page currently offers results and video, not a download.

Closing Thoughts

There is a pattern worth noticing in how Lucida is built. Its central insight is not a new architecture or a bigger model — it is a scheduling decision. The same three stages that everyone else uses, in the same order, but with the hard requirement moved from the front of the pipeline to the back, where there is enough accumulated evidence to actually satisfy it. That kind of contribution is easy to undersell in an abstract and hard to argue with in a benchmark table.

It also fits a broader shift in how research groups are using foundation models. A year ago the interesting question was what a large model could produce in one shot. Increasingly the interesting question is what it can converge on when handed a tool, a view of its own output, and permission to try again. GizmoAct is a small instance of that idea aimed at a very specific target, but the shape of it — perceive, act, look at the result, adjust — is showing up across enough of the field now that it looks less like a trick and more like a default.

한글 요약

바이트댄스 시드(ByteDance Seed) 연구팀이 8월 31일 arXiv에 공개한 Lucida는, 스마트폰으로 찍은 실내 영상 한 편을 로봇 시뮬레이터가 곧바로 쓸 수 있는 3D 장면으로 바꾸는 시스템입니다. 핵심은 방 전체를 하나의 덩어리로 복원하는 대신, 의자·컵·테이블 같은 물체를 각각 독립적으로 움직일 수 있는 개별 자산으로 되살린다는 점입니다. 베이징대와 저장대 인턴 연구자들이 참여했고, 논문 저자는 모두 9명입니다.

기존 방식은 분해·생성·배치라는 세 단계를 거치는데, 각 단계가 어수선한 실제 촬영본에서는 얻기 힘든 조건(정확한 형상, 가려지지 않은 시점, 관측과 일치하는 자산)을 앞단에서부터 요구한다는 문제가 있었습니다. Lucida는 순서는 유지하되 그 부담을 뒤로 미룹니다. 마지막 정렬 단계인 GizmoAct는 비전-언어 모델이 3D 편집기의 조작 핸들을 직접 잡고, 렌더링 결과를 보며 한 번에 한 동작씩 자세를 고친 뒤 스스로 멈출 시점을 판단하는 방식입니다.

성능은 R2S-Scene에서 탐지 mAP가 0.351에서 0.592로 약 69% 향상됐고, CA-1M에서 ADD-SB@0.05가 57.8%에서 83.4%로, 장면 복원 F-Score는 0.794에서 0.924로 올랐습니다. 다만 실험이 모두 실내 환경에 한정돼 있고, 물체의 질량·마찰·관절 같은 물리 속성은 아직 별도 과제로 남아 있습니다. 코드와 가중치 공개 여부도 아직 밝혀지지 않았습니다.

참고: arXiv:2608.30821 · Lucida project page · Hugging Face Papers