TI-0.1 Technical Report

#Abstract

We present TRACE (Telemetry-grounded Reconstruction, Alignment, and Closed-loop Evaluation), a real2sim2real agentic harness for training dexterous manipulation policies from a few Tutor demonstrations. Unlike prior real2sim methods that reconstruct from images or video, TRACE uses frontier coding agents, built on LLMs such as GPT-6 Astra, to reconstruct tasks in simulation. It uses the synchronized images, commands, and measured robot state from demonstrations to fit the simulator's appearance and physical parameters. We then train a privileged-state policy with reinforcement learning in the reconstructed simulator, with task progress specified as a hybrid Markov decision process (MDP), and distill it into TI-0.1, a visuomotor action-chunking policy that runs on Sonny. Hardware rollouts feed back into the reconstruction to close the loop, making robot learning a problem of deployment-time compute scaling. We also describe Golem, a robot software stack for performant, scalable data collection and on-robot policy inference.

#Introduction

Robot learning methods such as behavior cloning have yielded impressive demos (Chi et al., 2023Diffusion Policy: Visuomotor Policy Learning via Action DiffusionChi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., Song, S. (2023). arXiv preprint arXiv:2303.04137.; Zhao et al., 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareZhao, T. Z., Kumar, V., Levine, S., Finn, C. (2023). arXiv preprint arXiv:2304.13705.). However, these methods require large amounts of real-world data to produce fast and robust behavior, and collecting diverse demonstrations that include corrective behavior is slow and expensive. Furthermore, human demonstrations are often suboptimal, limited by human reaction times and the absence of force and tactile feedback during teleoperation (Tutor Intelligence, 2026Data Factory 1 Technical ReportTutor Intelligence (2026).). Simulation offers an alternative solution: it provides unlimited practice for reinforcement learning (RL) and direct access to forces and contacts, which can be used to train robust contact-aware policies that surpass teleoperated demonstrations. Until recently, building a simulator faithful enough for a policy to transfer required substantial manual effort for every task, making it difficult to scale real-to-sim-to-real (real2sim2real) learning.

Frontier large language models (LLMs) change that cost. Models such as GPT-6 Astra (OpenAI, 2026GPT-6 AstraOpenAI (2026).) can inspect images and telemetry, write scene and rendering code, and iterate on a reconstruction until it matches a recording. A few Tutor demonstrations can seed a simulator with scene understanding and task intent. Thereafter, RL policies practice the task across many parallel simulated environments, and each episode randomizes the task configuration, lighting, background, and dynamics. On the Lego pick task, our best policy succeeds in 99.8% of simulated episodes. This turns robot learning into a deployment-time compute problem: for a given task on a deployed robot, performance scales with simulation timesteps and LLM tokens.

Most existing real2sim works recreate how a scene looks from images or video (Torne et al., 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust ManipulationTorne, M., Simeonov, A., Li, Z., Chan, A., Chen, T., Gupta, A., Agrawal, P. (2024). arXiv preprint arXiv:2403.03949.; Qureshi et al., 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian SplattingQureshi, M. N., Garg, S., Yandun, F., Held, D., Kantor, G., Silwal, A. (2024). arXiv preprint arXiv:2409.10161.). For manipulation, appearance is not enough: the simulator must also reproduce how the robot responds to its commands and how objects respond to the robot for a given task and scene. Real-world data collection lets us ground the reconstruction in the recorded telemetry, so that visual and physical fidelity are fit against real robot behavior.

We introduce TRACE, an agentic harness in which frontier LLM agents rebuild a robot's task in simulation and train policies there. It adds to concurrent work on this new paradigm of robot learning (Wang et al., 2026bReconstruct, Practice, Go Real: Guided Self-Improvement for Embodied AgentsWang, Y.-J., Jiang, H., Deng, S., Xue, H., Ye, W., Duan, R., Haghtalab, N., Sastry, S. S., Abbeel, P., Qi, H. (2026b). arXiv preprint arXiv:2610.02204.; Moonlake AI, 2026A Simulation-enabled Scaling Paradigm for Robot LearningMoonlake AI (2026).), enabled by the recent step change in frontier LLM capabilities. Our contributions include:

• Reconstruction: a telemetry-grounded real2sim process in which agents recover scene appearance through inverse rendering and robot and contact dynamics through command replay.
• Alignment: a hybrid MDP task specification that captures a Tutor demonstration's semantic task understanding and intent, written by agents with minimal human supervision.
• Closed-loop evaluation: distillation of privileged-state policies into TI-0.1, a visuomotor policy deployed on the bimanual Sonny manipulator, with hardware rollouts closing the loop.
• Infrastructure: Golem, a robot software stack for performant and scalable data collection and on-robot policy inference.

Seed

Tutor demonstrationsImages, commands, state

1Agent

ReconstructVisual and physical, done by agents

2Agent

Train policyPrivileged-state RL, done by agents

3

DistillInto TI-0.1

4

DeployOn Sonny

Real recording

SeedTutor demonstrationsImages, commands, state

ReconstructionNext: Reconstruct

#Related work

Learning from demonstration. Diffusion policies (Chi et al., 2023Diffusion Policy: Visuomotor Policy Learning via Action DiffusionChi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., Song, S. (2023). arXiv preprint arXiv:2303.04137.) and action chunking (Zhao et al., 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareZhao, T. Z., Kumar, V., Levine, S., Finn, C. (2023). arXiv preprint arXiv:2304.13705.) made it practical to turn synchronized robot telemetry into capable visuomotor policies through behavior cloning. The Universal Manipulation Interface (Chi et al., 2024Universal Manipulation Interface: In-the-Wild Robot Teaching Without In-the-Wild RobotsChi, C., Xu, Z., Pan, C., Cousineau, E., Burchfiel, B., Feng, S., Tedrake, R., Song, S. (2024). arXiv preprint arXiv:2402.10329.) extended data collection to handheld grippers matching a robot's form factor, and Surendran et al. (2026Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated ExpertsSurendran, V., Peri, N., Watkins, D. (2026). arXiv preprint arXiv:2606.26603.) show how handheld and teleoperated supervision can be combined around contact. Data Factory 1 (DF1; Tutor Intelligence, 2026Data Factory 1 Technical ReportTutor Intelligence (2026).) scaled robot teleoperation and human-gated dataset aggregation (HG-DAgger) (Kelly et al., 2019HG-DAgger: Interactive Imitation Learning with Human ExpertsKelly, M., Sidrane, C., Driggs-Campbell, K., Kochenderfer, M. J. (2019). IEEE International Conference on Robotics and Automation (ICRA), arXiv:1810.02890.) to large robot data collection fleets. These works set the field on a path of building datasets of robot manipulation collected through teleoperation, handheld devices, simulation, and egocentric video.

Real-to-sim reconstruction. Prior real2sim2real work builds simulated replicas of real scenes for policy training and evaluation (Torne et al., 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust ManipulationTorne, M., Simeonov, A., Li, Z., Chan, A., Chen, T., Gupta, A., Agrawal, P. (2024). arXiv preprint arXiv:2403.03949.). Video-based geometry estimation such as ViPE (Huang et al., 2025ViPE: Video Pose Engine for 3D Geometric PerceptionHuang, J., Zhou, Q., Rabeti, H., Korovko, A., Ling, H., Ren, X., Shen, T., Gao, J., Slepichev, D., Lin, C., et al. (2025). arXiv preprint arXiv:2508.10934.) recovers camera poses and scene structure from recordings. Recent toolkits give language-model agents direct control over reconstruction (RoboSimStudio; Chen et al., 2026RoboSimStudio: An Interactive Toolkit for Human and Agent Rapid Real2Sim of Robot Manipulation TasksChen, K., Xie, S., Yoo, U., Zhu, X., Magner, C., Zhu, A., Kim, D., Heiden, E., Coumans, E., Macklin, M., et al. (2026). arXiv preprint arXiv:2610.18835.), and frontier models such as GPT-6 Astra can now carry out long multi-step reconstruction tasks. Unlike these works, TRACE reconstructs from synchronized images, commands, and measured robot state, constraining the simulator to reproduce both the observations and the robot's motion.

Simulator fidelity. Policies trained in simulation must cross the reality gap (Jakobi et al., 1995Noise and the Reality Gap: The Use of Simulation in Evolutionary RoboticsJakobi, N., Husbands, P., Harvey, I. (1995). Advances in Artificial Life: Third European Conference on Artificial Life (ECAL), LNCS 929.). Common strategies include randomizing over simulator parameters (Tobin et al., 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real WorldTobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P. (2017). arXiv preprint arXiv:1703.06907.; Peng et al., 2018Sim-to-Real Transfer of Robotic Control with Dynamics RandomizationPeng, X. B., Andrychowicz, M., Zaremba, W., Abbeel, P. (2018). arXiv preprint arXiv:1710.06537.) and fitting the simulator to the robot through system identification and learned residuals (Hwangbo et al., 2019Learning Agile and Dynamic Motor Skills for Legged RobotsHwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., Hutter, M. (2019). arXiv preprint arXiv:1901.08652.; Gruenstein et al., 2021Residual Model Learning for Microrobot ControlGruenstein, J., Chen, T., Doshi, N., Agrawal, P. (2021). arXiv preprint arXiv:2104.00631.). Contact modeling is especially important for sim2real transfer of dexterous manipulation policies (OpenAI et al., 2019Solving Rubik's Cube with a Robot HandOpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al. (2019). arXiv preprint arXiv:1910.07113.; Jing et al., 2026Contact-Aware Neural DynamicsJing, C., Bandi, J. K., Ye, J., Duan, Y., Abbeel, P., Wang, X., Yi, S. (2026). arXiv preprint arXiv:2601.12796.), and it is often where physics engines diverge most from reality (Acosta et al., 2022Validating Robotics Simulators on Real-World ImpactsAcosta, B., Yang, W., Posa, M. (2022). arXiv preprint arXiv:2110.00541.; Wang et al., 2026aGAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World ModelsWang, S., Feng, Y., Jiang, X., Tian, S., Yan, N., Shen, X., Lyu, C., Wang, H., Zhou, Y., Wang, H., et al. (2026a). arXiv preprint arXiv:2608.05948.). Recent work such as VIRAL (He et al., 2025VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-ManipulationHe, T., Wang, Z., Xue, H., Ben, Q., Luo, Z., Xiao, W., Yuan, Y., Da, X., Castañeda, F., Sastry, S., et al. (2025). arXiv preprint arXiv:2511.15200.) pairs large-scale randomization with real-to-sim alignment. TRACE likewise fits the robot's controller response and contact behavior to recorded telemetry. As a verification step, it checks that an attempt that succeeds (resp., fails) on the real robot also succeeds (resp., fails) in simulation, rather than relying on success in simulation alone.

Reinforcement learning and distillation. We train policies in simulation with PPO (Schulman et al., 2017Proximal Policy Optimization AlgorithmsSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. (2017). arXiv preprint arXiv:1707.06347.), using an asymmetric actor-critic with privileged critic inputs (Pinto et al., 2017Asymmetric Actor Critic for Image-Based Robot LearningPinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., Abbeel, P. (2017). arXiv preprint arXiv:1710.06542.). The PPO policy has access to privileged state, which speeds up learning in simulation, and we distill it into a visuomotor policy, as in prior work (Chen et al., 2019Learning by CheatingChen, D., Zhou, B., Koltun, V., Krähenbühl, P. (2019). arXiv preprint arXiv:1912.12294.; Lee et al., 2020Learning Quadrupedal Locomotion over Challenging TerrainLee, J., Hwangbo, J., Wellhausen, L., Koltun, V., Hutter, M. (2020). arXiv preprint arXiv:2010.11251.). Domain randomization (Tobin et al., 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real WorldTobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P. (2017). arXiv preprint arXiv:1703.06907.) and success-guided curricula (SGS; Zhang et al., 2026aA Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot ControlZhang, O., Guaman Castro, M., Yin, P., Dagnino, I., Gupta, A., Scalise, R., Boots, B. (2026a). arXiv preprint arXiv:2610.12465.) support robustness and learning from sparse rewards, and AlphaNIST (Zhang et al., 2026bAlphaNIST: Emergent Assembly Skills from Binary Task RewardsZhang, O., Akinola, I., Scalise, R., Guaman Castro, M., Yin, P., Widmer, T., Macklin, M., Amevor, C., Turpin, D., Van Gelder, D., et al. (2026b). arXiv preprint arXiv:2610.13354.) shows that a binary task reward with enough task diversity can let large-scale RL discover contact-rich assembly skills. TRACE specifies each task as a hybrid MDP, whose structure enables agentic reward shaping with minimal human supervision and keeps the learned behavior Markovian, so that it survives the sim2real gap.

Agentic simulation for robot learning. Concurrent work also uses agents to build simulated practice for real robots. In RPG (Wang et al., 2026bReconstruct, Practice, Go Real: Guided Self-Improvement for Embodied AgentsWang, Y.-J., Jiang, H., Deng, S., Xue, H., Ye, W., Duan, R., Haghtalab, N., Sastry, S. S., Abbeel, P., Qi, H. (2026b). arXiv preprint arXiv:2610.02204.), an agent reviews videos and text descriptions from an offline dataset and builds simulation environments for practicing the relevant manipulation capabilities. Moonlake AI (2026A Simulation-enabled Scaling Paradigm for Robot LearningMoonlake AI (2026).) builds environments from scene captures and teleoperated trajectories, then fine-tunes a vision-language-action model in them with RL and by distilling a privileged-state code policy. In contrast to both, TRACE trains a privileged-state policy with RL on an agent-written hybrid MDP, then distills it into a visuomotor policy.

#TRACE

#Overview

TRACE turns a few Tutor demonstrations into a simulator faithful enough to train policies that run on real hardware. It has three stages, described in the sections below. Reconstruction fits the simulator's appearance and dynamics to the demonstrations' synchronized images, commands, and measured robot state. Alignment specifies the task as a hybrid MDP and trains a privileged-state policy with RL in the reconstructed simulator. Closed-loop evaluation distills that policy into TI-0.1, a visuomotor action-chunking diffusion policy that runs on Sonny. Agents built on frontier LLMs such as GPT-6 Astra and Claude Opus 5.5 (Anthropic, 2026Claude Opus 5.5Anthropic (2026).) drive the pipeline: they match the scene to the recordings, write each task's hybrid MDP and reward, and diagnose the behavior of the privileged-state policy. Hardware rollouts of transferred policies collect further data to verify simulator behavior or highlight real2sim gaps, feeding the next round of reconstruction.

#Reconstruction

Real2sim reconstruction is an inverse problem: recover the scene and physical properties needed to reproduce what the robot sees and how its actions affect the world. A few Tutor demonstrations provide synchronized images, commands, and measured robot state, tying what we see to the inputs that caused it and the motion that followed. These recordings seed two parts of the reconstruction:

• Visual fidelity (inverse rendering) recovers the geometry, cameras, materials, and lighting that explain the recorded images.
• Physical fidelity (inverse physics) recovers the controller response and contact behavior that explain the recorded motion and task outcome.

We use Pick Lego as the main example throughout this section. Below, the real recording appears beside the reconstructed scene from the robot's cameras.

#Visual fidelity: inverse rendering

We start from robot assets, camera calibration, and recordings, then build an approximate scene and render it through the robot's cameras. Let $O_{t,\mathrm i}\in\mathbb R^{H_{\mathrm i}\times W_{\mathrm i}\times3}$ be the recorded RGB frame from camera $\mathrm i$ at time $t$, and $q_t\in\mathbb R^n$ the corresponding arm joints and gripper positions. We use $C$ to denote prior information about the system, which includes robot kinematics and camera parameters.

We seek scene parameters $\theta_{\mathrm{vis}}$ describing scene geometry, object placement, camera corrections, materials, and lighting. The renderer $R_{\mathrm i}$ predicts the observation from camera $\mathrm i$ at the recorded robot configuration. The reconstruction target is to make the rendered observation $\hat O_{t,\mathrm i}$ match the recording $O_{t,\mathrm i}$:

Agents help close this loop by comparing real and rendered views, investigating discrepancies, and proposing changes to the scene or rendering code. We render again and compare across cameras and time. CAD and calibration constrain the reconstruction, while images guide the remaining adjustments.

#Refining geometry and appearance

Agents use real images to guide changes to the scene's materials and geometry. In these examples, they replaced generic wood grain with a texture assembled from photographed strips and rebuilt the brick's underside with a hollow cavity, support tubes, and ribs.

Real reference

Earlier render

Refined render

Surface texture

Real reference

Wood strips and grain in the recorded wrist view.

Earlier render

Generic grain misses the tabletop's characteristic strip pattern.

Refined render

Rebuild the material from photographed wood strips at a consistent scale.

Geometry detail

Real reference

Photograph of the brick's underside.

Earlier render

A closed surface hides the brick's internal structure.

Refined render

Model the cavity, three support tubes and reinforcing ribs.

#Reconstructing clutter across cameras

Reconstructing the pile is a small part of the scene, but a demanding test of the agents' ability to match many partly occluded objects across cameras. A placement that fits one view must also explain the shapes, colors, and occlusions in the other. The agents use these constraints to refine one shared 3D arrangement.

Real recording

Generic arrangement

Reconstructed arrangement

Right wrist

Real recording

Generic arrangement

Reconstructed arrangement

Left wrist

Real recording

Generic arrangement

Reconstructed arrangement

#Fitting lighting and materials to DF1

We reconstructed DF1 (Tutor Intelligence, 2026Data Factory 1 Technical ReportTutor Intelligence (2026).), home of our 100-robot fleet, in simulation. Agents fit the room's lighting and layout using photos and floor plans, and use the fleet's chest-camera recordings to fit the table's and objects' materials. During training, we use domain randomization to capture how the lighting changes over the course of a day.

#Extending to a workcell

The same process applies beyond the tabletop. Robot assets and setup imagery anchor the reconstruction of the surrounding room, shelving, stocked containers, and surfaces.

Real

Early render

Final render

Real

Early render

Final render

#Physical fidelity: inverse physics

Inverse physics asks which dynamics explain the consequences of recorded commands. We reconstruct each demonstration's initial state, replay its commands, and adjust physical parameters to bring simulated observations closer to the recording:

Here, $i$ indexes demonstrations, $u_t$ is the recorded command, and $\Phi$ maps state to sensor observations. The weights $W$ account for units and measurement uncertainty. Holding the commands and initialization fixed lets agents trace how a parameter change affects contact, transport, and release.

We reconstruct two things: how commands move the robot and how the robot moves objects.

#Matching robot response

A pose target passes through the robot's control path before it becomes motion. We port this control path into simulation, then fit the actuator delay and response against measured joint motion.

Calibrating when queued commands execute, and at what rate, brought simulated motion closer to the recording. The plots below show position error before and after this timing correction.

Complete replays then compare predicted and measured arm motion on the same timeline.

#Matching object interaction

Contact is central to manipulation and one of the hardest parts to reconstruct. Small differences in geometry, friction, or compliance can determine whether an object stays in the grasp or slips out.

We constrain object and gripper properties using CAD, hardware information, and recordings.

The Lego privileged-state policy exposed one such mismatch: it learned to hold bricks by their studs, exploiting contact behavior that did not transfer to the real gripper. We revised the pads' effective stiffness, travel limits, and friction. In the revised model, a shallow pinch produces little supporting force, while a body grasp reaches the pads' travel limit and transmits the motor's squeeze.

Evaluating the same frozen policies under both models confirmed the intended effect: stud-picking success fell sharply, while body-grasping success was preserved.

To test the contact model, we drive the simulated arm with measured joint motion and replay the recorded gripper commands, leaving the brick free to move under contact forces. This removes most of the arm-control mismatch from the comparison. Differences in when the brick lifts, whether it slips, and how it releases then guide revisions to geometry, friction, and compliance.

#Alignment

With the simulator reconstructed, we use RL to train a privileged-state policy that controls the arm and gripper. Its rollouts provide demonstrations for a visuomotor policy.

The alignment problem is to specify a reward whose optimization produces the task behavior we intend. We address it through the task definition, reward shaping, and training procedure.

We formulate the task as a hybrid Markov decision process (MDP): the state $s_t$ describes robot configuration, object poses, velocities, contacts, and task-relevant history, while a discrete mode $m_t$, defined by predicates on the simulator state, identifies the current subtask. Guard conditions specify when the robot can transition between modes, as in hybrid automata (Henzinger, 1996The Theory of Hybrid AutomataHenzinger, T. A. (1996). IEEE Symposium on Logic in Computer Science (LICS), doi:10.1109/LICS.1996.561342.). Reward machines (Toro Icarte et al., 2018Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement LearningToro Icarte, R., Klassen, T. Q., Valenzano, R., McIlraith, S. A. (2018). International Conference on Machine Learning (ICML), PMLR 80.) similarly expose task structure to an RL agent through discrete states.

Here, $a_t$ is an arm and gripper action, and $g$ applies the task's transition conditions. This lets us specify the required sequence of interactions while the policy learns the movements.

#Defining progress through modes

Consider Unpick Lego, the reverse of the pick-and-place task: remove one brick from a heap inside a bin, place it on a target region on the table, and return the arm home. The remaining bricks should stay in the bin, leaving the scene ready for another pick.

A binary reward for the final arrangement gives no progress signal until success and leaves the route to that arrangement unspecified. An early success condition checked that a brick reached the target, the bin was upright, and the arm was home. The policy learned to tip out the heap and stand the bin back up, an instance of reward hacking (Amodei et al., 2016Concrete Problems in AI SafetyAmodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D. (2016). arXiv preprint arXiv:1606.06565.).

We make the intended sequence explicit:

Each transition requires measurable conditions. Placement requires the brick on the target and exactly one brick outside the bin; completion additionally requires the scene at rest and the arm home.

Because modes share this structure across tasks, agents write each new task's modes and reward with minimal human supervision. They also check whether each mode can be inferred from what the robot observes, so that the task stays Markovian for the visuomotor policy. The same formulation can express other sequences, such as reorienting a cup on the table before lifting it.

#Training policies in simulation

We train a privileged-state policy for each task with Proximal Policy Optimization (PPO) across parallel randomized simulations. The actor $\pi_\psi$ maps robot state and visible object poses $o_t$ to arm and gripper commands. The critic $V_\phi$ also receives privileged information $c_t$, including contacts and task modes:

Rollouts provide return targets $\hat R_t$ and advantage estimates. We update the actor to improve the PPO objective and the critic to predict those returns, schematically:

Here, $J_{\mathrm{PPO}}$ is the clipped policy objective, and $\eta_\pi,\eta_V$ are learning rates.

#Verifying the simulator

The policy learns from the mechanics the simulator permits. Our environment runs in Isaac Lab (Mittal et al., 2025Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot LearningMittal, M., Roth, P., Tigue, J., Richard, A., Zhang, O., Du, P., Serrano-Muñoz, A., Yao, X., Zurbrügg, R., Rudin, N., et al. (2025). arXiv preprint arXiv:2511.04831.) with OvPhysX, where timestep, solver iterations, and contact settings affect both fidelity and training throughput. We test these choices with scripted stress trajectories, including fast approaches, off-center grasps, direction changes, and sustained holds.

If fingertips penetrate a rigid brick and still earn a lifting reward, training can reinforce an impossible grasp. We inspect contact throughout each test and compare the same trajectory across solver settings, rather than judging fidelity by task success alone.

#Shaping behavior for transfer

A privileged-state policy can use exact object locations outside the camera view or make abrupt corrections that are difficult for the visuomotor policy to reproduce. When training our privileged-state policy in simulation, the actor receives an object's simulator pose only while the wrist camera sees it. We also add noise to that pose to model the error of estimating it from RGB.

For example, an earlier policy released the brick while sweeping sideways. It could succeed in simulation while producing demonstrations of a throw we did not want the visuomotor policy to imitate. Adding release penalties encouraged a lower, slower release.

Fast sideways releases magnify the sim2real gap. Small errors in arm tracking or gripper-opening timing can produce large changes in where the brick lands, making transfer harder. Lower, slower releases reduce this sensitivity.

We also shape the privileged-state policy's motion to make it easier to imitate. An unconstrained policy that predicts actions step by step can make abrupt corrections using feedback at every decision. The visuomotor policy predicts an action chunk and executes part of it before replanning, making those rapid corrections difficult to reproduce. We address this by giving the privileged-state policy a smooth motion parameterization, ensuring that the visuomotor policy has a temporally coherent learning target.

In a comparison, the smooth policy's commands had roughly 4–5× lower RMS jerk during the approach than step-by-step commands. In later distillation experiments, the visuomotor policy trained on smooth commands converged faster and achieved higher success rates. The smooth motion parameterization was also essential for Real-Time Chunking (RTC; Black et al., 2025Real-Time Execution of Action Chunking Flow PoliciesBlack, K., Galliker, M. Y., Levine, S. (2025). arXiv preprint arXiv:2506.07339.) to work well in practice, since RTC relies on temporal consistency across action chunks.

#Learning with a sparse reward

The modes and shaping above make the desired behavior explicit. We also explored how much task-specific reward design we could replace with a reverse curriculum (Florensa et al., 2017Reverse Curriculum Generation for Reinforcement LearningFlorensa, C., Held, D., Wulfmeier, M., Zhang, M., Abbeel, P. (2017). Conference on Robot Learning (CoRL), arXiv:1707.05300.): begin episodes near completion, then allocate more practice to earlier parts of the task as the policy improves.

For Pick Lego, we build a fixed bank of physical starting states spanning approach, lift, transport, release, and return home. Scripted motion and physics generate the states, and grasp and placement checks filter them. These stages organize the bank; every episode still has the same final completion objective.

Success Guided Sampling (SGS) (Zhang et al., 2026aA Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot ControlZhang, O., Guaman Castro, M., Yin, P., Dagnino, I., Gupta, A., Scalise, R., Boots, B. (2026a). arXiv preprint arXiv:2610.12465.) adapts each state's sampling probability using its recent completion rate. States near 50% success receive more practice. As later behaviors become reliable, sampling can shift toward harder, earlier starts, allowing the policy to learn the full sequence without a separate reward for each stage.

For Pick Lego, this approach, with later improvements to our training pipeline, reached 99.8% success in simulation on held-out evaluation configurations, using a binary task reward and motion regularization. It was less effective on Unpick, where specifying the sequence through modes and shaping remained necessary, as described above.

#Closed-loop evaluation

We use policy distillation to turn the privileged-state policy's simulated rollouts into a visuomotor policy that acts from the robot's sensors.

Here, $h_t$ contains the sensor history and $A_t$ is the corresponding action chunk derived from the privileged-state policy's rollout. The visuomotor policy's parameters $\omega$ are learned from these pairs; privileged simulator state is not an input at deployment.

#From simulator state to camera-based control

We run the privileged-state policy across simulated episodes and record chest and wrist images alongside robot state and action targets.

The visuomotor policy, TI-0.1, uses action-chunking diffusion (Chi et al., 2023Diffusion Policy: Visuomotor Policy Learning via Action DiffusionChi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., Song, S. (2023). arXiv preprint arXiv:2303.04137.; Zhao et al., 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareZhao, T. Z., Kumar, V., Levine, S., Finn, C. (2023). arXiv preprint arXiv:2304.13705.) to predict a chunk of arm and gripper actions from recent camera images and robot state. We reuse the training pipeline used for real demonstrations, keeping coordinate frames and observation–action timing consistent with the robot controller.

#Domain randomization

Reconstruction gives us a reference scene. When we train the privileged-state policy and generate data for the visuomotor policy, each simulated episode draws three classes of properties from its own random seed:

• Task configuration: brick and bin poses, clutter arrangement, table height, and the arm's starting pose.
• Appearance: one of three backgrounds (a DF1 lab plate, an indoor HDRI panorama, or a textured room), lights with random placement, brightness, and color, and object colors. The table finish and bin color stay fixed.
• Dynamics: object mass and friction, gripper damping and gripping force, and controller delays.

We also perturb the visuomotor policy's proprioceptive inputs during training to improve sim2real transfer. Empirically, adding a small amount of noise produced more robust behavior.

#Learning from hardware trials

Robot trials test the complete chain, from perception through commanded motion to contact. The shallow stud grasps described above illustrate why successful simulated lifts alone are insufficient. Recordings from deployment let us compare what the policy observed, what it commanded, and what actually moved, then investigate the responsible part of the reconstruction or training process.

#Golem

Collecting demonstrations and evaluating policies across a fleet requires a fast and consistent robot software stack. Every demonstration depends on camera images, robot state, and actions agreeing about what happened and when. Across a fleet, differences in timing, calibration, and command processing can become differences in the data we train on and confound the experiments we use to measure progress.

We built Golem to raise the quality floor of data collection and shorten the cycle from collecting demonstrations to evaluating policies on hardware. That meant designing for consistent timing, traceable recordings, and repeatable experiments alongside efficient control and inference.

Golem runs Sonny's camera processing and arm control in Rust, and policy inference in Python on the robot's GPU. Compared with our previous Python stack, Golem uses 90% less CPU and 79% less RAM, with 68% lower camera-streaming latency.

#Parallel arm control

Separating target calculation from command sending lets each arm keep receiving commands while the next target is computed. Each arm uses two threads: one solves inverse kinematics and checks collisions; the other sends the latest target at a fixed rate. Both arms plan in parallel. Camera capture and recording run independently of arm commands.

#Timing and synchronization

Camera images, robot state, and commands arrive through different paths, with different delays. We use NTP-based clock calibration to place these streams on a common timeline, so their timestamps can be compared consistently.

This lets us relate what the cameras saw to the robot's measured motion and the commands that preceded it. That alignment matters when learning from demonstrations and reconstructing those interactions in simulation.

#Local GPU inference

The inference loop stays on the robot, from assembling camera and arm inputs to sending predicted movements. The policy predicts an action chunk from recent camera images and measured arm positions. Rust combines new predictions with queued actions and schedules the next arm targets.

Across 300 paired replays on one robot, preparing and scheduling arm commands averaged 6.57 ms in Python and 0.04 ms in Rust, about 156× faster. The measurement excludes model inference.

#Loading and reusing models

Golem can stop and restart a session without reloading the model. Loading a model also warms up the GPU. Two policies can stay loaded at once: a Do policy and an Undo policy that reverses it. Running them in a cycle removes the need for manual scene resets.

For the table below, the Python stack and Golem used the same policy checkpoint and recorded camera and arm inputs.

#Camera delivery, CPU, and RAM

“Ingest to send” is the delay from receiving a camera frame in the robot's software to handing it to the outgoing video stream. It excludes network and display time.

#Scaling up

Real-world robot fleets deployed across diverse settings and tasks can collect the synchronized images, commands, and measured robot state to reconstruct high-fidelity simulators at massive scale. Below, agents reconstruct a Tutor demonstration recorded on a deployed Sonny in a real production environment.

Where deployments provide diversity and scale, DF1 provides a kindergarten for robots: a safe, flexible, and forgiving environment in which failure modes that would take hours to surface on a single robot appear within minutes (Tutor Intelligence, 2026Data Factory 1 Technical ReportTutor Intelligence (2026).). Running simulation-trained policies across the fleet turns each failure into a targeted correction. It also lets us scale these methods to new tasks, with concurrent real-world evaluation on up to 100 robots.

Our goal is to continue to develop these pipelines to be more performant and more autonomous, supporting reconstruction, training, and evaluation that runs continuously across large fleets both in DF1 and in the field.

#Conclusion

TRACE trains dexterous manipulation policies in simulation from a few Tutor demonstrations. Usable simulators for real-world transfer must meet a high bar for both visual and physical fidelity: TRACE fits how the robot responds to its commands and how objects respond to the robot against the robot's own recorded telemetry. Frontier models make this reconstruction practical to repeat for new tasks, with real-world rollouts closing the loop. TI-0.1, distilled from privileged-state RL policies trained in simulation, picks and places Lego bricks on Sonny autonomously and in real time. Alongside real-world data collection and evaluation in DF1, TRACE turns recorded robot demonstrations into far more experience than the recordings alone provide.

What we learned.

• Visual fidelity is not enough. Manipulation simulators need dynamics fit to synchronized commands and measured robot state, not just images.
• Verify contact directly. Task success can hide physically impossible grasps. Replaying recorded commands and inspecting contact catches what success rates miss.
• Shape privileged-state policies for distillation. We let the policy see only objects in the wrist camera's view and give it a smooth motion parameterization, so that its behavior can be successfully distilled into a visuomotor policy.
• Hardware is the ground truth. Real rollouts expose gaps, like the stud grasp, that no simulated metric revealed.

Important gaps remain. High-precision tasks demand finer geometry and contact fidelity than our current reconstructions provide. Calibration errors in cameras and robots propagate into both rendering and dynamics. Deformable objects remain difficult for physics engines in general (Wang et al., 2026aGAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World ModelsWang, S., Feng, Y., Jiang, X., Tian, S., Yan, N., Shen, X., Lyu, C., Wang, H., Zhou, Y., Wang, H., et al. (2026a). arXiv preprint arXiv:2608.05948.), and compliant elements such as springs and soft gripper pads are hard to identify from telemetry alone.

If you are interested in working on scaling robot learning and deployment in the real world, please consider applying to join our team.

#Acknowledgments

We thank the people across Tutor Intelligence who make real-world robot learning possible: our team of Tutors, whose demonstrations seed this work; the manufacturing and maintenance technicians and operations managers who build our robots and keep DF1 and our field fleet running; and our deployment teams across go-to-market, implementation, and customer success, who bring robots into real facilities. We also thank Tutor's customers, our long-term partners in bringing generally capable robots into the world for the common good.

#Citation

@techreport{tutor2026ti01,
  title       = {{TI-0.1} Technical Report},
  author      = {{Tutor Intelligence}},
  institution = {Tutor Intelligence},
  year        = {2026},
  url         = {https://research.tutorintelligence.com/ti0-1}
}

#References

1. Acosta, B., Yang, W., Posa, M. (2022). Validating Robotics Simulators on Real-World Impacts. arXiv preprint arXiv:2110.00541.
2. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D. (2016). Concrete Problems in AI Safety. arXiv preprint arXiv:1606.06565.
3. Anthropic (2026). Claude Opus 5.5.
4. Black, K., Galliker, M. Y., Levine, S. (2025). Real-Time Execution of Action Chunking Flow Policies. arXiv preprint arXiv:2506.07339.
5. Chen, D., Zhou, B., Koltun, V., Krähenbühl, P. (2019). Learning by Cheating. arXiv preprint arXiv:1912.12294.
6. Chen, K., Xie, S., Yoo, U., Zhu, X., Magner, C., Zhu, A., Kim, D., Heiden, E., Coumans, E., Macklin, M., et al. (2026). RoboSimStudio: An Interactive Toolkit for Human and Agent Rapid Real2Sim of Robot Manipulation Tasks. arXiv preprint arXiv:2610.18835. Project page.
7. Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., Song, S. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv preprint arXiv:2303.04137.
8. Chi, C., Xu, Z., Pan, C., Cousineau, E., Burchfiel, B., Feng, S., Tedrake, R., Song, S. (2024). Universal Manipulation Interface: In-the-Wild Robot Teaching Without In-the-Wild Robots. arXiv preprint arXiv:2402.10329.
9. Florensa, C., Held, D., Wulfmeier, M., Zhang, M., Abbeel, P. (2017). Reverse Curriculum Generation for Reinforcement Learning. Conference on Robot Learning (CoRL), arXiv:1707.05300.
10. Gruenstein, J., Chen, T., Doshi, N., Agrawal, P. (2021). Residual Model Learning for Microrobot Control. arXiv preprint arXiv:2104.00631.
11. He, T., Wang, Z., Xue, H., Ben, Q., Luo, Z., Xiao, W., Yuan, Y., Da, X., Castañeda, F., Sastry, S., et al. (2025). VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation. arXiv preprint arXiv:2511.15200.
12. Henzinger, T. A. (1996). The Theory of Hybrid Automata. IEEE Symposium on Logic in Computer Science (LICS), doi:10.1109/LICS.1996.561342.
13. Huang, J., Zhou, Q., Rabeti, H., Korovko, A., Ling, H., Ren, X., Shen, T., Gao, J., Slepichev, D., Lin, C., et al. (2025). ViPE: Video Pose Engine for 3D Geometric Perception. arXiv preprint arXiv:2508.10934. Project page.
14. Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., Hutter, M. (2019). Learning Agile and Dynamic Motor Skills for Legged Robots. arXiv preprint arXiv:1901.08652.
15. Jakobi, N., Husbands, P., Harvey, I. (1995). Noise and the Reality Gap: The Use of Simulation in Evolutionary Robotics. Advances in Artificial Life: Third European Conference on Artificial Life (ECAL), LNCS 929.
16. Jing, C., Bandi, J. K., Ye, J., Duan, Y., Abbeel, P., Wang, X., Yi, S. (2026). Contact-Aware Neural Dynamics. arXiv preprint arXiv:2601.12796.
17. Kelly, M., Sidrane, C., Driggs-Campbell, K., Kochenderfer, M. J. (2019). HG-DAgger: Interactive Imitation Learning with Human Experts. IEEE International Conference on Robotics and Automation (ICRA), arXiv:1810.02890.
18. Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., Hutter, M. (2020). Learning Quadrupedal Locomotion over Challenging Terrain. arXiv preprint arXiv:2010.11251.
19. Mittal, M., Roth, P., Tigue, J., Richard, A., Zhang, O., Du, P., Serrano-Muñoz, A., Yao, X., Zurbrügg, R., Rudin, N., et al. (2025). Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. arXiv preprint arXiv:2511.04831.
20. Moonlake AI (2026). A Simulation-enabled Scaling Paradigm for Robot Learning.
21. OpenAI (2026). GPT-6 Astra.
22. OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al. (2019). Solving Rubik's Cube with a Robot Hand. arXiv preprint arXiv:1910.07113.
23. Peng, X. B., Andrychowicz, M., Zaremba, W., Abbeel, P. (2018). Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. arXiv preprint arXiv:1710.06537.
24. Pinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., Abbeel, P. (2017). Asymmetric Actor Critic for Image-Based Robot Learning. arXiv preprint arXiv:1710.06542.
25. Qureshi, M. N., Garg, S., Yandun, F., Held, D., Kantor, G., Silwal, A. (2024). SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting. arXiv preprint arXiv:2409.10161.
26. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.
27. Surendran, V., Peri, N., Watkins, D. (2026). Bridging Handheld and Teleoperated Supervision for Contact-Rich Manipulation via State-Gated Experts. arXiv preprint arXiv:2606.26603.
28. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv preprint arXiv:1703.06907.
29. Torne, M., Simeonov, A., Li, Z., Chan, A., Chen, T., Gupta, A., Agrawal, P. (2024). Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation. arXiv preprint arXiv:2403.03949.
30. Toro Icarte, R., Klassen, T. Q., Valenzano, R., McIlraith, S. A. (2018). Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning. International Conference on Machine Learning (ICML), PMLR 80.
31. Tutor Intelligence (2026). Data Factory 1 Technical Report.
32. Wang, S., Feng, Y., Jiang, X., Tian, S., Yan, N., Shen, X., Lyu, C., Wang, H., Zhou, Y., Wang, H., et al. (2026a). GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models. arXiv preprint arXiv:2608.05948.
33. Wang, Y.-J., Jiang, H., Deng, S., Xue, H., Ye, W., Duan, R., Haghtalab, N., Sastry, S. S., Abbeel, P., Qi, H. (2026b). Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents. arXiv preprint arXiv:2610.02204. Project page.
34. Zhang, O., Akinola, I., Scalise, R., Guaman Castro, M., Yin, P., Widmer, T., Macklin, M., Amevor, C., Turpin, D., Van Gelder, D., et al. (2026b). AlphaNIST: Emergent Assembly Skills from Binary Task Rewards. arXiv preprint arXiv:2610.13354. Project page.
35. Zhang, O., Guaman Castro, M., Yin, P., Dagnino, I., Gupta, A., Scalise, R., Boots, B. (2026a). A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control. arXiv preprint arXiv:2610.12465. Project page.
36. Zhao, T. Z., Kumar, V., Levine, S., Finn, C. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. arXiv preprint arXiv:2304.13705.