What is hindsight labeling?

Google's Gemini 4 Argon for Robotics release put hindsight labeling at the center of its training data story. Hindsight labeling is the process of labeling a recorded episode after it ends. It uses the known outcome to mark the goal, the steps over time, and which spans to train on.

The term has a nine-year history in robot learning. In 2017, researchers used it to rewrite goal coordinates inside reinforcement learning replay buffers. Today, frontier multimodal models inspect multi-camera robot video and output subtask hierarchies, contact primitives, and sample weights.

The field has come far from the tools available a year ago. Modern models make labeling a single trajectory practical. The hard work that follows is the data curation for physical AI. Teams must tie those labels to an opinionated taxonomy across a corpus. Then they need a system to find, grade, and select the episodes that produce a working capability on a real robot.

What is hindsight labeling?

In robot learning pipelines, hindsight labeling produces three outputs on a recorded trajectory:

1. The achieved episode goal: What the robot or human demonstrator completed by the end of the clip, even if it differed from the prompt.
2. Time-aligned steps and interactions: The start and end times for each subtask, hand-object contact, grasp, manipulation phase, and release.
3. Execution quality and training weights: Which spans show clean task progress, and which spans contain hesitations, slips, resets, or retries.

Dimension	Labeling during collection	Labeling in hindsight
Source of truth	The scripted prompt handed to the operator before recording	The recorded video and sensor log after the episode ends
Failed or imperfect demos	Discarded or mislabeled under the intended instruction	Relabeled with the achieved state or split into failure and recovery spans
Granularity	One episode-level task string	Episode goal, contiguous subtasks, contact primitives, and per-span weights
Taxonomy updates	Fixed at collection time	Re-run across the archive when the training ontology changes

Where hindsight labeling came from

Relabeling trajectories in hindsight moved through four eras over the past nine years. Each era expanded the label from a single goal vector to richer temporal structure.

Sparse-reward reinforcement learning (2017–2019)

In classic reinforcement learning, a robot arm that tries to push a puck to goal g and misses gets zero reward. Because early rollouts almost always fail, the policy learns nothing.

Hindsight Experience Replay (HER) (Andrychowicz et al., OpenAI, NeurIPS 2017) solved this problem. HER looks at the state s_T that the robot reached at the end of the episode. It rewrites the goal in the replay buffer as g' = s_T and replays the trajectory as a success for that goal.

A year later, Visual Reinforcement Learning with Imagined Goals (RIG) (Nair et al., UC Berkeley, NeurIPS 2018) took the next step. RIG extended hindsight relabeling from low-dimensional state coordinates to camera images in a latent space.

Unscripted teleoperated play and human language (2019–2022)

As robotics labs scaled imitation learning, scripted single-task demonstrations slowed data collection down. Resetting a scene hundreds of times for one narrow instruction wasted operator hours.

Learning Latent Plans from Play (Play-LMP) (Lynch et al., Google Brain, CoRL 2019) replaced scripted resets with continuous, unscripted teleoperated play. An operator interacted with objects on a desk without a fixed script. The pipeline sliced short windows out of the log and treated each window's final frame as its hindsight goal.

LangLMP / MCIL (Lynch & Sermanet, Robotics at Google, RSS 2021) paired those play windows with natural-language instructions. Human annotators watched the sliced clips and typed what the robot accomplished. Interactive Language (Lynch et al., IEEE RA-L / CoRL 2022) scaled human hindsight language relabeling to more than 600,000 trajectories.

Automated model relabeling and action hierarchies (2022–2024)

Human hindsight annotation worked well, but labeling hundreds of thousands of clips by hand cost too much time. Researchers turned to pretrained vision-language models to relabel offline robot datasets and break episodes into action hierarchies.

DIAL (project page; Xiao et al., Robotics at Google, RSS 2023) fine-tuned CLIP on 2,800 human-labeled episodes. The team then used that model to relabel 80,000 offline robot demonstrations with richer instructions. SPRINT (Zhang et al., USC / UC Berkeley / AI2, ICRA 2024) used language and vision models to relabel sub-trajectories and chain skills in offline reinforcement learning.

In 2024, two papers pushed hindsight labeling below the episode level into intermediate steps:

• RT-H: Action Hierarchies Using Language (Belkhale et al., Google DeepMind / Stanford, RSS 2024). RT-H split tasks into intermediate language motions, such as "move arm forward" and "rotate wrist", between the instruction and motor actions.
• ECoT: Robotic Control via Embodied Chain-of-Thought Reasoning (Zawalski et al., UC Berkeley / Google DeepMind / Stanford, CoRL 2024). ECoT built a synthetic hindsight pipeline that labels plans, subtasks, 2D boxes, and gripper positions before predicting actions.

Era	Period	Representative Works	What Hindsight Labeling Produced
1. Sparse-Reward RL	2017–2019	HER, RIG	Achieved state coordinates g' = s_T or latent goal images
2. Teleoperated Play	2019–2022	Play-LMP, LangLMP, Interactive Language	Crowd-sourced natural-language descriptions on sliced play windows
3. Model Relabeling	2022–2024	DIAL, SPRINT, RT-H, ECoT	VLM-generated episode instructions, language motions, and chain-of-thought features
4. Multi-Level Video Segmentation	2025–2026	GVL, Gemini 4 Argon, GPT-6 Astra, Claude Opus 5.5	Time-aligned subtask boundaries, contact primitives, progress curves, and sample weights

Gemini 4 Argon and today's frontier models

The Gemini team's Gemini 4 Argon for Robotics release honors that nine-year lineage. Their reference demo runs on a 136-second bimanual ALOHA 2 episode. Argon inspects the camera streams and produces a four-level hindsight hierarchy:

1. G1_episode_instruction: The top-level goal of the recording, such as folding a paper sheet into an origami fox.
2. G2_subtasks: Six contiguous subtask phases. Each phase records which arm anchors the paper and which arm folds or rotates it.
3. G3_atomic_primitives: Seventeen contact-level segments, each 3 to 15 seconds long. These align to grasp closure, crease sweeps, release, and retries.
4. quality_control_and_reweighting: Automatic detection of pauses, slips, and delamination retry loops. During a fumble, Argon holds task_progress_pct flat and assigns 0.15x–0.25x sample weights so behavior cloning does not copy the mistake.

Google pairs this hierarchy with physical understanding benchmarks and dense value functions. On Embodied Reasoning Question Answering (ERQA), Argon scores 87%, up from 53.3% for Gemini Robotics-ER in March 2025. For value estimation, Argon builds on Generative Value Learning (GVL) across synchronized UMI cameras.

Argon is a natural anchor for this post because it launched on October 13, 2026, and leads several of the benchmarks it cites. However, Argon is not the only frontier model that performs hindsight trajectory labeling. GPT-6 Astra and Claude Opus 5.5 also do strong work on video annotation and subtask decomposition. We do not maintain an opinion on which model is best. Every model has trade-offs in latency, cost, and boundary precision, and performance depends on the task.

From labeling a trajectory to curating a dataset

Still, running a prompt on one episode does not give a robotics team a training-ready dataset. Labeling a single trajectory with a frontier model is a major improvement over manual review, but it is only the first step.

Data curation for physical AI builds the training set that best represents the world for the task a model is learning. Sammy Sidhu, Eventual's cofounder, defines it as "having something ready to train, that actually gives you the best possible model accuracy."

Robot data is harder to curate than LLM data. In LLM data, most of the junk is predictable: copied text, contract boilerplate, and the license header at the top of every code file. Standard deduplication and quality filters remove most of it. In robot data, what counts as redundant depends on the task. Two demonstrations can repeat the same environment, the same trajectory, the same object, or a combination of them. Each repeat gives the model less than the one before. Curation balances the dataset and finds the rare examples that teach the model something new.

Quality is a separate question. Sammy puts it this way: "Quality means that I feel confident that this will make my model smarter at the thing I care about." A robot model often predicts several things at once, such as the next frame, the next action, and properties of an object, so quality depends on the target. For signals other than video, reliability comes first: if you cannot trust a sensor, do not train a prediction head on it. A wrong label also costs more than a redundant one. Training on wrong labels is like grading a quiz with the wrong answer key, so the model learns the wrong lessons.

Moving from single-trajectory labeling to corpus-scale curation takes three things that a single model prompt does not provide:

1. An opinionated, consistent taxonomy. When you prompt a model for free-text captions across 10,000 episodes, the phrasing drifts. The same motion appears as "pick up mug", "grasp cup handle", or "lift ceramic vessel". To balance a dataset or search for a failure mode, you need a consistent ontology of tasks, steps, hand roles, contact states, and outcomes.
2. Grading trajectories for a specific training goal. Whether an episode helps depends on what you are training:
   • Pretraining and world-model labs curate huge human and multi-embodiment corpora. At that scale, teams filter bad supervision and balance interactions across the mix.
   • Post-training and fleet robotics teams need targeted demonstrations, failures, and recoveries. A slip-and-retry span that you downweight for behavior cloning is the exact recovery sequence a world model needs to learn failure dynamics.
3. Separating source defects, label accuracy, and training relevance. A curation pipeline must keep three questions distinct:

Question	What it asks	Example
Source reliability	Is the raw recording or original instruction defective for this training use?	An occluded hand breaks 3D hand-pose supervision, while the clip remains valid for task progression.
Label accuracy	How often does the predicted label match human-reviewed ground truth on a held-out set?	Measuring segment F1 and boundary overlap on reviewed clips instead of trusting model confidence.
Training relevance	Does this span teach the capability that the next training run targets?	Selecting two-handed handoffs or contact-rich insertion retries for a manipulation policy.

Finding and selecting what to train on at scale

Once recordings share a consistent ontology of hindsight signals, a team can search the corpus for the behaviors a training run needs. That process is scenario mining: searching recorded video and telemetry for the spans where a described behavior happens, such as a failed grasp followed by a retry.

We built our REASSEMBLE scenario search benchmark on the public REASSEMBLE dataset (Sliwowski et al., 2025) to test how search over structured labels compares with watching frames:

• The corpus: All 149 teleoperated Franka arm recordings, totaling 13.3 hours including idle time. The robot assembles and disassembles 17 objects on the NIST Assembly Task Board 1, recorded with a wrist camera, two scene cameras, and force and joint telemetry.
• How Eventual searches: Our models label every recording with an ontology of pseudo-labels: task, step, predicted outcome, held part, in-contact twist, wrist yaw, and force. A language model turns a plain-language scenario description into a query over those labels. For "robot fails to pick up a gear, then retries and succeeds", the query matches a pick predicted to fail for at least 2 s, followed by a successful pick, holding a gear, within 20 s of the failed pick.
• Evaluation setup: We trained our labeling models on the dataset's public annotations with five-fold cross-fitting by episode. No model ever scores an episode it saw in training, and the search never reads the dataset annotations. On a customer's corpus, we label their recordings and train on those first. We compared against GPT-6 Astra (reasoning effort high) and Gemini 3.8 Flash (thinking level low). Both frontier models watched every episode at 1 fps across all three cameras via time-stamped contact sheets.
• Results: Across 10 plain-language failure and retry scenarios (scored at temporal IoU ≥ 0.5 against human-verified clips), Eventual reaches a mean F1 of 0.80. GPT-6 Astra scores 0.32 (0.325 unrounded), and Gemini 3.8 Flash scores 0.14. Eventual has the top score on 9 of the 10 scenarios. GPT-6 Astra scores higher on one scenario: gear push, realign, push.
• The cost of asking again: Eventual's search is one language-model call plus a scan of existing labels. The frontier models watch every episode again for each new question. Gemini 3.8 Flash cost about $1.70 per scenario across the corpus.

How hindsight labeling fits the robot training loop

Across four generations of research, hindsight labeling has grown from a replay-buffer trick into a core building block for robot data pipelines. Putting it to work inside a data curation loop takes four steps:

1. Define an opinionated ontology of tasks, steps, contact states, and quality flags for the capabilities you want to train.
2. Label and grade trajectories in hindsight across video and telemetry. Keep source reliability, label accuracy, and training relevance separate.
3. Mine scenarios and select training slices across your corpus. Downweight fumbles for behavior cloning, retain recoveries for world models, and balance interactions for pretraining.
4. Test on a held-out reviewed set and on the robot. Use the failure modes from each training run to guide the next selection query.

---

Bring one training-data question. Pick a behavior, failure mode, or dataset slice your team needs to curate. We will enrich a representative sample, measure the signal quality, and deliver outputs your researchers can evaluate in their own workflow. Get in touch.