Data Curation for Physical AI

researcher one-pager ↗

Choose better data, train better models.

Eventual turns video and sensor data into signals about actions, hand–object interactions, trajectories, and data quality. Use those signals to find moments in demonstrations that show skills your robot policy can reuse on its target task.

[01] Dexterity

Sample for the manipulation skill.

Compare a fingertip pinch with a movement driven by the arms.

Fine manipulation

pinch · 0.60
A0.80

Watch the thumb–index pinch as one hand steadies the bead and the other threads the cord.

thumb–index pinch
0.60
fingertip contact
0.20
hand quality
0.99
view full episode →

Whole-arm motion

D0.00

Once the door moves, the hands hold their shape as the arms guide it down.

thumb–index pinch
0.00
precise hold
0.00
hand quality
0.80
view full episode →

A ≥ 0.75 · B ≥ 0.50 · C ≥ 0.25 · D < 0.25. These demo bands describe the named property, not overall clip value or confidence. Set your selection rubric around the capabilities your model needs.

How it’s measured

Dexterity combines pinch, precise hold, fingertip contact, tool use, and in-hand manipulation. The span score uses its top 10% of seconds (at least one second); component rollups are independent and need not add up to it.

The bead clip shows tracked thumb and index fingertips. The pinch overlay reports the sampled score at that moment.

training decision

Sample more fine manipulation when your model struggles with precision.

[02] Objects & physics

Select the interaction your model needs.

Compare high and low values of the same signal: object difficulty first, then independent object motion.

Object difficulty · high vs low

Flexible, thin material versus a large rigid object. Both grades measure object difficulty.

high object difficulty

held-object region
A0.92

Watch the thin liner bend as it peels away.

bendability
1.00
thinness
1.00
smallness
1.00
view full episode →

low object difficulty

held-object region
D0.17

The ball keeps its shape as the hands move it.

bendability
0.00
thinness
0.00
smallness
0.00
view full episode →

Free motion · high vs low

An object moving after release versus an object supported by the hands. Both grades measure free motion.

high free motion

falling event
A0.82

Watch the pens keep moving after they leave the holder.

falling
0.82
hand quality
0.99
event cut off
no
view full episode →

low free motion

no free motion flagged
D0.00

Both hands stay in contact with the tray throughout the carry.

falling
0.00
rolling
0.00
hand quality
0.94
view full episode →
How it’s measured

Object attributes use 0 / 0.5 / 1. Cards show three attributes; grades use the full stored object-difficulty score. Free motion measures falling, rolling, sliding, toppling, and oscillation on a separate scale.

The held-object overlays are coarse regions. Visual review removed clear spillover onto neighboring objects. Uncertain boundaries remain approximate. The motion label follows the stored event interval. Open the full episode to inspect its timing.

training decision

Sample harder-to-handle materials for manipulation and complete free-motion events for dynamics.

[03] Quality signals

Detect unreliable hand-pose supervision.

Eventual checks recorded poses for implausible hand configurations. Red marks a detected problem in the input pose, even when the video looks clear.

pose checks pass

no invalid pose detected
A0.99

The hands and tool stay visible during tightening.

left hand visible
96%
left pose flagged
0/36
image quality
1.00
view full episode →

invalid pose detected

invalid left hand pose detected
D0.10

The hand is mostly visible. The red skeleton marks a flagged pose in the recording.

left hand visible
96%
left pose flagged
35/36
image quality
1.00
view full episode →
How it’s measured

Hand quality combines visibility and pose quality. Image quality, rapid motion, and idleness are separate axes. Visibility is the mean of stored samples; flag counts refer to the stored 4 Hz quality samples.

The red warning and red joints identify a recorded pose flagged by Eventual’s detector. The warning uses the nearest stored quality sample. A detector flag is not a claim of task failure.

training decision

Keep usable video and mask flagged hand poses in your training loss.

[04] Repetition & variation

Spend fewer samples on the same experience.

Two separate episodes capture the same bottle-opening task at the same station. The matrix shows where the motion aligns across the recordings.

Recording A · reference

Recording A · wrist paths
D0.11

Watch one hand steady the bottle as the other turns the cap.

hand quality
0.97
view full episode →

Recording B · separate episode

Recording B · wrist paths
D0.09

Look for the same grip and turn at a later moment.

hand quality
0.98
view full episode →

See where the recordings align.

View as table
Temporal similarity: recording A seconds down, recording B seconds across
Recording A ↓ / Recording B →2 s3 s4 s5 s6 s7 s8 s9 s
2 s0.910.910.890.890.000.000.000.00
3 s0.900.910.900.890.880.000.000.00
4 s0.000.900.910.900.900.890.000.00
5 s0.000.000.900.910.910.900.900.00
6 s0.000.000.000.900.910.910.910.90
7 s0.000.000.000.000.900.890.900.91

Recording A at 7 s matches recording B at 9 s.

0.681.00

Brighter cells indicate greater similarity. Outlines mark each row’s best match. Select a cell to compare both clips at those times.
How it’s measured

Novelty compares scene and hand-motion embeddings within a task and environment. Similar sequences contribute fractions of a copy; novelty is 1 / (1 + copies). The right clip is in the left clip’s retrieved matches.

Wrist trails show motion in the image. Each matrix cell uses the lower of scene and hand-motion similarity, averaged along a forward path of up to five seconds. Dark cells have a stored similarity of zero. The axes show which moments were sampled.

training decision

Sample near-duplicates less often to make room for meaningful variation.

Train with selected clips, sampling weights, labels, and loss masks.

try it on your data