An operator from behind holding two handheld controllers while a robot arm mirrors the motion over a block on a table, inside a dark camera-lined capture cell

Start with the experience your physical system is missing.

The gap is a task, object, environment or recovery state that the training data does not represent, or years of logs that nobody can load, compare or reuse with confidence.

Scope a Physical AI data project Read the six data paths
Representative episode
Grasp Failure detected Operator recovery Accepted episode

A raw frame records the motion, not the task it served.

A camera frame shows what the system observed. A state log shows a joint position. An action trace shows a command.

A small robot arm on a living-room table at night, gripper open above a shallow dish and a wooden block, a camera on a tripod at the table edge.
Gripper open
The same kind of table-top arm lowered onto the wooden block in the dish, the tripod camera recording from the table edge.
Gripper closed
Two states of the same task. A camera frame records both. Neither of them says which task applied, what changed next, why execution failed, when a person intervened, or whether the final state was usable.

The delivered episode keeps observations, robot state, actions, task context, timing and outcome connected.

That relationship is what lets your team train, compare, review and reproduce a decision.

Agree what counts as one delivered item.

An hour of teleoperation may contain several useful episodes, one long episode or mostly setup and invalid attempts. A short failure-and-recovery sequence may carry more decision value than repeated clean execution.

Your unit may be:

  • an accepted episode
  • a task demonstration
  • a synchronised period
  • a teleoperation session
  • a failure-and-recovery sequence
  • an evaluated policy run
  • a dataset release

Count price, progress, quality and acceptance in the same unit.

Start from the bottleneck you have.

YPAI helps robotics teams recover existing data, collect missing real-world demonstrations, structure and annotate episodes, extend defined gaps with synthetic data and evaluate the complete task.

  1. Data path: Existing-log recovery

    Start here when: Your logs exist, but the team cannot reuse them.

    Typical output: A source map, exclusion record, normalised episode set, alignment results, manifest and loader-tested export.

  2. Data path: Real-world demonstration collection

    Start here when: The required experience has never been captured.

    Typical output: Accepted demonstrations with observations, state, actions, task context, outcomes, rights records and delivery metadata.

  3. Data path: Episode structuring and annotation

    Start here when: Recordings exist, but task meaning is missing.

    Typical output: Episodes carrying tasks, phases, interactions, contact, object-state changes, outcomes, failures, interventions and recovery.

  4. Data path: Synthetic and simulated gap-fill

    Start here when: Real data reveals a measurable coverage gap.

    Typical output: Generated variants against a named gap, with generation lineage recorded and the combined set tested on held-out real scenarios.

  5. Data path: Physical AI evaluation

    Start here when: The team cannot tell whether the system improved.

    Typical output: A held-out set, run protocol, failure taxonomy, metric report, acceptance record and regression suite.

  6. Data path: A repeatable data operation

    Start here when: These stages need to become a repeatable operation.

    Typical output: A controlled data and evaluation workflow across capture, archive intake, curation, annotation, evaluation, versioning and model feedback.

When capture is the whole job, continue to Physical AI data collection. When annotation is the primary buying job, continue to the specialist robotics and industrial vision annotation service.

A robotic picker on a rail in a narrow, cold-lit archive aisle, between two tall racks of storage cartridges

EXISTING DATA

Audit existing data before commissioning more.

Robot and sensor archives often contain useful experience alongside inconsistent schemas, undocumented fields, mixed frequencies, missing calibration, truncated episodes or unreliable outcome labels.

Inventory  →  decode and reconcile  →  loader-tested sample

The audit decides whether to reuse the archive, repair part of it, exclude unusable records or collect only the conditions that remain missing.

An episode keeps the physical decision intact.

The final structure follows the actual robot, sensors, task, policy and receiving loader.

The streams depend on the system and are drawn from:

A large dark capture volume: a circular camera truss overhead and one person standing on a floor marked with grid and boundary lines.
A rig like this records the streams below, and the schema keeps the ones that change the decision.

Representative episode structure

Observations
  • fixed-camera video
  • egocentric video
  • wrist-camera video
  • multi-camera streams
  • RGB
  • depth
  • point cloud
  • audio where relevant
Robot and controller state
  • joint state
  • end-effector pose
  • gripper state
  • base pose
  • object state
  • tool state
  • controller state
  • safety state
Actions
  • translation
  • rotation
  • gripper commands
  • joint commands
  • navigation actions
  • tool use
  • control mode
  • intervention
  • stop or retry action
Force, contact and telemetry
  • force signals
  • tactile signals
  • contact events
  • positioning
  • equipment telemetry
Task context
  • task instruction
  • structured task goal
  • target object
  • destination
  • constraints
  • prohibited actions
  • success definition
  • failure definition
Time model
  • source clocks
  • timestamp units
  • expected frequencies
  • missing samples
  • interpolation
  • calibration references
  • permitted cross-stream offset
Outcome
  • successful
  • partially successful
  • failed
  • interrupted
  • unsafe
  • invalid
  • human intervention required
  • retry initiated
  • recovered
  • abandoned
Failure and recovery
  • precursor state
  • attempted action
  • failure point
  • cause category
  • intervention
  • recovery action
  • final disposition
Provenance and acceptance
  • source
  • rights status
  • collection protocol
  • calibration record
  • synchronisation status
  • annotation version
  • reviewer status
  • acceptance status
  • dataset version

The time model is defined before production scale, not implied by the word synchronised.

Source clocks, timestamp units, expected frequencies, missing samples, interpolation, calibration references and permitted cross-stream offset are written down.

Failure and intervention
are part of the evidence.

A clean success trajectory shows preferred execution. It does not show how a failure begins, which state preceded it, when a person should intervene, which recovery is valid or when retry becomes unsafe.

A useful failure record preserves the precursor state, attempted action, failure point, cause category, intervention, recovery action and final disposition. A takeover shows where autonomy stopped being sufficient and whether the episode returned to autonomous execution.

The safety model decides which failures may be induced, which may only be observed and when the task must stop.

A demonstration recorded once is recorded in one condition.

The system has to act in the rest of them. A policy that fails at dusk and a capture that failed at dusk are different problems, and only the record can tell them apart.

A warehouse aisle: an industrial arm holding a plastic crate, a worker in a high-visibility jacket walking away between racking bays.
The same aisle under strong overhead glare.
Overhead glare
The same aisle in low light.
Low light
The same aisle with motion blur.
Motion
The same aisle with the view partly occluded by racking.
Occlusion

The condition stays attached to the episode. Evaluation can then separate a policy that does not generalise from material that was never captured under the condition being tested.

Keep real and generated
evidence distinguishable.

Real data establishes the task, environment and failure distribution. Synthetic or simulated data then extends a named edge case or rare condition.

A grid of twelve dark panels, each holding the same sculptural form photographed at a different orientation. Real baseline Synthetic coverage
  1. Real baseline
  2. Named gap
  3. Generate
  4. Record lineage
  5. Evaluate on held-out real
  6. Accept for the stated use

Synthetic data is accepted for a defined job, not for volume by itself.

A multi-monitor review desk, the same road scene open on several screens above a timeline scrubber.
Scenario-level acceptance

Evaluate the complete task under named conditions.

A single aggregate score can hide which objects, scenes, embodiments, operators or environments drive failure.

Task completionInterventionRecoveryConstraint violationContact qualityData integrity

The contract defines the scenario distribution, metric formula, hardware and policy version, exclusions, human review and acceptance threshold for the actual use.

Ships with the accepted version  /  accepted episodes · derivatives · identifiers · episode boundaries · task and outcome states · calibration references · source lineage · quality results · exclusions · manifest · version history

The receiving team also gets a loader-tested sample or reference export that opens and reads the agreed structure in the target toolchain. Format support is checked against the source data, loader and customer requirement before it becomes a delivery commitment.

A person from behind loading plates into a dishwasher in a dim kitchen at night, a camera on a tripod at the counter end recording the task.
Applications

Physical AI spans
more than an
industrial robot cell.

Embodiment, object geometry, environment, viewpoint, sensor configuration, operator, control mode, payload, weather, clutter, failure condition and recovery path matter only when they can change the system decision being trained or evaluated.

A robotic lawn mower on a wet lawn at night, a fixed camera on a short mast at the border recording it.
Outdoor machines
A six-wheeled delivery robot on wet pavement at night, a data collector from behind on the far kerb, one street lamp.
Mobile and service robots

The same data logic carries across embodiments.  Generalist policies · manipulation · humanoid · mobile robots · logistics · service robots · outdoor machines · industrial inspection · human activity · world models.

Evidence your team can inspect.

Technical and operational evidence makes the data path inspectable. Depending on scope, an engagement produces a task and decision specification, data-gap map, stream schema, sample episode, synchronisation report, failure taxonomy, annotation guide, evaluation scenario, acceptance matrix, loader test and versioned manifest.

These artefacts let robotics, ML and operations teams inspect how the data was created, what was excluded and why the accepted material is fit for the stated task.

Define hardware, site and data controls in the operating contract.

Physical AI records can expose people, workplaces, proprietary equipment and operational behaviour. Each engagement assigns robot and sensor access, operator roles, site permissions, safety procedure, rights, permitted use, security review, retention, deletion, issue handling and handover.

This control layer makes the data operation inspectable.

Bring the task, embodiment and data bottleneck.

Your first brief can be short. Include the robot or embodiment, task, deployment environment, existing data, available sensors, target model or policy, known failure, required variation and timing if you know them. YPAI uses that brief to identify the first data path, the delivered unit, the pilot question and the evidence needed for a delivery decision.

Scope a Physical AI data project Request an archive review

Questions before the first conversation

We already have robot logs. Do we need new collection?
Not necessarily. Start with an archive audit. The right answer may be to use the existing data, repair part of it, add missing context or collect only the task conditions that remain absent.
How is this different from robotics annotation?
Physical AI data covers the broader operation across source recovery, collection, episode construction, annotation, gap-fill, evaluation and delivery. The specialist annotation route owns detailed label and review work when that is the primary purchase.
Can failures and recovery be included?
They can be included when the task and safety model allow collection, or when failures already exist in source logs. The scope defines which failures may be induced, which may only be observed and when a person must intervene.
Can real and synthetic data be combined?
Yes, for a defined coverage gap. Real and generated sources remain distinguishable, and the combined set is tested against held-out real scenarios before acceptance for the intended use.
Which delivery format will we receive?
The format follows the source data, target loader and receiving workflow. YPAI checks format feasibility during scoping and can include a loader-tested sample or reference export in the acceptance package.

Related

Scope a Physical AI data project

The form adapts to the work, asks only for relevant details and sends your brief to the person who can act on it.

Service required

Your selection routes the brief to the right person.

A named project lead reviews every enquiry and replies within one business day