Start with the experience your physical system is missing.
Your robot may fail on a task, object, environment or recovery state that the training data does not represent. You may instead have years of logs that nobody can load, compare or reuse with confidence.
Raw assets do not preserve the learning problem.
A camera frame can show what the system observed. A state log can show a joint position. An action trace can show a command.
The accepted record often has to keep observations, robot state, actions, task context, timing and outcome connected.
That relationship is what lets your team train, compare, review and reproduce a decision.
Make the accepted unit match the work.
An hour of teleoperation may contain several useful episodes, one long episode or mostly setup and invalid attempts. A short failure-and-recovery sequence may carry more decision value than repeated clean execution.
Your unit may be:
- an accepted episode
- a task demonstration
- a synchronized period
- a teleoperation session
- a failure-and-recovery sequence
- an evaluated policy run
- a dataset tranche
The commercial, progress, quality and acceptance unit should describe the same delivery.
Choose the data path from the bottleneck.
YPAI helps robotics teams recover existing data, collect missing real-world demonstrations, structure and annotate episodes, extend defined gaps with synthetic data and evaluate the complete task.
-
Data path: Existing-log recovery
Start here when: Your logs exist, but the team cannot reuse them.
The output can include A source map, exclusion record, normalized episode set, alignment results, manifest and loader-tested export.
-
Data path: Real-world demonstration collection
Start here when: The required experience has never been captured.
The output can include Accepted demonstrations with observations, state, actions, task context, outcomes, rights records and delivery metadata.
-
Data path: Episode structuring and annotation
Start here when: Recordings exist, but task meaning is missing.
The output can include Episodes carrying tasks, phases, interactions, contact, object-state changes, outcomes, failures, interventions and recovery.
-
Data path: Synthetic and simulated gap-fill
Start here when: Real data reveals a measurable coverage gap.
The output can include Generated variants against a named gap, with generation lineage recorded and the combined set tested on held-out real scenarios.
-
Data path: Physical AI evaluation
Start here when: The team cannot tell whether the system improved.
The output can include A held-out set, run protocol, failure taxonomy, metric report, acceptance record and regression suite.
-
Data path: One repeatable operating loop
Start here when: These stages need to become a repeatable operation.
The output can include A controlled data and evaluation workflow across capture, archive intake, curation, annotation, evaluation, versioning and model feedback.
When capture is the whole job, continue to Physical AI data collection. When annotation is the primary buying job, continue to the specialist robotics and industrial vision annotation service.
EXISTING DATA
Audit existing data before commissioning more.
Robot and sensor archives often contain useful experience alongside inconsistent schemas, undocumented fields, mixed frequencies, missing calibration, truncated episodes or unreliable outcome labels.
Inventory → decode and reconcile → loader-tested sample
The result may be to reuse the archive, repair part of it, exclude unusable records or collect only the conditions that remain missing.
An episode keeps the physical decision intact.
The final structure follows the actual robot, sensors, task, policy and receiving loader.
The required streams depend on the system. They may include:
Representative episode structure
- Observations
-
- fixed-camera video
- egocentric video
- wrist-camera video
- multi-camera streams
- RGB
- depth
- point cloud
- audio where relevant
- Robot and controller state
-
- joint state
- end-effector pose
- gripper state
- base pose
- object state
- tool state
- controller state
- safety state
- Actions
-
- translation
- rotation
- gripper commands
- joint commands
- navigation actions
- tool use
- control mode
- intervention
- stop or retry action
- Force, contact and telemetry
-
- force signals
- tactile signals
- contact events
- positioning
- equipment telemetry
- Task context
-
- task instruction
- structured task goal
- target object
- destination
- constraints
- prohibited actions
- success definition
- failure definition
- Time model
-
- source clocks
- timestamp units
- expected frequencies
- missing samples
- interpolation
- calibration references
- permitted cross-stream offset
- Outcome
-
- successful
- partially successful
- failed
- interrupted
- unsafe
- invalid
- human intervention required
- retry initiated
- recovered
- abandoned
- Failure and recovery
-
- precursor state
- attempted action
- failure point
- cause category
- intervention
- recovery action
- final disposition
- Provenance and acceptance
-
- source
- rights status
- collection protocol
- calibration record
- synchronization status
- annotation version
- reviewer status
- acceptance status
- dataset version
The time model is defined before production scale, not implied by the word synchronized.
Source clocks, timestamp units, expected frequencies, missing samples, interpolation, calibration references and permitted cross-stream offset are written down.
Failure and intervention
are part of the evidence.
A clean success trajectory shows preferred execution. It may not show how a failure begins, which state preceded it, when a person should intervene, which recovery is valid or when retry becomes unsafe.
A useful failure record can preserve the precursor state, attempted action, failure point, cause category, intervention, recovery action and final disposition. A takeover can show where autonomy stopped being sufficient and whether the episode returned to autonomous execution.
The safety model decides which failures may be induced, which may only be observed and when the task must stop.
A demonstration recorded once is recorded in one condition.
The system has to act in the rest of them. A policy that fails at dusk and a capture that failed at dusk are different problems, and only the record can tell them apart.
The condition stays attached to the episode. Evaluation can then separate a policy that does not generalize from material that was never captured under the condition being tested.
Keep real and generated
evidence distinguishable.
Real data establishes the task, environment and failure distribution. Synthetic or simulated data can then extend a named edge case or rare condition.
Real baseline Synthetic coverage - Real baseline
- Named gap
- Generate
- Record lineage
- Evaluate on held-out real
- Accept for the stated use
Synthetic data is accepted for a defined job, not for volume by itself.
Evaluate the complete task under named conditions.
A single aggregate score can hide which objects, scenes, embodiments, operators or environments drive failure.
The contract defines the scenario distribution, metric formula, hardware and policy version, exclusions, human review and acceptance threshold for the actual use.
Ships with the accepted version / accepted episodes · derivatives · identifiers · episode boundaries · task and outcome states · calibration references · source lineage · quality results · exclusions · manifest · version history
The receiving team also gets a loader-tested sample or reference export that proves the agreed structure can be opened and interpreted through the target toolchain. Format support is confirmed against the source data, loader and customer requirement before it becomes a delivery commitment.
Physical AI spans
more than an
industrial robot cell.
Embodiment, object geometry, environment, viewpoint, sensor configuration, operator, control mode, payload, weather, clutter, failure condition and recovery path matter only when they can change the system decision being trained or evaluated.
The same data logic carries across embodiments. Generalist policies · manipulation · humanoid · mobile robots · logistics · service robots · outdoor machines · industrial inspection · human activity · world models.
Use proof the team can inspect.
Technical and operational evidence makes the data path inspectable. A scoped engagement can produce a task and decision specification, data-gap map, stream schema, sample episode, synchronization report, failure taxonomy, annotation guide, evaluation scenario, acceptance matrix, loader test and versioned manifest.
These artefacts let robotics, ML and operations teams inspect how the data was created, what was excluded and why the accepted material is fit for the stated task.
Define hardware, site and data controls in the operating contract.
Physical AI records can expose people, workplaces, proprietary equipment and operational behavior. Each engagement assigns robot and sensor access, operator roles, site permissions, safety procedure, rights, permitted use, security review, retention, deletion, issue handling and handover.
This control layer makes the data operation inspectable.
Bring the task, embodiment and data bottleneck.
Your first brief can be short. Include the robot or embodiment, task, deployment environment, existing data, available sensors, target model or policy, known failure, required variation and timing if you know them. YPAI can use that brief to identify the first data path, the accepted unit, the pilot question and the evidence needed for a delivery decision.
Questions before the first conversation
- We already have robot logs. Do we need new collection?
- Not necessarily. Start with an archive audit. The right answer may be to use the existing data, repair part of it, add missing context or collect only the task conditions that remain absent.
- How is this different from robotics annotation?
- Physical AI data covers the broader operation across source recovery, collection, episode construction, annotation, gap-fill, evaluation and delivery. The specialist annotation route owns detailed label and review work when that is the primary purchase.
- Can failures and recovery be included?
- They can be included when the task and safety model allow collection, or when failures already exist in source logs. The scope defines which failures may be induced, which may only be observed and when a person must intervene.
- Can real and synthetic data be combined?
- Yes, for a defined coverage gap. Real and generated sources remain distinguishable, and the combined set is tested against held-out real scenarios before acceptance for the intended use.
- Which delivery format will we receive?
- The format follows the source data, target loader and receiving workflow. YPAI confirms format feasibility during scoping and can include a loader-tested sample or reference export in the acceptance package.
Related
- Physical AI data collectionEpisode capture, demonstrations and synchronized streams.
- Robotics annotationPerception, task-phase and acceptance labelling.
- Video dataBroad video and multimodal collection.
- Sensor fusionSynchronized multimodal annotation.
- LiDAR and 3DPoint-cloud annotation and spatial data.
- Data collectionCustom video, image, sensor and multimodal collection.
- Connected deliveryData and evaluation joined to implementation.
- PilotsValidate the task, output and acceptance plan before scale.