---
title: "Video Annotation: Tracking, VOS, Action, Pose | YPAI"
url: https://ypai.ai/annotation/video-annotation-services/
description: "Multi-object tracking, video object segmentation, action localisation, pose tracking, re-ID. Assisted tracking with human review, HOTA and IDF1 reported."
source: "src/copy/routes (route /annotation/video-annotation-services/)"
---

# Video annotation, tracked across every frame.

> Multi-object tracking, video object segmentation, action localisation, pose tracking, re-ID. Assisted tracking with human review, HOTA and IDF1 reported.

Multi-object tracking, video object segmentation, action localisation, pose, re-ID. SAM 2 assisted, kappa-gated, HOTA-reported per delivery.

## A clean frame can hide a broken track.

- A bounding box can be correct at one timestamp while the sequence fails around it.
- The same object may disappear behind an occluder, leave the field of view, return under different lighting or move into another camera. A useful annotation must preserve the right identity, boundary and event state throughout.

Review the sequence as a track, not as a stack of images.

crossing F 0195 - F 0263

Reference frame f 0208 · all tracks valid

Illustrative annotation readout. The engagement defines the actual classes, thresholds, confidence fields and output schema.

Window f 0182 - f 0271 · 90 frames · single camera

The failure often appears between two individually correct frames.

- Does the same object keep the same track ID through visibility changes?
- Record f 0232 · identity discontinuity
- Does the record distinguish partial visibility, full occlusion and disappearance from the scene?
- Record f 0217 · full occlusion, occluder ID 04
- When the object returns, is it linked to the correct existing identity or assigned a new one?
- Record f 0232 · re-entry, new identity assigned
- Can the reviewer locate the exact point where continuity broke?
- Record f 0232 · ID switch, ID 03 -> ID 17
- Does the action begin and end where the annotation contract says it does?
- Record f 0195 - f 0263 · action crossing
- Which frame or transition required judgement rather than automated propagation?
- Record f 0232 · reviewer adjudication

## For regulated video, the annotation record is part of the delivery.

Ontology, de-identification, reviewer access and change history are not administrative details added after annotation. They determine whether the resulting data can be understood, reviewed and used within the intended workflow.

### Data governance

- These records can support the customer’s own data-governance and technical documentation.

### Sensitive and identifiable content

- Where footage contains faces, licence plates, patients, employees or other identifiable people, the engagement defines:
- what must be removed or transformed
- The method is selected against the intended use and residual risk. The project does not assume that one generic de-identification method is sufficient for every video type.

### Processing responsibilities

- Where YPAI processes personal data on the customer’s behalf, the engagement can define:
- EEA residency by default. Residency, transfers and subprocessors are established per project.

YPAI supplies the agreed annotation operation, quality records and delivery evidence.

## Five annotation primitives. Five different definitions of correct.

The primitive determines what is labelled, how temporal consistency is reviewed, which metrics are meaningful and what the final delivery contains.

Same footage, different definition of correct

YPAI scopes the annotation object before selecting the tooling.

5 alternatives · 5 common fields

A project can combine primitives when the model needs more than one definition of correct in the same footage.

### Multi-object tracking

- The annotation follows the object through visibility changes and retains its persistent identity.
- Objects are located frame by frame and assigned persistent identities.
- Identity must survive partial visibility, full occlusion, scene exit and re-entry.
- Bounding boxes, track IDs, visibility states and frame-level attributes.
- Vehicles, people, equipment, animals and other moving objects.

### Video object segmentation

- The annotation follows the object boundary at pixel level while preserving the same instance across the sequence.
- Each object receives a pixel-level mask across the sequence, linked to a continuous identity.
- The mask must preserve both geometric accuracy and instance continuity as shape, lighting and visibility change.
- Per-frame masks, object IDs, visibility states and exception records.
- , boundary quality, missed objects and identity continuity.
- Fine-grained object boundaries, surgical scenes, content editing and foreground extraction.

### Action localisation

- The annotation defines where the action begins, continues and ends under the agreed temporal rule.
- Actions or events receive a class and a temporal start and end point.
- The first visible sign of an action, full action onset and task completion are different boundaries.
- Temporal segments, action classes, actor links and overlapping-event rules.
- at the agreed temporal overlap thresholds.
- Surgical phases, sports actions, industrial tasks, safety events and content understanding.

### Pose tracking

- The annotation links keypoints to a coherent skeleton and persistent person identity through motion.
- Joint keypoints are linked across frames and associated with persistent person identities.
- Keypoints must remain anatomically coherent through motion, occlusion, truncation and viewpoint change.
- Keypoints, skeletons, visibility flags and person track IDs.
- , missing joints, identity continuity and temporal consistency.
- Sports, rehabilitation, animation, ergonomics and human activity analysis.

### Re-identification

- The annotation links the same identity across separated cameras or sequences and records the handoff.
- The same person or object is linked across non-overlapping cameras or separated sequences.
- Clothing, angle, lighting, camera quality and elapsed time may all change while identity must remain consistent.
- Cross-camera identity links, handoff events, timestamps and exception records.
- , false links, missed links and project-specific retrieval or tracking metrics.
- Multi-camera operations, retail dwell analysis, sports and authorised security workflows.

## The ontology changes with the footage.

Automotive, surgical, surveillance, sports and retail video may all use tracks, masks or event labels. They do not use the same object definitions, privacy boundary or review logic.

Recognise the footage first. Define the annotation contract from its operating reality.

### Automotive

- Drivers, occupants, gaze, gesture, vehicles, pedestrians, cyclists, lanes and temporal events.
- In-cabin and outward-facing footage require different identity, visibility and scene rules. Privacy and safety requirements are scoped against the actual project.
- Driver monitoring, occupant monitoring, multi-object tracking, action and gesture taxonomies.

### Surveillance and security

- People, objects, movement, cross-camera identity, zones and events.
- Lawful basis, access, retention, signage and proportionality must be resolved for the intended deployment. A DPIA may be required for systematic monitoring.
- MOT, re-identification, event windows, entry and exit states, camera handoffs.

### Surgical video

- Tools, anatomy, surgical phase, actions and interactions.
- Health data, patient identifiers, reviewer qualifications and restricted access are defined before annotation begins.
- Phase, tool and anatomy taxonomies. Public research datasets may inform schema design, subject to their access and licensing restrictions.

### Sports analytics

- Players, ball, pose, actions, formations, possession and spatial events.
- Footage rights, athlete data, league agreements and intended commercial use must be established for the engagement.
- Tracking, pose and action taxonomies such as those used in sports-research benchmarks.

### Retail and content

- People, queues, dwell zones, movement, interactions, shots and content events.
- The project may require a legitimate-interest assessment, defined retention and a documented moderation or loss-prevention purpose.
- Dwell events, queue states, zone entry and exit, shot boundaries and content labels.

## Production starts at one gate.

Schema, guideline and calibration all serve the same decision:

Is the annotation contract stable enough to scale?

One gate. One recorded decision. A visible route back when the contract is not ready.

Where the metrics apply to the task, the reference production gate is:

The statement of work confirms which metrics apply and may define stricter or task-specific thresholds.

- One or more applicable requirements remain below the gate.
- The annotation contract returns to schema, guideline or calibration.
- All applicable requirements clear the agreed gate.

### Schema and sensitive-content policy

### Annotation guideline

### Calibration round

- A shared subset is annotated and reviewed before production.
- The round is used to expose:
- Disagreement feeds back into the schema and guideline.

### The gate

### Production

- Production may combine keyframe annotation, assisted tracking, interpolation and human review where the method fits the footage and acceptance plan.
- Assisted tracking handles the frames between keyframes. Correctness is decided at the agreed review gate.
- Annotated video, tracks, masks, events or keypoints in the agreed format.

### QA, adjudication and delivery

- Accepted delivery, quality record and known-limitations statement.

## The annotation is the sequence, not the screenshot.

A production track records what happened between the selected frames.

The diagram is schematic. Production work runs against the customer’s footage and approved ontology.

Where assistive tooling proposes masks or tracks, human review focuses on the cases that determine acceptance: occlusion, identity continuity, boundary drift, scene handoff and ambiguous events.

### Entry

- The object becomes observable and receives its first project-defined identity.

### Partial occlusion

- Visibility changes, but the identity and boundary rules remain active.

### Full occlusion

- The object is temporarily unobservable. The record distinguishes occlusion from confirmed exit.

### Re-entry

- The object returns. The reviewer confirms whether the earlier identity remains valid.

### ID-switch event

- The system or annotator links the object incorrectly. The exact transition is flagged for correction or adjudication.

### Action boundary

- The sequence records where the defined event begins, continues and ends.

### Human review

- Low-confidence or ambiguous transitions move to a reviewer under the project’s escalation rules.

## A benchmark schema can enter the project. The rights to its footage do not.

- Public benchmarks can help define taxonomy, output format and evaluation.
- Research access, annotation licences and familiar file formats do not automatically grant the right to use the underlying footage for commercial model training or deployment.
- YPAI treats benchmark structure and source-data rights as separate questions.
- A familiar schema is not evidence that the footage behind it may be used.
- Every source, licence and permitted use is assessed for the engagement.

### MOTChallenge

- Research and non-commercial terms must be checked against the intended use.

### DAVIS

- Public research access does not establish commercial training rights.

### YouTube-VOS

- Underlying video rights and annotation rights must be assessed separately.

### Kinetics-700

- An annotation licence does not necessarily confer rights to the source video.

### AVA

- Source footage, annotations and downstream use must each be checked.

### Cholec80

- Research, clinical-access and patient-data restrictions remain outside a generic annotation licence.

## The evidence package is defined before production.

- The delivery is more than labelled frames.
- The statement of work identifies the records the customer will receive, how they relate to the data and which technical or governance questions they answer.
- The exact evidence package is defined in the statement of work and applicable data-processing terms.
- These records support the customer’s own technical, procurement and regulatory review.

### Annotation contract

### Calibration and quality record

### Track-continuity record

### Sensitive-content and de-identification record

### Processing and delivery record

## Bring the footage, the model objective and the acceptance problem.

- A short brief is enough to start.
- YPAI will identify the appropriate annotation primitive, the open ontology decisions, the data and rights boundary and the first calibration step.
- the objects, actions or identities of interest
- the expected volume and volume unit

We reply inside one EU business day with the questions that must be resolved before a pilot or production scope.

## Answers that belong in the first conversation.

### How do you handle faces, licence plates and patient identifiers?

- The handling is agreed before annotation begins, against the intended use and the residual risk.
- Depending on that assessment the project may use exclusion, masking, blurring, replacement, restricted access, metadata removal or a reduced representation such as skeleton-only output.
- What was chosen, which exceptions were granted and what limitations remain are documented for the engagement.

### Can YPAI handle multi-camera re-identification?

- The delivery can include cross-camera identity links, handoff events, false-link review and project-specific continuity reporting.

### Do you use SAM 2 or other assisted annotation tools?

- Assistive segmentation and tracking can be used where they fit the footage, ontology and acceptance plan.
- The tool does not define correctness.
- Human reviewers remain responsible for the agreed checks, especially around occlusion, identity continuity, boundary drift, scene changes and ambiguous events.
- Tooling is confirmed during scoping rather than promised as a fixed method for every project.

### Where is the video processed?

- The engagement defines the processing environment, access model, residency, subprocessors and transfer controls.
- EEA residency by default; subprocessors and transfers are agreed per engagement.
- Customer-controlled environments or other restricted workflows can be assessed during scoping, but feasibility depends on the technical, security and operational requirements.

### How is annotation quality measured?

- The metric follows the annotation job.
- Tracking may use HOTA, IDF1, ID switches and fragmentation.
- Segmentation may use overlap and boundary measures.
- Action localisation may use mean average precision at temporal overlap thresholds.
- Pose may use keypoint metrics plus identity continuity.
- No universal accuracy percentage is meaningful across all of these tasks.

### What happens when annotators disagree?

- Disagreement is recorded, not silently overwritten.
- Recurring disagreement is evidence about the annotation contract, not merely worker performance.

### Which output formats can you deliver?

- The exact format is defined in the statement of work.
- Every delivery includes a manifest that identifies the agreed files, versions and fields.

### Can YPAI work in our existing annotation environment?

- Yes, once the engagement has defined how we work in it.
- YPAI can assess a customer-controlled environment or propose a managed workflow.

## Start with the sequence your model must understand.

- Bring the footage, annotation objective, classes, output requirements and acceptance problem.
- YPAI will return the proposed project scope, the open decisions, the calibration plan and the evidence package required to evaluate the first delivery.

First scoping response inside one EU business day. Production begins only after the annotation contract and applicable acceptance gate are agreed.

## What video annotation tasks does YPAI deliver?

YPAI delivers object detection and tracking across frames, instance segmentation with temporal IDs, action recognition and event-segment labelling, multi-camera re-identification, 3D tracking with sensor fusion, pose and gesture annotation, and dense per-frame keypoint markup. The deliverable is structured per-frame and per-track so downstream training pipelines can choose the granularity they need without re-running annotation.

## Which video formats and resolutions does YPAI handle?

Standard codecs (H.264, H.265, VP9, AV1) and container formats (MP4, MOV, MKV) are supported. Working resolution is matched to the customer pipeline rather than fixed at 1080p, so 4K vehicle footage, drone imagery, and surveillance-grade lower-resolution streams are all handled. Time-stamped sensor traces (LiDAR, radar, IMU) can be annotated synchronously with video for autonomous-vehicle engagements.

## How does YPAI handle long-form video annotation?

Long-form video is split into review-friendly segments with frame-level cross-segment validation so track IDs remain consistent across the cut points. Annotators specialise by domain (automotive scene-understanding, retail loss-prevention, industrial vision, surgical recording) rather than rotating through arbitrary content. IAA on tracking metrics (HOTA, MOTA) is reported per batch.

## Can YPAI annotate autonomous-vehicle perception data?

Yes. Multi-camera, LiDAR, and radar fusion annotation for ADAS and autonomous-vehicle perception stacks is a recurring engagement type. The delivery includes project-specific taxonomies, traceability, edge-case documentation, and EU AI Act Article 10 evidence for the customer audit trail. See <a href="/solutions/automotive/">automotive solutions</a> for the wider engagement model.

## What output formats does YPAI deliver for video?

Deliveries are shipped in COCO-Video, CVAT, KITTI tracking format, custom JSON with frame and track schemas, and any customer-specified output. Frame-accurate timestamps and sensor-trace synchronisation are preserved through the export. The delivery bundle includes a calibration record where multi-sensor alignment was performed.

## How are video annotation engagements priced?

Pricing is per-project after scoping. Drivers include hours of source video, target IAA, label density per frame, and any multi-sensor fusion requirements. A flat rate per video-minute rarely reflects real cost. The scoping call at <a href="/contact-us/">/contact-us/</a> produces a written proposal with the cost drivers itemised so the customer can see which dimensions trade off against the others.

## Is video annotation GDPR-compliant for footage containing people?

Footage with identifiable individuals is processed under GDPR on a documented lawful basis. Where the use case requires it, YPAI runs blurring or face-obfuscation passes on identifiable subjects who did not provide consent for downstream model training. The decision tree is documented in the SOW and the DPA covers the processing terms.