--- title: Writing a Video Data Collection Specification url: https://ypai.ai/blog/data-engineering/video-data-collection-specification-model-requirements/ category: Data Engineering published: 2026-09-06T00:00:00.000Z modified: 2026-09-06T00:00:00.000Z author: YPAI Engineering tags: [Video Data Collection, Computer Vision, Dataset Design, Data Quality, Training Data] --- # Writing a Video Data Collection Specification > How to turn a model requirement into a video collection brief: coverage dimensions, two-layer QA, subject-independent splits, and a coverage matrix. Most video collection briefs start with a number of hours and a list of scenarios. That ordering is what produces a dataset that scores well on its own test set and then fails on a customer's phone in a dim kitchen. A collection specification is written backwards. Start from the failures the model is not allowed to have in deployment, derive the capture conditions that would produce those failures, then write the acceptance criteria and evaluation slices that let you see whether the data actually covered them. This walkthrough shows that derivation on a worked scenario and gives the matrix that connects each step. It is written for computer-vision and data leads who have to hand a specification to a collection partner. ## Start from the failure, not the format The useful first question is not "what should we film" but "what will this model get wrong in a way that matters, and what would have to be in the data for us to find out." Those failures are usually conditional rather than general. A gesture classifier does not simply fail. It fails at 1.5 meters, or against a bright window, or on the hand it saw less of, or on a natural movement that resembles a target gesture. Each of those is a capture condition. Each of them is also an evaluation slice, and if it is not both, you cannot tell whether the collection worked. This is where a published result is worth borrowing, carefully. ObjectNet, presented at NeurIPS 2019, was collected by asking crowd workers to photograph objects in their own homes to instructions that deliberately varied background, rotation and viewpoint. The authors reported a 40 to 45 percent performance loss for object detectors on the classes shared with ImageNet once those priors were removed, and that the loss persisted after fine-tuning. Two limits travel with that number. ObjectNet is a test set with no training set, and it is object recognition rather than gesture, so transfer to video is by analogy rather than by measurement. What it establishes is narrower and still useful: coverage of conditions exposes brittleness that a larger image count does not. ## An illustrative collection brief The following is an illustrative brief for a gesture-video scenario. **Every number in it is illustrative and must be replaced by pilot-derived values.** They are written as concrete figures because a vague specification cannot be tested, not because they are proven requirements. **Task.** Per-clip classification of a fixed set of upper-body and hand gestures from a front-facing smartphone stream, for on-device UI control. Illustratively, twelve dynamic gestures such as swipe left, swipe right, thumbs up, wave, palm-stop and pinch-zoom, plus a "no gesture" contrast class of natural movement. **Deployment assumptions.** User seated or standing 0.3 to 1.5 meters from a handheld or propped phone. Indoor home or office. Mixed daylight and artificial light. One or two hands. Varied clothing including long sleeves. **Capture protocol.** Unmoderated remote capture as the default, because it gives scale and it gives natural devices and backgrounds. A moderated tranche is added for the hard slices, meaning low light, distance and occlusion, where a supervised session is the only way to guarantee the condition is actually present. Per participant: on-screen text and a reference example clip per gesture, a countdown, self-review with the option to re-record, and an instruction to keep the gesturing hands in frame. Roughly three to six seconds per clip. The "no gesture" task asks for several distinct everyday hand movements that resemble the targets, which is the part most briefs omit. **Coverage dimensions.** Subjects across age bands and self-reported gender. Skin tone stratified on a published scale such as Fitzpatrick or Monk, with a low-light and darker-skin cell deliberately covered rather than left to chance. At least two device tiers, older and newer. Distances across the deployment range. Lighting including backlit and window extremes. Left hand, right hand and both. Sleeve length. Geographies only where the deployment actually justifies them. **Acceptance criteria, in two layers.** Layer one is task-independent and rejects: corrupt or unreadable files, wrong duration, the specified action absent, missing or invalid metadata, wrong orientation. Layer two is task-conditional: minimum resolution, both hands in frame when the label requires it, and an independent verifier confirming the labeled gesture by picking it from the label set without seeing the original assignment. Low light and partial occlusion pass layer two as hard examples. They are not rejects. That last sentence is the one that most often goes missing, and it is the one that decides whether the collection is worth running. A blanket quality filter deletes exactly the clips the specification was written to obtain. **Metadata.** Subject ID, session ID, device and OS and camera model, capture-distance bucket, lighting condition, hands used, consent basis and permitted-use scope, timestamp, and collection-tool version. Metadata that is not captured at recording time cannot be reconstructed later, and every field here is load-bearing for some slice in the matrix below. **Split strategy.** Subject-independent by default. Additionally hold out by device tier, and by scene if scenes recur. Perceptual-hash de-duplication across the split boundary. Reserve both a deployment-representative test slice and an oversampled hard-condition stress slice, labeled separately so their numbers never get averaged together. **Delivery contents.** Accepted clips, per-clip metadata, label and verification record, rights and consent record with an explicit commercial-use grant, the QA and acceptance report, checksums, a datasheet, and a versioned manifest. ## The coverage matrix This is the artifact that makes a collection brief auditable. Each row starts with a deployment failure and ends with the number you will report. If a row cannot be completed, either the condition is not being collected or its effect will be invisible. | Model risk in deployment | Capture condition to collect | Acceptance or QA check | Evaluation slice to report | |---|---|---|---| | Fails on unseen people | Many distinct subjects, enough to hold some out | Subject ID recorded, subjects not shared across splits | Subject-independent test accuracy | | Fails in low light or backlight | Deliberate dark and window-backlit captures | Low light accepted as a hard example, not rejected | Accuracy on low-light slice against normal | | Fails at distance | Captures across the deployment distance range | Distance bucket recorded | Accuracy by distance bucket | | Confuses look-alike non-gestures | A "no gesture" class of natural movements | Verifier confirms the non-gesture label | False-positive rate on the contrast class | | Fails for darker skin tones | Skin-tone-stratified recruitment, with a low-light and darker-skin cell | Skin-tone annotation present | Per-skin-tone accuracy gap | | Left or right hand bias | Both hands, both sides | Hand-used field recorded | Accuracy by hand used | | Device or codec brittleness | At least two device tiers, natural compression | Device and OS metadata captured | Accuracy by device tier | | Inflated metrics from duplicates | Post-capture rather than a capture condition | Perceptual-hash de-duplication across splits | Duplicate rate removed, reported | Read the last column on its own and it is the evaluation plan. Read the first column on its own and it is the risk register. The matrix exists so that those two documents cannot drift apart, which is what happens when the collection brief and the evaluation plan are written by different people at different times. ## The split is part of the collection, not part of the analysis You cannot split by subject after the fact if nobody recorded a subject identifier. This is why the split strategy sits in the collection specification rather than in a downstream notebook. Subject-independent splitting is the documented norm in this field. Gesture and action datasets including Jester, Something-Something, HaGRID and EgoGesture split train and test by worker or by subject specifically so that the same person does not appear on both sides. The failure mode when they do not is well characterized. A re-analysis of a UAV gesture study noted that a 90/10 sample-level split over six subjects necessarily places every subject in both train and test, so the reported accuracy reflects subject-level leakage rather than generalization. Duplicate leakage does the same thing more bluntly: a leakage-safe re-evaluation of a CCTV violence-detection benchmark found 287 test clips that were byte-for-byte copies of training clips, producing a meaningless F1 of 1.000, and after cryptographic and perceptual-hash de-duplication an X3D-S model scored a mean F1 of 0.894 with a standard deviation of 0.020. That study is violence detection rather than gesture, so it illustrates the mechanism rather than sizing the effect for your project. Near-duplicate contamination is not exotic either: a datamodels-based analysis of CIFAR-10 found that a majority of annotators judged roughly 10 percent of test images to be same-scene near-duplicates of training images. The practical consequence for a specification is three lines: record a subject identifier, run perceptual-hash de-duplication across the split boundary, and report the duplicate rate you removed. The third line is the one that lets a buyer compare two vendors honestly. ## What the evidence supports about scale Volume is the first question most briefs ask and close to the last one the evidence answers. HaGRID, published at WACV 2024, is a useful reference point because it is large and its splitting is explicit: 554,800 images across 18 gesture classes plus one no-gesture class, 37,583 unique subjects, captured between 0.5 and 4 meters under varied lighting, split by subject in roughly 74/10/16 proportions. On its 90,000-image test set the best classifier reached an F1 of 98.3 and the best detector reached an mAP of 79.1. The more interesting result is the ablation, and it comes with a real limitation: it was published as figure trends rather than as a table, so the numbers cannot be quoted precisely. Qualitatively, classifier accuracy rose quickly with more samples per class and then saturated, and the authors describe a large share of samples as redundant for classification while still being useful for detection. Lighting diversity mattered little for same-dataset testing but helped when fine-tuning across datasets. Two things follow for a specification. First, the saturation point is task-dependent on the same data, so a volume target borrowed from a classification project can be wrong for a detection project on identical footage. Second, the value of diversity showed up in the cross-dataset condition rather than the same-dataset one, which is the condition that resembles deployment. Related findings point the same direction from other modalities and deserve their own caveats. A controlled study of visual compositional generalization found the effect driven by combinatorial diversity in the data rather than by raw quantity. Work on image-text pretraining found that mixing web sources did not beat the best single source and that combining sources sometimes diluted the robustness of the strongest one. Neither is video gesture, and neither transfers as a number. Together they argue against treating "more data" as the default answer to a robustness problem. ## Rights and consent are part of the specification A specification that does not state permitted use produces a dataset you cannot ship. Two points are worth writing into the brief explicitly. Research access is not commercial permission: WLASL, a widely used sign-language benchmark assembled from web video, is released under an agreement limited to academic and computational use with no commercial usage. Availability and licensing are separate questions, and the second one is the one that governs a product. On regulation, the classification depends on purpose rather than modality. Under GDPR, biometric data is special-category data when it is processed for the purpose of uniquely identifying a person, so plain gesture video is personal data without automatically becoming Article 9 data on that basis alone. In the United States, Illinois BIPA requires prior written notice and a signed release before collecting a scan of hand or face geometry, mandates a published retention schedule, and carries a private right of action, with a 2024 amendment permitting electronic signatures. Both of these are legal sources rather than engineering guidance and should be confirmed per project with counsel. What belongs in the specification is the consent basis, the permitted-use scope and the retention period, recorded per clip. ## What to hand a collection partner A specification a vendor can actually price and execute states the task and label set, the deployment assumptions, the capture protocol, the coverage dimensions with pilot-testable targets, both layers of acceptance criteria, the metadata schema, the split strategy, and the delivery package. Run a pilot before fixing any of the numbers. The pilot's job is not to produce data but to replace the illustrative values with measured ones: what the rejection rate actually is under layer one, whether the hard-condition cells can be filled at the recruitment rate you assumed, and whether the verifier agrees with the collector often enough for the label to mean anything. Then report by slice rather than in aggregate. A single accuracy number computed across a deliberately unbalanced dataset, part deployment-representative and part hard-condition stress, is not a meaningful quantity. The matrix already told you which numbers to publish. Every figure cited above is a third-party published result with its stated scope and limits. They are useful for reasoning about mechanism. None of them is a target for your dataset. ## YPAI managed video collection YPAI runs video collection against a written specification rather than a volume target. Scoping covers the coverage dimensions and their evaluation slices, both layers of acceptance criteria, the metadata schema recorded at capture time, the split strategy, and a delivery package with consent records, checksums, a datasheet and a versioned manifest. Computer-vision and data leads with a defined model requirement can [request a consultation](/contact-us/) to scope a collection, or read how we run [video data collection](/video-data-collection/) programs. --- ## Related Resources - [Data labeling QA thresholds](/blog/data-engineering/data-labeling-quality-assurance-thresholds/) - What published sources actually support on agreement metrics - [Computer vision applications from image annotation to production](/blog/infrastructure/computer-vision-applications-image-annotation-production-deployment/) - Where collected video goes after delivery - [AI training data guide](/blog/data-engineering/ai-training-data-guide/) - Sourcing, licensing and provenance across modalities - [EU AI Act Article 10 engineering requirements](/blog/compliance/eu-ai-act-article-10-engineering-requirements/) - Data-governance obligations and what they mean for engineering teams - [Video data collection](/video-data-collection/) - Managed collection programs with YPAI