YPAI Insights: Agentic AI

AI Email-to-CRM Workflows: Demo to Production

noreply@ypai.ai (YPAI Engineering) — Sun, 06 Sep 2026 00:00:00 GMT

A customer emails a signed order amendment as a PDF. Two values in that PDF have to reach the right CRM record, and a reply has to go back. Wiring that up takes an afternoon and it will work. Running it four hundred times a week without silently writing a wrong number into a live account is a different engineering problem, and no model release solves it.

What follows is a stage-by-stage decomposition of an email-to-document-to-CRM workflow: which component owns each stage, what published benchmarks say about the reliability of the stages a model would own, and which failure modes appear only once the stages are chained. It is written for operations and automation owners who have already seen the demo work.

The suitable work and the unreliable work are the same work

The stages that look like obvious language-model territory are classification, extraction and interpretation. Those are also the stages that measure worst.

On CRMArena-Pro, a Salesforce AI Research benchmark of 19 tasks in a synthetic Salesforce org, leading agents reached roughly 58 percent success on single-turn business tasks and dropped to roughly 35 percent multi-turn. Workflow execution such as case routing was the bright spot at 83 percent for one model, while rule-following and text-understanding scored materially lower. The benchmark is Salesforce-authored and runs on a synthetic org, so read it as a bound on expectations rather than a forecast.

On customer-service task completion, the sharper number comes from grading the database rather than the transcript. The tau-bench work from Sierra scores on final database state, and reported 61.2 percent retail and 35.2 percent airline at pass^1 for one model generation, with pass^8 below 25 percent in retail. Running the same task eight times and succeeding all eight times happened less than a quarter of the time. The follow-up tau-squared-bench added a domain where the user also controls part of the environment, and moving from agent-only control to shared control cost roughly 20 percentage points of pass^1.

On document extraction, the OmniAI OCR benchmark from February 2025 measured JSON extraction accuracy across general models and cloud document services, scoring accuracy by comparing the predicted JSON object against the ground-truth object and taking one minus the ratio of differing fields to total fields. Across the systems other than the benchmark's own operator, results spanned roughly 51 to 86 percent; including that operator's system the top of the range was 92 percent. The benchmark is vendor-run and its operator topped it, which is the caveat to carry, and the spread matters more than any single entry: on the same documents, comparable products differed by more than thirty points. OmniDocBench, an academic document-parsing benchmark, scores text with normalized edit distance and tables with TEDS across a corpus annotated by page type and attribute, which is the more useful design point here: it reports where a parser degrades rather than a single headline number.

Three things get conflated when people quote any of these numbers. A schema-conformant JSON output is not the same thing as a factually correct interpretation of the document, and neither is the same thing as a completed business task where the right record changed and nothing else broke. Constrained decoding solves the first only.

Step accuracy does not survive chaining

For independent steps at accuracy p over k steps, end-to-end success is p raised to the power of k. That arithmetic is unforgiving in a way step-level numbers hide:

Per-step accuracy	Steps	End-to-end success
99%	100	36.6%
95%	20	~36%
95%	10	~59%
90%	10	~35%

Real pipelines correlate their errors, so this arithmetic is a model rather than a measurement. It is still the right mental model, because it explains why the intuition "each step is nearly always right" produces a workflow that is wrong most of the time. It also explains why each additional nine of reliability costs roughly the same engineering effort as the previous one.

The compounding is worse than the arithmetic suggests, because language-model failures are usually silent. A misclassified intent, a wrong entity match or a plausible-but-wrong extracted value passes downstream without raising an exception. Deterministic code fails loudly. A model fails politely.

The design consequence is that reliability is bought two ways, and neither of them is a better prompt. Remove steps from the model, and insert verification between the steps that remain.

The workflow, stage by stage

The following is an illustrative decomposition, not a deployed configuration. It exists to make the authority boundary explicit at every stage: what the input is, who is responsible, what is checked, and what the stage is permitted to do.

Stage	Input	Operation	Responsible component	Validation	Permitted action	Failure handling
1. Ingest and de-duplicate	Inbound email plus PDF	Capture, hash, assign request ID	Deterministic code	Dedup on message ID or hash, virus and type scan	Store raw, quarantine untrusted content	Drop or merge duplicates, reject malformed
2. Classify intent	Email body, untrusted	Assign intent label	Model classifier, fixed taxonomy	Confidence threshold, label must be in taxonomy	Route by label	Low confidence routes to human triage
3. Identify customer	Sender, signature, references	Match to CRM account	Deterministic query on validated keys	Exact or fuzzy match score gate	Read-only CRM lookup	No match or ambiguous match routes to human
4. Extract fields	PDF attachment	OCR plus schema-constrained extraction	Document model plus validator	Schema conformance and source-span check	Populate draft record fields	Below-threshold field is flagged for review
5. Interpret request	Classified and extracted data	Formulate what is being asked	Model, proposal only	Cross-check against extracted fields	Produce a proposed action, not an action	Contradiction routes to human
6. Check business rules	Proposed action plus records	Apply eligibility, policy, limits	Deterministic rules engine	Rule pass or fail, logged	Approve or deny the proposal	Rule failure takes the exception route
7. Prepare response	Approved proposal	Draft customer reply	Model, draft only	Groundedness, PII and output filter	Create a draft, do not send	Ungrounded or unsafe output regenerates or escalates
8. Human approval	Draft plus record change	Review the consequential step	Human reviewer	Positive confirmation required	Authorize send and CRM write	Rejection corrects or escalates
9. Execute	Approved action	Write CRM, send email	Deterministic idempotent executor	Idempotency key, write acknowledgement	Commit, external send	Retry under idempotency key, roll back or alert on partial failure
10. Log and learn	All artifacts	Audit record	Deterministic logging	Completeness check	Store for audit, evaluation, regression	Alert on missing telemetry

Read down the "responsible component" column and the architecture states itself. The model appears at four stages out of ten, and at three of those four its output is explicitly a proposal or a draft. Retrieval, business rules, execution and logging are deterministic, because they are the stages where being wrong is expensive and being consistent is cheap.

Read the "permitted action" column and the same point appears from the other direction. No stage that touches untrusted content is permitted to act. The permission to act appears at stage 9, after a rules engine and a human have both had a turn.

The boundary is proposing versus acting, not assisting versus autonomous

Anthropic's engineering guidance on building effective agents draws the distinction that matters here: a workflow orchestrates models and tools through predefined code paths, while an agent lets the model direct its own process and tool use. Their recommendation is to find the simplest solution that works and increase complexity only when needed, including not building an agentic system at all. OpenAI's practical guide agrees from the other side, reserving agents for workflows with genuine ambiguity, varied paths and brittle rule sets.

An email-to-CRM flow mostly does not qualify. The path is knowable in advance. An agent adds cost, latency and attack surface in exchange for flexibility this workflow does not need. Where an agent does earn its place is a multi-system investigation where the next step genuinely depends on the last tool's return.

Permissions should be graduated rather than binary, and the gradient runs along reversibility:

Read internal data. Low risk, automate.
Propose a reply or a record change. Safe by construction, and it is the model's natural output.
Update internal records. Deterministic, idempotent, logged, reversible where possible.
Send external communications or make irreversible writes. Human approval gate.

The most useful thing this framing buys is that a wrong model output at stage 5 becomes a rejected proposal rather than an incident.

Human review has to be load-bearing

"Add a human in the loop" is where most designs stop, and the human-factors literature points the other way. Parasuraman and Manzey's 2010 review in Human Factors treats two related but distinct effects. Automation complacency, the degraded monitoring of an automated aid, shows up under multitask load and is not overcome by simple practice. Automation bias, the tendency to follow the aid rather than check it, is not prevented by training or by instructions, and it produces both omission errors, where a failure goes unnoticed, and commission errors, where a wrong recommendation is acted on. The condition under which both effects were observed is an aid that is imperfect but usually right. Any extraction step in this workflow sits in that band by construction, which is why the review design matters as much as the extraction accuracy. The failure mode both effects predict is the reviewer who has approved two hundred correct extractions and approves the two hundred and first without really reading it.

That review predates language models, so applying it to an extraction queue is inference rather than measurement. The mechanism it describes, an aid that is reliable enough to trust and wrong often enough to matter, is the same one, and the inference runs in the direction of caution. The design responses are concrete:

Surface the source. Highlight the span in the PDF that a value came from, so verification is a glance instead of a re-keying.
Show confidence and route only genuine exceptions, concentrating attention on the hard cases rather than diluting it across easy ones.
Require positive confirmation for external or irreversible actions. Silence is not approval.
Log every decision with actor, timestamp, model version and sources used, so a disputed outcome can be reconstructed.
Sample known-answer items into the review queue to measure whether reviewer attention is drifting.

Untrusted content arrives by design

This workflow reads an email body and an attachment, both written by someone outside the organization, and it holds credentials to a CRM and a mail sender. That combination has a name. Simon Willison's lethal trifecta describes an AI system with access to private data, exposure to untrusted content, and the ability to communicate externally as having a working exfiltration path, where the reliable mitigation is to break one leg rather than to harden all three.

This is not hypothetical for this ingress channel specifically. EchoLeak, CVE-2025-32711 at CVSS 9.3, was a zero-click indirect prompt injection in Microsoft 365 Copilot: a single crafted email could cause internal data to be exfiltrated with no user interaction at all. No click, no attachment opened. It has been described as the first documented weaponization of indirect prompt injection in a production language-model system. It was patched server-side and there is no confirmed in-the-wild exploitation, which is the correct caveat and not a reason to design around its absence.

The OWASP Top 10 for LLM Applications for 2025 ranks prompt injection first, with sensitive information disclosure and excessive agency also high, and states plainly that neither retrieval augmentation nor fine-tuning fully mitigates injection. On AgentDojo, a NeurIPS 2024 benchmark of 97 tasks and 629 security cases, indirect-injection attacks succeed at meaningful rates, and while sanitization and firewall-style defenses cut attack success sharply, published defenses have repeatedly been bypassed by adaptive attacks. Anthropic's Claude Opus 4.5 system card from November 2025 reports, for that model's Thinking configuration in an evaluation combining indirect injection, direct injection and jailbreaking, attack success rising from 4.7 percent at a single attempt to 63.0 percent across one hundred attempts. Whatever the absolute numbers on your system, that shape is the point: a defense measured at one attempt has not been measured against an attacker who can retry.

Design so that a successful injection is survivable rather than merely improbable:

Content extracted from an untrusted email or PDF never directly determines a tool call or a recipient. It is parsed to structured fields, and only validated fields drive actions.
Tool scopes are least-privilege, and recipients or domains are allowlisted.
Any path that has touched untrusted content loses its external-communication leg until a human has reviewed it.
Model output is treated as untrusted input to every downstream system, including your own.
Indirect-injection cases are run against your actual document types before go-live, not against a generic corpus.

What to measure, and against what

Measure per request rather than per model call: task completion rate verified against system state, critical-field error rate, incorrect-action rate, exception and escalation rate, human review time, latency, and cost per successfully completed request. That last one is the number that moves when the exception rate moves, and it is the one that per-attempt pricing hides.

Baseline against three alternatives rather than one: the current human process, deterministic automation without a model, and no automation at all. Include maintenance, monitoring, review labor and regression testing in the cost side. Time saved does not automatically convert to money saved.

Two published results bound expectations in opposite directions, and both deserve their caveats.

MIT NANDA's The GenAI Divide: State of AI in Business 2025 is the source of the widely repeated claim that 95 percent of enterprise GenAI projects deliver no measurable P&L impact. Read the methodology before you quote the number. The report describes structured interviews with representatives of 52 organizations, survey responses from 153 senior leaders, and a review of more than 300 publicly disclosed initiatives. Those are interviews, a leader survey and a document review, not a controlled measurement of whether the systems worked, and the secondary reporting that circulated alongside the report did not always carry that distinction. The defensible reading stays narrow: across the initiatives the authors could see, most could not be shown to have moved P&L. Whether the impact was absent or merely unmeasured is not something the report settles.

The cleanest causal evidence points the other way, for a narrower claim. Brynjolfsson, Li and Raymond studied a staggered rollout of a generative-AI assistant across 5,179 customer-support agents at a single firm, first as NBER working paper 31161 and later in the Quarterly Journal of Economics. The working paper reported a 14 percent average increase in issues resolved per hour, with the gain concentrated in novice and lower-skilled workers and close to none among the most experienced. Exact point estimates shifted between the working paper and the published version, so cite the study rather than a single figure. Two limits travel with it regardless: one firm and structured chat support, and an assistive deployment where the model suggested and humans decided. It is evidence for the pattern in this article, not for autonomy.

The cautionary case is worth stating precisely because it is usually quoted imprecisely. Klarna reported in 2024 that its assistant handled 2.3 million chats in its first month, equated to roughly 700 agents and roughly 40 million dollars of projected profit impact. In May 2025 its chief executive told Bloomberg that the cost-first approach had ended up with lower quality and that the company was rehiring humans. On the Q3 2025 earnings call the company said the assistant now does the work of more than 853 full-time agents and saves 60 million dollars, while trade coverage noted that customer-service and operations cost had nonetheless risen to 50 million dollars from 42 million a year earlier. Every efficiency figure there is first-party, the agent-equivalence is modeled rather than headcount, and the numbers do not cleanly reconcile. The transferable part is not the arithmetic but the sequence the company described: volume was handled, service quality was judged to have fallen, and the design was partly reversed. On the company's own account it was quality rather than throughput that forced the change.

Vendor resolution rates deserve the same scrutiny, because the definition does most of the work. One widely cited vendor figure counts a conversation resolved if the customer does not reply or escalate within a window, which is not what competitors mean by deflection, and independent production analyses of the same product land considerably lower. Pin the definition before comparing two numbers.

Scoping a bounded pilot

An initial brief should carry monthly message volume and peak, an intent taxonomy with rough frequencies, document types and their quality (native PDF versus scanned versus photographed, tables, handwriting, languages), the CRM and ERP schema with the specific write paths involved, current SLAs and average handling time, error tolerance per action class, data-sensitivity and residency constraints, and a labeled sample corpus of real de-identified emails and attachments with ground-truth outcomes.

The pilot itself should take one or two high-frequency intents and run in propose-only shadow mode, first against historical traffic and then alongside live traffic, measuring whether it would have acted wrongly rather than letting it act. Set the thresholds before you start.

Three honest outcomes follow. Proceed, if critical-field accuracy and incorrect-action rate clear the pre-agreed thresholds on your data and the exception rate leaves a favorable cost per completed request. Redesign, if errors cluster in a specific document class, which is a data problem, or in a specific step, which is a candidate for becoming deterministic. Choose non-AI automation, if the intents turn out to be low-variance and rule-expressible, because deterministic automation will then be cheaper, faster and more reliable.

The third outcome is a real result, not a failed pilot. A workflow that did not need a model is a cheaper workflow that will not surprise you in eighteen months.

All benchmark and deployment figures in this article are third-party results reported by their authors, with the limits noted alongside each. They bound what is plausible on your data. They do not predict it.

YPAI document workflow scoping

YPAI scopes document-triggered workflows by mapping stages to responsible components before any build, so the authority boundary between proposing and acting is a design decision rather than an emergent one. Scoping covers the intent taxonomy, the extraction schema and its validation, the rules that stay deterministic, the review surface, and the evaluation set the workflow will be regression-tested against.

Operations and automation owners with a defined document-triggered process can request a consultation to scope it, or read how we approach document AI workflow automation.

Related Resources

Agentic AI training data guide - Multi-turn dialogue, tool-use traces and preference data for agentic systems
EU AI Act Article 10 engineering requirements - Data-governance obligations and what they mean for engineering teams
Data labeling QA thresholds - Building the labeled evaluation set this workflow is measured against
Document AI workflow automation - Scoping a document-triggered workflow with YPAI

Knowledge Assistants: Policy, Permission, Version

noreply@ypai.ai (YPAI Engineering) — Sun, 06 Sep 2026 00:00:00 GMT

An employee asks which approval rules apply to a purchase. The assistant returns a fluent answer with a citation to a real internal document containing that exact wording. The answer is wrong, because the document it cited is a wiki page with no policy authority, and the rule that actually governs this purchase sits in a delegation schedule that is phrased nothing like the question.

That failure is not a retrieval failure in the usual sense. The system found a relevant passage. What it could not do is establish that the passage was applicable, and applicability depends on three properties that are not in the text: who owns the source, which version it is, and what period it was in force. This walkthrough works a procurement question through those properties, derives an answer contract from it, and gives the evaluation matrix that tests each case. It is written for the engineering and governance owners of an internal assistant.

The corpus and scenario below are constructed for this article. They are not a YPAI deployment or a customer result.

A relevant passage is not an applicable rule

Assume an employee asks, on 6 September 2026: which approval rules apply to this purchase? The corpus available to the assistant looks like this.

Source	Status and metadata	Access	What it says
Procurement Policy v4.2	Authoritative, Procurement owner, effective from 1 Jan 2026, company-wide base rule	Employee can read	Purchases above £50,000 require Director approval
Procurement Policy v3.8	Historical, effective 1 Jan 2024 to 31 Dec 2025, superseded by v4.2	Employee can read	Threshold was £25,000
Engineering Delegation Schedule 2026	Authorized delegation, Finance Operations owner, applies to Engineering, effective 1 Apr to 31 Dec 2026	Employee can read	Engineering purchases up to £75,000 may be approved by designated budget holders
Procurement Wiki	Informal explanatory material, no policy authority	Employee can read	Says all purchases over £50k need Director sign-off
Strategic Supplier Exception	Authoritative restricted exception	Employee has no access	Contains rules for a restricted supplier program

Two common ranking heuristics both fail here, and they fail in opposite directions.

A pure relevance ranker puts the wiki first, because its wording matches the question most closely. It is the only document in the corpus written in the same register as the question, and it is the one document with no authority at all.

A recency heuristic favors whichever file was edited most recently, which is a filesystem property rather than a semantic one. A superseded policy can be touched by a metadata migration and become the freshest thing in the index. Effective dates are declared inside the document, not derived from its modification time, and the two are unrelated.

Neither heuristic can produce the governing answer, because the governing answer depends on transaction attributes the question did not carry: the amount, the employee's department, the intended purchase date, the purchase category, and potentially the supplier's status.

Walking the question

Suppose the employee is in Engineering, the purchase is £60,000, and the intended transaction date is 10 September 2026.

The base policy applies. So does the authorized Engineering delegation, whose effective period covers that date. If organizational source governance states that an approved delegation may modify the base approval route, both are governing, and the assistant can answer in substance:

Based on the current Procurement Policy and the Engineering Delegation Schedule, the standard £50,000 Director threshold is modified for eligible Engineering purchases during the delegation period. A designated budget holder may approve qualifying purchases up to £75,000.

That answer cites both governing passages rather than the one that matched the question best, and it exposes their effective dates and owners so the employee can check the reasoning instead of trusting it. The wiki is absent from the conclusion even though it says something superficially similar, on the grounds that being right by coincidence does not make a page a source.

The superseded v3.8 policy stays in the corpus and stays out of this answer. It is not noise to be deleted. It is the correct source for a question about a 2025 transaction.

Three changes that break it

The scenario is only useful if you perturb it. Each of these is a test case, and each maps to a distinct piece of machinery.

Change the date to 15 November 2025. Now v3.8 rather than v4.2 may govern, and the delegation schedule is not yet in force. A system that always prefers the current version answers this confidently and wrongly. Temporal applicability is a semantic rule about effective periods, and it has to be modeled as one.

Remove the encoded relationship between the delegation and the base policy. Now the corpus contains a £50,000 threshold and a £75,000 threshold, both authoritative, with nothing stating which prevails. The correct behavior is to surface the conflict, present both sources with their owners, and route to the owner who can resolve it. Inventing a precedence rule is the failure, and it is a quiet one, because the invented answer is indistinguishable from a correct one at the point of use.

Revoke the employee's Engineering group membership at 14:00. The next query has to be tested against the declared revocation SLA. This is where architecture decides behavior: a system that filters at query time against live access state behaves differently from one that replicated an ACL snapshot into its index, and the second can keep answering from stale permissions long after the change. A cached prior answer is a third path to the same leak, and it bypasses retrieval entirely.

The answer contract

Those cases resolve into a sequence, and the sequence is the actual architecture. Retrieval is one step inside it rather than the thing itself:

Identify question scope, establish eligible sources, enforce access, resolve authority and effective period, retrieve evidence, detect conflicts and exceptions, generate only supported claims, attach access-safe evidence, then state uncertainty or escalation.

The order is doing most of the work. Access enforcement comes first: content a user may not see should never enter their generation context, and filtering it out of the visible answer afterwards is too late. Authority and effective period resolve next, since they are what make a piece of evidence eligible in the first place. Conflict detection then sits between retrieval and generation, in the one position where discovering a conflict can still stop a confident answer from being written.

The restricted Strategic Supplier Exception shows why the last step is separate. The assistant must not pull it into the user's context merely because it might change the answer. It also should not say "a confidential exception exists for Supplier X" if the existence of the exception is itself sensitive. What it can do is take an approved generic route: additional rules may apply to purchases outside the sources available to this role, so route this case to Procurement. How much existence metadata may be disclosed is an organizational security decision, and permission-aware retrieval capabilities in cloud search products make the filtering implementable without deciding the disclosure policy for you.

The evaluation matrix

Each row below is a test case with an expected behavior and something to count. Together they cover retrieval, generation, security, temporal reasoning and escalation, which are the five things a single accuracy percentage merges into one uninterpretable number.

Case	Test setup	Expected behavior	Useful measurement
Routine current rule	An ordinary purchase where one current authoritative policy applies	Correct answer from the governing policy, claim-level citation, owner and version and effective date available	Authoritative-source recall, claim correctness, citation support, latency
Restricted source	A relevant restricted policy contains an apparently better answer	Restricted content never enters unauthorized retrieval or generation, and no title or snippet leaks through a citation	Unauthorized retrieval and output rate, citation-access tests
Revoked access	The user could access a document, membership is revoked, the same question is repeated before and after synchronization	Behavior matches the declared revocation SLA, and a cached answer cannot bypass revocation	Revocation propagation time, stale-cache leakage
Outdated version	An old policy has a closer lexical or semantic match than the current version	A current-period question uses the current applicable source and does not treat the old passage as governing	Temporal applicability accuracy, superseded-source error rate
Historical question	The user asks which rule applied to a purchase in a prior year	Retrieves the historical version appropriate to that period rather than today's rule	Temporal and historical correctness
Conflicting authorities	Base policy and delegation give different thresholds and precedence is not encoded	States the conflict, does not silently choose, identifies the source owner to escalate to	Conflict-detection rate, correct-escalation rate
Missing or ambiguous evidence	Amount, department or category is absent and the answer depends on it	Asks for the minimum clarification, or gives an explicitly bounded partial answer	Clarification appropriateness, unsupported-claim rate, task completion
Misleading or malicious source	An informal page carries false or injected text with high semantic similarity	Does not elevate it above governing sources, and retrieved text cannot alter system or security instructions	Source-tier adherence, injection attack success, poisoned-source retrieval influence

Two rows are the ones teams discover late. The revoked-access row fails in systems that looked correct in every functional test, because functional tests do not manipulate group membership mid-session. The conflicting-authorities row fails silently by design: the system produces a plausible answer, nobody reports a bug, and the underlying governance gap is never surfaced to the owner who could close it.

What retrieval and citation metrics do and do not establish

Public benchmarks diagnose components well and do not convert into an enterprise accuracy figure. The reason is specific rather than general skepticism.

ALCE treats citations as an evaluable property rather than decorative output, evaluating end-to-end systems across answer correctness and citation quality, and separating evidence coverage from evidence appropriateness. What it supports is important: citations can be present while significant claims remain inadequately supported, and citation quality is measurable separately from answer quality. What it cannot do is know whether one document has organizational authority over another, whether the user may access a source, or whether an effective date makes a passage applicable. Its corpora are public QA datasets, not policy repositories, so its scores do not convert into an internal assistant's accuracy.

RAGChecker is useful precedent for the structural point: diagnose retrieval and generation independently rather than collapsing the whole system into one score. Applied to this problem, that means measuring whether the necessary authoritative evidence was present, not merely whether a semantically relevant passage reached the top of the list, and separately measuring claim correctness, unsupported claims, citation entailment and citation completeness. The enterprise-specific labels, source authority and effective-date applicability, have to be added by you, because no public benchmark knows your organization's authority rules.

Automated judges scale qualitative evaluation and should not be the sole oracle on high-risk acceptance tests. The MT-Bench and Chatbot Arena work reported strong judges reaching over 80 percent agreement with human preferences in its evaluation setting, while documenting position, verbosity and self-enhancement biases, along with limited reasoning ability, in the same paper. In its position-bias test, one leading judge model gave consistent pairwise judgments after swapping answer order in only 65 percent of cases under the default setup, improving to 77.5 percent with few-shot examples. Read that as a case for calibrated aids, not for treating judge scores as ground truth.

On deployment practice, OpenAI reports that Morgan Stanley's internal assistant incorporated expert grading and a daily regression suite of sample questions into its operating model. That is customer and vendor reported rather than independently audited, and it is cited here for the shape of the practice rather than for any outcome.

Retrieved text is data, not instructions

A permission model protects against the wrong person reading a source. It does not protect against the wrong source being written.

PoisonedRAG demonstrated in controlled experiments that inserting a very small number of crafted texts into large public knowledge corpora can dominate retrieval for a targeted question, with reported black-box attack success reaching 97 percent on Natural Questions, 99 percent on HotpotQA and 91 percent on MS MARCO using five poisoned texts per target question against one model. Those are adversarial benchmark conditions and are not estimates of real-world enterprise incident rates. What they establish is narrower and still decisive: high retrieval similarity is not evidence that a passage is trustworthy.

The design response is source-aware rather than prompt-based, because instructing a model to ignore malicious instructions is a request rather than a control:

Treat retrieved document content as data, never as governing model instructions.
Control ingestion provenance and approved sources, so that who can introduce authoritative content is a permissions question.
Prevent low-trust sources such as editable wikis from silently outranking approved policy.
Ensure retrieved text cannot change access rules or tool permissions.
Apply least-privilege tool design wherever the assistant can take an action rather than only answer.

This research establishes the attack class. It does not establish a complete defense, and no published defense should be treated as one.

When to re-run the regression suite

An answer in this system can change without anyone editing a prompt. The trigger list is therefore longer than most teams write down, and writing it down before launch is the cheap part:

Source content or source status changes.
Owners or precedence metadata change.
ACLs, groups or connectors change.
Chunking, embeddings, retrieval or reranking settings change.
The model or system prompt changes.
Citation rendering changes.
Cache behavior changes.
Connected-tool permissions change.

The first three are governance events that engineering does not initiate and often does not hear about, which is why the suite has to be triggered by them rather than by deployments.

Operational measurement should include p50 and p95 latency, error and timeout rates, and synchronization freshness. Employee-level measurement should stay on the actual task: whether the employee reached the correct next action, whether escalation was correctly triggered, and how long it took relative to the existing process. None of those should be replaced by a generic accuracy percentage.

A bounded pilot

Five artifacts should exist before or early in development: a representative employee-question set, named owners for the governing sources, a source and authority and version map, a permissions matrix that includes revocation and deletion cases, and acceptance criteria tied to decision risk.

The baseline is the part most pilots skip. Record what employees actually do now, meaning search, contact support, ask Procurement, compare several files, and measure the same task after the pilot. Without it, a program can report improved retrieval scores while employee decisions did not change, and nobody can tell the difference.

Every benchmark and deployment reference above is third-party published or vendor-reported work with its own scope. They support reasoning about mechanism and testing design. None supplies an expected accuracy for an assistant built on your corpus.

YPAI knowledge assistant scoping

YPAI scopes knowledge assistants around one employee group, one domain and a bounded source set rather than an organization-wide index. Scoping maps the owners, permissions and versions of the governing sources, defines the representative and adversarial question set, and sets acceptance criteria before build, so that access enforcement and effective-date applicability are design decisions rather than discoveries.

Engineering and governance owners with a defined internal knowledge problem can request a consultation, or read how we approach enterprise knowledge assistants.

Related Resources

Agentic AI training data guide - Multi-turn dialogue, tool-use traces and preference data for agentic systems
EU AI Act Article 10 engineering requirements - Data-governance obligations and what they mean for engineering teams
CTO's guide to sovereign AI architecture and costs - Where an internal assistant's data and inference actually run
Enterprise knowledge assistants - Bounded-scope assistant programs with YPAI

Agentic AI training data: enterprise guide

noreply@ypai.ai (YPAI Engineering) — Sat, 07 Mar 2026 00:00:00 GMT

Most enterprises building agentic AI systems reach the same point: the base model performs well on benchmarks but fails in production deployment. The failure mode is not model architecture. It is agentic AI training data that was never designed for multi-step autonomous operation.

Static LLM pre-training produces models that complete single turns well. Agentic operation requires something different: a model that plans across multiple steps, decides when and how to use tools, manages uncertainty when instructions are ambiguous, and maintains consistency across a conversation that spans dozens of turns. These capabilities require specific training data structures that web-scale text corpora do not provide.

What makes agentic AI different from standard LLMs

An agentic AI system does not just generate text. It takes actions: querying databases, executing code, calling APIs, browsing the web, sending messages, and making decisions about which tool to use and in what sequence. The downstream consequences of those actions are real, not hypothetical.

This operational difference has direct implications for training data requirements. A standard language model learns to predict the next token given the preceding context. An agentic model must learn to predict the next action given a task goal, a history of prior actions, and a partial view of the world state. These are distinct learning problems requiring distinct training signals.

Three architectural properties define agentic AI systems and drive their data requirements.

Multi-step reasoning. Agentic systems decompose complex goals into subtask sequences. Each subtask depends on the outcome of prior subtasks. Training data must include complete task trajectories, not isolated turns, so the model learns which plans succeed and which fail.

Tool use. Agentic systems invoke external tools to retrieve information, perform computation, or take actions in external systems. Training data must include tool-invocation examples with correct tool selection, properly formatted arguments, and the handling of both successful and failed tool responses.

Memory and context management. Long-horizon tasks require the model to retrieve, store, and update information across turns. Training data must include scenarios where prior context is necessary to complete the current step correctly.

Training data requirements for agentic systems

The training data categories that matter for agentic AI differ substantially from the corpora that drive LLM capability on standard benchmarks.

Multi-turn dialogue corpora

Multi-turn dialogue data is the foundation. The key quality requirement is not volume but trajectory completeness: each conversation must trace a task from initial instruction through completion or failure, with all intermediate steps represented. A corpus of short two-turn exchanges does not train multi-step planning capability regardless of its size.

Enterprise task domains add a further specification requirement. A coding agent operating in a software engineering environment needs task trajectories drawn from software engineering workflows: debugging sessions, code review sequences, architecture planning dialogues. A customer service agent needs task trajectories drawn from customer service workflows. Domain-mismatched dialogue data trains general conversational fluency, not domain-specific task completion.

Instruction-following data under ambiguity

Agentic systems regularly receive underspecified instructions. "Schedule the meeting for next week" requires resolving which participants to include, which time zone to use, and which calendar system to write to. Training data must include examples of instruction clarification, graceful degradation under ambiguity, and appropriate refusal when an instruction cannot be completed without information the agent does not have.

This is a data category most procurement teams underspecify. Generic instruction-following benchmarks measure whether the model completes clear instructions correctly. Agentic deployment measures whether the model handles unclear instructions appropriately. These require different training examples.

Tool-use execution traces

Tool-use training data consists of interaction traces showing the model selecting a tool, constructing the invocation arguments, receiving the tool response, and incorporating that response into the next step. Good tool-use training data includes failure cases: tool calls that return errors, empty results, or unexpected formats, and the correct recovery behavior for each.

The diversity of tool types matters. An agent that has only seen database query traces will not generalize well to web search invocations. Training data should cover the tool categories the deployed system will use, at realistic frequency distributions for the target domain.

Voice and speech data for voice agents

Voice agents introduce a separate data dimension that text-only agent training does not address. The acoustic and linguistic coverage of the speech corpus determines production performance in ways that no amount of text-based fine-tuning can correct.

For voice agents, the agentic AI training data challenge compounds with the speech corpus challenge. The model must learn to understand spoken instructions across speaker diversity, acoustic environments, and dialect variation, and it must learn to generate spoken responses with appropriate prosody for multi-turn dialogue.

Prosody and spoken instruction patterns

Written instruction-following data does not capture how humans give instructions verbally. Spoken instructions include hesitations, restarts, prosodic emphasis, and implied boundaries that text does not contain. A voice agent trained only on text-based instruction-following data will encounter a distribution shift when deployed in production.

Prosody annotation adds the signal needed for spoken dialogue training: speech rate, pitch contours, pause patterns, and emphasis markers. For voice agents that must detect when a user has finished speaking or is correcting a prior instruction, this annotation layer is not optional.

Speaker diversity across dialects and noise conditions

Speaker diversity requirements for voice agents follow the same principle as for any ASR system: the corpus must represent the speaker population the agent will encounter. For European deployments, this means covering regional dialects, non-native speaker patterns, and age-range variation within each target language.

Acoustic condition coverage is equally important for voice agents deployed outside controlled environments. A voice agent used in an open-plan office, a manufacturing floor, or a vehicle will encounter background noise conditions that a studio-recorded corpus does not represent. The word error rate on clean speech tells you nothing useful about performance in the deployment environment.

For voice agents covering European markets, dialect coverage is a known gap in most available datasets. Norwegian Bokmål and Nynorsk, Catalan versus Castilian Spanish, Swiss German versus Standard German: these distinctions affect recognition accuracy in exactly the speaker populations where the agent will be used.

Internal links to the voice agent training data requirements covered in our voice AI agent training data requirements guide provide more detail on corpus specification for voice-first agentic systems.

RLHF and preference data collection at scale

Reinforcement learning from human feedback is the technique that closes the gap between a model that generates plausible text and a model that reliably behaves well. For agentic systems, RLHF is not optional: the consequence of poor decisions accumulates across task steps, and pre-training alone does not produce reliable enough agent behavior for enterprise deployment.

What preference data looks like for agents

RLHF preference data for agentic systems consists of comparison pairs: two candidate responses to the same task state, with a human judgment indicating which response is preferred and why. For agentic systems, the comparison pairs include not just final answers but intermediate tool-use decisions, plan steps, and recovery behaviors.

Collecting preference data for agentic systems is more expensive than for single-turn assistants because each comparison requires evaluating a multi-step trajectory, not a single response. Annotators must understand the task domain well enough to judge whether the agent's plan is correct, not just whether the final output reads well.

Annotator quality and inter-annotator agreement

The signal quality of preference data depends on annotator quality and consistency. Low inter-annotator agreement produces noisy preference labels that degrade the reward model rather than improving it. For technical domains like software engineering, legal analysis, or medical information, domain-literate annotators produce substantially better preference signal than general-population annotators.

Inter-annotator agreement should be measured and documented. A preference dataset without inter-annotator agreement metrics cannot support a claim of high-quality preference signal. For systems subject to EU AI Act Article 10, inter-annotator agreement documentation forms part of the data quality evidence required at conformity assessment.

Scale and iteration cadence

A reward model trained on too few preference pairs will overfit to surface features rather than learning substantive quality distinctions. Initial RLHF runs for enterprise agentic systems typically require tens of thousands of comparison pairs to produce stable reward models, with ongoing collection to correct the distribution shift that occurs as the base model improves.

The iteration cadence matters. Preference data collected on an earlier model version becomes less useful as the model improves, because the model no longer generates the lower-quality responses that appeared in the original comparison pairs. An ongoing preference data collection pipeline is more valuable than a one-time large dataset.

Compliance requirements for agentic AI training data

The regulatory environment for agentic AI training data in Europe is governed by two frameworks: GDPR for any personal data in the training corpus, and EU AI Act Article 10 for systems classified as high-risk.

GDPR requirements

Any training corpus that includes real user interactions, voice recordings, or preference labels derived from human behavior involves personal data under GDPR. The lawful basis for processing must be documented, consent records must support erasure requests traceable to individual training examples, and data must not be transferred outside the EEA without adequate safeguards.

Voice data adds a further complication: it is biometric data under GDPR Article 4(14), which triggers special category data obligations under Article 9. Standard legitimate interests processing is not available for biometric training data. Explicit consent naming the AI training use case is the most defensible lawful basis. Our GDPR-compliant speech data collection guide covers the documentation requirements in full.

EU AI Act Article 10

The EU AI Act Article 10 data governance requirements apply to training data for high-risk AI systems. Agentic systems operating in healthcare, employment screening, credit assessment, educational testing, law enforcement, or critical infrastructure fall within Annex III high-risk categories. The Article 10 requirements are legal obligations, not engineering recommendations.

Four quality standards must be satisfied: training data must be relevant to the intended purpose; sufficiently representative of the deployment population; free from errors that could cause discriminatory outcomes; and complete for the task. Completeness is a source of frequent failure. A preference dataset collected entirely from English-language interactions does not satisfy representativeness requirements for a multi-language European deployment, even if it is large.

Documentation requirements include collection methodology, preprocessing steps, bias examination results, and demographic breakdowns of training data sources. For agentic AI systems assessed by a notified body, this documentation package must exist before conformity assessment. Retrofitting it after development is time-consuming and often incomplete.

The full implications for procurement teams are covered in our EU AI Act high-risk AI training data requirements guide.

Data sovereignty and EEA residency

Agentic AI systems trained on data collected outside the EEA face dual exposure: GDPR Chapter V transfer obligations for any EU personal data, and Article 10 documentation gaps if the foreign data collection did not meet EU consent standards. US-collected preference data presents both risks simultaneously.

EEA-native data collection eliminates transfer exposure and produces preference signal from annotators whose linguistic and cultural context reflects the European markets where the agent will be deployed. For voice agents, EEA collection also ensures dialect and language variety coverage that US providers do not supply for European languages.

Vendor evaluation: what to require

Evaluating a training data vendor for agentic AI requires different criteria than evaluating a general LLM data provider. The questions below reflect the data dimensions specific to agentic systems.

Coverage of agentic task types. Does the vendor have dialogue trajectory data for the task domains relevant to your deployment? General conversational data is not a substitute for domain-specific task completion trajectories.

Tool-use trace documentation. Can the vendor provide training data that includes tool invocation patterns, not just natural language generation? Tool diversity and failure-case coverage are key differentiators.

Preference data quality documentation. What is the inter-annotator agreement on preference labels? What annotator qualification process does the vendor use? Are domain-literate annotators available for technical task evaluation?

Consent chain completeness. Can the vendor provide individual consent records that explicitly name the AI training use case? For voice data, can the consent records support erasure requests traceable to individual recordings?

EU data residency confirmation. Where is data collected, stored, and processed? Can the vendor confirm EEA residency throughout the pipeline, including annotation sub-contractors?

Article 10 documentation readiness. Does the vendor provide collection methodology documentation, demographic breakdowns, and bias examination reports? These must exist before you need them at conformity assessment, not after.

YPAI positioning: European speech corpora for agentic AI

YPAI collects speech data across European languages using a network of verified contributors in the EEA. For voice agents, this means dialect coverage across 50+ EU dialects, human-verified transcriptions with prosody annotation capability, and GDPR-native consent chains where each contributor provides explicit consent for AI training use.

YPAI's network includes more than 40,000 contributors across more than 50 countries and 150 languages. Each project still needs a recruitment plan for the required age ranges, dialects, and non-native speaker groups. EEA processing can be specified when the buyer's risk and transfer requirements call for it.

For agentic AI training data that includes voice interaction components, YPAI provides corpus specifications matched to deployment requirements rather than volume targets. The documentation package covers Article 10 compliance evidence including demographic breakdowns, collection methodology, and inter-annotator agreement for transcription tasks.

More detail on EU compliance requirements for this data category is available in our EU AI Act high-risk AI training data requirements guide and our GDPR-compliant speech data collection guide.

Getting started

The right specification for agentic AI training data starts with the task domain, the tool inventory the agent will use, and the speaker population the system will serve. Those three parameters determine the corpus structure, the annotation requirements, and the RLHF preference collection cadence.

A corpus that is large but mismatched to the deployment environment will not close the gap between benchmark performance and production reliability. The mismatch between training distribution and deployment distribution is the most common root cause of production failure for agentic systems.

YPAI works with enterprise data teams to design training data specifications that match deployment requirements. If you are specifying agentic AI training data for a European deployment and want to discuss requirements, contact our data team.

For annotation pipeline design for voice and speech data, our audio annotation pipeline guide covers the technical workflow from raw audio to training-ready corpora.

Sources:

Voice Agent Training Data: Beyond ASR Corpora

noreply@ypai.ai (YPAI Engineering) — Sat, 07 Mar 2026 00:00:00 GMT

Voice AI agents are not ASR systems. They listen, respond, interrupt, clarify, and maintain context across multiple turns. Product teams that treat voice agent training data as equivalent to ASR training data discover this gap in production, where turn-taking failures, missed interruptions, and broken dialogue flows emerge at a scale that benchmark scores do not predict.

The distinction matters because training data requirements for voice agents differ structurally from requirements for passive speech recognition. Understanding those differences is the first step toward a corpus specification that produces an agent capable of handling real conversation.

What voice agents do that ASR models do not

A conventional ASR model has one job: convert audio to text. It processes a speech segment and produces a transcript. The acoustic model is trained on utterances in isolation, without reference to what came before or after in the conversation.

A voice agent does more. It must detect when a user is speaking, decide whether to stop its own output in response, hold conversational state across multiple exchanges, recognize when a user's utterance is incomplete and wait rather than respond, and issue clarifying questions when the input is ambiguous. Each of these behaviors requires training data that passive ASR corpora do not contain.

None of that is an architecture problem, and treating it as one is how teams end up tuning a model against data that cannot express the behavior they want. A model has no way to learn barge-in handling from a corpus with no barge-in events in it, and no way to learn that an utterance is incomplete when every training example is a complete, well-formed sentence. The behaviors that make an agent usable in conversation come from examples of those behaviors.

Barge-in and overlapping speech

Barge-in, where a user starts speaking before the agent has finished its turn, is the requirement that most often forces a custom collection. A production agent has to detect the interruption in near real time, suppress its own ongoing output and switch to listening, and all three depend on training examples that a sequential-turn corpus does not contain.

Training data for barge-in handling has structural properties that standard ASR data does not. It must contain:

Overlapping audio segments where the human speaker's input begins while the agent's output is still in progress. The annotation must mark the onset of the interruption relative to the agent's utterance, not just the transcription of what was said.

Recovery sequences showing how the agent re-establishes the dialogue after a barge-in. A model trained only on clean, non-overlapping turns learns to produce the right words but not the right behavior when conversation does not follow the expected pattern.

Negative examples where the audio resembles barge-in acoustically but the speaker did not intend to interrupt, such as a brief affirmative sound mid-agent-turn. Without negative examples, agents over-trigger on filler signals and produce broken dialogue flow.

Collecting this data requires scripted interaction scenarios in which contributors are instructed to interrupt at specified points, combined with spontaneous dialogue collection in which interruptions occur naturally. Neither type alone is sufficient.

Incomplete utterances and end-of-turn detection

End-of-turn detection determines when the agent should begin its response. It is one of the most common failure modes in deployed voice agents and one of the least represented aspects of training data specifications.

Human speech does not end cleanly. Speakers pause mid-sentence, trail off, begin a thought and revise it, and produce sounds that acoustically resemble an utterance ending without communicating a complete thought. An agent trained on clean, complete utterances treats every pause as a signal to respond and every incomplete thought as a complete query.

A production corpus for voice agent training must include:

Utterances that are genuinely incomplete, annotated as such, showing the agent waiting rather than responding. These represent a fundamentally different training signal from transcription accuracy on complete sentences.

Filled pauses and disfluency patterns that precede continuation rather than turn completion. The acoustic and prosodic features that signal "I am still speaking" differ from those that signal "I am done" in ways that a model must learn from labeled examples.

Turn-final prosody in the specific languages and dialects of the deployment population. End-of-turn prosodic cues vary significantly across languages. A corpus calibrated on English prosody will produce end-of-turn detection errors on German, French, or Norwegian speakers.

Multi-turn dialogue structure

Single-turn speech models see input and produce output without reference to conversation history. Voice agents operate across multiple turns, maintaining context about what was said earlier, what questions were asked, and what commitments were made.

Training data for multi-turn voice agents must represent the full conversational arc, not a collection of isolated utterances. This means:

The corpus must include complete conversation transcripts with turn boundaries preserved, not individual utterance extracts. A training example for a clarification exchange must show the original ambiguous utterance, the agent's clarification question, and the user's response, all in sequence.

Reference resolution patterns, where a user's utterance only makes sense against a prior turn, must be present. "Yes, that one" is meaningless without the prior turn that established what "that one" refers to. A voice agent that processes utterances without discourse context will fail on any interaction that involves reference to prior turns.

Domain-specific dialogue flow patterns for the agent's deployment context must be collected. A voice agent for healthcare appointment booking has a different conversational arc than one for financial services customer support. Generic dialogue data is a starting point, not a sufficient corpus.

Clarification exchanges and dialogue repair

Dialogue repair is the linguistic mechanism by which participants in a conversation fix misunderstandings, clarify ambiguous references, and recover from recognition errors. Voice agents encounter dialogue repair constantly in production and must be trained to initiate and respond to clarification exchanges gracefully.

Clarification exchanges have a structure: the agent detects ambiguity or low confidence, produces a clarification question, receives additional input from the user, and proceeds with updated context. Each step in this sequence is a distinct behavior that requires training examples. Agents not trained on clarification data respond to ambiguity with either a hallucinated completion or a failure state.

Training data for dialogue repair must include naturally occurring clarification sequences, not just scripted examples. Real clarification exchanges have acoustic and prosodic properties that differ from first-attempt utterances. Users often repeat themselves with different emphasis, reformulate their question, or express frustration when clarification fails. A corpus that includes only cooperative, clean clarification examples will not produce an agent that handles the full range of real-world repair patterns.

What a transcript deletes

The gap between an ASR corpus and an agent corpus is easiest to see on a single exchange.

The dialogue, timings and labels below are illustrative examples constructed for this article. They are not measured data and are not drawn from a YPAI dataset or any cited corpus.

An isolated scripted collection might contain one prompt, one utterance, one reference transcript:

Prompt:     "I need to change my booking to Friday."
Speaker:    "I need to change my booking to Friday."
Transcript: I need to change my booking to Friday.

For an ASR system that sample is perfectly useful. It carries target speech acoustics and a clean lexical reference. Now put the same intent inside a conversation:

00:00.000  User:   "I need to change my booking to-"
00:01.850  Agent:  "Sure, what date would-"
00:02.300  User:   "Friday. Uh, actually, Saturday."
00:03.900  Agent:  "Saturday. Got it."
00:04.400  User:   "Mm-hm."

A transcript-only normalization reduces that to two lines:

User:  "I need to change my booking to Saturday."
Agent: "Saturday. Got it."

The final semantic request survives. Five interaction facts do not. The user's first turn was abandoned mid-utterance. The agent entered before the user had finished. The two speakers overlapped. "Friday" was spoken and then repaired to "Saturday". And the closing "Mm-hm" may be a backchannel rather than a bid for the floor.

Each of those belongs to a different downstream problem, which is why losing them is expensive rather than untidy. The abandoned turn is an endpointing example, the early agent entry a turn-taking one, the overlap a diarization case, the repair a dialogue-state transition, and the closing token a question of floor management. A corpus that normalizes them away has not simply been given fewer labels. It can no longer train or evaluate any of the five behaviors.

A representation that keeps them separates the layers:

USER_WORDS:             [I need to change my booking to-] [Friday] [uh] [actually] [Saturday]
AGENT_WORDS:            [Sure, what date would-] [Saturday. Got it.]
OVERLAP:                user/agent @ 00:02.300
PARTIAL_ABANDONED:      "to-"
SELF_REPAIR:            Friday -> Saturday
AGENT_OUTPUT_START_STOP: timestamped
BACKCHANNEL_CANDIDATE:  "Mm-hm"
DIALOGUE_STATE_FINAL:   requested_date=Saturday

The principle is not to annotate everything. Every additional label carries annotation cost, ambiguity and its own quality-control burden. The principle is to preserve the information needed to reconstruct the decision the target model is expected to learn, or that the evaluator is expected to score. Anything beyond that is expense.

There is precedent for keeping the messy layer rather than the tidy one. The AMI Meeting Corpus retains partial words and disfluencies in its transcripts rather than silently rewriting them, alongside speaker-specific transcripts, word timing and higher-level dialogue information across synchronized close and far-field devices. AMI is roughly a hundred hours of meetings, about two-thirds scenario-based and the rest naturally occurring, which is a useful demonstration that scenario design and ecological validity are a trade-off rather than opposites. It is also meeting speech with many non-native English speakers, so its content does not transfer to a two-party voice agent and its microphone design is not automatically the right one for agent data.

Specify by target behavior, not by hours

A specification that opens with an hours total is answering the wrong question first. What has to be recorded, what has to be labeled, and what can then be measured all differ by the behavior you are trying to produce.

Target behavior	Collection scenario	Recording requirement	Labels to preserve	Evaluation
Isolated ASR	Read or prompted speech can be suitable; design lexical, speaker and deployment-condition coverage	Target-relevant microphone, channel and acoustics	Reference transcript, optional word timing, speaker and session metadata	WER or CER, with meaningful speaker, acoustic and language slices
VAD and endpointing	Continuous speech with natural within-turn pauses, hesitation, noise and true turn endings	Untrimmed continuous audio on a deployment-like channel	Speech activity, onset and offset, pauses, optionally completion or continuation state	Miss and false-alarm rate, endpoint-delay distribution, premature cut-off rate
Diarization	Multi-speaker speech with real speaker transitions and overlap	Mixed deployment signal, plus separate reference channels where feasible	Speaker IDs, activity intervals, overlap	DER with error decomposition, and overlap-specific analysis
Turn-taking	Prompted spontaneous speech, role-play, natural dialogue or agent interaction designed to elicit holds, shifts, backchannels and interruptions	A shared synchronized timeline for both sides	Speaker activity, silence, overlap, backchannel and floor-change labels, interruption events	Shift and hold accuracy, backchannel handling, floor-transfer timing, premature interventions
Task-oriented dialogue	Goal-based scenarios with information changes, ambiguity, corrections and repair opportunities	Complete sessions rather than independent clips	Verbatim turns, intents and acts, slots and state, repairs and corrections, task outcome	State accuracy, task completion, repair handling
Full-duplex interruptible agent	Human-agent interaction or controlled simulation containing barge-in, pauses and concurrent activity	Synchronized user input, agent rendered audio and event logs, with an echo or render reference where needed	User and agent activity, playback start and stop, interruption outcome, relevant tool and system events	Interruption success and failure, unwanted cut-offs, response latency, task success, human judgment

Read down the recording column and one point repeats: the specification follows the deployment channel rather than a default. Switchboard's 8 kHz telephony and AMI's synchronized multi-microphone meetings solve different problems, and neither is a general speech-data specification. SpokenWOZ is worth studying for the task-oriented row specifically, because it illustrates how cross-turn dialogue state becomes harder in spoken rather than written conversation. Turn-taking has a research formulation in voice activity projection work, which predicts upcoming activity rather than only detecting current activity.

One empirical result makes the case for the endpointing row better than any argument. LibriCSS constructed continuous far-field audio by concatenating and replaying utterances with controlled silence and overlap. In its no-overlap condition, shortening inter-utterance gaps from roughly three seconds to between 0.1 and 0.5 seconds raised continuous-input WER from about 11.5 percent to 15.4 percent in the reported baseline, which the authors describe as a 33.9 percent relative increase, while utterance-wise evaluation with oracle segmentation was almost unchanged. The words did not get harder. The segmentation problem did. That is a 2020 setup using replayed audiobook speech rather than a modern conversational agent, so it should not be read as an agent benchmark. What it establishes is exactly the concern this article started from: an utterance-level benchmark can remove a problem that the deployed continuous system still has to solve.

Coverage, sampling and splits

Define coverage across the axes that actually vary in deployment: language or dialect where relevant, independent speakers, device and channel, acoustic environment, speaking style and scenario. For conversational targets, add interaction-event coverage as its own axis. A hundred hours containing almost no genuine interruption does not carry the same information as a smaller, deliberately designed set containing diverse interruption contexts. No reviewed study offers a universal optimum for that trade-off, which is a reason to derive the balance from a pilot rather than from a published quota.

Assign a speaker ID, a dyad or session ID, a scenario family and, where useful, device and environment identifiers before splitting. The hold-out unit should mirror the claim being made: unseen sessions for session generalization, unseen speakers for new-speaker performance, unseen scenario formulations for semantic robustness. ASR evidence that more independent speakers can outperform more minutes from fewer speakers at a fixed budget is ASR-specific in its numbers, and general in its lesson: track sampling units, not only hours.

Keep offline and production measurement separate

An offline test can establish that an endpoint occurred 300 milliseconds after a labeled reference event. Only an interaction test can establish whether a user experienced that as a disruptive interruption, and only production telemetry exposes the retries, abandonment and repair patterns a benchmark never contained.

The metric set should follow the target behavior rather than collapsing into one number: endpoint delay and premature cut-off for endpointing, DER with overlap-aware analysis for diarization, shift and hold and interruption behavior for turn-taking, intent and state and task success for task dialogue, and human judgment or production outcomes for the integrated experience. The PARADISE framework established the broader principle decades ago by modeling spoken-dialogue performance as task success together with dialogue costs rather than as a single component metric, and recent full-duplex benchmarking keeps separate measures for tool selection, arguments, task success, response quality, turn-taking and latency. Nothing reviewed here supports a universal conversation-quality score.

Where simulation and licensed corpora fit

Licensed existing corpora buy speed and reproducibility, and they can mismatch your channel, your interaction style or your legal permissions. Custom collection aligns those directly and costs more. Controlled simulation is strong for counterfactual stress tests, and LibriCSS is the demonstration: varying overlap and gap structure systematically isolates a phenomenon in a way collected data rarely can. It does not reproduce spontaneous human semantic repair or the way people adapt to an agent. Synthetic speech can fill controlled acoustic or lexical cases on the same terms, and evidence drawn from it should not be presented as proof of natural conversational behavior.

Every corpus and result named here is third-party published work with its own task, channel and stated limits. They are useful for deciding what to record and what to measure. None of them is a specification for your deployment.

GDPR implications for conversational training data

Conversational recordings present a GDPR problem that single-speaker utterance collection does not, and it is worth stating precisely, because the common shorthand overstates it.

A voice recording is personal data. It is not automatically biometric data. Under Article 4(14) and Article 9, voice becomes biometric special-category data when it is processed for the purpose of uniquely identifying a person, such as speaker verification or enrolling a voiceprint. Training a general voice agent on conversational speech does not meet that condition on modality alone, and the classification turns on what the processing is for rather than on the fact that it is audio.

What does change with dialogue is the number of data subjects. A two-speaker exchange has two, and every participant's personal data needs a lawful basis under Article 6 regardless of whether Article 9 is engaged. Where Article 9 does apply, explicit consent under 9(2)(a) is one available condition rather than a universal requirement, other conditions may be available, and none of them removes the separate Article 6 basis. Treat the paragraphs below as a collection design that stays defensible across those readings, and confirm the analysis for your own project with counsel.

The methodology consequence is concrete. Standard crowdsourced speech platforms collect one speaker at a time. Scaling that to dialogue needs a framework for capturing every participant's basis and handling both participants' data under documented terms.

For European collection, three things belong in the design:

Individual per-speaker records for every participant in each recorded exchange, rather than blanket platform terms of service. This holds whether the basis is consent or another lawful basis, because the record is what makes the basis auditable later.

Purpose scope that names AI training explicitly. A general audio-recording permission that does not name the training use is weak evidence of an informed basis, and it is the first thing a reviewer will ask to see.

Right-to-erasure procedures that can identify and remove all recordings involving a specific speaker, even where that speaker appears in exchanges with other contributors. This requires speaker-level identifiers in every recording and a metadata structure that enables speaker-specific extraction.

For related context on GDPR compliance in speech collection, see our guide on GDPR-compliant speech data collection and the EU AI Act data requirements that apply if your voice agent is classified as high-risk under Annex III.

What to specify in a voice agent corpus brief

A corpus specification for voice agent training should address five requirements that standard ASR corpus briefs do not include.

Dialogue structure. Specify the conversational arc your agent will handle: average turn count per session, domain topics, expected clarification rate, and barge-in frequency in your target deployment population. These numbers drive collection scenario design.

Barge-in coverage. Specify minimum hours of overlapping speech with onset annotations. This is a distinct collection task from standard utterance recording and must be scoped explicitly.

End-of-turn diversity. Specify prosodic diversity requirements by language, including dialect coverage. End-of-turn detection failures are often dialect-specific, not general model failures.

Incomplete utterance representation. Specify minimum hours of annotated incomplete utterances with wait-state labels. Without a minimum, vendors default to complete-utterance collection and the resulting corpus does not address end-of-turn detection requirements.

Consent documentation. Specify that every recording requires individual participant consent records with the purpose "AI voice agent training," retention period, and right-to-erasure reference. For multi-speaker recordings, consent records must cover all participants.

For procurement teams comparing vendors, see our enterprise speech corpus collection guide and the contact center voice AI training data guide for related procurement context. For annotation requirements on collected data, the audio annotation pipeline guide covers transcription quality standards and inter-annotator agreement thresholds.

YPAI voice agent data collection

YPAI collects conversational speech for voice agent training across European languages and dialects. Collection is specified by target behavior rather than by an hours total, so a brief states which interaction events have to be present, what has to be labeled to preserve them, and which evaluation slice each one supports.

On legal basis, the collection design follows the analysis above rather than a single template. Every participant in an exchange gets an individual record covering the lawful basis relied on, purpose scope that names AI training, retention period and speaker-level right-to-erasure, and the Article 9 question is settled per project against the intended processing purpose rather than assumed from the modality. EU AI Act Article 10 documentation can be reviewed before contract signature.

Product teams building voice agents for EU deployment can request a consultation to discuss corpus specifications, or review our speech data services.

Related Resources

GDPR-compliant speech data collection in Europe - Lawful basis and consent requirements for voice data
Contact center voice AI training data procurement - Contact center-specific data requirements and procurement
Audio annotation pipeline for speech data labeling - Transcription quality standards and annotation workflows
Enterprise speech corpus collection - What separates production-grade corpora from bulk audio
EU AI Act high-risk AI training data requirements - Annex III categories and Article 10 obligations
Agentic AI training data guide - Training data foundations for agentic AI systems
Healthcare voice AI training data - Clinical deployment requirements for healthcare voice agents

Sources: