--- title: "AI Evaluation: Human and Automated Model Testing | YPAI" url: https://ypai.ai/data-solutions/evaluation/ description: "Human experts and an automated engine test models, agents, RAG, speech and multimodal systems. Calibrated judges, red teaming and release evidence." source: "src/copy/routes (route /data-solutions/evaluation/)" --- # Know where your AI works. Find where it fails. > Human experts and an automated engine test models, agents, RAG, speech and multimodal systems. Calibrated judges, red teaming and release evidence. YPAI tests AI models, agents, retrieval systems, speech and multimodal products against the tasks, languages and workflows that matter to your product. Specialist human judgment and an automated evaluation engine run as one operation. Contributors create test data and use your system. Evaluators, domain experts, red teamers and adjudicators judge it. The engine runs versioned test suites on Amazon Bedrock, Amazon Bedrock AgentCore and Amazon SageMaker AI. Everything meets in YPAI Platform. - [Client login](https://portal.ypai.ai/login) ## Start with the decision you need to make. ### Choose between model versions. - Run the same versioned tasks across models, prompts and configurations. See which version performs better, where neither meets your requirements, and which differences deserve closer review. ### Check performance in a specific language or market. - Examine meaning, terminology, tone and speech quality with reviewers selected for the language and task. Find the failures a single overall score can hide. ### Review a customer-facing workflow. - Evaluate answers and conversations against your source material and operating rules. Separate a helpful-sounding response from one that meets the brief and completes the task. ### Release an agent. - Test tool selection, arguments, recovery and task completion in a controlled environment. The end state the agent claims to have reached is read from the system. ### Red-team before launch. - Human red teamers and automated adversarial runs attempt misuse, boundary cases and the failures particular to your workflow. The report shows what was tried, what the system did and how severe the outcome was. ### Keep testing after launch. - Maintained regression suites run against each new model, prompt, retrieval setting and tool configuration. Permitted production traces feed targeted review and new test cases. ## Bring the outputs, or connect the system. ### Models and responses - Factual accuracy, reasoning, source use, instruction following, terminology and task-specific requirements. One response against a rubric, or two candidates compared with their model names withheld. Repeated trials across models, prompts and configurations show the variability. ### Retrieval and grounded answers - Retrieval and response are measured separately: relevance and coverage of the retrieved material, use of sources, citation accuracy, unsupported claims, and behaviour when the material does not answer the question. A failure is traced to the collection, retrieval, ranking, context or generation. ### Agents, tools and code - Multi-step tasks in controlled environments: tool selection, arguments, execution results, recovery and task completion. Browser workflows and code tasks run in isolated environments. The end state is read from the system, and code is judged by specialist review and executable tests. ### Speech, conversation and voice agents - Speech recognition, generated speech and conversational agents. Human assessment of meaning, intelligibility, pronunciation and interaction quality, with measurements of recognition errors, latency, interruptions and turn-taking, including full-duplex voice. ### Multimodal and specialist tasks - Tasks that combine text, audio, images, video or documents, and domain work in software, science, engineering, healthcare, law and finance, judged by specialists against explicit criteria and observable anchors. Start with existing responses, recordings or traces, or give the project access to the running system. Access, environments and processing arrangements are defined for the engagement. ## A result you can inspect, example by example. ### A documented evaluation protocol - Task, scenario coverage, rubric, reference material, human-review design, automated checks and result definitions, agreed before evaluation begins. ### Judgments at the level of each example - Human and automated judgments with ratings, decisions, supporting evidence and failure category. Adjudicated reference and preference sets carry their review history. ### A report that explains the findings - Results by model version, language and task group, with reviewer agreement, judge calibration, uncertainty, critical failures and prioritised findings. ### Agent traces and outcome evidence - The agreed traces, tool results and observable end-state evidence, with latency and resource use where the task calls for it. ### Test assets for the next version - Versioned test sets, rubrics and regression tests, and CSV or JSONL results. Each delivery is linked by identifier to its dataset, model configuration, rubric and run. ## A useful evaluation shows the failure. Illustrative example. Not a customer result. A customer asks to change a delivery address after the order has been dispatched. Delivery addresses can be changed before dispatch. After dispatch, refer address-change requests to support. ### Response A - I've updated the delivery address. Your order will arrive at the new address. ### Response B - Your order has already been dispatched, so an address change needs support review. Please contact support to check the available options. Response A conflicts with the supplied instruction and claims an action without supporting evidence. Response B follows the instruction and gives the appropriate next step. This is a content evaluation. It does not verify an action in a delivery system. A short brief is enough to start. We read it and come back with the scope, the team and the deliverables. Get this result for your system. Run your system through this protocol. This is the report your release gets. ## Six roles, one project team. 50,000 contributors registered with YPAI, and 210,000+ across the wider network. We select and qualify people for the language, subject area, interaction or judgment the task requires. Calibration examples and reference criteria align the team before and during evaluation. ### Contributors - Create prompts, conversations, recordings, scenarios and reference data for the test set. ### Test participants - Use the system: assistants, voice agents, retrieval systems and tool-using agents, across accents, language switching, interruptions and follow-up questions. ### Evaluators - Make independent first-pass judgments against explicit criteria and observable anchors, and record the evidence and the failure category. ### Domain experts - Judge correctness, reasoning and domain requirements in software, science, engineering, healthcare, law and finance. ### Red teamers - Attempt adversarial inputs, misuse scenarios and boundary cases, and record what was tried, what the system did and how severe the outcome was. ### Adjudicators - Resolve reviewer disagreement and produce the reference judgments that calibrate the automated judges. Relay and the YPAI Freelancer platform match contributor profiles, languages and work types to the project. Collect brings the selected team into it with assignments, instructions, agreements, task access, submissions and review. ## Versioned test suites, calibrated judges, recorded runs. - The evaluation engine executes versioned test suites across models, prompts, retrieval configurations and agent workflows. Each run records its inputs, configuration, evaluator versions, outputs, timing and resource use. Repeated trials show variability, and comparisons use consistent scenario definitions and task-appropriate statistics. - It combines deterministic checks, specialist scoring models and frontier-model judges, selected and benchmarked for the task. Human reference judgments anchor every model-based score: calibration checks agreement, error patterns and performance by language and task group, and pairwise scoring is tested for order effects. Invalid model output, a technical failure and a valid judgment that the system failed are recorded as three different outcomes. - Human and automated adversarial testing, repeated trials and maintained regression suites run before release and after it. Reporting separates attack success, harmful or incorrect responses, excessive refusal and loss of task utility. Connected to AWS. Independent of any single model. ### Amazon Bedrock - Candidate-model execution, model-based evaluation and RAG evaluation. The demonstrations on this page run their judge on Bedrock in eu-west-1. ### Amazon Bedrock AgentCore - Agent execution, tool-connected testing, browser and code environments, trajectory evaluation and observability. ### Amazon SageMaker AI - Hosted and customised models, evaluation processing jobs and model-evaluation pipelines. ### Inspect AI and model adapters - Evaluation across AWS, other model providers and customer-managed environments, in one result model. YPAI owns the evaluation protocol, the contributor operation, the rubric and quality framework, result interpretation and delivery. AWS and other providers supply execution. Services, regions and customer-account resources follow the engagement's technical and data-handling requirements. ## Two streams, one record. - The protocol defines the task, scenario coverage, rubric, reference material, human-review design, automated checks and result definitions. Contributors create and exercise scenarios while the engine runs the corresponding tests. - Human evaluators make their first-pass judgments independently; model identities, machine scores and other reviewers' verdicts are withheld under the blinding protocol. Reference comparisons and adjudication then connect the two streams, and judge calibration is measured on every run. - Collect links scenarios, run records, human judgments, automated scores and the final findings. A reviewer opens the evidence behind any failure, and the customer reads the same record. ### Human stream ### Automated stream - uncertain scores go to human review One evaluation record on YPAI Platform ## See how a judgment is made. Explore the task, source material and criteria behind a model comparison. The demonstrations below show how individual judgments, human references and automated scores contribute to an evaluation record. Demonstration data. These examples explain the method; they are not customer results or a benchmark for every model, language or task. the frame both models were shown - every object named is in the frame, with its count - an object is missed, doubled or invented - each claim points at something visible in the frame - a claim goes beyond what the frame shows - the response stays inside the task and the policy - the response advises, speculates or discloses beyond the task - uncertainty is named and passed to a person - uncertainty is hidden behind a confident sentence ## reviewer A ## reviewer B Two reviewers grade every item without seeing each other's verdict. Disagreement goes to an adjudicator, and the adjudicated verdict is the record. ## Check the claim against the source. Follow a response back to the material it was given. Separate supported observations from details the answer adds without evidence. the frame, closer · the marking the van is waiting at the crossing the frame shows a van stopped, not waiting ## Keep the test set identifiable across versions. Record which examples, source versions and criteria belong to each round. Keep development material and final evaluation material separate where the project requires an independent comparison. Each item carries a hash in the set manifest, so a later round can show it ran the same items. ## Check the judge against human reference judgments. Compare automated decisions with the agreed human references. Inspect disagreement and order sensitivity before deciding how much of the task to automate. the pair graded A then B, and B then A; a verdict that flips is position bias an illustrative floor; Landis and Koch call it substantial agreement In this demonstration, a judge above the floor and consistent under the swap test grades the task alone, with a human slice re-graded every run. A project sets its own floor. ## Compare performance without hiding uncertainty. Report the observed preference or criterion results with the method, sample definition and uncertainty appropriate to the task. Preserve ties, both-unacceptable outcomes and insufficient evidence as different results. tied · the interval includes 0.50 ## Distinguish what the agent says from what the records show. Review the messages, tool records and the end state together. The agent runs in a controlled environment, and the action it claims is checked against the system state. file the inspection record for frame A8 and flag it for remediation - You are the inspection assistant. Tools: read_frame, write_record, flag. - File the record for frame A8 and flag it if it fails the guideline. - { blur: 0.62, guideline_max: 0.20, objects: [robot, pallet] } - Blur 0.62 is above the guideline maximum of 0.20. Writing the record and flagging. - write_record(id: "A8", verdict: "fail", reason: "motion blur 0.62 > 0.20") - Frame A8 recorded as failed for motion blur and flagged for remediation. - record A8 exists · verdict fail · queue remediation - 3 of 3 correct, in order - read in full · agrees with the scorer The end state is read from the environment the project gives access to, and the reviewer's share of trajectories is agreed per task. ## The same test set, every version. - Evaluation runs before release and through the product lifecycle. Each new model, prompt, retrieval setting or tool configuration is compared against the maintained test set, and the report shows what changed and how the results moved with it. - Permitted production traces and scheduled runs surface emerging failures. Those feed targeted human review, new regression cases and the next comparison round. ## Ten methods, each with a named artefact. The report names the model version, the set, the thresholds and the verdict per criterion. Each method produces an artefact the buyer's technical file can cite. Together they are the testing evidence Annex IV asks for, and the evaluation record Articles 53 and 55 ask of a general-purpose model. - Items written after the model's training cutoff, stratified by scenario, language, population and difficulty, versioned and frozen with a hash per item. - Pass or fail per criterion, each verdict anchored to an observable behaviour. Likert scales are kept for continuous qualities such as tone. - For every item, the output, the verdict per criterion, the reviewer, the rubric version and the run id. - Two reviewers per item and an adjudicator; Cohen's Kappa for two, Fleiss' Kappa for three or more, reported against the floor agreed for the task. - The judge's Kappa against the adjudicated human verdict on a held-out slice, its consistency under the swap test and under length control. It grades alone above the floor. - Blind A/B comparison ranked by Bradley-Terry with bootstrap confidence intervals. Overlapping intervals are reported as a tie. - Every message, tool call and result logged in the Inspect format; the end state checked in the test environment; a reviewer reads the trajectories the scorer flags. - Adversarial prompts across the twelve AILuminate hazard categories, human red-teaming on the margins. Attack success rate and over-refusal rate are reported together. - Speech graded on content, voice quality and paralinguistics; MOS by ITU-T P.808, P.835 where background noise matters; latency and turn-taking for full-duplex agents. - The frozen set runs before every release. The report carries the model version, prompt configuration, set id, judge Kappa, thresholds, per-criterion pass or fail and the sign-off. Testing against prior defined metrics and probabilistic thresholds is what Article 9(8) asks of a high-risk provider. The ten methods are that test, written down. ## Every line, mapped to a named statute. Procurement and legal teams can verify each line against the standard DPA, included with every evaluation engagement. YPAI evaluates. The provider runs the conformity assessment, and a notified body where the Act requires one. The evaluation report enters that file as supplier evidence, named as such. ### [Article 15](https://artificialintelligenceact.eu/article/15/) - Outputs are graded against the accuracy metrics the provider declares, under perturbation and adversarial input. Metrics, thresholds and results are recorded per run. - Evaluation report with metrics and thresholds ### [Article 9 + Annex IV](https://artificialintelligenceact.eu/annex/4/) - Testing against prior defined metrics and probabilistic thresholds, documented as Annex IV asks: the procedures, the sets and their characteristics, the metrics, and dated test logs. - § 9(6-8) + Annex IV 2(g) ### [Articles 53 and 55](https://artificialintelligenceact.eu/article/55/) - Article 53 documentation carries the evaluation results; Article 55 asks a systemic-risk model for state-of-the-art evaluation and documented adversarial testing. The Code of Practice of 10 July 2025 is the route to showing it. The Commission's enforcement powers apply from 2 August 2026. ### [Chapter V](https://gdpr-info.eu/chapter-5/) - YPAI is a Norwegian limited company (AS). For EEA-pinned engagements, YPAI's directly controlled processing chain stays inside the EEA; where a transfer is required, SCCs are in place. Sub-processor list and jurisdictions itemised in the DPA. - Chap. V + DPA Art. 28 ### [Harmonised standards](https://www.cencenelec.eu/areas-of-work/cen-cenelec-topics/artificial-intelligence/) - The European standards written to the Act's high-risk chapter are in drafting and formal vote. Presumption of conformity under Article 40 follows citation in the Official Journal. Every method above is named so it can be cross-referenced to the clauses when they are cited. ## The evidence is built before the date lands. General-purpose obligations are in force and enforceable. The Digital Omnibus on AI moved the high-risk dates. The dates below are when each obligation applies. ### AI Act in force. - Regulation (EU) 2024/1689 enters into force; the high-risk chapter and the GPAI chapter are written. ### GPAI obligations apply. - Articles 53 and 55 bind providers of general-purpose models; the Code of Practice was published on 10 July 2025. The Commission's enforcement powers apply from 2 August 2026. ### Digital Omnibus on AI in force. - Regulation (EU) 2026/1744 defers the high-risk dates. ### Annex III high-risk obligations apply. - Articles 9 and 15 bind providers of Annex III systems from this date. ### Annex I high-risk obligations apply. - Systems under existing EU harmonisation legislation follow a year later. ## Before you start What teams ask before they send a brief. ### Do we need a finished test set? Start with the use case, existing examples and the decision you need to make. Contributors create prompts, conversations, recordings, scenarios and reference data as part of the programme, and reviewed synthetic material serves its own purpose in the set. Development material, judge-calibration data and held-out evaluation material are kept separate. ### Can you compare two versions of the same model or agent? Yes. We agree the tasks and comparison method, then run the same versioned test suite across the models, prompts, retrieval configurations or agent workflows. The report identifies the versions assessed and any changes in source material or test conditions. ### Can we use our own reviewers or rubric? Bring your rubric, reference answers or internal judgments into scoping. We agree how they are used, what additional expertise is needed and how disagreements are handled. ### Who does the human work? Contributors, test participants, evaluators, domain experts, red teamers and adjudicators, selected and qualified for each project through Relay and the YPAI Freelancer platform: 50,000 contributors registered with YPAI, 210,000+ across the wider network. They work on YPAI Platform under the project's agreements, instructions and quality framework. ### Which AWS services do you integrate with? Amazon Bedrock for model execution, model-based evaluation and RAG evaluation. Amazon Bedrock AgentCore for agent execution, tool-connected testing, browser and code environments, trajectory evaluation and observability. Amazon SageMaker AI for hosted and customised models and model-evaluation pipelines. Inspect AI and model adapters cover other providers and customer-managed environments. ### Do we have to move to a different evaluation tool? Your model infrastructure, AWS account and existing evaluation tools stay as they are. YPAI Platform is the workspace for the engagement, and the deliverables and export format are agreed for your workflow. ### Does every project need an automated model judge? Human review, deterministic checks and model-based scoring are selected according to the task. Where a judge is used, its agreement with human reference judgments is measured and reported with the results. ### Do you test live voice agents and verify agent actions? Yes. Test participants use voice agents and tool-using agents in sessions that exercise accents, interruptions, ambiguous requests and follow-up questions. Agents run in controlled environments, on Amazon Bedrock AgentCore or in your own stack, and the evaluation reads the end state the agent claims to have reached. Access, environments and permissions are defined in the protocol. ### How is sensitive material handled? Describe the handling requirements in the initial brief and keep the records and credentials themselves for later. Access, processing arrangements, confidentiality and delivery terms are agreed before the relevant project material is shared. ### What does an evaluation cost? The price covers the agreed human work, automated testing, technical integration, quality control, project operation and deliverables. It is scoped by task, languages, specialist expertise, number of examples, environments and review method. Infrastructure and model usage in your own AWS account stay on that account. ## Your evaluation project, on YPAI Platform. - YPAI Platform is the workspace for the engagement: the brief and scope, task material, contributor assignments and agreements, automated runs, human submissions, review and delivery. Your team follows progress and reads the published reports and result sets in the same project. - Role- and project-based access, confidentiality, processing grounds, dataset rights and retention follow the agreed engagement. Model access and test-environment permissions are defined for the work, and sensitive source material stays within the project arrangement. - Your model infrastructure stays where it is. Evaluation connects to it and returns results in the formats your workflows already use. ## What does your AI need to get right? Tell us what you are building, which languages matter, and whether we evaluate supplied outputs or connect to the running system. We review the brief and discuss the scope, team, environments, deliverables and commercial terms with you. A short description is enough to start. Anyone can send this brief; an account, a test set and model access come later, if the work needs them. Leave passwords, API keys and sensitive personal information out of this form. Describe any special handling requirements, and we agree with you how the records are shared. ### Your evaluation brief has been received. We review the information you sent and follow up using the contact details in your brief. The next step is to discuss the scope, approach, deliverables and commercial terms. Please keep sensitive files and credentials out of general email. Your project contact can discuss the appropriate way to share project material. Your brief reached us, and part of our handling did not finish. If you have not heard from us when you expect to, write to Check the highlighted fields and try again. We couldn't confirm whether your brief was received. Please check your connection and try again. Too many requests. Wait a few minutes and try again. ## From a project brief to a usable result. ### Tell us what needs testing. - Share the system, use case, languages, available material and the decision the evaluation needs to support. An initial brief can start before a complete dataset or model access exists. ### Agree the programme. - We define the scenarios, criteria, team, environments, automated checks, deliverables, data handling, price and timeline with you. ### Evaluate and resolve. - The human and automated streams run on the agreed protocol. Disagreement, uncertain scores, missing evidence and technical failures are examined and recorded as their own outcomes. ### Deliver and compare. - Receive the report, example-level judgments, traces and structured results, each linked to its run. Use the findings to prioritise changes. ### Run the next version. - The maintained test set and regression suite run against the next model, prompt or configuration, and the report shows the difference. - [Data Solutions](https://ypai.ai/data-solutions/) - [Data Collection](https://ypai.ai/data-collection/) - [Annotation](https://ypai.ai/data-solutions/annotation/) - [Validation](https://ypai.ai/data-solutions/validation/) - [Speech Evaluation](https://ypai.ai/speech-data/evaluation-program/) - [Clinical model evaluation](https://ypai.ai/solutions/healthcare/clinical-model-evaluation/) - [ASR benchmark](https://ypai.ai/research/asr-benchmark/) - [AI companies and model developers](https://ypai.ai/solutions/ai-companies/) - [AI evaluation and acceptance plan](https://ypai.ai/ai-implementation/discovery-and-architecture/evaluation-plan/) - [Scandinavian dialect ASR benchmark](https://ypai.ai/research/asr-benchmark/) - [Speech evaluation programme](https://ypai.ai/speech-data/evaluation-program/) - [DPA overview](https://ypai.ai/speech-data/dpa/) - [EEA data residency](https://ypai.ai/data-residency-eea/) - [Ethical framework](https://ypai.ai/data-solutions/ethical-framework/) Where your system falls under the Act - [AI Act risk classification](https://ypai.ai/speech-data/ai-act-risk-classification/) - Article 10 checker ## Ask both models about this frame Two vision models read the frame above and answer the prompt you pick. The judge grades the pair in both orders and runs on Amazon Bedrock, the same engine the programme uses. The answers and the verdicts appear beside the frame. reviewers on the inspection task: B meets the threshold, A fails on source use ## Run the judge on the sixteen items The judge grades the same sixteen items the reviewers graded, then the pair above in both orders. Its row in the strip and the four readouts update from the real verdicts.