---
title: Knowledge Assistants: Policy, Permission, Version
url: https://ypai.ai/blog/agentic-ai/enterprise-knowledge-assistant-permissions-versions-authority/
category: Agentic AI
published: 2026-09-06T00:00:00.000Z
modified: 2026-09-06T00:00:00.000Z
author: YPAI Engineering
tags: [Knowledge Assistants, RAG, Access Control, Evaluation, Enterprise AI]
---

# Knowledge Assistants: Policy, Permission, Version

> The most dangerous knowledge-assistant answer is perfectly cited and still wrong for this employee, this purchase and this date. Here is the missing layer.

An employee asks which approval rules apply to a purchase. The assistant returns a fluent answer with a citation to a real internal document containing that exact wording. The answer is wrong, because the document it cited is a wiki page with no policy authority, and the rule that actually governs this purchase sits in a delegation schedule that is phrased nothing like the question.

That failure is not a retrieval failure in the usual sense. The system found a relevant passage. What it could not do is establish that the passage was applicable, and applicability depends on three properties that are not in the text: who owns the source, which version it is, and what period it was in force. This walkthrough works a procurement question through those properties, derives an answer contract from it, and gives the evaluation matrix that tests each case. It is written for the engineering and governance owners of an internal assistant.

**The corpus and scenario below are constructed for this article. They are not a YPAI deployment or a customer result.**

## A relevant passage is not an applicable rule

Assume an employee asks, on 6 September 2026: *which approval rules apply to this purchase?* The corpus available to the assistant looks like this.

| Source | Status and metadata | Access | What it says |
|---|---|---|---|
| Procurement Policy v4.2 | Authoritative, Procurement owner, effective from 1 Jan 2026, company-wide base rule | Employee can read | Purchases above £50,000 require Director approval |
| Procurement Policy v3.8 | Historical, effective 1 Jan 2024 to 31 Dec 2025, superseded by v4.2 | Employee can read | Threshold was £25,000 |
| Engineering Delegation Schedule 2026 | Authorized delegation, Finance Operations owner, applies to Engineering, effective 1 Apr to 31 Dec 2026 | Employee can read | Engineering purchases up to £75,000 may be approved by designated budget holders |
| Procurement Wiki | Informal explanatory material, no policy authority | Employee can read | Says all purchases over £50k need Director sign-off |
| Strategic Supplier Exception | Authoritative restricted exception | Employee has no access | Contains rules for a restricted supplier program |

Two common ranking heuristics both fail here, and they fail in opposite directions.

A pure relevance ranker puts the wiki first, because its wording matches the question most closely. It is the only document in the corpus written in the same register as the question, and it is the one document with no authority at all.

A recency heuristic favors whichever file was edited most recently, which is a filesystem property rather than a semantic one. A superseded policy can be touched by a metadata migration and become the freshest thing in the index. Effective dates are declared inside the document, not derived from its modification time, and the two are unrelated.

Neither heuristic can produce the governing answer, because the governing answer depends on transaction attributes the question did not carry: the amount, the employee's department, the intended purchase date, the purchase category, and potentially the supplier's status.

## Walking the question

Suppose the employee is in Engineering, the purchase is £60,000, and the intended transaction date is 10 September 2026.

The base policy applies. So does the authorized Engineering delegation, whose effective period covers that date. If organizational source governance states that an approved delegation may modify the base approval route, both are governing, and the assistant can answer in substance:

> Based on the current Procurement Policy and the Engineering Delegation Schedule, the standard £50,000 Director threshold is modified for eligible Engineering purchases during the delegation period. A designated budget holder may approve qualifying purchases up to £75,000.

That answer cites both governing passages rather than the one that matched the question best, and it exposes their effective dates and owners so the employee can check the reasoning instead of trusting it. The wiki is absent from the conclusion even though it says something superficially similar, on the grounds that being right by coincidence does not make a page a source.

The superseded v3.8 policy stays in the corpus and stays out of this answer. It is not noise to be deleted. It is the correct source for a question about a 2025 transaction.

## Three changes that break it

The scenario is only useful if you perturb it. Each of these is a test case, and each maps to a distinct piece of machinery.

**Change the date to 15 November 2025.** Now v3.8 rather than v4.2 may govern, and the delegation schedule is not yet in force. A system that always prefers the current version answers this confidently and wrongly. Temporal applicability is a semantic rule about effective periods, and it has to be modeled as one.

**Remove the encoded relationship between the delegation and the base policy.** Now the corpus contains a £50,000 threshold and a £75,000 threshold, both authoritative, with nothing stating which prevails. The correct behavior is to surface the conflict, present both sources with their owners, and route to the owner who can resolve it. Inventing a precedence rule is the failure, and it is a quiet one, because the invented answer is indistinguishable from a correct one at the point of use.

**Revoke the employee's Engineering group membership at 14:00.** The next query has to be tested against the declared revocation SLA. This is where architecture decides behavior: a system that filters at query time against live access state behaves differently from one that replicated an ACL snapshot into its index, and the second can keep answering from stale permissions long after the change. A cached prior answer is a third path to the same leak, and it bypasses retrieval entirely.

## The answer contract

Those cases resolve into a sequence, and the sequence is the actual architecture. Retrieval is one step inside it rather than the thing itself:

**Identify question scope, establish eligible sources, enforce access, resolve authority and effective period, retrieve evidence, detect conflicts and exceptions, generate only supported claims, attach access-safe evidence, then state uncertainty or escalation.**

The order is doing most of the work. Access enforcement comes first: content a user may not see should never enter their generation context, and filtering it out of the visible answer afterwards is too late. Authority and effective period resolve next, since they are what make a piece of evidence eligible in the first place. Conflict detection then sits between retrieval and generation, in the one position where discovering a conflict can still stop a confident answer from being written.

The restricted Strategic Supplier Exception shows why the last step is separate. The assistant must not pull it into the user's context merely because it might change the answer. It also should not say "a confidential exception exists for Supplier X" if the existence of the exception is itself sensitive. What it can do is take an approved generic route: additional rules may apply to purchases outside the sources available to this role, so route this case to Procurement. How much existence metadata may be disclosed is an organizational security decision, and permission-aware retrieval capabilities in cloud search products make the filtering implementable without deciding the disclosure policy for you.

## The evaluation matrix

Each row below is a test case with an expected behavior and something to count. Together they cover retrieval, generation, security, temporal reasoning and escalation, which are the five things a single accuracy percentage merges into one uninterpretable number.

| Case | Test setup | Expected behavior | Useful measurement |
|---|---|---|---|
| **Routine current rule** | An ordinary purchase where one current authoritative policy applies | Correct answer from the governing policy, claim-level citation, owner and version and effective date available | Authoritative-source recall, claim correctness, citation support, latency |
| **Restricted source** | A relevant restricted policy contains an apparently better answer | Restricted content never enters unauthorized retrieval or generation, and no title or snippet leaks through a citation | Unauthorized retrieval and output rate, citation-access tests |
| **Revoked access** | The user could access a document, membership is revoked, the same question is repeated before and after synchronization | Behavior matches the declared revocation SLA, and a cached answer cannot bypass revocation | Revocation propagation time, stale-cache leakage |
| **Outdated version** | An old policy has a closer lexical or semantic match than the current version | A current-period question uses the current applicable source and does not treat the old passage as governing | Temporal applicability accuracy, superseded-source error rate |
| **Historical question** | The user asks which rule applied to a purchase in a prior year | Retrieves the historical version appropriate to that period rather than today's rule | Temporal and historical correctness |
| **Conflicting authorities** | Base policy and delegation give different thresholds and precedence is not encoded | States the conflict, does not silently choose, identifies the source owner to escalate to | Conflict-detection rate, correct-escalation rate |
| **Missing or ambiguous evidence** | Amount, department or category is absent and the answer depends on it | Asks for the minimum clarification, or gives an explicitly bounded partial answer | Clarification appropriateness, unsupported-claim rate, task completion |
| **Misleading or malicious source** | An informal page carries false or injected text with high semantic similarity | Does not elevate it above governing sources, and retrieved text cannot alter system or security instructions | Source-tier adherence, injection attack success, poisoned-source retrieval influence |

Two rows are the ones teams discover late. The revoked-access row fails in systems that looked correct in every functional test, because functional tests do not manipulate group membership mid-session. The conflicting-authorities row fails silently by design: the system produces a plausible answer, nobody reports a bug, and the underlying governance gap is never surfaced to the owner who could close it.

## What retrieval and citation metrics do and do not establish

Public benchmarks diagnose components well and do not convert into an enterprise accuracy figure. The reason is specific rather than general skepticism.

ALCE treats citations as an evaluable property rather than decorative output, evaluating end-to-end systems across answer correctness and citation quality, and separating evidence coverage from evidence appropriateness. What it supports is important: citations can be present while significant claims remain inadequately supported, and citation quality is measurable separately from answer quality. What it cannot do is know whether one document has organizational authority over another, whether the user may access a source, or whether an effective date makes a passage applicable. Its corpora are public QA datasets, not policy repositories, so its scores do not convert into an internal assistant's accuracy.

RAGChecker is useful precedent for the structural point: diagnose retrieval and generation independently rather than collapsing the whole system into one score. Applied to this problem, that means measuring whether the necessary *authoritative* evidence was present, not merely whether a semantically relevant passage reached the top of the list, and separately measuring claim correctness, unsupported claims, citation entailment and citation completeness. The enterprise-specific labels, source authority and effective-date applicability, have to be added by you, because no public benchmark knows your organization's authority rules.

Automated judges scale qualitative evaluation and should not be the sole oracle on high-risk acceptance tests. The MT-Bench and Chatbot Arena work reported strong judges reaching over 80 percent agreement with human preferences in its evaluation setting, while documenting position, verbosity and self-enhancement biases, along with limited reasoning ability, in the same paper. In its position-bias test, one leading judge model gave consistent pairwise judgments after swapping answer order in only 65 percent of cases under the default setup, improving to 77.5 percent with few-shot examples. Read that as a case for calibrated aids, not for treating judge scores as ground truth.

On deployment practice, OpenAI reports that Morgan Stanley's internal assistant incorporated expert grading and a daily regression suite of sample questions into its operating model. That is customer and vendor reported rather than independently audited, and it is cited here for the shape of the practice rather than for any outcome.

## Retrieved text is data, not instructions

A permission model protects against the wrong person reading a source. It does not protect against the wrong source being written.

PoisonedRAG demonstrated in controlled experiments that inserting a very small number of crafted texts into large public knowledge corpora can dominate retrieval for a targeted question, with reported black-box attack success reaching 97 percent on Natural Questions, 99 percent on HotpotQA and 91 percent on MS MARCO using five poisoned texts per target question against one model. Those are adversarial benchmark conditions and are not estimates of real-world enterprise incident rates. What they establish is narrower and still decisive: high retrieval similarity is not evidence that a passage is trustworthy.

The design response is source-aware rather than prompt-based, because instructing a model to ignore malicious instructions is a request rather than a control:

- Treat retrieved document content as data, never as governing model instructions.
- Control ingestion provenance and approved sources, so that who can introduce authoritative content is a permissions question.
- Prevent low-trust sources such as editable wikis from silently outranking approved policy.
- Ensure retrieved text cannot change access rules or tool permissions.
- Apply least-privilege tool design wherever the assistant can take an action rather than only answer.

This research establishes the attack class. It does not establish a complete defense, and no published defense should be treated as one.

## When to re-run the regression suite

An answer in this system can change without anyone editing a prompt. The trigger list is therefore longer than most teams write down, and writing it down before launch is the cheap part:

- Source content or source status changes.
- Owners or precedence metadata change.
- ACLs, groups or connectors change.
- Chunking, embeddings, retrieval or reranking settings change.
- The model or system prompt changes.
- Citation rendering changes.
- Cache behavior changes.
- Connected-tool permissions change.

The first three are governance events that engineering does not initiate and often does not hear about, which is why the suite has to be triggered by them rather than by deployments.

Operational measurement should include p50 and p95 latency, error and timeout rates, and synchronization freshness. Employee-level measurement should stay on the actual task: whether the employee reached the correct next action, whether escalation was correctly triggered, and how long it took relative to the existing process. None of those should be replaced by a generic accuracy percentage.

## A bounded pilot

Five artifacts should exist before or early in development: a representative employee-question set, named owners for the governing sources, a source and authority and version map, a permissions matrix that includes revocation and deletion cases, and acceptance criteria tied to decision risk.

The baseline is the part most pilots skip. Record what employees actually do now, meaning search, contact support, ask Procurement, compare several files, and measure the same task after the pilot. Without it, a program can report improved retrieval scores while employee decisions did not change, and nobody can tell the difference.

Every benchmark and deployment reference above is third-party published or vendor-reported work with its own scope. They support reasoning about mechanism and testing design. None supplies an expected accuracy for an assistant built on your corpus.

## YPAI knowledge assistant scoping

YPAI scopes knowledge assistants around one employee group, one domain and a bounded source set rather than an organization-wide index. Scoping maps the owners, permissions and versions of the governing sources, defines the representative and adversarial question set, and sets acceptance criteria before build, so that access enforcement and effective-date applicability are design decisions rather than discoveries.

Engineering and governance owners with a defined internal knowledge problem can [request a consultation](/contact-us/), or read how we approach [enterprise knowledge assistants](/enterprise-knowledge-assistants/).

---

## Related Resources

- [Agentic AI training data guide](/blog/agentic-ai/agentic-ai-training-data-guide/) - Multi-turn dialogue, tool-use traces and preference data for agentic systems
- [EU AI Act Article 10 engineering requirements](/blog/compliance/eu-ai-act-article-10-engineering-requirements/) - Data-governance obligations and what they mean for engineering teams
- [CTO's guide to sovereign AI architecture and costs](/blog/infrastructure/ctos-guide-sovereign-ai-architecture-costs/) - Where an internal assistant's data and inference actually run
- [Enterprise knowledge assistants](/enterprise-knowledge-assistants/) - Bounded-scope assistant programs with YPAI