Auditing a knowledge base for AI: who owns the knowledge, which source wins, and what to fix first
Last updated:
An AI assistant inherits the state of the knowledge base it reads. If the same policy exists in several versions, an old page stays discoverable, or nobody knows which source is authoritative, the assistant can receive conflicting or outdated context. Auditing the knowledge base before a rollout is a way to find those problems and decide who resolves them. It does not guarantee more accurate answers.
Why do AI assistants give wrong answers from a knowledge base?
Because the assistant answers from whatever the knowledge base contains. Three kinds of problems in the knowledge system itself can lead to a wrong or inconsistent answer:
- An outdated source. A page that was true once and was never updated, but is still available to the assistant.
- A conflicting source. Two pages that give different values for the same rule.
- Ambiguous authority. Pages that look equally official, with nothing to say which one replaces the other.
Example (fictional): an old FAQ page says refunds can be requested within 30 days. The current policy page says 14 days. Both are in the knowledge base the assistant reads. Which one it uses is not something you can predict from outside, and neither answer comes with a warning.
None of these is a model problem. Each one is a question about who is responsible for the content and which page counts.
How does knowledge drift happen?
Knowledge drift is the gradual divergence of what a knowledge base says from what is currently true, or from what its other pages say. It usually has ordinary causes:
- Copied pages. A page is duplicated into another space for convenience, and from then on the two copies are edited separately.
- Release-specific documentation. Pages written for an earlier version stay published after a newer version changes the behavior they describe.
- Policy changes. A rule changes in the policy page, but the onboarding guide, the FAQ and the team wiki still quote the old one.
- Reorganizations. Pages are moved or merged, and the old locations, with their old content, remain reachable.
- Local team variants. A team adapts a shared procedure for its own use, and the variant slowly diverges from the original.
- Owners leaving. The person who kept a page current moves on, and nobody takes over the review.
Pages updated independently of each other are the common thread. Each edit is reasonable on its own, and the inconsistency appears only when the pages are read together.
Who owns knowledge quality, and when should it be reviewed?
Ownership works best when it is assigned to a topic and a subject-matter owner, not only to individual pages. A topic owner is responsible for what the organization says about, for example, refunds, wherever it appears. A page owner is responsible for one page. When a topic spans several pages, page-level ownership alone leaves the gaps between them unowned.
Review can be triggered by events, by risk, or by the calendar. A fixed periodic review is useful where content is stable and a change of ownership is the main risk. For content that changes with the business, an event is usually a better prompt than a date.
| Content type | Example review triggers |
|---|---|
| Policy / compliance | A policy change, or a scheduled review |
| Pricing / commercial rules | A product or pricing change |
| Operational procedure | A workflow or tool change |
| Stable reference content | A periodic review, or a change of owner |
The table shows examples. It is not a standard, and the right triggers depend on your content and your risks.
Owner and review-date findings come from the export. KnowledgeReady works with an export of your documentation. If owner or review information is missing from the exported content, the finding means that the audit could not find it there. It does not mean the page has no owner in Confluence, Notion or whichever system the content came from.
How do you audit a knowledge base before an AI rollout?
Treat the audit as a process with decisions and owners, not as a one-off scan. A sequence that works for most teams:
- 1.Define which sources the assistant will be allowed to use.
- 2.Identify the high-risk knowledge areas, where a wrong answer would cost the most.
- 3.Run structural and semantic checks across the sources in scope.
- 4.Review conflicts and duplicates with the subject owners.
- 5.Mark authoritative and superseded sources.
- 6.Resolve ownership and review gaps.
- 7.Decide what belongs in the corpus the AI can access.
- 8.Then evaluate the AI and retrieval system separately.
The audit and the evaluation of the AI system answer different questions. The audit is about the knowledge the assistant will draw on. For the technical side, see what a documentation audit checks before retrieval-augmented generation.
What does KnowledgeReady contribute?
KnowledgeReady helps find candidates for review in an export of your documentation: stale content, missing owner or review information in the exported content, potential contradictions and duplicates, and structure that is weak or depends on context.
- Semantic findings, such as contradictions and duplicates, are potential findings.
- People decide which source is correct and what to do about it.
- The product currently works from exports and does not rewrite or delete source pages.
How uploaded exports are handled is described on the security page.
What should you fix first?
Not every finding deserves the same urgency. These principles help decide where to start:
- Conflicting high-impact policies before stylistic issues.
- Wrong or outdated commercial or operational rules before cosmetic structure.
- Widely referenced sources before isolated pages.
- Unresolved source-of-truth conflicts before duplicate cleanup.
- Ownership gaps in high-risk content before low-risk reference pages.
One way to think about it is to weigh the impact of a wrong answer, how likely the content is to be used, and how uncertain you are about which version is right. The product does not compute this combination. It is a way to organize your own judgment.
What can an audit not tell you?
An audit cannot decide which of two conflicting pages is factually correct, and it cannot measure your assistant's end-to-end accuracy or guarantee better answers. Metadata held in the source system may also not survive the export. For the limits of the evidence behind KnowledgeReady, see the research.