KnowledgeReady

Auditing a knowledge base for AI: who owns the knowledge, which source wins, and what to fix first

Last updated:

An AI assistant inherits the state of the knowledge base it reads. If the same policy exists in several versions, an old page stays discoverable, or nobody knows which source is authoritative, the assistant can receive conflicting or outdated context. Auditing the knowledge base before a rollout is a way to find those problems and decide who resolves them. It does not guarantee more accurate answers.

Why do AI assistants give wrong answers from a knowledge base?

Because the assistant answers from whatever the knowledge base contains. Three kinds of problems in the knowledge system itself can lead to a wrong or inconsistent answer:

  • An outdated source. A page that was true once and was never updated, but is still available to the assistant.
  • A conflicting source. Two pages that give different values for the same rule.
  • Ambiguous authority. Pages that look equally official, with nothing to say which one replaces the other.

Example (fictional): an old FAQ page says refunds can be requested within 30 days. The current policy page says 14 days. Both are in the knowledge base the assistant reads. Which one it uses is not something you can predict from outside, and neither answer comes with a warning.

None of these is a model problem. Each one is a question about who is responsible for the content and which page counts.

Which page should an AI assistant trust?

An authoritative source is the page your organization has decided is the source of truth for a given rule or fact. Deciding this is a governance practice, and it is made by people who know the subject. In practice it usually involves:

  • One authoritative source per important rule, where practical.
  • Marking superseded pages as superseded, so a replaced page is not mistaken for the current one.
  • Deprecating or removing pages that no longer apply, instead of leaving them in place.
  • Showing effective dates on rules that change over time.
  • Labelling the canonical or source-of-truth page where several pages mention the same rule.
  • Watching duplicate pages, which tend to drift apart as each copy is edited on its own.

KnowledgeReady can surface potential conflicts and duplicates between pages. It cannot decide which version is true. That decision belongs to the person or team who owns the knowledge.

These are governance practices. We have not tested whether any of them improves the accuracy of an AI assistant, and we do not claim that they do.

How does knowledge drift happen?

Knowledge drift is the gradual divergence of what a knowledge base says from what is currently true, or from what its other pages say. It usually has ordinary causes:

  • Copied pages. A page is duplicated into another space for convenience, and from then on the two copies are edited separately.
  • Release-specific documentation. Pages written for an earlier version stay published after a newer version changes the behavior they describe.
  • Policy changes. A rule changes in the policy page, but the onboarding guide, the FAQ and the team wiki still quote the old one.
  • Reorganizations. Pages are moved or merged, and the old locations, with their old content, remain reachable.
  • Local team variants. A team adapts a shared procedure for its own use, and the variant slowly diverges from the original.
  • Owners leaving. The person who kept a page current moves on, and nobody takes over the review.

Pages updated independently of each other are the common thread. Each edit is reasonable on its own, and the inconsistency appears only when the pages are read together.

Who owns knowledge quality, and when should it be reviewed?

Ownership works best when it is assigned to a topic and a subject-matter owner, not only to individual pages. A topic owner is responsible for what the organization says about, for example, refunds, wherever it appears. A page owner is responsible for one page. When a topic spans several pages, page-level ownership alone leaves the gaps between them unowned.

Review can be triggered by events, by risk, or by the calendar. A fixed periodic review is useful where content is stable and a change of ownership is the main risk. For content that changes with the business, an event is usually a better prompt than a date.

Content typeExample review triggers
Policy / complianceA policy change, or a scheduled review
Pricing / commercial rulesA product or pricing change
Operational procedureA workflow or tool change
Stable reference contentA periodic review, or a change of owner

The table shows examples. It is not a standard, and the right triggers depend on your content and your risks.

Owner and review-date findings come from the export. KnowledgeReady works with an export of your documentation. If owner or review information is missing from the exported content, the finding means that the audit could not find it there. It does not mean the page has no owner in Confluence, Notion or whichever system the content came from.

How do you audit a knowledge base before an AI rollout?

Treat the audit as a process with decisions and owners, not as a one-off scan. A sequence that works for most teams:

  1. 1.Define which sources the assistant will be allowed to use.
  2. 2.Identify the high-risk knowledge areas, where a wrong answer would cost the most.
  3. 3.Run structural and semantic checks across the sources in scope.
  4. 4.Review conflicts and duplicates with the subject owners.
  5. 5.Mark authoritative and superseded sources.
  6. 6.Resolve ownership and review gaps.
  7. 7.Decide what belongs in the corpus the AI can access.
  8. 8.Then evaluate the AI and retrieval system separately.

The audit and the evaluation of the AI system answer different questions. The audit is about the knowledge the assistant will draw on. For the technical side, see what a documentation audit checks before retrieval-augmented generation.

What does KnowledgeReady contribute?

KnowledgeReady helps find candidates for review in an export of your documentation: stale content, missing owner or review information in the exported content, potential contradictions and duplicates, and structure that is weak or depends on context.

  • Semantic findings, such as contradictions and duplicates, are potential findings.
  • People decide which source is correct and what to do about it.
  • The product currently works from exports and does not rewrite or delete source pages.

How uploaded exports are handled is described on the security page.

What should you fix first?

Not every finding deserves the same urgency. These principles help decide where to start:

  1. Conflicting high-impact policies before stylistic issues.
  2. Wrong or outdated commercial or operational rules before cosmetic structure.
  3. Widely referenced sources before isolated pages.
  4. Unresolved source-of-truth conflicts before duplicate cleanup.
  5. Ownership gaps in high-risk content before low-risk reference pages.

One way to think about it is to weigh the impact of a wrong answer, how likely the content is to be used, and how uncertain you are about which version is right. The product does not compute this combination. It is a way to organize your own judgment.

What can an audit not tell you?

An audit cannot decide which of two conflicting pages is factually correct, and it cannot measure your assistant's end-to-end accuracy or guarantee better answers. Metadata held in the source system may also not survive the export. For the limits of the evidence behind KnowledgeReady, see the research.