KnowledgeReady

Research at KnowledgeReady: what we tested, what held up, what did not

Last updated:

KnowledgeReady audits documentation for structural problems that can lead AI assistants and RAG (retrieval-augmented generation) systems to wrong, outdated or hard-to-find answers. This page summarizes the evidence behind the product: synthetic benchmarks, a real-world pilot, a real-world retrieval experiment, a cross-corpus replication on a second documentation set (same team, design frozen in advance), and a check of one detector across several corpora.

The results are mixed. Some early findings did not replicate, and several effects are small or specific to one corpus. We run this research to test product assumptions, not to confirm a result chosen in advance, and we report neutral and negative outcomes alongside positive ones.

What we are trying to learn

  • How well does KnowledgeReady detect structural defects in documentation?
  • Do proposed documentation transformations improve retrieval and the quality of AI answers?
  • How far do those effects carry over between different documentation corpora?

Definitions

Structural defect
A property of a document or knowledge base that can mislead retrieval or readers, such as a broken link, a missing owner, stale content or an oversized page. Benchmarks use planted defects with known locations.
RAG
Retrieval-augmented generation: a system retrieves passages from a document collection and a language model answers using them.
Evidence coverage
The share of the text that answers a question (the gold evidence) that appears in the passages retrieved for it. 1.0 means all of it was retrieved.
Guardrail
A check defined in advance that an intervention does not make a measure worse than an agreed margin. In the GitLab replication, answer quality had to stay within 3.0 percentage points of the control.
Replication
Repeating a test on a different corpus, with a design frozen in advance, to see whether an earlier result holds.
NEUTRAL result
The predefined label for an experiment in which no reliable improvement and no material harm were detected. It is neither a success nor a failure.

Summary of findings

EvidenceResultInterpretation
Benchmark #5 (synthetic, 120 documents)Structural checks found 44 of 44 planted defects, with no false positives.Perfect on this benchmark (Benchmark #5 only). Not evidence about real documentation.
Benchmark #6 (synthetic, 150 documents)Structural checks found 35 of 44 planted defects (precision 94.6%, recall 79.5%).Lower than Benchmark #5. The semantic labels were later found to be partly invalid.
Prometheus pilot (real documentation)326 structural findings across 140 pages. 35 of 60 reviewed candidate pairs were judged relevant.Real documentation produces many candidates. Accuracy was not estimated.
PostHog experiment (real, 33 pages)Table-row reformatting gave the best single retrieval result, with a small effect. Section splitting lowered coverage. The answer-level result is inconclusive.Suggestive, and resting on very few questions.
GitLab replication (real, 174 pages)Table-row reformatting: no retrieval gain (−0.027). The answer-quality guardrail passed. Classification: NEUTRAL.The PostHog retrieval effect did not replicate on this corpus.
Detector cross-corpus checkThe table-dependency detector flagged 0 to 3 documents per corpus.Low coverage on real documentation. A limitation of the detector, not a finding about the transformation.

Synthetic benchmarks

Benchmark #5: 120 documents

On a blind synthetic benchmark of 120 documents, the deterministic checks found all 44 planted defects with no false positives. The ground truth was sealed before the run and the analyzer version was frozen first.

MeasureValue
Documents120
Planted structural defects44
True positives / false positives / false negatives44 / 0 / 0

The same benchmark also scored a semantic verifier. One configuration, DeepSeek Flash with prompt V2, with the deterministic and semantic layers combined, reached an overall F1 of 97.7% on this benchmark (precision 97.0%, recall 98.5%). That figure belongs to that configuration and that synthetic benchmark only. It is not a general accuracy figure for KnowledgeReady or for the current production verifier, and the configuration did not pass every strict gate set in advance. After the ground truth was opened, the benchmark was also used for development (a later prompt and a retrieval cutoff), so it is not an unseen test for those changes.

Benchmark #6: 150 documents, a later blind run

On a later blind benchmark of 150 documents, the deterministic layer found 35 of 44 planted defects, with 94.6% precision and 79.5% recall.

MeasureValue
Documents150
Planted structural defects44
True positives / false positives / false negatives35 / 2 / 9
Precision / recall / F194.6% / 79.5% / 86.4%

The misses were concentrated in two defect types: weak heading structure (0 of 5 found) and low contextual independence (0 of 4 found). Both false positives were oversized-page flags.

After the ground truth was revealed, we found that its semantic part was partly invalid. Every duplicate-positive pair had been built by inserting the same generic sentence into pages about different subjects, which conflicts with the verifier's frozen definition of a duplicate. Four of the ten contradiction labels were not contradictions under that definition, and one pair was labelled both a contradiction and a hard negative. The original strict result is preserved, but we do not use the semantic results of Benchmark #6 as a performance estimate and they are not a headline result here.

What synthetic benchmarks cannot show

  • A synthetic corpus is not production documentation, and its wording and structure differ from real knowledge bases.
  • Planted defects are defined in advance, so they say little about how often, or in what form, defects occur in real documentation.
  • A benchmark of this kind validates a detector. It does not show an effect on retrieval or on the quality of AI answers.

Real-world pilot: Prometheus documentation

MeasureValue
Documents140
Deterministic findings326
Semantic candidate pairs502
Human-reviewed stratified sample60 pairs, 35 judged meaningfully relevant (58.3%)

The pilot showed that real documentation produces many candidate findings, and that a little over half of a human-reviewed stratified sample of candidate pairs was judged relevant. Only retrieved candidates were annotated, so corpus-wide recall was not estimated. The pilot is not an accuracy benchmark and its figures should not be compared directly with the synthetic benchmarks above.

Real-world RAG experiment: PostHog documentation

Setup

The corpus was 33 PostHog documentation pages and 36 frozen questions: 32 answerable questions with gold evidence and 4 unanswerable ones. Retrieval was BM25 over 1,800-character chunks (200 overlap), top 5 results, with one language model answering, three runs per question. The baseline scored 162 of 192 points (84.375%) on the 32 answerable questions. Three transformations were tested: section splitting with a context header (R1), heading-level repair (R3) and table-row reformatting (R4).

Answer quality: strict and outage-adjusted analyses

AnalysisBaselineWith transformationsDifference
Strict (pre-registered)84.375%47.917%−36.46 pp
Outage-adjusted (protocol amendment)85.417%92.708%+7.29 pp

The strict result was heavily distorted by a network outage: 51 of 108 jobs in the transformed condition failed, probably because the host entered sleep cycles (the cause was not proven). It is not a measure of the transformations. The outage-adjusted analysis retried those jobs once under a protocol amendment made after the fact.

In the adjusted analysis, 6 of the 14 points of improvement were differences in empty-response failures. The non-technical gain was +4.17 percentage points, below the 6-point threshold set in advance, and the sign test gave p = 0.22. In a small outage-affected experiment, the full transformation did not show a reliable answer-level improvement. The result is suggestive, not conclusive.

R4: table-row reformatting

R4 rewrites each row of a table as plain text of the form "Column: value; Column: value". Table-row reformatting was the strongest single retrieval intervention in the PostHog corpus, but the effect was small and concentrated in very few questions.

Retrieval (32 answerable questions)BaselineR4 only
Questions with any evidence retrieved2829
Questions with all evidence retrieved2526
Mean evidence coverage0.8480.877

One question improved (coverage 0 to 1.0) and one got slightly worse (0.55 to 0.45). Each effect rests on one or two questions, so it is not robust enough to support a general claim.

R1: section splitting

Section splitting did not help overall and reduced mean evidence coverage across the tested variants.

VariantMean evidence coverage
Baseline0.848
Split only0.837
Context header only0.842
Split with context header0.835
Split with a neutral marker0.835

The one question that improved did so because of the split itself. The context header added nothing, and every split variant lost coverage on at least one other question.

R3: heading-level repair

In the tested setup R3 changed two headings and changed no retrieval hit or coverage result. It was not tested on its own, and nothing here supports a broader claim about it.

Limitations of this experiment: one corpus, 36 questions, a single AI grader who had seen baseline answers earlier, and rules chosen after the failure types were known. Retrieval is deterministic, so no variance estimate exists for it.

Cross-corpus replication: GitLab documentation (same team, design frozen in advance)

Why we replicated

The PostHog result was small and came from one corpus. To test whether the R4 retrieval effect holds elsewhere, we ran a replication on a different corpus with the design, questions, scoring rules and decision thresholds frozen before any treated result existed.

Setup

The corpus was the GitLab CI/CD documentation subset (version 19.4.0): 174 pages, of which 70 contain tables, with 132 tables and 1,057 table rows in total. R4 was applied to every table found by its syntax, without using any detector to choose where. The pipeline matched the PostHog experiment (BM25, top 5 results, 1,800 characters with 200 overlap). There were 112 questions: 60 whose answer is in a table row, 20 about prose on pages that contain tables, 20 about pages without tables, and 12 unanswerable. A copy of the pipeline run on the original corpus checked that the experimental build itself changed nothing.

Retrieval result

On GitLab documentation, R4 produced no retrieval gain. The PostHog retrieval effect did not replicate on this corpus.

Measure (60 table questions)Result
Mean evidence coverage, control to R40.969 to 0.942
Difference−0.027 (95% interval −0.098 to +0.019)
Document-level sign testp = 0.75
Questions improved / worsened / unchanged1 / 2 / 57

Questions about prose on pages with tables showed no coverage difference (0.000). Questions about pages without tables moved by +0.050 (one question), although those pages were not modified; changed corpus-wide index statistics are a likely cause. The control already retrieved most table evidence (0.969), so there was little room to improve. This does not show that R4 does not work. It shows no gain under this corpus and this pipeline.

Answer-quality guardrail

Measure (100 answerable questions, 3 runs each)ControlR4
Answer score (maximum 600 points per arm)92.50% (555)93.33% (560)
Difference+0.83 pp
Guardrail: no worse than −3.0 ppPASS

Under the guardrail defined in advance, there was no evidence of material answer-quality harm. Technical failures (empty responses, one malformed response and HTTP errors) were specified in advance to score zero within the fixed 600-point denominator: 5 answers in the control and 3 in the R4 arm, costing 10 and 6 points. Most of the observed +0.83 pp difference came from this imbalance in technical failures rather than from answer-content differences. Both arms abstained correctly on every unanswerable question that was answered.

Human re-grade and final classification

Answers were first graded by AI models (a Claude-family workflow) using a pooled, shuffled set in which arm and run labels were not included; all initial answer grading was performed by AI models. The project owner then re-graded a frozen 20% sample of 75 unique answers while blind to arm labels and model grades. This is not an external, independent review. The owner's grades agreed with the locked model grades on 73 of 75 answers (97.3%), with Cohen's kappa of about 0.79 and linear-weighted kappa of about 0.88. In both disagreements the owner gave 1 where the model gave 0, and both were control answers. Using the owner's scores would move the guardrail difference from +0.83 to +0.50 points. The two disagreements do not change the NEUTRAL classification.

Final classification: NEUTRAL. No reliable retrieval improvement was detected, and no material answer-quality harm was detected. We do not treat this as a success or as a failure.

Production detector findings

This check concerns the detector, which decides where table reformatting would be recommended, and is separate from whether the reformatting helps. We ran the production detector for table-dependent information on four public documentation subsets and on PostHog.

CorpusPages with tablesPages flagged
GitLab CI/CD docs (174 pages)703
Argo CD operator manual (142 pages)182
Kubernetes concepts (172 pages)301
MDN games (62 pages)00
PostHog (33 pages)not tallied2

The current production table-dependency detector flags very few documents in real-world documentation corpora.

The detector flags a page only when it has at least five table lines and either little prose (under 220 words) or a cue phrase. On GitLab, 56 of the 70 pages with tables had at least five table lines but enough prose to avoid the trigger. A flagged-page count of at least 30 had been set as the exposure requirement for continuing a detector-driven test, and no corpus came close. This is a limitation of the detector. It is not evidence against the R4 transformation, and the transformation was tested separately in the GitLab replication above.

What replicated, and what did not

Held up

  • KnowledgeReady detected most planted structural defects in synthetic settings, with some defect types missed on Benchmark #6.
  • Real-world corpora exposed limitations that synthetic benchmarks did not.
  • The predefined answer-quality guardrail did not trigger in the GitLab replication, indicating no material answer-quality degradation under the specified threshold.
  • Replicating on a second corpus was necessary: it changed how we read the PostHog result.

Did not hold up

  • The PostHog retrieval gain from table-row reformatting did not replicate on GitLab documentation.
  • Synthetic benchmark performance did not carry over fully to the later Benchmark #6.
  • The current table-dependency detector has low coverage on real documentation.

What we can claim

  • KnowledgeReady detects structural documentation issues that may matter for AI and RAG systems.
  • Performance varies by defect type and by corpus.
  • Some documentation transformations can change retrieval behavior.
  • Effects are corpus- and pipeline-dependent.
  • A replication on a second corpus (same team, design frozen in advance) did not confirm the PostHog retrieval improvement.
  • Our current evidence does not support a universal claim that KnowledgeReady remediation improves RAG accuracy.

What we cannot claim yet

  • That KnowledgeReady improves RAG accuracy.
  • That table normalization improves retrieval in general.
  • That remediation makes AI assistants more accurate.
  • That 97.7% F1 is the current general accuracy of the product.
  • That these results hold for dense or hybrid retrieval.
  • That these results hold across different answer models.
  • That the current table detector is complete for production use.

Product implications

KnowledgeReady currently detects and reports issues and recommends actions. The transformations described here are research interventions run with separate scripts, not remediation features of the product. Research findings decide which rules deserve to become product features, following one path:

hypothesis → experiment → validation → product rule

So far, no transformation has cleared that path.

Current implementation limitations

  • A broad heuristic for loosely formatted metadata produced false positives on PostHog (ordinary prose lines read as metadata).
  • Lines inside code blocks that start with # could be read as headings and trigger the weak-heading finding.
  • The production service runs analyzer version 0.4 and Benchmark #6 used version 0.5. The two versions have identical structural checks and differ only in one semantic claim-similarity cutoff (0.18 versus 0.175), so the 35 of 44 structural result applies to the same structural logic. The version difference matters for the semantic layer.

Methodology and reproducibility

  • Designs, inputs, questions and grades were locked (hashed) before unblinding where applicable. Analysis scripts were hashed only after they had run; they were not pre-registered.
  • Blinding was procedural. In the benchmarks, the ground truth was sealed before the run. In the GitLab replication, answers were graded in a shuffled pool and the grading packets did not include arm or run labels. Graders were instructed not to access the unblinded mapping, but that file existed on disk before grading and nothing technically prevented access.
  • The cross-corpus replication used frozen inputs: the corpus, questions, scoring rules, thresholds and pipeline settings were fixed before any treated result existed.
  • The project owner re-graded a frozen 20% sample of the unique answers while blind to arm labels and model grades. This is not an external, independent review.

Designs, inputs and grades were hash-locked as described above. Full research artifacts are not yet publicly released, so this page cannot be reproduced by outside readers from public files alone.

Limitations

  • The real-world experiments used three documentation corpora (Prometheus, PostHog, GitLab). The two RAG experiments used one retrieval method (BM25) and one answer model.
  • The answer model was qwen3.5-397b-a17b (via Scaleway). Questions, gold evidence and all initial answer grading came from a Claude-family AI workflow. A second model grading pass agreed on all 75 sample answers, but it came from the same workflow, so that agreement is not independent. The only human check was the project owner's re-grade of a 20% sample.
  • Effects in the PostHog experiment rest on one or two questions each.
  • Benchmarks #5 and #6 are synthetic and author-generated.

Ongoing research

Not yet tested: a redesigned table detector, other retrieval methods (dense and hybrid), more answer models, and more corpora. A broader test is planned only if it is justified by the evidence above.

Research integrity: artifact identifiers

These are SHA-256 identifiers of frozen research artifacts. The artifacts themselves are not published on this site. Listing the identifiers lets a future release be checked against what was locked at the time.

ExperimentArtifactDateStatusHeld inSHA-256
Benchmark #5Freeze record2026-09-30Frozen before the runResearch archive (not public)54a2c0133cda12cabf57da303b90638f008c89139df87e67162592f5f0b0c7cd
Benchmark #5Strict score report2026-10-01Final strict scoringResearch archive (not public)b2544d94062c9c48717bd71f16a63c76f08bff582811046490319bb6d01351e0
Benchmark #6Freeze protocol2026-10-02Frozen before dataset generationProject repository8b22ceed87b62a4f65507675075c3225960f9d8d5bcf9ada584698234aade6eb
Benchmark #6Strict score report2026-10-02Final strict scoringProject repository861d5f1f233d3f0d74a49d87ebba4cfc7ddd8c8861155e2ccf27894573b39730
Benchmark #6Post-reveal validity analysis2026-10-02Documents invalid semantic labelsProject repositoryd43b3370673aa8dea1eac09c2ab426b5f81df726ac1401e1316fd6fedfe719f1
PostHog experimentSeries closing lock2026-10-06Series closedResearch archive (not public)3c3ca762e0abcd5ab2a9b3d51e11a47d0044fc45d77c9a142afa337bb02563ba
PostHog experimentBlind comparison grading lock2026-10-06Locked before unblindingResearch archive (not public)be334c61d3f5ca5e8a391f11569cbb503f24d8471f4f1dc5fe823f6b84c77522
Detector cross-corpus checkCross-corpus summary2026-10-06FinalResearch archive (not public)c56fec78e812d5089a607783300a31c4eca16ab6cae9c2540dac2005a3842971
GitLab replicationDesign lock2026-10-06Frozen before questionsResearch archive (not public)489c538c01a9ca7b3816d9437b426ac8b8b6ce66b40a2603e8fc6a32581dd917
GitLab replicationQuestion set lock2026-10-06Frozen before any treatmentResearch archive (not public)b9d644f01eb21eaaafccd88d12db3786806807450625c10bd79e4fb6e3ab4ec5
GitLab replicationBuild lock2026-10-06Control, treatment and null armResearch archive (not public)3063f6a30c3efd3c9931cd02649a1375be4758bca8f5ec77a1ff4f6832b447f4
GitLab replicationRetrieval result lock2026-10-06Locked before answer generationResearch archive (not public)fb8cd7cbc6777320c5bc6b2088aeaaa8b1e0818e28c223f6f07bc69a4a8838d3
GitLab replicationAnswer-run lock2026-10-06672 of 672 jobs completedResearch archive (not public)6be9322687285525930ceaa4852126aa59c0ba00deb04a76ef2fcdd96d78037d
GitLab replicationBlind grading lock2026-10-07Locked before unblindingResearch archive (not public)650467b35ed5242c2cbe44e3c12f91574d7b294d24c22414ecaa4c674c855ff1
GitLab replicationFinal package manifest2026-10-07Final, includes human re-gradeResearch archive (not public)d41b2dfa3ccaf3ede20a54db238b739cd077f98b14e9f751237f3a47334b5e12