Research at KnowledgeReady: what we tested, what held up, what did not
Last updated:
KnowledgeReady audits documentation for structural problems that can lead AI assistants and RAG (retrieval-augmented generation) systems to wrong, outdated or hard-to-find answers. This page summarizes the evidence behind the product: synthetic benchmarks, a real-world pilot, a real-world retrieval experiment, a cross-corpus replication on a second documentation set (same team, design frozen in advance), and a check of one detector across several corpora.
The results are mixed. Some early findings did not replicate, and several effects are small or specific to one corpus. We run this research to test product assumptions, not to confirm a result chosen in advance, and we report neutral and negative outcomes alongside positive ones.
What we are trying to learn
- How well does KnowledgeReady detect structural defects in documentation?
- Do proposed documentation transformations improve retrieval and the quality of AI answers?
- How far do those effects carry over between different documentation corpora?
Definitions
- Structural defect
- A property of a document or knowledge base that can mislead retrieval or readers, such as a broken link, a missing owner, stale content or an oversized page. Benchmarks use planted defects with known locations.
- RAG
- Retrieval-augmented generation: a system retrieves passages from a document collection and a language model answers using them.
- Evidence coverage
- The share of the text that answers a question (the gold evidence) that appears in the passages retrieved for it. 1.0 means all of it was retrieved.
- Guardrail
- A check defined in advance that an intervention does not make a measure worse than an agreed margin. In the GitLab replication, answer quality had to stay within 3.0 percentage points of the control.
- Replication
- Repeating a test on a different corpus, with a design frozen in advance, to see whether an earlier result holds.
- NEUTRAL result
- The predefined label for an experiment in which no reliable improvement and no material harm were detected. It is neither a success nor a failure.
Summary of findings
| Evidence | Result | Interpretation |
|---|---|---|
| Benchmark #5 (synthetic, 120 documents) | Structural checks found 44 of 44 planted defects, with no false positives. | Perfect on this benchmark (Benchmark #5 only). Not evidence about real documentation. |
| Benchmark #6 (synthetic, 150 documents) | Structural checks found 35 of 44 planted defects (precision 94.6%, recall 79.5%). | Lower than Benchmark #5. The semantic labels were later found to be partly invalid. |
| Prometheus pilot (real documentation) | 326 structural findings across 140 pages. 35 of 60 reviewed candidate pairs were judged relevant. | Real documentation produces many candidates. Accuracy was not estimated. |
| PostHog experiment (real, 33 pages) | Table-row reformatting gave the best single retrieval result, with a small effect. Section splitting lowered coverage. The answer-level result is inconclusive. | Suggestive, and resting on very few questions. |
| GitLab replication (real, 174 pages) | Table-row reformatting: no retrieval gain (−0.027). The answer-quality guardrail passed. Classification: NEUTRAL. | The PostHog retrieval effect did not replicate on this corpus. |
| Detector cross-corpus check | The table-dependency detector flagged 0 to 3 documents per corpus. | Low coverage on real documentation. A limitation of the detector, not a finding about the transformation. |
Synthetic benchmarks
Benchmark #5: 120 documents
On a blind synthetic benchmark of 120 documents, the deterministic checks found all 44 planted defects with no false positives. The ground truth was sealed before the run and the analyzer version was frozen first.
| Measure | Value |
|---|---|
| Documents | 120 |
| Planted structural defects | 44 |
| True positives / false positives / false negatives | 44 / 0 / 0 |
The same benchmark also scored a semantic verifier. One configuration, DeepSeek Flash with prompt V2, with the deterministic and semantic layers combined, reached an overall F1 of 97.7% on this benchmark (precision 97.0%, recall 98.5%). That figure belongs to that configuration and that synthetic benchmark only. It is not a general accuracy figure for KnowledgeReady or for the current production verifier, and the configuration did not pass every strict gate set in advance. After the ground truth was opened, the benchmark was also used for development (a later prompt and a retrieval cutoff), so it is not an unseen test for those changes.
Benchmark #6: 150 documents, a later blind run
On a later blind benchmark of 150 documents, the deterministic layer found 35 of 44 planted defects, with 94.6% precision and 79.5% recall.
| Measure | Value |
|---|---|
| Documents | 150 |
| Planted structural defects | 44 |
| True positives / false positives / false negatives | 35 / 2 / 9 |
| Precision / recall / F1 | 94.6% / 79.5% / 86.4% |
The misses were concentrated in two defect types: weak heading structure (0 of 5 found) and low contextual independence (0 of 4 found). Both false positives were oversized-page flags.
After the ground truth was revealed, we found that its semantic part was partly invalid. Every duplicate-positive pair had been built by inserting the same generic sentence into pages about different subjects, which conflicts with the verifier's frozen definition of a duplicate. Four of the ten contradiction labels were not contradictions under that definition, and one pair was labelled both a contradiction and a hard negative. The original strict result is preserved, but we do not use the semantic results of Benchmark #6 as a performance estimate and they are not a headline result here.
What synthetic benchmarks cannot show
- A synthetic corpus is not production documentation, and its wording and structure differ from real knowledge bases.
- Planted defects are defined in advance, so they say little about how often, or in what form, defects occur in real documentation.
- A benchmark of this kind validates a detector. It does not show an effect on retrieval or on the quality of AI answers.
Real-world pilot: Prometheus documentation
| Measure | Value |
|---|---|
| Documents | 140 |
| Deterministic findings | 326 |
| Semantic candidate pairs | 502 |
| Human-reviewed stratified sample | 60 pairs, 35 judged meaningfully relevant (58.3%) |
The pilot showed that real documentation produces many candidate findings, and that a little over half of a human-reviewed stratified sample of candidate pairs was judged relevant. Only retrieved candidates were annotated, so corpus-wide recall was not estimated. The pilot is not an accuracy benchmark and its figures should not be compared directly with the synthetic benchmarks above.
Real-world RAG experiment: PostHog documentation
Setup
The corpus was 33 PostHog documentation pages and 36 frozen questions: 32 answerable questions with gold evidence and 4 unanswerable ones. Retrieval was BM25 over 1,800-character chunks (200 overlap), top 5 results, with one language model answering, three runs per question. The baseline scored 162 of 192 points (84.375%) on the 32 answerable questions. Three transformations were tested: section splitting with a context header (R1), heading-level repair (R3) and table-row reformatting (R4).
Answer quality: strict and outage-adjusted analyses
| Analysis | Baseline | With transformations | Difference |
|---|---|---|---|
| Strict (pre-registered) | 84.375% | 47.917% | −36.46 pp |
| Outage-adjusted (protocol amendment) | 85.417% | 92.708% | +7.29 pp |
The strict result was heavily distorted by a network outage: 51 of 108 jobs in the transformed condition failed, probably because the host entered sleep cycles (the cause was not proven). It is not a measure of the transformations. The outage-adjusted analysis retried those jobs once under a protocol amendment made after the fact.
In the adjusted analysis, 6 of the 14 points of improvement were differences in empty-response failures. The non-technical gain was +4.17 percentage points, below the 6-point threshold set in advance, and the sign test gave p = 0.22. In a small outage-affected experiment, the full transformation did not show a reliable answer-level improvement. The result is suggestive, not conclusive.
R4: table-row reformatting
R4 rewrites each row of a table as plain text of the form "Column: value; Column: value". Table-row reformatting was the strongest single retrieval intervention in the PostHog corpus, but the effect was small and concentrated in very few questions.
| Retrieval (32 answerable questions) | Baseline | R4 only |
|---|---|---|
| Questions with any evidence retrieved | 28 | 29 |
| Questions with all evidence retrieved | 25 | 26 |
| Mean evidence coverage | 0.848 | 0.877 |
One question improved (coverage 0 to 1.0) and one got slightly worse (0.55 to 0.45). Each effect rests on one or two questions, so it is not robust enough to support a general claim.
R1: section splitting
Section splitting did not help overall and reduced mean evidence coverage across the tested variants.
| Variant | Mean evidence coverage |
|---|---|
| Baseline | 0.848 |
| Split only | 0.837 |
| Context header only | 0.842 |
| Split with context header | 0.835 |
| Split with a neutral marker | 0.835 |
The one question that improved did so because of the split itself. The context header added nothing, and every split variant lost coverage on at least one other question.
R3: heading-level repair
In the tested setup R3 changed two headings and changed no retrieval hit or coverage result. It was not tested on its own, and nothing here supports a broader claim about it.
Limitations of this experiment: one corpus, 36 questions, a single AI grader who had seen baseline answers earlier, and rules chosen after the failure types were known. Retrieval is deterministic, so no variance estimate exists for it.
Cross-corpus replication: GitLab documentation (same team, design frozen in advance)
Why we replicated
The PostHog result was small and came from one corpus. To test whether the R4 retrieval effect holds elsewhere, we ran a replication on a different corpus with the design, questions, scoring rules and decision thresholds frozen before any treated result existed.
Setup
The corpus was the GitLab CI/CD documentation subset (version 19.4.0): 174 pages, of which 70 contain tables, with 132 tables and 1,057 table rows in total. R4 was applied to every table found by its syntax, without using any detector to choose where. The pipeline matched the PostHog experiment (BM25, top 5 results, 1,800 characters with 200 overlap). There were 112 questions: 60 whose answer is in a table row, 20 about prose on pages that contain tables, 20 about pages without tables, and 12 unanswerable. A copy of the pipeline run on the original corpus checked that the experimental build itself changed nothing.
Retrieval result
On GitLab documentation, R4 produced no retrieval gain. The PostHog retrieval effect did not replicate on this corpus.
| Measure (60 table questions) | Result |
|---|---|
| Mean evidence coverage, control to R4 | 0.969 to 0.942 |
| Difference | −0.027 (95% interval −0.098 to +0.019) |
| Document-level sign test | p = 0.75 |
| Questions improved / worsened / unchanged | 1 / 2 / 57 |
Questions about prose on pages with tables showed no coverage difference (0.000). Questions about pages without tables moved by +0.050 (one question), although those pages were not modified; changed corpus-wide index statistics are a likely cause. The control already retrieved most table evidence (0.969), so there was little room to improve. This does not show that R4 does not work. It shows no gain under this corpus and this pipeline.
Answer-quality guardrail
| Measure (100 answerable questions, 3 runs each) | Control | R4 |
|---|---|---|
| Answer score (maximum 600 points per arm) | 92.50% (555) | 93.33% (560) |
| Difference | +0.83 pp | |
| Guardrail: no worse than −3.0 pp | PASS |
Under the guardrail defined in advance, there was no evidence of material answer-quality harm. Technical failures (empty responses, one malformed response and HTTP errors) were specified in advance to score zero within the fixed 600-point denominator: 5 answers in the control and 3 in the R4 arm, costing 10 and 6 points. Most of the observed +0.83 pp difference came from this imbalance in technical failures rather than from answer-content differences. Both arms abstained correctly on every unanswerable question that was answered.
Human re-grade and final classification
Answers were first graded by AI models (a Claude-family workflow) using a pooled, shuffled set in which arm and run labels were not included; all initial answer grading was performed by AI models. The project owner then re-graded a frozen 20% sample of 75 unique answers while blind to arm labels and model grades. This is not an external, independent review. The owner's grades agreed with the locked model grades on 73 of 75 answers (97.3%), with Cohen's kappa of about 0.79 and linear-weighted kappa of about 0.88. In both disagreements the owner gave 1 where the model gave 0, and both were control answers. Using the owner's scores would move the guardrail difference from +0.83 to +0.50 points. The two disagreements do not change the NEUTRAL classification.
Final classification: NEUTRAL. No reliable retrieval improvement was detected, and no material answer-quality harm was detected. We do not treat this as a success or as a failure.
Production detector findings
This check concerns the detector, which decides where table reformatting would be recommended, and is separate from whether the reformatting helps. We ran the production detector for table-dependent information on four public documentation subsets and on PostHog.
| Corpus | Pages with tables | Pages flagged |
|---|---|---|
| GitLab CI/CD docs (174 pages) | 70 | 3 |
| Argo CD operator manual (142 pages) | 18 | 2 |
| Kubernetes concepts (172 pages) | 30 | 1 |
| MDN games (62 pages) | 0 | 0 |
| PostHog (33 pages) | not tallied | 2 |
The current production table-dependency detector flags very few documents in real-world documentation corpora.
The detector flags a page only when it has at least five table lines and either little prose (under 220 words) or a cue phrase. On GitLab, 56 of the 70 pages with tables had at least five table lines but enough prose to avoid the trigger. A flagged-page count of at least 30 had been set as the exposure requirement for continuing a detector-driven test, and no corpus came close. This is a limitation of the detector. It is not evidence against the R4 transformation, and the transformation was tested separately in the GitLab replication above.
What replicated, and what did not
Held up
- KnowledgeReady detected most planted structural defects in synthetic settings, with some defect types missed on Benchmark #6.
- Real-world corpora exposed limitations that synthetic benchmarks did not.
- The predefined answer-quality guardrail did not trigger in the GitLab replication, indicating no material answer-quality degradation under the specified threshold.
- Replicating on a second corpus was necessary: it changed how we read the PostHog result.
Did not hold up
- The PostHog retrieval gain from table-row reformatting did not replicate on GitLab documentation.
- Synthetic benchmark performance did not carry over fully to the later Benchmark #6.
- The current table-dependency detector has low coverage on real documentation.
What we can claim
- KnowledgeReady detects structural documentation issues that may matter for AI and RAG systems.
- Performance varies by defect type and by corpus.
- Some documentation transformations can change retrieval behavior.
- Effects are corpus- and pipeline-dependent.
- A replication on a second corpus (same team, design frozen in advance) did not confirm the PostHog retrieval improvement.
- Our current evidence does not support a universal claim that KnowledgeReady remediation improves RAG accuracy.
What we cannot claim yet
- That KnowledgeReady improves RAG accuracy.
- That table normalization improves retrieval in general.
- That remediation makes AI assistants more accurate.
- That 97.7% F1 is the current general accuracy of the product.
- That these results hold for dense or hybrid retrieval.
- That these results hold across different answer models.
- That the current table detector is complete for production use.
Product implications
KnowledgeReady currently detects and reports issues and recommends actions. The transformations described here are research interventions run with separate scripts, not remediation features of the product. Research findings decide which rules deserve to become product features, following one path:
hypothesis → experiment → validation → product rule
So far, no transformation has cleared that path.
Current implementation limitations
- A broad heuristic for loosely formatted metadata produced false positives on PostHog (ordinary prose lines read as metadata).
- Lines inside code blocks that start with # could be read as headings and trigger the weak-heading finding.
- The production service runs analyzer version 0.4 and Benchmark #6 used version 0.5. The two versions have identical structural checks and differ only in one semantic claim-similarity cutoff (0.18 versus 0.175), so the 35 of 44 structural result applies to the same structural logic. The version difference matters for the semantic layer.
Methodology and reproducibility
- Designs, inputs, questions and grades were locked (hashed) before unblinding where applicable. Analysis scripts were hashed only after they had run; they were not pre-registered.
- Blinding was procedural. In the benchmarks, the ground truth was sealed before the run. In the GitLab replication, answers were graded in a shuffled pool and the grading packets did not include arm or run labels. Graders were instructed not to access the unblinded mapping, but that file existed on disk before grading and nothing technically prevented access.
- The cross-corpus replication used frozen inputs: the corpus, questions, scoring rules, thresholds and pipeline settings were fixed before any treated result existed.
- The project owner re-graded a frozen 20% sample of the unique answers while blind to arm labels and model grades. This is not an external, independent review.
Designs, inputs and grades were hash-locked as described above. Full research artifacts are not yet publicly released, so this page cannot be reproduced by outside readers from public files alone.
Limitations
- The real-world experiments used three documentation corpora (Prometheus, PostHog, GitLab). The two RAG experiments used one retrieval method (BM25) and one answer model.
- The answer model was qwen3.5-397b-a17b (via Scaleway). Questions, gold evidence and all initial answer grading came from a Claude-family AI workflow. A second model grading pass agreed on all 75 sample answers, but it came from the same workflow, so that agreement is not independent. The only human check was the project owner's re-grade of a 20% sample.
- Effects in the PostHog experiment rest on one or two questions each.
- Benchmarks #5 and #6 are synthetic and author-generated.
Ongoing research
Not yet tested: a redesigned table detector, other retrieval methods (dense and hybrid), more answer models, and more corpora. A broader test is planned only if it is justified by the evidence above.
Research integrity: artifact identifiers
These are SHA-256 identifiers of frozen research artifacts. The artifacts themselves are not published on this site. Listing the identifiers lets a future release be checked against what was locked at the time.
| Experiment | Artifact | Date | Status | Held in | SHA-256 |
|---|---|---|---|---|---|
| Benchmark #5 | Freeze record | 2026-09-30 | Frozen before the run | Research archive (not public) | 54a2c0133cda12cabf57da303b90638f008c89139df87e67162592f5f0b0c7cd |
| Benchmark #5 | Strict score report | 2026-10-01 | Final strict scoring | Research archive (not public) | b2544d94062c9c48717bd71f16a63c76f08bff582811046490319bb6d01351e0 |
| Benchmark #6 | Freeze protocol | 2026-10-02 | Frozen before dataset generation | Project repository | 8b22ceed87b62a4f65507675075c3225960f9d8d5bcf9ada584698234aade6eb |
| Benchmark #6 | Strict score report | 2026-10-02 | Final strict scoring | Project repository | 861d5f1f233d3f0d74a49d87ebba4cfc7ddd8c8861155e2ccf27894573b39730 |
| Benchmark #6 | Post-reveal validity analysis | 2026-10-02 | Documents invalid semantic labels | Project repository | d43b3370673aa8dea1eac09c2ab426b5f81df726ac1401e1316fd6fedfe719f1 |
| PostHog experiment | Series closing lock | 2026-10-06 | Series closed | Research archive (not public) | 3c3ca762e0abcd5ab2a9b3d51e11a47d0044fc45d77c9a142afa337bb02563ba |
| PostHog experiment | Blind comparison grading lock | 2026-10-06 | Locked before unblinding | Research archive (not public) | be334c61d3f5ca5e8a391f11569cbb503f24d8471f4f1dc5fe823f6b84c77522 |
| Detector cross-corpus check | Cross-corpus summary | 2026-10-06 | Final | Research archive (not public) | c56fec78e812d5089a607783300a31c4eca16ab6cae9c2540dac2005a3842971 |
| GitLab replication | Design lock | 2026-10-06 | Frozen before questions | Research archive (not public) | 489c538c01a9ca7b3816d9437b426ac8b8b6ce66b40a2603e8fc6a32581dd917 |
| GitLab replication | Question set lock | 2026-10-06 | Frozen before any treatment | Research archive (not public) | b9d644f01eb21eaaafccd88d12db3786806807450625c10bd79e4fb6e3ab4ec5 |
| GitLab replication | Build lock | 2026-10-06 | Control, treatment and null arm | Research archive (not public) | 3063f6a30c3efd3c9931cd02649a1375be4758bca8f5ec77a1ff4f6832b447f4 |
| GitLab replication | Retrieval result lock | 2026-10-06 | Locked before answer generation | Research archive (not public) | fb8cd7cbc6777320c5bc6b2088aeaaa8b1e0818e28c223f6f07bc69a4a8838d3 |
| GitLab replication | Answer-run lock | 2026-10-06 | 672 of 672 jobs completed | Research archive (not public) | 6be9322687285525930ceaa4852126aa59c0ba00deb04a76ef2fcdd96d78037d |
| GitLab replication | Blind grading lock | 2026-10-07 | Locked before unblinding | Research archive (not public) | 650467b35ed5242c2cbe44e3c12f91574d7b294d24c22414ecaa4c674c855ff1 |
| GitLab replication | Final package manifest | 2026-10-07 | Final, includes human re-grade | Research archive (not public) | d41b2dfa3ccaf3ede20a54db238b739cd077f98b14e9f751237f3a47334b5e12 |