Every second company we speak to has built or bought something that lets staff ask questions of internal documents. Roughly one in four is still using it a year later.
The failures are not failures of the interface. Chat has been fine since the first week. The failures are underneath.
Retrieval is the product
If the system returns the wrong three documents, no amount of model quality saves the answer. Most of the engineering effort in a system that works is spent on retrieval: how documents are chunked, how they are indexed, how a vague question gets expanded into something that actually matches, and how results are ranked when forty documents are plausibly relevant.
Permissions are not a phase two
A knowledge system that ignores access control will eventually surface a salary review to the wrong person, and that single incident ends the program.
The right model is simple. The brain shows a person exactly what they could have found by digging themselves, and nothing more. Anything else requires a policy conversation that is much larger than the project.
Freshness beats completeness
The instinct is to index everything. The result is a system that confidently returns a rate card from 2018.
Better to index less and mark what is current. Every source needs a freshness signal, and superseded material either gets excluded or gets returned with a visible warning. A brain that says “I found this, it is four years old, here is the newer version” is more useful than one with twice the coverage and no dating.
Somebody has to read the bad answers
This is the difference between the systems that survive and the ones that rot.
Every month, someone should read the list of questions that returned poor answers. That list tells you which knowledge domain is stale, which document is missing, and which question people keep asking that the company has never actually written down.
In our experience it is the highest value document in the entire program, and it is almost never read.
Why the redesign framing matters
When this works, the second order effects are larger than the search time saved. Onboarding shortens. Decisions get made further down the org chart, because the context needed to make them is available further down the org chart. The company’s dependence on a few long tenured people softens.
That is a change in how the organisation operates. Calling it a chatbot undersells it, and worse, it leads you to buy it like a chatbot: as a tool, with no owner, no curation and no review, which is why so many of them are gathering dust.