The most common request we get for an internal assistant is "answer questions from our documentation". The most common failure is that it answers questions the documentation does not cover, fluently and incorrectly.
This is not a prompt engineering problem. It is an architecture problem.
Refusal is a feature, not a failure
Most retrieval pipelines are built so that the model always produces an answer. Retrieval returns the top-k chunks by similarity, those chunks go into the context, and the model is asked to answer. If the chunks are irrelevant, the model answers anyway — because nothing in the pipeline gave it permission not to.
We invert that. Retrieval has a relevance floor. If nothing clears it, the pipeline short-circuits before the model is called at all and returns a refusal with a suggestion of who to ask instead. The model never sees a question it has no basis to answer.
That one change removes the majority of hallucinated answers, and it costs a token call rather than saving one.
Citations have to be verifiable, not decorative
An assistant that says "according to the safety procedure" is not cited. An assistant that links to section 4.2 of a specific document with a specific version date is cited, because a reader can check it and will occasionally discover the assistant was wrong.
We require every factual claim in a response to map to a retrieved chunk, and we render the chunk alongside the answer. This makes the system harder to build and much easier to trust. It also surfaces something valuable: when users click through and find the source contradicts the answer, that is a labelled failure case you can act on.
Chunking is where most quality is won or lost
Fixed-size chunking with a token overlap is the default because it is easy, and it is usually wrong for enterprise documents. A procedure split mid-step retrieves as two useless fragments. A table split across a boundary loses its header row and becomes meaningless.
Structure-aware chunking — splitting on headings, keeping tables intact, carrying the document title and section path into every chunk — consistently outperforms tuning the embedding model. It is unfashionable work and it is where the improvement is.
Knowledge needs governance, not just ingestion
The most dangerous failure mode is not an invented answer. It is a confidently correct answer sourced from a document that was superseded two years ago.
Every source in our knowledge base carries an owner, a review date and a visibility scope. Past its review date it is flagged in the interface and de-prioritised in retrieval. Answers drawn from ageing material carry a warning. This is boring information-management work and it is the difference between an assistant that helps and one that quietly propagates stale policy.
Measure refusal rate, not just accuracy
Two systems with identical accuracy on answered questions are not equivalent if one answers 95% of questions and the other answers 70%. The second may well be the better system, if the 25% it declines are genuinely outside its knowledge.
We track answer rate, refusal rate, citation click-through and escalation rate together. A rising refusal rate is usually a signal about the knowledge base, not the model — and it tells you exactly what to write next.