Key takeaways
- Answer quality depends more on retrieval than on the model
- Permissions must be checked when documents are retrieved
- Every answer should point to its source
When a knowledge assistant disappoints, the model is usually fine. The trouble is upstream: the wrong document was found, or an old one, or none at all. Work through this list before blaming the model.
Sources
- Every source has a named owner and a refresh schedule. If the price list changes every quarter, someone is responsible for re-indexing it every quarter.
- Outdated and duplicate documents are removed or clearly marked. Three versions of the leave policy will produce three different answers.
- Scanned PDFs are converted with a quality check on the OCR output.
Chunking and indexing
- Chunks follow the document's own structure, such as sections and clauses.
- Each chunk keeps its title, section heading and source link as metadata.
- Tables, such as rate cards and GST slabs, are stored in a form the model can read row by row.
Retrieval
- Hybrid search combines keyword and semantic search. Keywords catch part numbers and HSN codes; semantic search catches paraphrased questions.
- A re-ranking step puts the most relevant passages first.
- Metadata filters narrow results by department, product, region or date.
Permissions
- Retrieval respects the access rights the user already has.
- Sensitive sources, such as HR files and salary data, are kept separate and every access is logged.
- No answer can contain text the user could not have opened directly.
Answering
- The model answers only from retrieved text, and says plainly when it does not know.
- Every answer cites the passages it used.
- Answers are short by default, with a way to see more.
Evaluation
- A test set of real staff questions, each with the expected answer and source.
- Retrieval accuracy and answer accuracy are measured separately, so you know which half to fix.
- The test set is re-run on every change to prompts, models or sources.
Operations
- Questions that got no good answer are logged and reviewed every week. They usually point to a missing document.
- Dashboards for cost and response time.
- A one-click way for users to flag a wrong answer, and someone who reads the flags.
Have a question about this guide? Write to us or ask Colony, our AI guide.