Retrieval

Cited or declined: how Pragence decides not to answer

What happens between a question and a cited answer, and the three cases where the assistant says it does not know.

A support assistant that answers every question is easy to build. The hard part is the questions it should not answer. If your refund policy does not cover a case, a guess that sounds confident is worse than no answer. Your customer acts on it, and your team cleans up after it.

Pragence is built so that an answer points back to the passages it came from. When the passages are not there, it says so. This post walks through how that works today, including the setting you should check before you go live.

Step one: find the evidence

Every question starts with a search of the sources connected to the project. Two searches run side by side:

  • A vector search, which finds passages with a similar meaning even when the wording differs.
  • A keyword search (BM25, a standard ranking method for exact words), with spelling correction applied to the question first, so “refnd polcy” still finds “refund policy”.

The two result lists are merged with reciprocal rank fusion. That method rewards passages that rank well in either list. A reranker model then scores each candidate against the question, and only the top passages go to the language model.

The search only sees what the person asking is allowed to read. A visitor on your website gets documents shared with the whole organization. Documents that carry sharing mirrored from Google Drive, for example, are only readable by the people that sharing names. If a mirrored permission has not been confirmed within 24 hours by default, the document is treated as unreadable until the next sync. Cached answers are rechecked against current permissions before they are reused.

Step two: answer with numbered sources

The model receives the passages numbered [1], [2] and so on, with an instruction to cite them and to say so when the context does not contain enough information.

Each citation in the response carries the document and passage it refers to, the file name, the relevance score and an excerpt of the passage text. The website widget and the portal show the citations under each answer, so the reader can see which documents it relied on.

Step three: check the answer against the sources

After the answer is written, Pragence compares it to the cited passages sentence by sentence. Each sentence is matched to the closest span of text in its source. The match is graded as exact, fuzzy, partial or none, and the character positions of the matching text are returned. This lets the interface highlight the supporting words in the source.

Those grades roll up into a grounding score between 0 and 1. A second check asks how relevant the retrieved passages were to the question. The answer’s confidence is the lower of the two. If it falls below the uncertainty threshold, 0.5 by default, the answer is flagged as uncertain. The shared React chat component shows a low-confidence marker on those answers, with a note that the answer may not be fully supported by the sources.

When Pragence does not answer

There are three cases where the assistant declines or warns instead of answering normally.

The search could not run

If the database, the index or the embedding provider fails, retrieval returns nothing. Pragence treats this differently from an empty result. The assistant tells the person there was a temporary problem and asks them to try again. It does not fall back to the model’s general knowledge in this case, whatever the settings say. An outage should never look like a confident answer.

The sources have nothing relevant

If the search runs and finds nothing, the behaviour depends on one deployment setting, ALLOW_GENERAL_KNOWLEDGE_FALLBACK.

The setting is off by default. The assistant then replies that it could not find anything about the question in the knowledge base and cannot answer it reliably. It suggests rephrasing or checking the documentation directly.

With the setting on, the model may answer from its general knowledge. That answer is labelled. It starts with a note that no documents were found and a “General Knowledge” marker, it has no citations, its confidence is recorded as zero, and it is flagged as uncertain. Analytics records its source as general knowledge, so you can count how often it happens.

Turn it on only for an internal assistant where a labelled guess is more useful than no answer:

ALLOW_GENERAL_KNOWLEDGE_FALLBACK=true

The question is out of scope

An optional topic check, ENABLE_TOPIC_BOUNDARY, runs before retrieval. You describe the assistant’s domain in one line, and questions outside it are refused before any search runs. It is off by default.

If you build on the search API directly, you can set your own floor. Each request accepts min_relevance, a value from 0 to 1 that defaults to 0.1. Passages scored below it by the reranker are dropped. Set it to null to receive every candidate. The floor is only applied when reranker scoring succeeds, so a scoring failure does not silently empty your results.

What citations do and do not prove

A citation shows where an answer came from. It does not prove that your documents are complete or current. If your policy page is out of date, a well-cited answer will repeat the outdated policy. Our own documentation says the same thing: test important questions against your own material before anyone outside your team sees the answers.

The practical routine is short. Connect the sources, ask the questions your customers actually ask, open the citations, and look at the answers flagged as uncertain. Those flags usually point at a gap in the content. The fix is usually a new or updated document.

Next steps

See it answer from your sources.

30 minutes. Bring a real question and the document that answers it. We’ll show you the answer on chat, voice and WhatsApp, the citation, and what happens when the answer isn’t there.