An assistant can only be as good as what it reads. Most bad answers we see trace back to ingestion. A scanned PDF went in with no text. A page header was repeated on every passage. An old version of a policy sat next to the new one.
This post follows a single file from upload to the search index. It describes what the current code does, including the cases where Pragence refuses a file instead of indexing something useless.
Before the steps: uploads run in the background
When you upload a file through the API, Pragence runs the quick checks first, such as your plan’s quota. It then accepts the upload with HTTP 202 and returns a run ID. The processing runs in a background worker.
You can follow each run through the ingestion runs API. It lists runs with their status, the stage they are in and how many documents reached each stage. A run’s detail lists every document that was skipped, failed or deleted, with the reason. A failed run can be retried, from the copy of the upload the server keeps for a limited time, and a run that has not finished can be cancelled. Connector syncs appear in the same list.
Connectors such as Google Drive and Notion feed the same pipeline. Every source, whatever its format, is reduced to one standard document shape before ingestion. That shape requires a source_id: a stable identifier owned by the source system, such as a file path or a record ID. It is what lets Pragence tell an edited document apart from a new one.
Step 1: identify the file by its contents
Pragence reads the first bytes of a file to decide what it is. The file name and extension do not decide it. A PDF renamed to .txt is still read as a PDF.
This matters more than it sounds. Decoded as plain text, a PDF looks mostly printable. Without this check, raw PDF syntax such as %PDF-1.7 ... obj can end up in the index as if it were document text.
Step 2: extract the text, or refuse
For PDFs, Pragence reads the text layer and judges whether it is usable. A file is rejected, with a reason code, when:
- it is password-protected;
- it has no usable text layer, which usually means it is a scan (OCR is not configured, so you can supply the text through the manual ingest endpoint instead);
- the text layer is mostly unprintable characters;
- more than half of the text is unmapped glyph codes, the sign of a broken font encoding.
PDFs of three or more pages also lose their running headers and footers. A short line at the top or bottom of a page that repeats on most pages is removed, with page numbers normalised so “Page 3 of 10” and “Page 7 of 10” count as the same line. A page is never emptied this way.
Word, PowerPoint and Excel files are ZIP archives. Before anything is decompressed, Pragence checks the archive’s directory: at most 5,000 entries, 100 MB per entry, 300 MB in total, and a compression ratio no higher than 200 to 1. An archive that fails is refused instead of expanded. Spreadsheets are read up to 50,000 rows per sheet.
A file with no extractable text fails loudly. The run records it as a failed document with the reason. It never becomes an empty document.
Step 3: check whether it is new, changed or a duplicate
Each document gets a SHA-256 hash of its content. If a document with the same source_id already exists:
- the same hash means nothing changed, so it is skipped;
- a different hash means the document was edited, so the old version and all its passages are replaced.
A document with no source_id that matches the content of an existing document in your organization is reported as a duplicate. This keeps old and new versions of the same policy from competing in search results.
Step 4: split it into passages
The text is split into passages, the units that search returns and the answer cites. Markdown sources are split along their structure. Other text is split by word count with a small overlap between neighbours.
Each passage after the first also keeps the last 200 characters of the passage before it. That carry-over helps when a passage starts with “She” or “This fee” and the name or amount sits just before the split.
Step 5: index for meaning and for exact words
Each passage gets an embedding, a list of numbers that represents its meaning, for vector search. The passages are also searchable by exact words through a BM25 keyword index. Only when every passage is stored is the document marked as indexed. Search ignores documents that are still being processed, so a half-ingested file never appears in an answer.
Three things happen after indexing:
- Caches are invalidated. Your organization’s content version is bumped, so cached answers and search results built on the old content are not reused.
- The spelling vocabulary is refreshed. Query-side spelling correction works from the words in your own documents. New terms, like a product name, become correctable.
- Permissions travel with the document. By default a document is readable by everyone in your organization. Connectors that support it can mirror the source’s sharing (per person, per domain or per group) onto the document when you turn
mirror_permissionson. Search then applies those permissions on every request.
What this means when you set up a project
Upload a few representative files first and watch the runs. A rejected scan or a garbled PDF shows up as a failed run with a reason, which is easier to fix before launch than after. Give documents you will update a stable source_id, so a new version replaces the old one instead of sitting beside it. Then ask real questions and open the citations to check that the right passages come back.