Pragence

Sources

The Sources area of the console (/manage/sources) is where you add knowledge, see whether it is current and find out why a document is not being used. It replaces the earlier Ingestion, Documents, Connectors, Collections and Workflows pages.

What is listed

Each row is one source:

  • A connector: a website, drive or workspace that is kept in sync.
  • Uploads: the files and pasted text added by hand to one collection.

Rows that need someone come first. The Next step column names the one thing a source needs: reconnect, review changes, resume, or see failures.

Add a source

Choose Add source. Pick a type and fill in its sign-in and settings. Google Drive, SharePoint and OneDrive, and Confluence sign in with the provider; what you typed is kept while you are away, and the sign-in is offered only when the deployment has that provider set up. Then choose the destination collection and, if you want, a schedule. Test connection checks the sign-in before anything is created. A new connector does not sync until you choose Sync now or its schedule comes due.

To add documents by hand, upload files or paste text into a collection. Uploads larger than the server limit (INGESTION_MAX_UPLOAD_MB, 100 MB by default) are refused before any run starts.

A source's tabs

TabWhat it shows
RunsEvery sync, upload, reprocess and permission refresh, with what happened to each document
DocumentsThe documents the source brought in, with status
AccessWhether sharing at the source is mirrored, and group memberships
SettingsConnection, schedule, source settings, pause, delete

Statuses

Connection

StatusMeaning
ConnectedThe source accepted the sign-in
Needs attentionThe source refused the sign-in, or a source that signs in through its provider has no working sign-in. Replace the credentials under Settings, or sign in again
Not checkedThe sign-in has not been tested since it was saved
Not requiredThe source needs no sign-in

Freshness

Measured against the source's target age for its last successful sync.

StatusMeaning
FreshWithin twice the target
BehindOlder than twice the target
StaleOlder than four times the target, never synced, or paused
ManualNo schedule: it syncs only when someone chooses Sync now

Runs

StatusMeaningWhat to do
QueuedWaiting for a workerNothing
RunningIn progressNothing, or cancel
CompletedEverything was appliedNothing
Completed with errorsSome documents failed; the rest were appliedOpen the run and read the reasons
Held for reviewThe change was larger than allowed. Nothing was appliedReview and approve, or fix the source
FailedNothing was applied, or the source could not be readRead the reason, then retry
CancelledStopped by someoneRetry if needed

Documents

Queued, Processing, Indexed or Failed. Only indexed documents are used in answers.

Reading a run

Open a run to see how many documents reached each stage (discovered, fetched, parsed, chunked, embedded, indexed) and a list of every document that was skipped, failed or deleted. Each has the stage it reached, a reason and what to do about it. Common reasons:

ReasonWhat to do
No text layerThe PDF is a scanned image. Replace it with a text PDF or run OCR first
Password-protected PDFRemove the password and add the file again
Unsupported typeExport as PDF, Word, PowerPoint, Excel, text or Markdown
Source refused the sign-inReconnect the source
Source unreachableRetry; if it continues, check the address
Embedding pendingThe document is searchable by keyword. Open it and choose Repair embeddings
Document limit reachedRemove documents or upgrade, then retry
Removed at sourceIt was deleted at the source, so it was deleted here. No action
UnchangedAlready up to date. No action

Held for review

A sync is held when the documents it would change and delete, together, exceed a share of what the source already has indexed. The default threshold is 45% (CONNECTOR_CHANGE_THRESHOLD); a connector can override it with settings.change_threshold. The first sync of a source is never held.

A held sync applies nothing. This protects against a site redesign or a source that suddenly lists nothing. Open the run to see the figures. If the change is expected, choose Approve and sync (offered only on the source's newest run; once a later run exists, sync again instead); the sync runs again without the check and no extra credits are charged. If it is not expected, fix the source and sync again.

Retry and cancel

Retry starts a new run and keeps the old one as it was.

  • A failed upload is retried from the copy the server kept. That copy is removed after INGESTION_FAILED_INPUT_RETENTION_HOURS (24 by default); after that, add the document again.
  • A sync is started again from the source.

Cancel stops a run before its next document. What it already applied stays applied.

Schedules and pausing

A schedule is a cron expression in UTC. Scheduled syncs need ENABLE_SOURCE_SCHEDULES (on by default) and, when jobs are dispatched through Prefect, a running job worker.

A source that fails three syncs in a row is paused: scheduled syncs stop, what is indexed stays answerable and Sync now still works. Choose Resume once the cause is fixed. You can also pause a source yourself.

Document lookup

Document lookup (/manage/sources/lookup, organization administrators only) answers "why was this not used?". Enter a document id, its address at the source or part of its title. Optionally name a person and a project.

The trace runs these checks in order, and the first one that fails is the reason:

  1. Found in the index
  2. Indexed
  3. Searchable by meaning (every chunk has an embedding)
  4. Its source is connected to the project (when a project is named)
  5. Permissions are current (for documents that mirror source sharing)
  6. The person can read it (or, with no person named, public channels can)

A check can also come back as a caution: the document can be used, with a limit. The usual one is a document whose embeddings are incomplete, which is found only when a question shares its words until you choose Repair embeddings.

If the document is not in the index, the lookup shows the last attempt to bring it in and why that failed.

Live lookup

Records that should be read fresh at question time, not copied, are set up under Live lookup.

Deleting a source

Deleting a source deletes its run history. Its documents are kept unless you choose to delete them too.

API

Everything here is available through the Ingestion runs, Connectors and Documents APIs.

On this page