Sources
The Sources area of the console (/manage/sources) is where you add knowledge,
see whether it is current and find out why a document is not being used. It
replaces the earlier Ingestion, Documents, Connectors, Collections and Workflows
pages.
What is listed
Each row is one source:
- A connector: a website, drive or workspace that is kept in sync.
- Uploads: the files and pasted text added by hand to one collection.
Rows that need someone come first. The Next step column names the one thing a source needs: reconnect, review changes, resume, or see failures.
Add a source
Choose Add source. Pick a type and fill in its sign-in and settings. Google Drive, SharePoint and OneDrive, and Confluence sign in with the provider; what you typed is kept while you are away, and the sign-in is offered only when the deployment has that provider set up. Then choose the destination collection and, if you want, a schedule. Test connection checks the sign-in before anything is created. A new connector does not sync until you choose Sync now or its schedule comes due.
To add documents by hand, upload files or paste text into a collection. Uploads
larger than the server limit (INGESTION_MAX_UPLOAD_MB, 100 MB by default) are
refused before any run starts.
A source's tabs
| Tab | What it shows |
|---|---|
| Runs | Every sync, upload, reprocess and permission refresh, with what happened to each document |
| Documents | The documents the source brought in, with status |
| Access | Whether sharing at the source is mirrored, and group memberships |
| Settings | Connection, schedule, source settings, pause, delete |
Statuses
Connection
| Status | Meaning |
|---|---|
| Connected | The source accepted the sign-in |
| Needs attention | The source refused the sign-in, or a source that signs in through its provider has no working sign-in. Replace the credentials under Settings, or sign in again |
| Not checked | The sign-in has not been tested since it was saved |
| Not required | The source needs no sign-in |
Freshness
Measured against the source's target age for its last successful sync.
| Status | Meaning |
|---|---|
| Fresh | Within twice the target |
| Behind | Older than twice the target |
| Stale | Older than four times the target, never synced, or paused |
| Manual | No schedule: it syncs only when someone chooses Sync now |
Runs
| Status | Meaning | What to do |
|---|---|---|
| Queued | Waiting for a worker | Nothing |
| Running | In progress | Nothing, or cancel |
| Completed | Everything was applied | Nothing |
| Completed with errors | Some documents failed; the rest were applied | Open the run and read the reasons |
| Held for review | The change was larger than allowed. Nothing was applied | Review and approve, or fix the source |
| Failed | Nothing was applied, or the source could not be read | Read the reason, then retry |
| Cancelled | Stopped by someone | Retry if needed |
Documents
Queued, Processing, Indexed or Failed. Only indexed documents are used in answers.
Reading a run
Open a run to see how many documents reached each stage (discovered, fetched, parsed, chunked, embedded, indexed) and a list of every document that was skipped, failed or deleted. Each has the stage it reached, a reason and what to do about it. Common reasons:
| Reason | What to do |
|---|---|
| No text layer | The PDF is a scanned image. Replace it with a text PDF or run OCR first |
| Password-protected PDF | Remove the password and add the file again |
| Unsupported type | Export as PDF, Word, PowerPoint, Excel, text or Markdown |
| Source refused the sign-in | Reconnect the source |
| Source unreachable | Retry; if it continues, check the address |
| Embedding pending | The document is searchable by keyword. Open it and choose Repair embeddings |
| Document limit reached | Remove documents or upgrade, then retry |
| Removed at source | It was deleted at the source, so it was deleted here. No action |
| Unchanged | Already up to date. No action |
Held for review
A sync is held when the documents it would change and delete, together, exceed
a share of what the source already has indexed. The default threshold is 45%
(CONNECTOR_CHANGE_THRESHOLD); a connector can override it with
settings.change_threshold. The first sync of a source is never held.
A held sync applies nothing. This protects against a site redesign or a source that suddenly lists nothing. Open the run to see the figures. If the change is expected, choose Approve and sync (offered only on the source's newest run; once a later run exists, sync again instead); the sync runs again without the check and no extra credits are charged. If it is not expected, fix the source and sync again.
Retry and cancel
Retry starts a new run and keeps the old one as it was.
- A failed upload is retried from the copy the server kept. That copy is removed after
INGESTION_FAILED_INPUT_RETENTION_HOURS(24 by default); after that, add the document again. - A sync is started again from the source.
Cancel stops a run before its next document. What it already applied stays applied.
Schedules and pausing
A schedule is a cron expression in UTC. Scheduled syncs need
ENABLE_SOURCE_SCHEDULES (on by default) and, when jobs are dispatched through
Prefect, a running job worker.
A source that fails three syncs in a row is paused: scheduled syncs stop, what is indexed stays answerable and Sync now still works. Choose Resume once the cause is fixed. You can also pause a source yourself.
Document lookup
Document lookup (/manage/sources/lookup, organization administrators only)
answers "why was this not used?". Enter a document id, its address at the
source or part of its title. Optionally name a person and a project.
The trace runs these checks in order, and the first one that fails is the reason:
- Found in the index
- Indexed
- Searchable by meaning (every chunk has an embedding)
- Its source is connected to the project (when a project is named)
- Permissions are current (for documents that mirror source sharing)
- The person can read it (or, with no person named, public channels can)
A check can also come back as a caution: the document can be used, with a limit. The usual one is a document whose embeddings are incomplete, which is found only when a question shares its words until you choose Repair embeddings.
If the document is not in the index, the lookup shows the last attempt to bring it in and why that failed.
Live lookup
Records that should be read fresh at question time, not copied, are set up under Live lookup.
Deleting a source
Deleting a source deletes its run history. Its documents are kept unless you choose to delete them too.
API
Everything here is available through the Ingestion runs, Connectors and Documents APIs.