Pragence

Custom Connector

Connectors are part of the platform. A new source type is added to the server code and its catalogue; it is not uploaded or registered through the API. Once it is in the deployed server, it appears in GET /api/v1/connectors/types and organizations create instances of it like any other type.

There are two ways to add one.

1. A declarative source

For a REST API that lists content the whole workspace may read, add a manifest to connectors/rest_sources.py. A manifest states where to sign in, which endpoint lists the content, how it paginates and how one record becomes a document. dlt's REST source does the HTTP work. The nine help centre, site and tracker sources are built this way.

2. A connector class

For anything else, write a class that extends BaseConnector, register it with @connector_registry.register("<type>") and add its entry to the catalogue in connectors/catalogue.py. The class implements:

MethodPurpose
validate_credentials()Whether the sign-in works. Used by the connection test
list_sources()Folders, databases or other scopes the operator can choose
fetch_documents(since, limit)The documents, as envelopes
list_source_ids()Optional. Every source_id now at the source, so removed documents are deleted
fetch_deletions(since)Optional. source_ids the source reports as deleted
fetch_permissions()Optional. Source sharing, for a connector that mirrors permissions

A connector whose catalogue entry has auth="oauth2" gets a current token from await self.access_token(); it never handles a client secret or a refresh token.

If the source cannot be read, raise SourceError(reason, message). Do not return an empty list: a refused sign-in must fail the sync. Use self.report() to record a document that was left out or could not be turned into text.

The document envelope

Every document a connector returns must fit one shape:

{
  "title": "Document title",
  "content": "Document body as plain text or Markdown",
  "source_id": "stable-id-owned-by-the-source",
  "source_url": "https://source.example.com/doc/123",
  "metadata": {
    "content_type": "text/markdown",
    "modified_at": "2026-01-15T10:00:00Z",
    "language": "en",
    "tags": ["policy", "onboarding"]
  }
}
FieldRequiredNotes
titleYesNot blank
contentYesExtracted text or Markdown, never raw bytes
source_idYesStable and unique within the source. It is how an edit replaces the earlier version and how a removal is detected
source_urlNoAddress at the source; used as the citation link
metadataNoNamed keys are content_type, modified_at, language, acl, tags, source_type, uploaded_by, uploaded_at. Other keys are kept

The envelope has no top-level id or content_type: the identifier is source_id and the content type is metadata.content_type. The sync engine fills in a missing source_id from source_url, or from a hash of the content, so older connectors keep working; new ones should always set it.

What the framework does

  • Records each sync as an ingestion run with per-document outcomes
  • Skips unchanged documents and replaces changed ones
  • Deletes documents the source no longer lists, scoped to that connector
  • Holds an unusually large change for review
  • Runs the stored schedule and pauses a connector after three failed syncs in a row
  • Applies plan document limits

Create an instance

POST /api/v1/connectors/
Content-Type: application/json

{
  "connector_type": "my_source",
  "name": "My custom source",
  "collection_id": "<collection-id>",
  "credentials": {"api_key": "<source-key>"},
  "settings": {"api_url": "https://source.example.com"}
}

The type must already be in the deployed server's catalogue.

On this page