Custom Connector
Connectors are part of the platform. A new source type is added to the server
code and its catalogue; it is not uploaded or registered through the API. Once
it is in the deployed server, it appears in GET /api/v1/connectors/types and
organizations create instances of it like any other type.
There are two ways to add one.
1. A declarative source
For a REST API that lists content the whole workspace may read, add a manifest
to connectors/rest_sources.py. A manifest states where to sign in, which
endpoint lists the content, how it paginates and how one record becomes a
document. dlt's REST source does the HTTP work. The nine
help centre, site and tracker sources
are built this way.
2. A connector class
For anything else, write a class that extends BaseConnector, register it with
@connector_registry.register("<type>") and add its entry to the catalogue in
connectors/catalogue.py. The class implements:
| Method | Purpose |
|---|---|
validate_credentials() | Whether the sign-in works. Used by the connection test |
list_sources() | Folders, databases or other scopes the operator can choose |
fetch_documents(since, limit) | The documents, as envelopes |
list_source_ids() | Optional. Every source_id now at the source, so removed documents are deleted |
fetch_deletions(since) | Optional. source_ids the source reports as deleted |
fetch_permissions() | Optional. Source sharing, for a connector that mirrors permissions |
A connector whose catalogue entry has auth="oauth2" gets a current token
from await self.access_token(); it never handles a client secret or a refresh
token.
If the source cannot be read, raise SourceError(reason, message). Do not
return an empty list: a refused sign-in must fail the sync. Use self.report()
to record a document that was left out or could not be turned into text.
The document envelope
Every document a connector returns must fit one shape:
{
"title": "Document title",
"content": "Document body as plain text or Markdown",
"source_id": "stable-id-owned-by-the-source",
"source_url": "https://source.example.com/doc/123",
"metadata": {
"content_type": "text/markdown",
"modified_at": "2026-01-15T10:00:00Z",
"language": "en",
"tags": ["policy", "onboarding"]
}
}| Field | Required | Notes |
|---|---|---|
title | Yes | Not blank |
content | Yes | Extracted text or Markdown, never raw bytes |
source_id | Yes | Stable and unique within the source. It is how an edit replaces the earlier version and how a removal is detected |
source_url | No | Address at the source; used as the citation link |
metadata | No | Named keys are content_type, modified_at, language, acl, tags, source_type, uploaded_by, uploaded_at. Other keys are kept |
The envelope has no top-level id or content_type: the identifier is
source_id and the content type is metadata.content_type. The sync engine
fills in a missing source_id from source_url, or from a hash of the content,
so older connectors keep working; new ones should always set it.
What the framework does
- Records each sync as an ingestion run with per-document outcomes
- Skips unchanged documents and replaces changed ones
- Deletes documents the source no longer lists, scoped to that connector
- Holds an unusually large change for review
- Runs the stored schedule and pauses a connector after three failed syncs in a row
- Applies plan document limits
Create an instance
POST /api/v1/connectors/
Content-Type: application/json
{
"connector_type": "my_source",
"name": "My custom source",
"collection_id": "<collection-id>",
"credentials": {"api_key": "<source-key>"},
"settings": {"api_url": "https://source.example.com"}
}The type must already be in the deployed server's catalogue.