Connector framework
Overview
The connector framework is core to all connector types, including custom connectors and native connectors.
Permissions
A key feature of the connector framework is ensuring that users only see search results for content they have access to, matching their access in the external system. This relies on two components:
- A user's external identities (identity and group assignments)
- A document and which external identities have access to it (user identity and groups)
To support this generically for any connector, the framework introduces an external identities concept for users. This does not interfere with any other existing permissions or groups (main organisations, additional organisations, Power Users, and so on) within Interact.
External identities are also stored against indexed documents. When a search is performed, the existing security query (validating permissions and access) is extended to take the external identities of the user performing the query into account. Search results are only returned for documents that have one or more of the user's external identities, for example their user ID or the ID of a group or role they are in.
Note: External identities are automatically scoped to a specific Workplace Search connector, to keep them distinct from other Workplace Search external identities. This provides isolation between connectors.
Synchronised state
The connector framework can record state against users or docs, within the context of a connector, which the connector can use to optimise synchronisation. For example, it can record a delta token to query for changes to an item rather than fetching all content each time.
User state
For users, a collection of external identities (for each connector) is associated with them, as well as various state values used by the framework.
Doc state
For docs, several session values are recorded against any document. Each value can be set at a document level, and it is up to the connector implementation (or custom connector) to set them appropriately.
- Deletion Mode — Session or Explicit.
- State — arbitrary string content used by the connector to streamline synchronisation.
- Sync Session Deletion Candidate — used by the begin/end synchronisation session logic to identify which previously synchronised documents are no longer being synchronised and can be removed from the search index.
- Is Parent — used internally to improve performance when a parent document (such as a list or library) is no longer returned and the framework needs to delete all child documents efficiently.
- Materialised Path — used internally to improve performance in the same scenario. A child document's materialised path always starts with the parent materialised path, making it quick to discover large quantities of child documents.
- Allowed/Disallowed External Identities — used to limit access to users with the correct external identities.
Change detection
Native connectors typically use two change detection and synchronisation strategies. The SharePoint connector uses a combination of both.
Basic
Connectors that use the basic change detection strategy use the last updated date to track whether documents have changed. Documents synchronised this way typically have their deletion mode set to Session. Any documents not received during a synchronisation session, even if their last updated date has not changed, are automatically removed once the session completes.
Advanced (delta)
Connectors that use the advanced change detection strategy typically implement their own change detection and set the deletion mode of documents to Explicit. The connector explicitly tells the framework when to remove a document from the index, and the document is not automatically removed when the synchronisation session ends. This allows efficient use of delta queries (tracking changes only) for connectors that support it, such as SharePoint drives.
Digestion
Digestion is the process of synchronising content associated with a Workplace Search connector. For any tenant, each native connector is digested sequentially, in the following stages:
- Initialisation — set up the connection.
- Begin synchronisation session for users' external identities — flag any users with deletion mode set to Session as a synchronisation session deletion candidate.
- Digest users' external identities.
- End synchronisation session for users' external identities — delete any session deletion candidate users' external identities.
- Begin synchronisation session for docs — flag any docs with deletion mode set to Session as a synchronisation session deletion candidate.
- Digest docs.
- End synchronisation session for docs — delete any session deletion candidate docs.
- Finalise — close connections.
Initialisation and finalise
Use the initialisation and finalise stages to create in advance any services (and to open or test connections) used during digestion. They can also handle any synchronisation state management that the connector wants to handle itself.
Digest users
This stage returns users and their external identities.
Pairing Interact users with connector users: users are paired between external systems and Interact using the same mechanisms as other synchronisation features, based on usernames and emails.
External identities: a user may have multiple external identities (user ID, email, username, role ID, group ID, and so on). It is up to the connector to evaluate these comprehensively per user, while also evaluating the corresponding external identities for documents.
Digest docs
This stage returns the docs and a synchronisation result of either Add/Update or Delete.
During docs synchronisation, a connector can also influence the existing state of items beyond returning the Add/Update or Delete result:
- Error Message — displayed in logs and useful for diagnosing synchronisation problems.
- Deletion Mode — Session or Explicit.
- Synchronise State — an arbitrary string the connector can use as it likes, including during future synchronisation jobs.
Any doc returned includes a list of external identities allowed to access it.