Review, Start and Re-sync
Review
The Review step shows the connection, the records you chose, the document limit and a sample document built with your mapping. Use its links to go back and change the records or the mapping.
Then choose:
- Connect & Start Sync to save the source and start its first sync; or
- Connect & Approve, shown when your plan or the source's limit leaves no room for new documents. The source is saved and approved, but no documents are added.
What Start does
Start reads the first part of the list (up to 10 pages, within about 40 seconds) and adds the matching records it found, up to the room left under the limit and at most 10,000 at a time. The sync progress opens so you can follow the documents being processed.
- Credits. Start checks that you have enough credits for the new documents (20 credits each). If not, nothing is added and the message links to Billing. Documents are charged as they're processed, like any sync.
- Nothing is written on errors. If a page has a record without an ID, or two records share an ID, Start stops with nothing added. See Troubleshooting.
- No room. When there's no room, Start records the approval without a sync. Once there's room again (raise the limit or upgrade), the Review step shows Sync New Records.
After a Start, the Review step shows View Sync for that sync.
After the first Start
The source keeps going on its own; you don't need to press Start again.
- The import continues. Every 5 minutes, the app checks for sources with work to do. When the documents from the previous part have finished processing, it reads the next part of the list (up to 10 pages) from where the last part stopped.
- A pass ends when the list's last page has been read.
- The next pass starts 15 minutes after a pass ends, from the beginning of the list, and keeps repeating.
Each pass:
- adds new matching records, up to the limit;
- re-queues tracked records that changed (see below), whether or not they still match your conditions;
- leaves unchanged records alone: their details aren't fetched again.
Each reviewed version of the source's settings is started once. After that, the Review step shows View Sync for that sync (running or finished), and the rest of the work happens through scheduled checks. After an edit and its review, Start begins a new pass from the start of the list. Sync New Records, on a version approved without adding documents, continues the current pass from where it stopped if one is under way.
Following the source
After the first Start, the source's status chip on Sources shows where it is (Initial import, Up to date, Limit reached and so on), and Monitor opens scanning and processing in detail. Owners and Admins can pause and resume scanning, choose Check Now, and turn recurring checks and their interval (5 minutes to a day; 15 by default) on or off there. See Monitoring and scheduled checks.
See Monitoring and scheduled checks for what happens when a scheduled check fails.
How changes are detected
Changes are found from the list response, without fetching every record's detail:
- With a Last updated field, a record is re-queued when that value changes.
- Without one, a record is re-queued when its title, text fields or source link change in the list response. A change that appears only in the detail response isn't seen, so map Last updated if your list includes it.
A re-queued record's detail is fetched and its document is rebuilt:
- if the text, title and link are all unchanged, and the project's embedding settings and Enhanced Sync choice haven't changed since it was last processed, nothing is re-embedded and it costs 1 credit;
- otherwise it's processed and charged like any document sync.
What stays
- Documents stay when their records disappear from your API. Delete them from Documents if you don't want them.
- Documents stay when you change the conditions so their records no longer match, and they're still refreshed when those records change.
Review required and edits
Open the source's gear (Edit Custom API) to change it. Saving runs the same checks as setup, and a change to the request settings, the conditions, the mapping or the limit marks the source Review required on the Sources page. Nothing syncs from it until an Owner or Admin opens Review and chooses Start Sync, or Approve Without Adding when there's no room. Renaming the source or replacing its credentials doesn't need a review.
| You changed | After the review |
|---|---|
| The mapping or document text | Every tracked document is refreshed once, as later passes reach its record |
| The conditions | New matches are added; documents already imported stay |
| The limit | A higher limit lets more new matches be added on the following passes |
| Request settings (pagination, page size, fixed parameters) | Later requests use them |
The limit can't be set below the documents the source already tracks. Fields that identify the source can't be edited; see What can't change later.
If someone else saved the source while you were editing, you'll see This source changed since you opened it. The dialog reloads the latest version; nothing was started.
Re-syncing documents
Browse on the source opens Documents filtered to it. There:
- Re-sync on a document always fetches its detail and processes it in full, even if nothing changed, and is charged like any sync. Members can re-sync too.
- Tags work as for any document. Owners and Admins can delete documents.