Skip to main content
If you mirror Chift data into your own store — a warehouse, a reporting database, an AI context store — you don’t want to re-read everything on every cycle. The data layer is built for exactly this: reads are served from Chift’s store rather than the live source, so you can fetch often and cheaply, and you can ask for only what changed since last time. The recommended strategy combines two complementary patterns:

Incremental reads

On every cycle, fetch only the records created or changed since your last fetch, using the updated_after parameter.

Periodic full fetch

On a slower cadence (for example weekly), fetch everything to realign your store — a safety net that also recovers any missed deletes.
Everything below is about reading from the data layer. Make sure the data layer is enabled on the connection and that you send the x-chift-datalayer: true header on these reads — see Activating the data layer. All examples send that header.

​
The sync model

Run a full load once to seed your store, then keep it fresh incrementally and reconcile periodically:
1

Initial full load

The first time, page through each resource with no updated_after filter and upsert every record into your store. This seeds your dataset.
2

Incremental reads

On each subsequent cycle, request the same resources with updated_after set to the start of your previous fetch. You get back only the records that were created or changed since, and you upsert them.
3

Periodic full fetch

On a slower cadence (for example once a week), do a full fetch and reconcile as a safety net for deletes — the account.datalayer.data_deleted webhook reports removals in real time, and the full pass recovers any you might have missed.

​
Incremental reads with updated_after

Virtually every list resource the data layer serves accepts an updated_after query parameter — across all verticals, not just accounting. When set, the data layer returns only the records whose last-updated timestamp is after the value you pass — created or modified records — so each incremental cycle stays small and fast regardless of how much history exists. This works uniformly because the data layer stamps every stored row with its own last-updated timestamp: it can honor updated_after on any resource it serves, even ones whose source system has no native incremental filter.
Since these filters are served from the data layer, they only work with the x-chift-datalayer: true header. Sending updated_after on a live read (no header) for a resource the source can’t filter natively is rejected with a 400 error, rather than silently returning unfiltered data.
cURL
A few rules to get right:
  • Always pass UTC. Use an ISO 8601 timestamp in UTC (for example 2024-01-31T15:00:00Z). UTC is the only format supported consistently across connectors.
  • Use the start of your previous fetch, not its end. Record when each cycle began, and pass that value as updated_after next time. This overlaps the window slightly and avoids missing records that changed while the previous fetch was running.
  • Reads still paginate. A large incremental window can still span several pages — keep following pagination until the page is empty. See Pagination limits.
  • Upsert, don’t append. A record can come back because it was updated, so key your writes on the record id and upsert.
Small, rarely-changing catalog resources — chart of accounts, journals, tax codes, product catalogs, and the like — can just be re-fetched in full each cycle: incremental reads are optional there, and a full pass is cheap and just as correct.

​
React to refresh webhooks

The data layer is only as fresh as its last sync, so there’s no point fetching more often than it refreshes. Instead of polling on a timer, subscribe to the datalayer webhooks and react when a refresh happens:
  • account.datalayer.refresh_initiated — a refresh started for a consumer.
  • account.datalayer.refresh_executed — the refresh finished for that consumer, with a status of success or error.
  • account.datalayer.data_changed — which resources changed in that refresh.
  • account.datalayer.data_deleted — which records were removed from the data layer in that refresh.
When you receive account.datalayer.refresh_executed (or data_changed), trigger an incremental read with updated_after set to the start of your previous fetch and upsert the results — this keeps your store aligned without guessing the cadence. When you receive account.datalayer.data_deleted, delete the listed records from your store (see Handling deletes). See Webhooks for the payload format and how to verify the signature.

​
Handling deletes

updated_after returns created and changed records — it can never return a record that was deleted in the source, because a deleted record simply stops appearing in reads. Incremental reads alone will therefore let deleted records linger in your store. The data layer surfaces deletes in two ways:
  • The account.datalayer.data_deleted webhook (recommended). Each refresh emits this event with the records it removed, so you can drop exactly those from your store as soon as the refresh completes — no extra reads, and deletes stay as fresh as your incremental updates.
  • A periodic full fetch, as a safety net. On a slower cadence (weekly is a good default), fetch each resource in full and reconcile against your store: any record you hold that is absent from the full result set has been removed at the source and should be deleted (or soft-deleted) on your side. This also recovers any delete notification you might have missed.
Don’t reconcile deletes from an incremental result set. A response filtered by updated_after is only a partial slice of the data — treating absence from it as a delete would wipe every record that simply didn’t change. Only reconcile deletes against a full, unfiltered fetch.

​
Putting it together

A typical loop:
1

Seed

Full load once. Record the timestamp at which the load started.
2

Stay fresh

On each account.datalayer.refresh_executed webhook (or your chosen cadence), read each incremental resource with updated_after = the start of your previous fetch, upsert the results, then store the new fetch-start timestamp. Apply any account.datalayer.data_deleted events to drop removed records.
3

Reconcile

Once a week, full-fetch every resource and reconcile against your store as a safety net — this recovers any delete you might have missed.
This keeps day-to-day traffic tiny while guaranteeing your store fully realigns with the source on every full pass.

Data layer overview

What the data layer is and how it stays fresh.

Activating the data layer

Enable it on a connection and use the header.