> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chift.eu/llms.txt
> Use this file to discover all available pages before exploring further.

# Incremental synchronization

When you synchronize your end-users' financial data with the Unified API, you want to fetch only what's new or changed since your last synchronization. Incremental synchronization is the most efficient way to keep data up-to-date without fetching the entire dataset on every run.

## Why incremental synchronization matters

Fetching all records on every synchronization has real costs:

* **Slower synchronizations** — more data to transfer and process
* **Higher load** on the connected systems
* **Unnecessary processing** — comparing all records to detect changes is expensive
* **Rate limiting risk** — repeated full synchronizations can hit connector rate limits

With incremental synchronization, you only fetch what's changed since your last successful synchronization.

## How it works

Most Chift Unified API collection endpoints support the `updated_after` parameter, which returns only records created or updated on or after a specified timestamp.

1. Store the timestamp of your last successful synchronization
2. On the next synchronization run, pass that timestamp as `updated_after`
3. Process only the new and updated records
4. Store the new timestamp for the next run

### Example request

```http theme={null}
GET /accounting/clients?updated_after=2024-10-15T14:30:00Z
Authorization: Bearer <token>
```

**Response:** Only clients created or updated after `2024-10-15T14:30:00Z`.

***

## Implementing incremental synchronization

### 1. Check if `updated_after` is supported

The API reference for each endpoint documents whether `updated_after` is available. Not all connectors support it for every endpoint, and some newer connectors may have limited support.

### 2. Store your last synchronization timestamp

Keep the timestamp of your last successful synchronization for each consumer and each endpoint. This is the value you'll pass as `updated_after` on the next run.

```javascript theme={null}
// Example: storing the last synchronization time in your database
const lastSyncTime = new Date('2024-10-15T14:30:00Z');

const response = await fetch(
  `https://api.chift.eu/consumers/{consumer_id}/accounting/clients?updated_after=${lastSyncTime.toISOString()}`,
  {
    headers: {
      'Authorization': `Bearer ${token}`,
    }
  }
);

// After successful processing:
await db.updateLastSync(consumerId, 'customers', new Date());
```

***

## First synchronization: full initial load

On your first synchronization for a consumer, you don't have a previous timestamp to use. Instead, use the `date_from` and `date_to` parameters (if supported) to load historical data, or simply fetch all records without `updated_after`.

```http theme={null}
GET /accounting/clients
Authorization: Bearer <token>
```

After this initial load completes successfully, store the timestamp and switch to incremental synchronizations with `updated_after`.

```javascript theme={null}
// First synchronization: full load
if (!hasPreviousSyncTime) {
  // Fetch all records in the desired date range
  const firstSyncResponse = await fetch(
    `https://api.chift.eu/consumers/{consumer_id}/accounting/clients`,
    { headers: /* ... */ }
  );
  
  // Process all records...
  
  // Store the synchronization time for next run
  lastSyncTime = new Date();
} else {
  // Subsequent synchronizations: incremental only
  const incrementalResponse = await fetch(
    `https://api.chift.eu/consumers/{consumer_id}/accounting/clients?updated_after=${lastSyncTime.toISOString()}`,
    { headers: /* ... */ }
  );
  
  // Process only changed records...
}
```

***

## When `updated_after` is not supported

Some connectors don't support incremental synchronization for certain endpoints. When this happens, you'll need to manually track changes by comparing the current response with your stored records.

### Strategy: Compare and identify changes

1. **Fetch all records** from the endpoint (you'll need to paginate if there are many)
2. **Compare** each record with your stored version
3. **Identify** three types:
   * **New records** — in the API response but not in your system
   * **Updated records** — in your system but with changed values (check `updated_at` if available)
   * **Deleted records** — in your system but missing from the API response (optional: implement soft deletes or mark as archived)

### Performance considerations

Manually comparing all records is less efficient than true incremental synchronization. We recommend:

* Requesting higher [pagination limits](/guides/unified-api/pagination) to reduce API calls
* Running these synchronizations less frequently (e.g., once per day instead of every hour)
* Consider using caching headers to avoid unnecessary fetches — see [Caching in Chift API](/guides/unified-api/caching)

***

## Best practices

### 1. Always store timestamps in UTC

Always work with UTC timestamps. Many timezone bugs come from mixing timezones.

```javascript theme={null}
// ✅ Good
const syncTime = new Date().toISOString(); // "2024-10-15T14:30:00.000Z"

// ❌ Avoid
const syncTime = new Date().toString(); // Contains local timezone info
```

### 2. Be precise with timing

Store the exact timestamp of when the synchronization completed, not rounded times. This ensures you don't miss edge-case records created at exact second boundaries.

### 3. Handle synchronization failures gracefully

If a synchronization fails partway through, don't update your stored timestamp. Only update it after all records have been successfully processed and stored.

### 4. Monitor connector-specific behavior

Some connectors have quirks:

* Records might be updated within a brief window after creation
* Deleted records might still appear for a short time with a `deleted_at` field
* Timestamps might have millisecond precision on some systems and second precision on others

Test your synchronization logic with your integrated connectors to understand their behavior.

### 5. Plan for the first synchronization carefully

Initial synchronizations can be large. Consider:

* Running them at off-peak times
* Breaking them into smaller date ranges if the dataset is huge
* Using `date_from` and `date_to` to limit the initial load period

***

## Troubleshooting

### No records returned, but I know data was updated

Check:

* Is `updated_after` supported for this endpoint? (Check the API reference)
* Is your timestamp in UTC?
* Are you using the correct format? (ISO 8601: `YYYY-MM-DDTHH:MM:SSZ`)
* Is your stored timestamp actually *before* the data you expect? (Clocks can drift)

### Different results between connectors

Connector behavior varies:

* Some update `updated_at` when related objects change (e.g., marking an invoice as paid)
* Others only update `updated_at` when the record itself changes
* Some don't support `updated_at` at all

Always test incremental synchronization with each connector you plan to support.

### Timestamps are drifting

If your synchronizations keep missing recent data, you may have a clock drift issue between your system and Chift's servers. Monitor the `server_time` in API responses and adjust your next synchronization time accordingly.

### Memory issues with full synchronizations

If you're doing manual comparison (for connectors without `updated_after` support) and hitting memory limits with large datasets:

* Fetch and process records in smaller batches rather than loading everything at once
* Use streaming if your API client supports it

***

## Related documentation

* [Pagination limits](/guides/unified-api/pagination) — Understanding page sizes and throughput
* [Caching in Chift API](/guides/unified-api/caching) — How cached responses work and when to bypass them
* [Good practices for create and update requests](/guides/unified-api/good-practice-create-and-update-requests) — Handling mutations reliably
* [API reference](/api-reference) — See which endpoints support `updated_after`