Incremental synchronization

​

Why incremental synchronization matters

• Slower synchronizations — more data to transfer and process
• Higher load on the connected systems
• Unnecessary processing — comparing all records to detect changes is expensive
• Rate limiting risk — repeated full synchronizations can hit connector rate limits

​

How it works

1. Store the timestamp of your last successful synchronization
2. On the next synchronization run, pass that timestamp as updated_after
3. Process only the new and updated records
4. Store the new timestamp for the next run

​

Example request

GET /accounting/clients?updated_after=2024-10-15T14:30:00Z
Authorization: Bearer <token>

​

Implementing incremental synchronization

​

1. Check if updated_after is supported

​

2. Store your last synchronization timestamp

// Example: storing the last synchronization time in your database
const lastSyncTime = new Date('2024-10-15T14:30:00Z');

const response = await fetch(
  `https://api.chift.eu/consumers/{consumer_id}/accounting/clients?updated_after=${lastSyncTime.toISOString()}`,
  {
    headers: {
      'Authorization': `Bearer ${token}`,
    }
  }
);

// After successful processing:
await db.updateLastSync(consumerId, 'customers', new Date());

​

First synchronization: full initial load

GET /accounting/clients
Authorization: Bearer <token>

// First synchronization: full load
if (!hasPreviousSyncTime) {
  // Fetch all records in the desired date range
  const firstSyncResponse = await fetch(
    `https://api.chift.eu/consumers/{consumer_id}/accounting/clients`,
    { headers: /* ... */ }
  );
  
  // Process all records...
  
  // Store the synchronization time for next run
  lastSyncTime = new Date();
} else {
  // Subsequent synchronizations: incremental only
  const incrementalResponse = await fetch(
    `https://api.chift.eu/consumers/{consumer_id}/accounting/clients?updated_after=${lastSyncTime.toISOString()}`,
    { headers: /* ... */ }
  );
  
  // Process only changed records...
}

​

When updated_after is not supported

​

Strategy: Compare and identify changes

1. Fetch all records from the endpoint (you’ll need to paginate if there are many)
2. Compare each record with your stored version
3. Identify three types:
   • New records — in the API response but not in your system
   • Updated records — in your system but with changed values (check updated_at if available)
   • Deleted records — in your system but missing from the API response (optional: implement soft deletes or mark as archived)

​

Performance considerations

• Requesting higher pagination limits to reduce API calls
• Running these synchronizations less frequently (e.g., once per day instead of every hour)
• Consider using caching headers to avoid unnecessary fetches — see Caching in Chift API

​

Best practices

​

1. Always store timestamps in UTC

// ✅ Good
const syncTime = new Date().toISOString(); // "2024-10-15T14:30:00.000Z"

// ❌ Avoid
const syncTime = new Date().toString(); // Contains local timezone info

​

2. Be precise with timing

​

3. Handle synchronization failures gracefully

​

4. Monitor connector-specific behavior

• Records might be updated within a brief window after creation
• Deleted records might still appear for a short time with a deleted_at field
• Timestamps might have millisecond precision on some systems and second precision on others

​

5. Plan for the first synchronization carefully

• Running them at off-peak times
• Breaking them into smaller date ranges if the dataset is huge
• Using date_from and date_to to limit the initial load period

​

Troubleshooting

​

No records returned, but I know data was updated

• Is updated_after supported for this endpoint? (Check the API reference)
• Is your timestamp in UTC?
• Are you using the correct format? (ISO 8601: YYYY-MM-DDTHH:MM:SSZ)
• Is your stored timestamp actually before the data you expect? (Clocks can drift)

​

Different results between connectors

• Some update updated_at when related objects change (e.g., marking an invoice as paid)
• Others only update updated_at when the record itself changes
• Some don’t support updated_at at all

​

Timestamps are drifting

​

Memory issues with full synchronizations

• Fetch and process records in smaller batches rather than loading everything at once
• Use streaming if your API client supports it

​

Related documentation

• Pagination limits — Understanding page sizes and throughput
• Caching in Chift API — How cached responses work and when to bypass them
• Good practices for create and update requests — Handling mutations reliably
• API reference — See which endpoints support updated_after