API integration in detail: from source to destination

Connecting two programs through their APIs seems simple: fetch a piece of data here, send it there. In practice, a reliable flow involves a series of steps and special cases. Here is the detail, from source to destination.

1. The trigger

What starts the flow? Two approaches:

• Polling: you regularly ask the source whether there is anything new. Simple, but with a delay and useless calls;
• Webhook: the source notifies you itself, with an HTTP call, as soon as an event occurs. More responsive, but you need an address able to receive the call and check its authenticity.

2. Authentication

Each API requires proof of who is calling: an access key, a token carried in the request header, or the OAuth 2.0 protocol with user authorisation and tokens to renew. These secrets must be stored safely and limited to the rights needed.

3. The request

A web API (most often REST-style) is handled with HTTP methods: read (GET), create (POST), modify (PUT or PATCH), delete (DELETE). Data is generally exchanged in JSON format.

4. The transformation

Between source and destination, data is adapted: field mapping, conversion of types, dates and currencies, merging or splitting values, adding fixed values. This is the step that demands the most business thinking. See the role of middleware.

5. Sending and reading the response

The API response is first read through its status code: 2xx codes mean success, 4xx codes an error on the caller’s side (401: not authenticated, 403: not authorised, 404: not found, 429: too many requests), 5xx codes a problem on the server side. Each case calls for a different reaction.

6. Volumes: pagination and rate limits

• Pagination: an API returns results in pages; you must go through them all;
• Rate limits: the number of calls is capped; beyond it, the API answers 429 and you must wait;
• Batch processing: for large volumes, group the sends.

7. Duplicates and idempotency

A flow interrupted and then restarted may send the same data twice. To avoid this, rely on a shared identifier recognised on both sides, or on an idempotency key: sending the same request again then produces the same result, without creating a duplicate.

8. Errors and retries

• retry automatically, with a growing wait between attempts;
• tell temporary errors (network, 5xx) from permanent ones (invalid data);
• set aside failed exchanges to replay them later;
• alert when an exchange stays stuck.

9. Security and data protection

• encrypted communications (HTTPS);
• minimum rights for each access key;
• logs that avoid storing sensitive data;
• thought about retention: what is not kept cannot leak. This is the choice made for La Passerelle, one of my personal projects, which does not keep the data that passes through.

10. Monitoring

A flow that stops without anyone knowing is worse than no flow at all. You need to watch execution, errors and the availability of the connected systems (monitoring).

A checklist before launching a flow

1. the trigger is chosen (polling or webhook);
2. access is created with limited rights and stored safely;
3. the field mapping is documented;
4. error, duplicate and rate-limit cases are planned for;
5. an alert exists in case of failure;
6. the personal data involved is identified.

For the basics, see connecting your software with APIs: where to start.