Middleware: making data exchanges between systems compatible

Your online store, your invoicing software and your customer relationship tool hold the same information, but each describes it in its own way. To make them talk, you need a translator. In IT, it is called middleware.

What is middleware?

Middleware is a software layer placed between systems. It receives data from one system, adapts it, then hands it to one or more others. Each system no longer needs to know the others: it only talks to the middleware.

Why systems do not understand each other

• Formats: JSON, XML, CSV, fixed-width files;
• Data models: “client” here, “customer” there; a full name on one side, first and last name separated on the other; dates and currencies written differently;
• Protocols: REST API, SOAP, files dropped on an SFTP server;
• Authentication: access key, token, OAuth;
• Pace: real time for one, nightly batch processing for the other.

What middleware does

• Transform: map fields, convert formats, dates, units;
• Validate: reject incomplete or inconsistent data before it reaches the destination;
• Enrich and filter: add information, pass on only what is useful;
• Route: send data to the right destination, or to several;
• Handle errors: retry, queue, alert;
• Trace and secure: centralise access and log exchanges.

Point-to-point links or middleware?

Without an intermediary, each system is linked directly to every other one. With four systems, that means six links to build and maintain; with five, ten. With middleware, each system has just one connection: four connections for four systems. Adding a system takes one more connection, and a change in one system only affects its own connection.

When to use one?

• you connect more than two systems, or will add more;
• formats or business rules differ significantly;
• you need reliability: retry on error, alerts, traces;
• you want to limit dependence on any one system.

Conversely, to simply link two tools that already offer a native integration, a direct connection is often enough.

How to choose?

• Where the data goes: hosting country, possible retention, processors (a data protection issue);
• Available connectors for your tools, and the ability to add more;
• Transformation capabilities: simple mappings or more elaborate rules;
• Monitoring: knowing when an exchange fails;
• Cost model: subscription, volume, flow complexity;
• Reversibility: being able to leave the solution without rebuilding everything.

This is the role of middleware: connectors to link your systems through their APIs, data transformation in between, hosting in France and no retention of the data that passes through. This is the approach followed in La Passerelle, one of my personal projects. To understand the basics of these exchanges, also read connecting your software with APIs.