Most machine learning starts with the same step: collect all the data in one place, then train a model on it. For a lot of the data that matters most, that first step is impossible. Hospitals cannot send patient images to a shared server. Factories will not hand their process data to a competitor. Phones should not upload everything their owners type.
Federated learning turns the usual approach around. Instead of bringing the data to the model, it sends the model to the data.
How it works
Picture five hospitals that each hold their own chest X-rays and want a better model for detecting pneumonia.
- A coordinating server sends the same starting model to every hospital.
- Each hospital trains that model for a short while on its own images, behind its own firewall.
- Each hospital sends back only the model update: the changes to the model’s numbers, not the images.
- The server combines the updates into one improved model. The most common way, called FedAvg, is a weighted average.
- The improved model goes back out, and the cycle repeats until it stops getting better.
No image ever leaves a hospital. The final model has still learned from the X-rays of all five, which is usually far more than any single hospital could train on alone.
Where it is used
- Healthcare. In one of our studies, a federated model told acute lymphoblastic leukemia apart from normal cells with 95.6% accuracy, close to a model trained on pooled data, while every image stayed with the client that owned it (paper).
- Language. FedQAS trains question-answering models across organisations whose documents are private.
- Industry. In the EU project ProcTwin, we use federated learning so that steel plants can improve shared digital-twin models without exposing their process data.
- Phones. Keyboard suggestions and voice features on many phones are trained this way, across millions of devices.
The hard parts
Federated learning removes the need to share data. It does not make every problem go away.
The data is uneven. One hospital sees mostly older patients, another mostly children. Models trained on such different slices pull in different directions, and a plain average can end up worse than any of them. Much current research, including ours on clients that join training late (FedRecall), is about combining updates more cleverly.
Devices are small and networks are slow. Sending a full deep-learning model back and forth every round is expensive at the edge. We showed that training only a random subset of layers on each device cut data transfer by around 75% when a quarter of the layers were trained, without hurting the final model (paper).
Updates can leak. A model update is not the data, but it is not nothing either. With enough effort, attackers can sometimes recover details of the training data from it. Real deployments therefore add protections such as secure aggregation, where the server only ever sees the combined update, and differential privacy, which adds carefully measured noise.
Some clients cheat. A client can send poisoned updates, for example by training on deliberately wrong labels. We proposed an aggregation rule that weighs each update by how well it performs on a small clean validation set, which held off both label flipping and hidden backdoors in our experiments (paper).
From one server to many
The classic set-up has one central server, which becomes a bottleneck and a single point of failure as the number of clients grows. Hierarchical federated learning adds a middle layer. Clients report to a regional coordinator, the regions report to a global one, and each level aggregates what it receives.
This is the idea behind Decentrai, my upcoming open-source framework: clients train locally, regional coordinators aggregate their updates, and a global orchestrator builds the next model. It also includes 18 aggregation algorithms, differential privacy and LoRA fine-tuning of language models.
When to reach for it
Federated learning is worth the extra engineering when three things are true:
- the data cannot or should not be moved;
- it is spread across several owners or devices;
- each owner alone has too little data for a good model.
If you can legally and practically put the data in one place, centralised training is simpler and usually a little more accurate. If you cannot, federated learning is often the only way to learn from it at all.
Want to go further? Start with the publications filtered to federated learning, or look at the open thesis topics if you would like to work on these problems with us.