Introduction to Federated Learning
25 Aug 2026
When I was in high school preparing group projects, my classmates and I often faced a dilemma when compiling our notes. We each had personal study diaries full of private annotations, shorthand tricks, and thoughts that we did not want everyone else to read.
At first we tried sharing the entire notebooks, but people felt uncomfortable having their private pages scrutinized. So we changed strategy: each of us studied at home, extracted only the high-level conclusions from our chapters, and brought those summaries to class to assemble the master project. No personal notes left our desks, yet the collective project was smarter than what any single one of us could have produced alone.
In Data Science we do something almost identical when we train models on decentralized data without shipping the raw records around: Federated Learning. The data stays on local devices—phones, regional hospitals, factory sensors—and only model updates (gradients, weights, or similar summaries) travel to a coordinator, or sometimes only between peers.
There are several ways to partition the problem
Architectures are usually classified by how features and samples are split across participants:
- Horizontal Federated Learning (HFL): Same feature space, different sample IDs. In the study-group analogy, two students read different textbooks that cover the same syllabus. A classic industrial example is next-word prediction across thousands of phone keyboards.
- Vertical Federated Learning (VFL): Same sample IDs (or overlapping users), different feature spaces. Think of two teachers holding distinct records—academic scores versus athletic performance—for the exact same group of students. A bank and an e-commerce platform collaborating over shared clients sits here.
- Federated Transfer Learning (FTL): Little overlap in both samples and features. Students from different grades and subjects trying to transfer study habits rather than share identical notes. Domain adaptation does the bridging work between silos that barely touch.
Aggregation and orchestration
Local updates still need a way to become a global model:
- Federated Averaging (FedAvg) is the baseline most people start from. Clients run local SGD on private data; a server averages the resulting weights, usually weighted by how much data each client saw.
- FedProx adds a proximal term so local updates cannot drift too far from the global model. That matters when hardware and data distributions differ wildly across clients (the Non-IID headache everyone mentions in FL papers).
- Decentralized / peer-to-peer FL drops the central server. Clients talk to topological neighbors and reach consensus without a single orchestrator that holds the full picture.
None of these choices is free. FedAvg is simple and often good enough; FedProx buys robustness under heterogeneity at the cost of an extra hyperparameter to tune; peer-to-peer removes a single point of failure but makes debugging and auditing harder.
Trade-offs that show up outside the paper
- Privacy vs. utility: Keeping raw data local is the point, but gradients and embeddings can still leak information if you are careless. Real deployments often pair FL with secure aggregation, differential privacy, or both—and each of those moves the accuracy needle.
- Communication vs. compute: Many rounds of weight exchange can cost more than the local training itself, especially on constrained devices. Fewer rounds or heavier local epochs help the network bill and hurt convergence if clients are too dissimilar.
- Centralized simplicity vs. organizational reality: A single data lake is easier to train on. FL exists because hospitals, banks, and telecoms often cannot build that lake—even when they would like to.
Federated Learning is really a negotiation between what you are allowed to move and what you need the model to learn. Regulations tighten, warehouses remain attractive targets, and more teams discover that “just centralize everything” is not an option. The study-group trick still holds: share the conclusions, keep the diaries at home, and only then ask whether the assembled project is good enough to ship.
In later essays we can dig into horizontal versus vertical pipelines in more detail, and into what changes when the local models are not neural nets but random forests or gradient boosting—settings that show up surprisingly often once you leave the textbook examples.