Federated Fine-Tuning of Foundation Models for Prostate Cancer Detection

Prostate cancer is one of the most commonly diagnosed cancers worldwide, yet accurately identifying clinically significant cases from medical imaging remains a challenge, often requiring invasive biopsies. Data-driven approaches, particularly deep learning models trained on MRI data, offer a promising path toward less invasive diagnosis. However, training such models requires large volumes of sensitive patient data, raising serious privacy concerns.

Federated Learning (FL) addresses this challenge by enabling multiple institutions to collaboratively train models without sharing raw data. Yet, training large models across a federated network introduces significant communication overhead and performance trade-offs. Researchers from Siemens, as part of the FLUTE project’s Work Package 9, are developing new algorithms that combine foundation models with efficient federated fine-tuning and compression techniques to make privacy-preserving prostate cancer detection both scalable and performant.

Efficient federated fine-tuning with foundation models

Rather than training models from scratch, this work leverages foundation models — large models pre-trained on unlabeled medical imaging data — and fine-tunes them for the specific task of classifying prostate cancer from MRI. Pre-trained models converge faster and produce better results, particularly when using a Swin Transformer architecture, which outperformed both ResNet50 and Vision Transformers in the experiments.

To reduce the communication burden in FL, additive fine-tuning methods are employed: instead of updating and transmitting the entire model, only a small set of additional parameters is trained. Two approaches were explored — adapters in the final stage and Low-Rank Adaptation (LoRA). Both methods reduce the number of trainable parameters by up to 99%, translating to data transfers as low as 0.8 MB per client per training round. Crucially, these fine-tuning methods also preserve the quality of pre-trained embeddings, outperforming full-model fine-tuning in the federated setting, where averaging all parameters across heterogeneous data distributions can degrade performance.

Compression, distillation, and key insights

Hybrid approaches that combine LoRA with quantization further improve efficiency, significantly cutting communication costs with only minimal impact on accuracy. These techniques are especially relevant for real-world deployments where bandwidth and computational resources vary across clinical sites.

Experiments were conducted on a diverse dataset of approximately 2,040 patients from 17 clinical collections spanning North America, Europe, and Asia, using T2, ADC, and DWI MRI sequences. A lesion-based classification approach — focusing on cropped image regions around each lesion — proved significantly more effective than whole-organ analysis. Additionally, using FedAdam as the aggregation strategy yielded noteably better results than the commonly used FedAvg, suggesting that more sophisticated aggregation methods deserve greater attention in federated learning research.

These findings provide a clear roadmap for the FLUTE platform: integrating both final-stage and LoRA-based fine-tuning options, adopting advanced aggregators, and incorporating quantization to serve institutions with varying communication constraints — all while preserving patient privacy.

Alexandru Serban, Cosmin Hatfaludi