Our paper Federated Instruction Tuning with DeepSeek: Towards Scalable and Private LLM Adaptation, with Fatiha Ait Baali and Tarik Agouti, was published at the 3rd International Conference on Federated Learning Technologies and Applications (FLTA 2025).
We fine-tuned the DeepSeek-R1-Distill-Qwen-1.5B model across institutions with federated learning, so that each site trains on its own data and only model updates are combined. On the medical-o1-reasoning-SFT dataset, the federated model reached 56.9% accuracy, close to centralized training, and it stayed stable when the data was split unevenly between sites.