Author: Fatema Siddika (Iowa State University)
Advisor: Ali Jannesari (Iowa State University)
Abstract: This dissertation addresses two fundamental challenges in distributed machine learning: client heterogeneity and continual adaptation to new tasks. It develops efficient fine-tuning and representation-learning frameworks that let distributed models acquire new knowledge and align it across clients without communication bottlenecks or catastrophic forgetting.
First, we introduce a dual-distilled federated learning framework for clients with heterogeneous systems and data. Trainable global prototypes with adaptive margins align local representations and logits without requiring a shared model architecture, enabling robust knowledge transfer and stable convergence.
Next, we propose a communication-efficient federated framework for large language models (LLMs) based on representation fine-tuning. By intervening on hidden states rather than weights, and combining All-But-Me aggregation with dynamic test-time computing, it reduces communication cost while balancing local personalization and global generalization.
We then present SETA, a mixture of sparse experts for task-agnostic continual learning in LLMs. SETA resolves the plasticity–stability dilemma by structurally decomposing the parameter space into shared and task-specific experts, protecting shared knowledge from semantic drift while preserving task-specific features.
Finally, we introduce FedSEAM, which extends sparse experts to federated continual learning. Clients learn experts on private task streams, and the server merges them through subspace agreement: it combines only the update directions clients share, keeps disagreeing directions as client-private experts, and protects prior-task subspaces from being overwritten.
Together, these contributions chart a path toward scalable, resource-aware machine learning systems that bridge decentralized efficiency and continual real-world deployment, learning new tasks across privacy-constrained environments without forgetting.
Thesis Canvas: pdf