Doctoral Showcase Archive

Efficient Representation Learning for Heterogeneous Systems and Continual Task Adaptation


Author: Fatema Siddika (Iowa State University)

Advisor: Ali Jannesari (Iowa State University)

Abstract: This dissertation addresses two fundamental challenges in distributed machine learning: client heterogeneity and continual adaptation to new tasks. It develops efficient fine-tuning and representation-learning frameworks that let distributed models acquire new knowledge and align it across clients without communication bottlenecks or catastrophic forgetting.

First, we introduce a dual-distilled federated learning framework for clients with heterogeneous systems and data. Trainable global prototypes with adaptive margins align local representations and logits without requiring a shared model architecture, enabling robust knowledge transfer and stable convergence.

Next, we propose a communication-efficient federated framework for large language models (LLMs) based on representation fine-tuning. By intervening on hidden states rather than weights, and combining All-But-Me aggregation with dynamic test-time computing, it reduces communication cost while balancing local personalization and global generalization.

We then present SETA, a mixture of sparse experts for task-agnostic continual learning in LLMs. SETA resolves the plasticity–stability dilemma by structurally decomposing the parameter space into shared and task-specific experts, protecting shared knowledge from semantic drift while preserving task-specific features.

Finally, we introduce FedSEAM, which extends sparse experts to federated continual learning. Clients learn experts on private task streams, and the server merges them through subspace agreement: it combines only the update directions clients share, keeps disagreeing directions as client-private experts, and protects prior-task subspaces from being overwritten.

Together, these contributions chart a path toward scalable, resource-aware machine learning systems that bridge decentralized efficiency and continual real-world deployment, learning new tasks across privacy-constrained environments without forgetting.


Thesis Canvas: pdf



Back to Doctoral Showcase Archive Listing