Author: Mingzhe Yu (University of Tsukuba, Graduate School of Science and Technology)
Advisor: Osamu Tatebe (University of Tsukuba, RIKEN Center for Computational Science (R-CCS))
Abstract: Modern deep learning systems must balance optimization quality against computation, memory, communication, and reliability constraints, whose relative importance varies across supercomputing architectures. This dissertation develops controlled relaxation as a structured optimization strategy: preserve information essential to learning while relaxing costly requirements in synchronization, curvature representation, and parameterization.
The thesis develops this principle through three contributions. AD-KFAC relaxes global synchronization in distributed second-order optimization while retaining curvature information in asynchronous, versioned, and recoverable state. BD-KFAC replaces dense curvature representations with block-diagonal structure, exposing a controllable trade-off between statistical fidelity and resource cost. RRLoRA relaxes the requirement for an uninterrupted low-rank optimization trajectory by periodically refactorizing the learned update and restarting the learning-rate schedule, allowing the parameterization to evolve without discarding accumulated information.
Together, these methods address complementary bottlenecks in distributed second-order training, resource-efficient curvature estimation, and parameter-efficient fine-tuning while preserving the optimization structure that makes each approach effective. The results support controlled relaxation as a systems-aware design principle: preserve signal-carrying structure, relax constraints that impose disproportionate system cost, and keep the resulting approximation or asynchronous state observable and recoverable. This principle offers a practical framework for designing efficient and reliable machine-learning systems on large-scale supercomputing platforms.
Thesis Canvas: pdf