Doctoral Showcase Archive

Towards Resource-Efficient Parallel Concept Drift Detection in Data Streams


Author: Elias Werner (Technical University of Dresden, ScaDS.AI Dresden/Leipzig)

Advisor: Wolfgang E. Nagel (Technical University of Dresden, ScaDS.AI Dresden/Leipzig)

Abstract: Modern data science increasingly outpaces the efficient use of parallel hardware, creating a need for computational performance engineering. Because HPC tools are not embedded in exploratory data science environments, data scientists lack support for such investigations. This PhD project therefore bridges HPC and data science, using concept drift detection as a case study.

Concept drift—the change in data distributions after ML model deployment—must be detected to ensure robust streaming ML systems. However, the computational overhead of drift detectors (DDs) remains underexplored, even though directly affecting latency, energy consumption, and deployability. This gap motivates the thesis contributions.

First, a framework is introduced for systematically investigating the computational performance of DDs, including complexity analysis, benchmarking and performance analysis. Second, building on this framework, a survey and benchmark evaluate DDs bi-objectively, measuring both runtime and accuracy. The results reveal that computational performance is treated as secondary in the literature, no single detector dominates across scenarios, and higher accuracy consistently requires longer runtimes. Simultaneously, timely drift detection can reduce pipeline runtime by avoiding unnecessary model updates. Third, performance analysis identifies bottlenecks in state-of-the-art DD implementations, particularly inefficient BLAS and OpenMP usage in streaming scenarios. Consequently, the thesis presents MoPEDDs, a modular parallel ensemble of DDs that exploits unused parallel resources improving detection with negligible runtime overhead.

Finally, because performance engineering remains complex for most data scientists, the thesis introduces JUmPER (Jupyter meets Performance). It embeds performance engineering into Jupyter through coarse- and fine-grained monitoring, making HPC expertise accessible where data science happens.


Thesis Canvas: pdf



Back to Doctoral Showcase Archive Listing