Workshops Archive

Using ethtrace to debug AI networks


Workshop: Workshop on High Performance Fabrics for AI and HPC

Authors: Adrian Popa and Dragos Dumitrescu (Broadcom) and Costin Raiciu (Broadcom, Politehnica of Bucharest)

Abstract: We introduce ethtrace, a simple and lightweight on-demand network monitoring tool that generates probes from a source host at a specified frequency and relies on switch telemetry extensions such as the in-band flow analyzer to collect per hop and interactively displays or logs it at the destination host. ethtrace usage is as simple as that of established tools like ping and traceroute, but provides richer visibility and finer grained insights than existing approaches. ethtrace can be used to track the path of an existing queue-pair or TCP connection, or to monitor one or all paths between two hosts. We present several use-cases where ethtrace was used to quickly debug RoCEv2 performance anomalies such as PFC is- sues, DCQCN parameters settings, offering visibility into complex network behaviour that today is very difficult to understand and debug. ethtrace is deployable today, as a large fraction of deployed datacenter switches have telemetry support.


Back to Workshop on High Performance Fabrics for AI and HPC Archive Listing

Back to Full Workshop Archive Listing