ARGUS has run stably for over six months on a production cluster of more than 10,000 GPUs, and has been key to detecting and diagnosing performance failures like compute stragglers and communication degradation at that scale.
Tencent's ARGUS tracing system has been deployed on a 10,000+ GPU production cluster for six-plus months, diagnosing issues like compute stragglers, communication link degradation, and pipeline bubble amplification.
transcript
Tencent researchers: ARGUS has been deployed on a 10,000+ GPU production cluster for over six months, running stably alongside production training and playing a key role in rapid fail-slow detection and performance optimization.